Detect the depth of a ROI in RGB camera

NhanNgo

Hi all,

I have a ROI in RGB frame, please give me some advice to get the depth of that ROI.

Many thanks,

Nhan

jakaskerl

Hi NhanNgo
Are you able to use the stereo cameras? If so, check this example: https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_location_calculator/ and add
stereo.setLeftRightCheck(True) stereo.setDepthAlign(dai.CameraBoardSocket.RGB)
[source]

If not, you can use something like https://github.com/dwofk/fast-depth (we have it at luxonis model zoo for quick implementation using blobconverter).

Hope this helps,
Jaka

NhanNgo

jakaskerl

Hi Jakaskerl,

Yes, I'm using the OAK-D, so I able to use the stereo cameras.

So I just need to set the code "stereo.setLeftRightCheck(True) stereo.setDepthAlign(dai.CameraBoardSocket.RGB)" then put the ROI (in RGB frame) into the SpatialLocationCalculator, is it correct? Do I need to configure the other things?

Thanks for your help.

Simone

Hi jakaskerl, during my tests I seem to notice that a cropped version is more accurate but I am not sure is there a cropped version of this code you linked to test it? with depth_crop_control.py I seem to see the differences better but I can't implement the roi to detect the distance

jakaskerl

Hi NhanNgo
You might need to add some focus control. Just follow https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/rgb_depth_aligned/ and you should be good 🙂

Thanks,
Jaka

NhanNgo

jakaskerl

Thanks, I'll try and inform you.

jakaskerl

Hi @Simone
Could you add some code so I have a better understanding of what you are trying to achieve?

Thanks,
Jaka

Simone

Hi jakaskerl I'm trying to use OAK to avoid collision with things it sees, not pierce through detect. In the example I asked about earlier I was wondering if it was possible that a cropped version like this but implemented ad this code -> could be more accurate? In my search I also found this example -> collision_avoidance which however when I try to reproduce it on my computer is much less sharp you can't tell anything apart.

jakaskerl

Hi @Simone
Cropping the mono frames will only improve performance, not accuracy.

Simone In my search I also found this example -> collision_avoidance which however when I try to reproduce it on my computer is much less sharp you can't tell anything apart.

Any images you can add? Are you using the code as-is or are you modifying it.

Thanks,
Jaka

Simone

jakaskerl yes, in this screenshot I am positioned in front of the camera with a norlame ambient light, and the code is the same as that shown in the demo here, but I am not visible or even detected as in the demo.
depthai 2.22
depthai-sdk 1.13.1

jakaskerl

Hi @Simone
Uh the example is broken in the new versions. Nobody checked.

Here's the old code that should work on 1.12.1:

from depthai_sdk import OakCamera
from depthai_sdk.visualize.configs import StereoColor
from depthai_sdk.classes.packets import DepthPacket
import math
import depthai as dai
import cv2

# User-defined constants
WARNING = 1000 # 1m, orange
CRITICAL = 500 # 50cm, red

slc_data = []

def cb(packet: DepthPacket):
    global slc_data
    fontType = cv2.FONT_HERSHEY_TRIPLEX

    depthFrameColor = packet.visualizer.draw(packet.frame)

    for depthData in slc_data:
        roi = depthData.config.roi
        roi = roi.denormalize(width=depthFrameColor.shape[1], height=depthFrameColor.shape[0])

        xmin = int(roi.topLeft().x)
        ymin = int(roi.topLeft().y)
        xmax = int(roi.bottomRight().x)
        ymax = int(roi.bottomRight().y)

        coords = depthData.spatialCoordinates
        distance = math.sqrt(coords.x ** 2 + coords.y ** 2 + coords.z ** 2)

        if distance == 0: # Invalid
            continue

        if distance < CRITICAL:
            color = (0, 0, 255)
            cv2.rectangle(depthFrameColor, (xmin, ymin), (xmax, ymax), color, thickness=4)
            cv2.putText(depthFrameColor, "{:.1f}m".format(distance/1000), (xmin + 10, ymin + 20), fontType, 0.5, color)
        elif distance < WARNING:
            color = (0, 140, 255)
            cv2.rectangle(depthFrameColor, (xmin, ymin), (xmax, ymax), color, thickness=2)
            cv2.putText(depthFrameColor, "{:.1f}m".format(distance/1000), (xmin + 10, ymin + 20), fontType, 0.5, color)

    cv2.imshow('0_depth', depthFrameColor)

with OakCamera() as oak:
    stereo = oak.create_stereo('720p')
    # We don't need high fill rate, just very accurate depth, that's why we enable some filters, and
    # set the confidence threshold to 50
    config = stereo.node.initialConfig.get()
    config.postProcessing.brightnessFilter.minBrightness = 0
    config.postProcessing.brightnessFilter.maxBrightness = 255
    stereo.node.initialConfig.set(config)
    stereo.config_postprocessing(colorize=StereoColor.RGBD, colormap=cv2.COLORMAP_BONE)
    stereo.config_stereo(confidence=50, lr_check=True, extended=True)

    oak.visualize([stereo], fps=True, callback=cb)

    oak.build()

    slc = oak.pipeline.create(dai.node.SpatialLocationCalculator)
    for x in range(15):
        for y in range(9):
            config = dai.SpatialLocationCalculatorConfigData()
            config.depthThresholds.lowerThreshold = 200
            config.depthThresholds.upperThreshold = 10000
            config.roi = dai.Rect(dai.Point2f((x+0.5)*0.0625, (y+0.5)*0.1), dai.Point2f((x+1.5)*0.0625, (y+1.5)*0.1))
            # TODO: change from median to 10th percentile once supported
            config.calculationAlgorithm = dai.SpatialLocationCalculatorAlgorithm.MEDIAN
            slc.initialConfig.addROI(config)

    stereo.depth.link(slc.inputDepth)

    slc_out = oak.pipeline.create(dai.node.XLinkOut)
    slc_out.setStreamName('slc')
    slc.out.link(slc_out.input)

    oak.start() # Start the pipeline (upload it to the OAK)

    q = oak.device.getOutputQueue('slc') # Create output queue after calling start()
    while oak.running():
        if q.has():
            slc_data = q.get().getSpatialLocations()
        oak.poll()

Thanks,
Jaka

Simone

jakaskerl Thanks it works now, I wanted to ask if there was a version of this collision_avoidance program that was implemented directly with dephtai and not depthai-sdk, kind of like this program here? In case you need to crop the video so that it focuses only in certain bands

jakaskerl

Hi @Simone
Made one now, should make modifications easier:

import depthai as dai
import cv2
import numpy as np
import math

# User-defined constants
WARNING = 500  # 50cm, orange
CRITICAL = 300  # 30cm, red

# Create pipeline
pipeline = dai.Pipeline()

# Define source - stereo depth cameras
left = pipeline.create(dai.node.MonoCamera)
left.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
left.setBoardSocket(dai.CameraBoardSocket.LEFT)

right = pipeline.create(dai.node.MonoCamera)
right.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)
right.setBoardSocket(dai.CameraBoardSocket.RIGHT)

# Create stereo depth node
stereo = pipeline.create(dai.node.StereoDepth)
stereo.setConfidenceThreshold(50)
stereo.setLeftRightCheck(True)
stereo.setExtendedDisparity(True)

# Linking
left.out.link(stereo.left)
right.out.link(stereo.right)

# Spatial location calculator configuration
slc = pipeline.create(dai.node.SpatialLocationCalculator)
for x in range(15):
    for y in range(9):
        config = dai.SpatialLocationCalculatorConfigData()
        config.depthThresholds.lowerThreshold = 200
        config.depthThresholds.upperThreshold = 10000
        config.roi = dai.Rect(dai.Point2f((x+0.5)*0.0625, (y+0.5)*0.1), dai.Point2f((x+1.5)*0.0625, (y+1.5)*0.1))
        config.calculationAlgorithm = dai.SpatialLocationCalculatorAlgorithm.MEDIAN
        slc.initialConfig.addROI(config)

stereo.depth.link(slc.inputDepth)
stereo.setDepthAlign(dai.CameraBoardSocket.LEFT)

# Create output
slcOut = pipeline.create(dai.node.XLinkOut)
slcOut.setStreamName('slc')
slc.out.link(slcOut.input)

monoOut = pipeline.create(dai.node.XLinkOut)
monoOut.setStreamName('left')
left.out.link(monoOut.input)


# Connect to device and start pipeline
with dai.Device(pipeline) as device:
    # Output queues will be used to get the left mono frames and spatial location data
    qLeft = device.getOutputQueue(name="left", maxSize=4, blocking=False)
    qSlc = device.getOutputQueue(name="slc", maxSize=4, blocking=False)

    fontType = cv2.FONT_HERSHEY_TRIPLEX

    while True:
        inLeft = qLeft.get()  # Try to get a frame from the left mono camera
        inSlc = qSlc.get()  # Try to get spatial location data

        if inLeft is None:
            print("No left camera data")
        if inSlc is None:
            print("No spatial location data")

        leftFrame = None
        if inLeft is not None:
            leftFrame = inLeft.getCvFrame()  # Fetch the frame from the left mono camera
            # convert to rgb
            leftFrame = cv2.cvtColor(leftFrame, cv2.COLOR_GRAY2BGR)

        if inSlc is not None and leftFrame is not None:
            slc_data = inSlc.getSpatialLocations()
            for depthData in slc_data:
                roi = depthData.config.roi
                roi = roi.denormalize(width=leftFrame.shape[1], height=leftFrame.shape[0])

                xmin = int(roi.topLeft().x)
                ymin = int(roi.topLeft().y)
                xmax = int(roi.bottomRight().x)
                ymax = int(roi.bottomRight().y)

                coords = depthData.spatialCoordinates
                distance = math.sqrt(coords.x ** 2 + coords.y ** 2 + coords.z ** 2)

                if distance == 0:  # Invalid
                    continue

                # Determine color based on distance
                if distance < CRITICAL:
                    color = (0, 0, 255)  # Red
                elif distance < WARNING:
                    color = (0, 140, 255)  # Orange
                else:
                    continue  # Skip drawing for non-critical/non-warning distances

                # Draw rectangle and distance text on the left mono frame
                cv2.rectangle(leftFrame, (xmin, ymin), (xmax, ymax), color, thickness=2)
                cv2.putText(leftFrame, "{:.1f}m".format(distance / 1000), (xmin + 10, ymin + 20), fontType, 0.5, color)

            # Display the left mono frame
            cv2.imshow('Left Mono Camera', leftFrame)
            if cv2.waitKey(1) == ord('q'):
                break

Thanks,
Jaka

Simone

jakaskerl Yes thank you, this code also seems to work well, is there a specific reason why you used the left camera for the outputs? could you use the main camera? is it possible to change the saturation to make something reflective more explicit so that it can improve to the accuracy of depth detection

jakaskerl

jakaskerl stereo.setDepthAlign(dai.CameraBoardSocket.LEFT)

Alignment is currently set to left mono camera. You could also align depth to the RGB and then display the grid over the rgb frame instead of the mono frame.

jakaskerl

Hi @Simone
No real reason. You could use the color camera just as well, but you need to set the alignment.

Thanks,
Jaka

smoothVision

jakaskerl Hi , can you be more explicit on what you've said ?

"No real reason. You could use the color camera just as well, but you need to set the alignment."

how to use the central camera to do the same thing ?

smoothVision

@Simone

HI , Simone can you be more explit on what you are trying to say : " to change the saturation to make something reflective more explicit so that it can improve to the accuracy of depth detection" making it more saturate improve quality of the depth ?

i think i'm doing a similar project as you , do you mind sharing your code ? if you changed something from the shared one above ?

jakaskerl

smoothVision forgot to tag