• Hardware

  • Depth Map for Object Detection in Low Light Settings

I've been running the spatial_mobilenet_mono API example in low light/IR environments, and noticed that the depth map improves the mono footage and its rate of detection by a huge amount (in addition to providing depth data).

How would I do this with yolo 8 with the SDK? I've managed to run object detection with spatial data in bright rooms, but have not been able to get the depth map to improve the detection in low light environments. It seems like I have to feed the depth map into the NN_Component somehow, but I have not been able to find documentation on this.

I have included screenshots and code below:

spatial_mobilenet_mono in low light/IR environment
You can see that the mono footage is very clear and it detects people more than 90% of the time

yolo_8 in low light (I want it to work like the spatial_mobilenet_mono)
The mono footage is a lot blurrier and you can barely see anything. Detects people less than 10% of the time.

yolo_8 code

from depthai_sdk import OakCamera
import depthai as dai

with OakCamera() as oak:
    left = oak.create_camera("left")
    right = oak.create_camera("right")
    stereo = oak.create_stereo(left=left, right=right, fps=30)

    nn = oak.create_nn('yolov8n_coco_640x352', left, spatial=stereo)

    nn.config_spatial(
        bb_scale_factor=0.5, # Scaling bounding box before averaging the depth in that ROI
        lower_threshold=300, # Discard depth points below 30cm
        upper_threshold=10000, # Discard depth pints above 10m
        #Average depth points before calculating X and Y spatial coordinates:
        calc_algo=dai.SpatialLocationCalculatorAlgorithm.AVERAGE
    )
    oak.visualize([nn.out.main, stereo.out.depth], fps=True)
    #oak.show_graph()
    oak.start(blocking=True)

API code of spatial_mobilenet_mono
https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_mobilenet_mono/

Appreciate your time and thanks in advance!

    Hi jsiic
    Does mono camera resolution affect the low light performance in your case?

    Thanks,
    Jaka

        jakaskerl In the two situations above, I don't think it's the resolution that is affecting the performance. For example, the mono_mobilenet script also has the same issue I am running into on yolo_8, despite being the same ML model as spatial_mobilenet_mono (which doesn't have the issue). Also, spatial_mobilenet_mono has lower resolution than yolo8 on mono...

        Also to clarify, I am just displaying the depth map in my yolo_8 script, but it doesn't affect the mono footage at all, whereas the depth map in spatial_mobilenet_mono seems to affect the mono footage.

            Hi jsiic
            Could be that the IR laser dot projector that gets set to 800mA in the SDK example causes the mono image to be difficult for the yolo model to run inference on (the dots most likely disturb it). I don't see the laser projector enabled in the first case. Try lowering the dot projector strength.

            Thoughts?
            Jaka

                9 days later

                jakaskerl
                I looked into this, and I think you are right. Thank you!

                However, I cannot lower the intensity of the dot projectors. I added two lines of code:
                stereo.set_auto_ir(False, continuous_mode=False)
                stereo.set_ir(dot_projector_brightness=200, flood_brightness=1000)

                The flood lights work fine, which helps improve the detection. But the dot projector is still set to 800ma, which is too strong and reduces the accuracy. Any thoughts on how to fix this?

                    Hi jsiic
                    Despite receiving the message that IR has been set to 800mA, the code above should work properly. Lower the intensity to eg. 10 and try also lowering the floodlight, maybe it causes the same problem.

                    Thanks,
                    Jaka

                        jakaskerl Hi! I think you were right about the IR dots, because if I put them at 0mA, you cannot see any dots on the screen.

                        However, in complete darkness, the floodlights fluctuate in brightness even though I set auto_IR to false. I also tried to comment out the auto_IR code and still the same issue. It only goes away when I turn on more light in my environment. I do not have this issue when I am using the spatial_mobile_net API example.

                        Both models are running in almost no light environment with Dot: 200mA, Flood Lights 1000mA.

                        Code:

                        from depthai_sdk import OakCamera
import depthai as dai

with OakCamera() as oak:
    left = oak.create_camera("left")
    right = oak.create_camera("right")
    stereo = oak.create_stereo(left=left, right=right, fps=30)
    stereo.set_auto_ir(auto_mode=False, continuous_mode=False)
    stereo.set_ir(dot_projector_brightness=200, flood_brightness=1000)

    nn = oak.create_nn('yolov8n_coco_640x352', left, spatial=stereo)


    nn.config_spatial(
        bb_scale_factor=0.5, # Scaling bounding box before averaging the depth in that ROI
        lower_threshold=300, # Discard depth points below 30cm
        upper_threshold=10000, # Discard depth pints above 10m
        #Average depth points before calculating X and Y spatial coordinates:
        calc_algo=dai.SpatialLocationCalculatorAlgorithm.AVERAGE
    )


    oak.visualize([nn.out.main, stereo.out.depth], fps=True)
    #oak.show_graph()
    oak.start(blocking=True)

                          erik Looks interesting... I will look into it!

                          Although for the current project, I am happy with the performance with 1000mA illumination and 200ma dot projector for now... I would just like to know how to stop the illumination from fluctuating like in my video.

                              Hi jsiic
                              Try displaying the frames using a callback function to see if there is any difference. I was testing the flood and laser brightness settings and it looks to me like the visualization normalization (or something similar) could be the culprit and not automatic IR adjustment.

                              Thanks,
                              Jaka