IR (for night-vision) + rgb + stereo (disparity) + Mobilnet

npeditto

I have created this pipeline:

def create_pipeline(SETTINGS):

    # Create pipeline

    pipeline = dai.Pipeline()

    # Define INPUT RGB camera

    camRgb = pipeline.create(dai.node.ColorCamera)

    # camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

    camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)

    camRgb.setPreviewSize(300, 300)    # NN input

    camRgb.setInterleaved(False)

    camRgb.setPreviewKeepAspectRatio(False)

    camRgb.setFps(int(SETTINGS['NN']['fps']))

    # Define INPUT MONO-RIGHT camera

    monoRight = pipeline.create(dai.node.MonoCamera)

    monoRight.setCamera("right")

    monoRight.setFps(int(SETTINGS['NN']['fps']))

    # monoRight.setNumFramesPool(8)

    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

    # Define INPUT MONO-LEFT camera   

    monoLeft = pipeline.create(dai.node.MonoCamera)

    monoLeft.setCamera("left")

    monoLeft.setFps(int(SETTINGS['NN']['fps']))

    # monoLeft.setNumFramesPool(8)

    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

    

    # Define OUTPUTS

    nnOut = pipeline.create(dai.node.XLinkOut)

    nnOut.setStreamName("nn")

    disparityOut = pipeline.create(dai.node.XLinkOut)

    disparityOut.setStreamName("disparity")

    xoutRight = pipeline.create(dai.node.XLinkOut)

    xoutRight.setStreamName("rectifiedRight")

    xout_color = pipeline.create(dai.node.XLinkOut)

    xout_color.setStreamName("video")

    # Define COMPUTE-NODES

    stereo = pipeline.create(dai.node.StereoDepth)

    manip = pipeline.create(dai.node.ImageManip)

    nn = pipeline.create(dai.node.MobileNetDetectionNetwork)

    

    stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.DEFAULT)

    # stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)

    stereo.setRectifyEdgeFillColor(0) 

    

    manip.initialConfig.setResize(300, 300)

   
    manip.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p)



    nn.setConfidenceThreshold(float(SETTINGS['NN']['confidence']))

    nnPath = str((Path(__file__).parent / Path(str(SETTINGS['NN']['blob_path']))).resolve().absolute())

    nn.setBlobPath(nnPath)

    nn.setNumInferenceThreads(2)

    nn.input.setBlocking(False)

    # Linking

    camRgb.video.link(xout_color.input)

    monoRight.out.link(stereo.right)

    monoLeft.out.link(stereo.left)

    stereo.rectifiedRight.link(manip.inputImage)

    stereo.disparity.link(disparityOut.input)

    manip.out.link(nn.input)

    manip.out.link(xoutRight.input)

    nn.out.link(nnOut.input)

    return pipeline

and I'm using it in this way in order to process with NN the mono frame with IR enabled (for night-vision):


def Tracking(SETTINGS, selected_devices):


    DETECTED_OBJ=json.loads(SETTINGS['NN']['object'])

    pipeline = create_pipeline(SETTINGS)
    print("PIPELINE CREATED")

    devices = {}


    # Connect to device and start pipeline
    with contextlib.ExitStack() as stack:

        for device_info in selected_devices:
            openvino_version = dai.OpenVINO.Version.VERSION_2021_4
            usb2_mode = False
            device = stack.enter_context(dai.Device(openvino_version, device_info, usb2_mode))

            # Note: currently on POE, DeviceInfo.getMxId() and Device.getMxId() are different!
            print("\n==> Connected to " + device_info.getMxId())
            mxid = device.getMxId()


            device.startPipeline(pipeline)


            device.setIrLaserDotProjectorIntensity(0) # in %, from 0 to 1
            device.setIrFloodLightIntensity(0.9) # in %, from 0 to 1 


            devices[mxid] = {
                'nn': device.getOutputQueue("nn", maxSize=4, blocking=False),
                'mono': device.getOutputQueue("rectifiedRight", maxSize=4, blocking=False),
                'rgb':device.getOutputQueue("video", maxSize=4, blocking=False),
                'disparity':device.getOutputQueue("disparity", maxSize=4, blocking=False),
            }


...

I notice that the IR intensity is enabled only for the first frames.

Is it correct the pipeline and the IR intensity parameters?

Thank you

jakaskerl

npeditto
Hmm, should work.. Which device is this?

Thanks,
Jaka

npeditto

It is an OAK-D Pro…

I realized that, unlike what I wrote in this post yesterday, I was using the following configuration in my code:

device.setIrLaserDotProjectorIntensity(0.5) 
device.setIrFloodLightIntensity(0.9)

where I specified a value other than zero for both settings.

From what I understand, is it true that I should enable only one at a time?

At the moment I'm using this setting:

device.setIrLaserDotProjectorIntensity(0) 
device.setIrFloodLightIntensity(0.5)

and now It is working.

I noticed that I can't increase the value 0f 0.5 in setIrFloodLightIntensity if I use NN + stereo-disparity + IR at the same time, because the code or the camera crashes; instead if I use in my pipeline only the IR + mono-camera I am able to set it to 1 and the video preview is smoother. Is it a performance issue, a power supply issue, or a problem with my pipeline configuration?

The oak is connected via USB Thunderbolt 5

Thank you

jakaskerl

npeditto
It's most likely a power issue. We've seen power peaks when using the IR laser that exceed the usual limits. This might be the cause. We'll need to investigate more. Generally, using the Y-adapter to deliver power should solve the issue if supply is strong enough.
Maybe check the cable you are using, it might not be able to handle the peaks.

Thanks,
Jaka

npeditto

jakaskerl

Thanks for the suggestions!

Could you confirm if I can enable only one of these two parameters (setIrLaserDotProjectorIntensity setIrFloodLightIntensity) at a time?

jakaskerl

npeditto
No, both should work at once.