RTSP Demo: OAK-D Pro IR Illumination in H.265 encoded Mono Frames

aschlieb

Hi,

I'm currently exploring the new V3 API using the rtsp-streaming example. RGB and Mono streaming work fine, but for some reason, I cannot see (or enable) the IR flood light in dark scenes. This is the code I'm using (copied together into a single code block):

import sys
import depthai as dai

from utils.host_stream_output import StreamOutput
from utils.arguments import initialize_argparser

def enable_ir_flood_light(pipeline: dai.Pipeline) -> None:
    dots_status = pipeline.getDefaultDevice().setIrLaserDotProjectorIntensity(0.0)
    ir_status = pipeline.getDefaultDevice().setIrFloodLightIntensity(1.0)
    print("Dots/IR status:", dots_status, ir_status)

def create_encoded_rtsp_stream(
    pipeline: dai.Pipeline,
    frame_source: dai.Node.Output,
    fps: int,
    rtsp_host: str,
    rtsp_port: int,
    rtsp_mount: str = "stream",
    encoder_profile: dai.VideoEncoderProperties.Profile = dai.VideoEncoderProperties.Profile.H265_MAIN,
) -> None:
    encoder = pipeline.create(dai.node.VideoEncoder)
    encoder.setDefaultProfilePreset(fps=fps, profile=encoder_profile)
    frame_source.link(encoder.input)

    rtsp_stream = pipeline.create(StreamOutput).build(
        stream=encoder.bitstream,
        host=rtsp_host,
        port=rtsp_port,
        mount=rtsp_mount,
        fps=fps,
    )
    rtsp_stream.inputs["stream"].setBlocking(True)
    rtsp_stream.inputs["stream"].setMaxSize(fps)

_, args = initialize_argparser()

try:
    device = dai.Device(dai.DeviceInfo(args.device)) if args.device else dai.Device()
    device_info = device.getDeviceInfo()
except RuntimeError as e:
    print(str(e))
    sys.exit(1)

with dai.Pipeline(device) as pipeline:
    enable_ir_flood_light(pipeline=pipeline)

    create_encoded_rtsp_stream(
        pipeline=pipeline,
        frame_source=pipeline.create(dai.node.Camera)
        .build(dai.CameraBoardSocket.CAM_A)
        .requestOutput(
            size=(args.width, args.height), type=dai.ImgFrame.Type.NV12, fps=args.fps
        ),
        fps=args.fps,
        rtsp_host=args.rtsp_host,
        rtsp_port=args.rtsp_port,
        rtsp_mount="rgb",
    )

    create_encoded_rtsp_stream(
        pipeline=pipeline,
        frame_source=pipeline.create(dai.node.Camera)
        .build(dai.CameraBoardSocket.CAM_C)
        .requestFullResolutionOutput(type=dai.ImgFrame.Type.NV12, fps=args.fps),
        fps=args.fps,
        rtsp_host=args.rtsp_host,
        rtsp_port=args.rtsp_port + 1,
        rtsp_mount="ir",
    )

    pipeline.run()

Currently, I'm getting this rather dark output from the ir (Mono) stream in my Frigate live viewer:

Some help would be greatly appreciated! Thanks!

OskarSonc

Hey @aschlieb - did you take a look at our premade example for flood lights? Also you can see how to correctly implement it here:
https://docs.luxonis.com/software-v3/depthai/examples/misc/projectors/

Thanks,
Oskar

aschlieb

Hi OskarSonc ,

yes indeed! I copied the call to

 pipeline.getDefaultDevice().setIrFloodLightIntensity(...)

from that example. I see that the flood light intensity is set after calling pipeline.start(). Is it strictly nessecary to do it that way or should it be possible to set the intensity the way I tried to do it? Note that the RTSP example and your IR Projectors example have quite different program flows (one calls pipeline.run() and the other pipeline.start() followed by a while loop).

OskarSonc

afaik it’s effectively required to set flood/dot after the graph has started, because pipeline.start() initializes the sensors and resets emitter state to 0. Also I think you have use CAM_B for the IR stream so the emitter is driven.

aschlieb

OskarSonc

Awesome, just using CAM_B has fixed my issue. It also seems like I had to reduce the intensity as the image faded back to dark after some time, probably because of overheating?

OskarSonc

Happy to hear it worked! Yes that can happen especially if running for longer times with maxium intesity in environments with limited cooling...

aschlieb

Just for the record, I'm now setting the max. flood light intensity (1.0) directly after device creation:

device = dai.Device(dai.DeviceInfo(args.device)) if args.device else dai.Device()
if args.ir_intensity is not None and device.setIrFloodLightIntensity(
    args.ir_intensity
):
    print(f"IR intensity set to {args.ir_intensity}.")

This also seems to fix the "fading to black" issue. Maybe it was not overheating but something software related. Not entirely sure though. Thanks again for your help!