Issues with OAK-D-SR POE ToF camera with depthai

TamirLousky

I'm trying to add OAK-D-SR-POE preview streams to the depthai demo python QT app.

I've managed to read data from the camera and have conditional connections so that if another USB camera is connected it takes precedent, and if not it detects an attached POE camera.

The problem is with viewing and showing all the streams from the camera at all, and properly.
When iterating over available cameras I get the following results:

Connected Camera Features:

Camera on Socket: CameraBoardSocket.CAM_A

  Sensor Name: S5K33D

  Supported Types (Resolutions):

Camera on Socket: CameraBoardSocket.CAM_B

  Sensor Name: OV9782

  Supported Types (Resolutions):

{'TOF': {'width': 1280, 'height': 3848}, 'COLOR': {'width': 640, 'height': 400}}

So it detects two camera sensors on the device, but one of them (S5K33D) doesn't have any supported resolutions, and the other (OV9782) has the TOF and COLOR cameras (both of which have RGB output).

When trying to add a preview of both using:

PreviewManager.createColorCam(args = self._conf.args).setBoardSocket(rgbSocket)

PreviewManager.createColorCam(args = self._conf.args).setBoardSocket(tofSocket)

I get a conflict error:

ValueError: Streams have duplicate name 'color_control'

I can't connect the TOF socket as any other preview camera (depth requires left and right streams to which I don't have access, and the TOF has RGB rather than mono data so I can't attach it to either the left or right previews).

Also, when I do connect the TOF as the color camera I get a very strange preview that seems shifted on the Y axis, so that the top half of the video is down and the top is above.

Please advise how to properly set up BOTH streams on the preview and why I can't see any other available stream on this camera (it does have left and right cameras but I can't see them when reading the device's features using the API).

Thanks!

OskarSonc

Hey TamirLousky - S5K33D is the ToF sensor, so you can’t treat it like a normal ColorCamerapreview (the “shifted/garbled” image is just raw ToF data being displayed wrong). Instead, follow the ToF examples and wire it as Camera(CAM_A) → dai.node.ToF → preview tof.depth/tof.amplitude. The duplicate color_controlerror comes from creating two color previews that reuse the same control stream name in the demo app.
Also ToF camera has some additional functionality in dai v3 so I would suggest switching if possible...

TamirLousky

OskarSonc Thanks for the detailed answer!
Looking into the proper implementation. I'm currently using the example from the ToF examples and it has conflicts with the ordinary pipeline creation part since both are trying to connect to the same camera socket. Hope I manage to easily fix that one. Either way thanks and will update here.

TamirLousky

@OskarSonc Can't seem to get around the socket in use error:
RuntimeError: ColorCamera(4) - Specified camera board socket already used

I've patched the pipeline_manager.py to add a TOF pipeline with similar code to the tof example from the dedicated repo.

I've even made sure the tof pipeline uses the CAM-A socket while the color camera uses CAM-B.

However when the entire pipeline is created in self._device.startPipeline(self._pm.pipeline)

This error arises. Any help will be appreciated, thanks!

TamirLousky

TamirLousky OK figure that one out, it appears the default setup was still creating left and right streams even though they aren't functionally available. Removing them solved the conflict.

Finally got both color and tof to work and display on the preview, however the TOF preview stalls and stops after a few seconds. Probably some streaming or data or communication error.

TamirLousky

OK so I played with mjpeg compression, lowered the FPS and resolution, and still the preview gets stuck only for the TOF camera after around 1-2 seconds.

Here's my code for creating the TOF camera pipeline, will appreciate suggestions to fix this issue:

    def createTofCam(self,
        xout=False,
        pipeline=None,
        args=None):
        """
        Creates :obj:`depthai.node.ToF` node based on specified attributes

        Args:
            xout (bool, Optional): If set to :code:`True`, a dedicated :obj:`depthai.node.XLinkOut` will be created for this node
            pipeline (depthai.Pipeline, Optional): Pipeline instance
            args (Object, Optional): Arguments from the ArgsManager
        """
        if pipeline is None:
            pipeline = self.pipeline

        if args is not None:
            return self.createTofCam(
                tofSocket=args.tofSocket,
                xout=Previews.tofDepth.name in args.show,
                pipeline=pipeline,
                args=args
            )
            
            pipeline = dai.Pipeline()

        tof = pipeline.create(dai.node.ToF)

        # Configure the ToF node
        tofConfig = tof.initialConfig.get()

        # Optional. Best accuracy, but adds motion blur.
        # see ToF node docs on how to reduce/eliminate motion blur.
        tofConfig.enableOpticalCorrection = True
        tofConfig.enablePhaseShuffleTemporalFilter = True
        tofConfig.phaseUnwrappingLevel = 4
        tofConfig.phaseUnwrapErrorThreshold = 300

        tofConfig.enableTemperatureCorrection = False # Not yet supported

        xinTofConfig = pipeline.create(dai.node.XLinkIn)
        xinTofConfig.setStreamName("tofConfig")
        xinTofConfig.out.link(tof.inputConfig)

        tof.initialConfig.set(tofConfig)

        cam_tof = pipeline.create(dai.node.Camera)
        cam_tof.setFps(60) # ToF node will produce depth frames at /2 of this rate
        # cam_tof.setVideoSize((640, 400))
        cam_tof.setBoardSocket(dai.CameraBoardSocket.CAM_A)
        cam_tof.raw.link(tof.input)

        xout = pipeline.create(dai.node.XLinkOut)
        xout.setStreamName("depth")
        tof.depth.link(xout.input)

        self.nodes.tof = tof
        self.nodes.cam_tof = cam_tof
        self.nodes.tof_xout = xout

        self.nodes.tofConfig = tof.initialConfig.get()

        if xout:
            self.nodes.xoutTofDepth = pipeline.createXLinkOut()
            self.nodes.xoutTofDepth.setStreamName(Previews.tofDepth.name)
            self.nodes.tof.depth.link(self.nodes.xoutTofDepth.input)

        return self.nodes.tof

OskarSonc

Hey @TamirLousky - sorry for late reply.
Likely the ToF preview stalls because there’s an extra ToF XLinkOut stream ("depth") that isn’t being read, so its queue fills and back‑pressures the pipeline. In your snippet you create both "depth" and the preview stream; if the app only drains the preview, the other one will freeze the ToF node. Only create/link the stream you consume, or make any unused streams non‑blocking and small, and ensure the host drains them.

Thanks,
Oskar