Hi Luxonis Team, I would like to ask if it is possible to setup multiple stereo streams one one Oak-SoM-Pro. We have a board that has three IR Cameras connected. Baseline between Camera 1 and 2 is 2.5cm, Baseline between Camera 1 and 3 is 10cm. Is it possible to configure two depth stream pipelines on one Device? If yes, is there any documentation, demo code, etc on how to do this. To calibrate the Cameras for the depth pipeline. We would follow the procedure described here https://docs.luxonis.com/en/latest/pages/calibration/ . Would we have to do that for both pipelines individual? Thank you so much for your help. Christoph

Hi christoph , Yes it's possible! In fact, our upcoming OAK-D-LR is very similar to what you are describing. You can just create multiple StereoDepth nodes, but I think for now you might need to apply rectification matrix to them separately, so a bit manual work. I believe we are currently working on this 2-stereo system feature inside calibration file. How far are you with the implementation, so we know the ETA? As we should have everything ready in a few months, so the UX for you would be easier then (but still possible now, with a bit of manual work/custom implementations). Thoughts? Thanks, Erik

Hi christoph , I apologize for the delay, we still don't have the full OAK-D-LR calibration script in place, will try to get the answer to 1st question asap. Already included in the matrix. Please see this PR - it will calculate the rectification matrix based on the calibration, so after getting intrinsics/extrinsics of all cameras you can connect any 2 to the same StereoDepth node and it will automatically set it up correctly. Using the branch above should take care of all configurations from the user side. Thoughts? Thanks, Erik

Hi christoph Your board config looks fine to me. The specTranslation should be positive. What error are you getting? Could you run the device.getAllAvailableDevices() to see which cameras are successfully connected? This shouldn't be the case, but I'm not sure whether this was tested. Do you errors go away if you move the color camera inside? Thanks, Jaka

Hi Jaka, thanks for the reply. My error is that the calibration does not work. I get through all 39 calibration pictures and then it tells me that the error is to big, so the calibration failed. I do not seem to be able to run , but when I run the camera_preview.py I get the following output: Device name: OAK-FFC-4P Bootloader version: 0.0.18+c555ac2fb184b801291c95f7f73d23bf4dd42cf1 Usb speed: SUPER Connected cameras: [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}] {socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L} CameraBoardSocket.RGB CameraBoardSocket.RGB {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L} CameraBoardSocket.LEFT CameraBoardSocket.LEFT {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L} CameraBoardSocket.RIGHT CameraBoardSocket.RIGHT So it seems like it is detecting them correctly. We have three similar cameras, no dedicated color camera. We are trying to get three stereo pairs CamA/CamB, CamA/CamC and CamB/CamC .We use three of the https://shop.luxonis.com/collections/oak-modules/products/oak-ffc-ov9782-w . So yeah the issue is that I am getting through the calibration, but then I get the error that it failed due to high error between the cameras. I also watched the calibration video on your website, but it also has no obvious pointers that I do wrong. Best, Christoph

Multiple Stereo Streams

christoph

Hi Luxonis Team,
I would like to ask if it is possible to setup multiple stereo streams one one Oak-SoM-Pro. We have a board that has three IR Cameras connected. Baseline between Camera 1 and 2 is 2.5cm, Baseline between Camera 1 and 3 is 10cm.

Is it possible to configure two depth stream pipelines on one Device? If yes, is there any documentation, demo code, etc on how to do this.
To calibrate the Cameras for the depth pipeline. We would follow the procedure described here https://docs.luxonis.com/en/latest/pages/calibration/. Would we have to do that for both pipelines individual?

Thank you so much for your help.
Christoph

erik

Hi christoph ,
Yes it's possible! In fact, our upcoming OAK-D-LR is very similar to what you are describing.

You can just create multiple StereoDepth nodes, but I think for now you might need to apply rectification matrix to them separately, so a bit manual work.
I believe we are currently working on this 2-stereo system feature inside calibration file.

How far are you with the implementation, so we know the ETA? As we should have everything ready in a few months, so the UX for you would be easier then (but still possible now, with a bit of manual work/custom implementations). Thoughts?
Thanks, Erik

christoph

Hi Erik,

thanks for the quick reply. The new LR Model looks cool, really like the idea. I am definitely excited for this product. I am just starting. So my ETA is as soon as possible, however I have limited timeslots to work on the project, so it might be a couple weeks out. I would really like to try to manually get it working.

I have three follow up question.

In the calibration example I give the calibration program the following the information

{
    "board_config":
    {
        "name": "ACME01",
        "revision": "V1.2",
        "swap_left_and_right_cameras": true,
        "left_fov_deg": 73.5,
        "rgb_fov_deg": 68.7938,
        "left_to_right_distance_cm": 2.5,
        "left_to_rgb_distance_cm": 5.0
    }
}

Would my output of the calibration be the rectificaion matrix I have to configure the stereo stream manually with?

Is there any other configurations I need for the stereo stream like baseline between cameras, fov, etc or is that already included in the rectifiction matrix?
Is there any example or documentation on how to configure the depth stream manually? I guess the API would be a good place to look.

Thank you for your help.

Best,
Christoph

erik

Hi christoph ,
I apologize for the delay, we still don't have the full OAK-D-LR calibration script in place, will try to get the answer to 1st question asap.

Already included in the matrix. Please see this PR - it will calculate the rectification matrix based on the calibration, so after getting intrinsics/extrinsics of all cameras you can connect any 2 to the same StereoDepth node and it will automatically set it up correctly.
Using the branch above should take care of all configurations from the user side.

Thoughts?
Thanks, Erik

christoph

Hi Erik,

thank you for your reply. Could it be that the mentioned branch above from you was merged and I should be able to use the devel branch at this point to get it to work? I guess the example to get it working is examples/StereoDepth/depth_preview_lr.py. Only thing I am not seeing yet is how to calibrate the module. Would the standard calibration be enough for this?

Best,
Christoph

erik

Hi christoph ,
Yep, it got merged. And we also just created docs on multi-stereo calibration; PR here. I hope it helps.
Thanks, Erik

christoph

Hi Erik,

thanks for all your help. I got my setup now and I am trying the calibration, I tried it and it constantly fails. I seem to have some issues. My setup looks as follows.

In red are the connections to the FFC and the black numbers on bottom show the baseline in cm.

My calibration board file looks as follows

   {
       "board_config":
       {
           "cameras": {
               "CAM_A": {
                   "name": "rgb",
                   "hfov": 127.0,
                   "type": "color"
               },
               "CAM_B": {
                   "name": "left",
                   "hfov": 127.0,
                   "type": "color",
                   "extrinsics": {
                       "to_cam": "CAM_A",
                       "specTranslation": {
                           "x": 2.5,
                           "y": 0,
                           "z": 0
                       },
                       "rotation":{
                           "r": 0,
                           "p": 0,
                           "y": 0
                       }
                   }
               },
               "CAM_C": {
                   "name": "right",
                   "hfov": 127.0,
                   "type": "color",
                   "extrinsics": {
                       "to_cam": "CAM_A",
                       "specTranslation": {
                           "x": 10.0,
                           "y": 0,
                           "z": 0
                       },
                       "rotation":{
                           "r": 0,
                           "p": 0,
                           "y": 0
                       }
                   }
               }
           },
           "stereo_config":{
               "left_cam": "CAM_B",
               "right_cam": "CAM_C"
           }
       }
   }

Does that seem correct. I am not sure about the specTranslation, should the distance be negative or positive.

When I change my type to "mono" in the board config file I get an error saying

   ~/Downloads/depthai$ python3 calibrate.py -s 3.1 -db -brd OrionAI -iv
   [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}]
   Cam: rgb and focus: False
   Cam: left and focus: False
   Cam: right and focus: False
   Traceback (most recent call last):
     File "/home/christoph/Downloads/depthai/calibrate.py", line 1171, in <module>
       Main().run()
     File "/home/christoph/Downloads/depthai/calibrate.py", line 358, in __init__
       pipeline = self.create_pipeline()
     File "/home/christoph/Downloads/depthai/calibrate.py", line 416, in create_pipeline
       cam_node.setResolution(camToMonoRes[cam_info['sensorName']])
   KeyError: 'OV9782'

Not sure why this happens, since the cameras we have are mono.

Is it necessary that camA is the center camera, as you can see camA is the right camera in my case. Can this be changed via board config file or is this not important anyways?

Best,

Christoph

jakaskerl

Hi christoph
Your board config looks fine to me. The specTranslation should be positive. What error are you getting?

Could you run the device.getAllAvailableDevices() to see which cameras are successfully connected?
This shouldn't be the case, but I'm not sure whether this was tested. Do you errors go away if you move the color camera inside?

Thanks,
Jaka

christoph

Hi Jaka,

thanks for the reply. My error is that the calibration does not work. I get through all 39 calibration pictures and then it tells me that the error is to big, so the calibration failed.

I do not seem to be able to run , but when I run the camera_preview.py I get the following output:

Device name: OAK-FFC-4P

Bootloader version: 0.0.18+c555ac2fb184b801291c95f7f73d23bf4dd42cf1

Usb speed: SUPER

Connected cameras: [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}]

{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L} CameraBoardSocket.RGB CameraBoardSocket.RGB

{socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L} CameraBoardSocket.LEFT CameraBoardSocket.LEFT

{socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L} CameraBoardSocket.RIGHT CameraBoardSocket.RIGHT

So it seems like it is detecting them correctly.
We have three similar cameras, no dedicated color camera. We are trying to get three stereo pairs CamA/CamB, CamA/CamC and CamB/CamC .We use three of the https://shop.luxonis.com/collections/oak-modules/products/oak-ffc-ov9782-w.

So yeah the issue is that I am getting through the calibration, but then I get the error that it failed due to high error between the cameras. I also watched the calibration video on your website, but it also has no obvious pointers that I do wrong.

Best,

Christoph

jakaskerl

Hi christoph,
I looked at the image of your setup you sent above. It looks to me that CamA and CamB have a few millimeters of shift on the y axis. Can you confirm this is the case? This could potentially be the cause for the calibration error. It's more likely that the error stems from the fact you are using wide FOV cameras, but it might be worth checking.

Thanks,
Jaka

erik

( I am also not sure whether we already support stereo camera pairs not being on the same axis)

christoph

Hi Jaka and Erik,

so I just started 3d printing a mount. That should definitely help mking sure that everything is correct mechanically. I will keep you up to date with new information as soon as I tested it. @jakaskerl I wanted to ask what you mean by "It's more likely that the error stems from the fact you are using wide FOV cameras, but it might be worth checking." Could it be that the wide FOV cameras can not be used for the stereo pair?

Best,

Christop

erik

FWIW we do sell OAK FFC Kit, so users don't need to 3d print their own mounts; https://shop.luxonis.com/collections/early-access/products/oak-ffc-kit
Wide FOV can be used as stereo pairs, they are just a bit more finicky when calibrating. So you need to make sure to have large calibration board (+5cm of square sizes) and many images (I use default - 39 of them).

christoph

Hi Jaka and Erik,

thanks for the patience and advice. I got it to calibrate, it does work now. Thank you for that. One point that I realized is that when I connect the same camera to RGB/CAM_A or CAM_D on the FFC it does detect different sensors. I attached an image of my terminal output here. In the top I connect the camera Sensor to CAM_D and it is detected correctly as OV9282, in the bottom I connect the same camera to RGB/CAM_A FFC input and it is detected as OV9782. Why is that, is there a bug in the OAK-SOM Pro Firmware that forces input on RGB/CAM_A to be an RGB camera. Connecting it to CAM_D also allows me to configure the calibration file correctly with mono as color for all cameras.

erik

Hi christoph ,
Could you try with cam_test.py script instead?

christoph

Hi Erik,

This shows similar output. Here the output from OV9282 plugged into CAM_D

and here is the output from OV9282 plugged into RGB/CAM_A

erik

christoph looks like it's working to me - in terminal it's printed 30FPS for either CAM_A or CAM_D, just where the OV9282 is plugged into.

christoph

Hi Eric,

that is correct. It does show me the stream, but it detects it as the incorrect camera. It is not an OV9782. Plugging it in CAM_A gives me an error when I want to calibrate the setup with color set to "mono" for all three cameras. I have to set it to color for CAM__A, which is incorrect setting. So I wanted to ask if the firmware does on the OAK SOM has a bug there?

The it gives is

~/Downloads/depthai$ python3 calibrate.py -s 3.1 -db -brd OrionAI -iv [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}] Cam: rgb and focus: False Cam: left and focus: False Cam: right and focus: False Traceback (most recent call last): File "/home/christoph/Downloads/depthai/calibrate.py", line 1171, in <module> Main().run() File "/home/christoph/Downloads/depthai/calibrate.py", line 358, in __init__ pipeline = self.create_pipeline() File "/home/christoph/Downloads/depthai/calibrate.py", line 416, in create_pipeline cam_node.setResolution(camToMonoRes[cam_info['sensorName']]) KeyError: 'OV9782'

erik

Hi christoph ,
The FW itself can't know the difference between OV9282 and OV9782 - they are exactly the same sensor, the only difference is the bayer pattern filter. Which cameras are you using on which ports?

christoph

Hi Erik,

We designed a PCB for the OAK SOM PRO. On that one we connect OV9282 to input RGB/CAM_A, CAM_B and CAM_C. Is there any chance we get a Firmware Version that we can flash which would recognize camera sensor OV9282 correctly on RGB/CAM_A?

erik

Hi christoph ,
You can just create MonoCamera instead of ColorCamera node for the CAM_A to tell it its monochrome. For actual calibration, you'd need to specify that in your board json; https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration.html#modular-cameras-calibration