Multiple Stereo Streams

erik · Feb 8, 2023

Hi christoph ,
Yep, it got merged. And we also just created docs on multi-stereo calibration; PR here. I hope it helps.
Thanks, Erik

Cchristoph · May 14, 2023

Hi Erik,

thanks for all your help. I got my setup now and I am trying the calibration, I tried it and it constantly fails. I seem to have some issues. My setup looks as follows.

In red are the connections to the FFC and the black numbers on bottom show the baseline in cm.

My calibration board file looks as follows

   {
       "board_config":
       {
           "cameras": {
               "CAM_A": {
                   "name": "rgb",
                   "hfov": 127.0,
                   "type": "color"
               },
               "CAM_B": {
                   "name": "left",
                   "hfov": 127.0,
                   "type": "color",
                   "extrinsics": {
                       "to_cam": "CAM_A",
                       "specTranslation": {
                           "x": 2.5,
                           "y": 0,
                           "z": 0
                       },
                       "rotation":{
                           "r": 0,
                           "p": 0,
                           "y": 0
                       }
                   }
               },
               "CAM_C": {
                   "name": "right",
                   "hfov": 127.0,
                   "type": "color",
                   "extrinsics": {
                       "to_cam": "CAM_A",
                       "specTranslation": {
                           "x": 10.0,
                           "y": 0,
                           "z": 0
                       },
                       "rotation":{
                           "r": 0,
                           "p": 0,
                           "y": 0
                       }
                   }
               }
           },
           "stereo_config":{
               "left_cam": "CAM_B",
               "right_cam": "CAM_C"
           }
       }
   }

Does that seem correct. I am not sure about the specTranslation, should the distance be negative or positive.

When I change my type to "mono" in the board config file I get an error saying

   ~/Downloads/depthai$ python3 calibrate.py -s 3.1 -db -brd OrionAI -iv
   [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}]
   Cam: rgb and focus: False
   Cam: left and focus: False
   Cam: right and focus: False
   Traceback (most recent call last):
     File "/home/christoph/Downloads/depthai/calibrate.py", line 1171, in <module>
       Main().run()
     File "/home/christoph/Downloads/depthai/calibrate.py", line 358, in __init__
       pipeline = self.create_pipeline()
     File "/home/christoph/Downloads/depthai/calibrate.py", line 416, in create_pipeline
       cam_node.setResolution(camToMonoRes[cam_info['sensorName']])
   KeyError: 'OV9782'

Not sure why this happens, since the cameras we have are mono.

Is it necessary that camA is the center camera, as you can see camA is the right camera in my case. Can this be changed via board config file or is this not important anyways?

Best,

Christoph

jakaskerl · May 15, 2023

Hi christoph
Your board config looks fine to me. The specTranslation should be positive. What error are you getting?

Could you run the device.getAllAvailableDevices() to see which cameras are successfully connected?
This shouldn't be the case, but I'm not sure whether this was tested. Do you errors go away if you move the color camera inside?

Thanks,
Jaka

Cchristoph · May 18, 2023

Hi Jaka,

thanks for the reply. My error is that the calibration does not work. I get through all 39 calibration pictures and then it tells me that the error is to big, so the calibration failed.

I do not seem to be able to run , but when I run the camera_preview.py I get the following output:

Device name: OAK-FFC-4P

Bootloader version: 0.0.18+c555ac2fb184b801291c95f7f73d23bf4dd42cf1

Usb speed: SUPER

Connected cameras: [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}]

{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L} CameraBoardSocket.RGB CameraBoardSocket.RGB

{socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L} CameraBoardSocket.LEFT CameraBoardSocket.LEFT

{socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L} CameraBoardSocket.RIGHT CameraBoardSocket.RIGHT

So it seems like it is detecting them correctly.
We have three similar cameras, no dedicated color camera. We are trying to get three stereo pairs CamA/CamB, CamA/CamC and CamB/CamC .We use three of the https://shop.luxonis.com/collections/oak-modules/products/oak-ffc-ov9782-w.

So yeah the issue is that I am getting through the calibration, but then I get the error that it failed due to high error between the cameras. I also watched the calibration video on your website, but it also has no obvious pointers that I do wrong.

Best,

Christoph

jakaskerl · May 18, 2023

Hi christoph,
I looked at the image of your setup you sent above. It looks to me that CamA and CamB have a few millimeters of shift on the y axis. Can you confirm this is the case? This could potentially be the cause for the calibration error. It's more likely that the error stems from the fact you are using wide FOV cameras, but it might be worth checking.

Thanks,
Jaka

erik · May 19, 2023

( I am also not sure whether we already support stereo camera pairs not being on the same axis)

Cchristoph · May 20, 2023

Hi Jaka and Erik,

so I just started 3d printing a mount. That should definitely help mking sure that everything is correct mechanically. I will keep you up to date with new information as soon as I tested it. @jakaskerl I wanted to ask what you mean by "It's more likely that the error stems from the fact you are using wide FOV cameras, but it might be worth checking." Could it be that the wide FOV cameras can not be used for the stereo pair?

Best,

Christop

erik · May 20, 2023

FWIW we do sell OAK FFC Kit, so users don't need to 3d print their own mounts; https://shop.luxonis.com/collections/early-access/products/oak-ffc-kit
Wide FOV can be used as stereo pairs, they are just a bit more finicky when calibrating. So you need to make sure to have large calibration board (+5cm of square sizes) and many images (I use default - 39 of them).

Cchristoph · Jun 4, 2023

Hi Jaka and Erik,

thanks for the patience and advice. I got it to calibrate, it does work now. Thank you for that. One point that I realized is that when I connect the same camera to RGB/CAM_A or CAM_D on the FFC it does detect different sensors. I attached an image of my terminal output here. In the top I connect the camera Sensor to CAM_D and it is detected correctly as OV9282, in the bottom I connect the same camera to RGB/CAM_A FFC input and it is detected as OV9782. Why is that, is there a bug in the OAK-SOM Pro Firmware that forces input on RGB/CAM_A to be an RGB camera. Connecting it to CAM_D also allows me to configure the calibration file correctly with mono as color for all cameras.

erik · Jun 4, 2023

Hi christoph ,
Could you try with cam_test.py script instead?

Cchristoph · Jun 5, 2023

Hi Erik,

This shows similar output. Here the output from OV9282 plugged into CAM_D

and here is the output from OV9282 plugged into RGB/CAM_A

erik · Jun 6, 2023

christoph looks like it's working to me - in terminal it's printed 30FPS for either CAM_A or CAM_D, just where the OV9282 is plugged into.

Cchristoph · Jun 7, 2023

Hi Eric,

that is correct. It does show me the stream, but it detects it as the incorrect camera. It is not an OV9782. Plugging it in CAM_A gives me an error when I want to calibrate the setup with color set to "mono" for all three cameras. I have to set it to color for CAM__A, which is incorrect setting. So I wanted to ask if the firmware does on the OAK SOM has a bug there?

The it gives is

~/Downloads/depthai$ python3 calibrate.py -s 3.1 -db -brd OrionAI -iv [{socket: RGB/CENTER/CAM_A, sensorName: OV9782, width: 1280, height: 800, orientation: AUTO, supportedTypes: [COLOR, MONO], hasAutofocus: 0, name: CAMA-4L}, {socket: LEFT/CAM_B, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMB-2L}, {socket: RIGHT/CAM_C, sensorName: OV9282, width: 1280, height: 800, orientation: AUTO, supportedTypes: [MONO, COLOR], hasAutofocus: 0, name: CAMC-2L}] Cam: rgb and focus: False Cam: left and focus: False Cam: right and focus: False Traceback (most recent call last): File "/home/christoph/Downloads/depthai/calibrate.py", line 1171, in <module> Main().run() File "/home/christoph/Downloads/depthai/calibrate.py", line 358, in __init__ pipeline = self.create_pipeline() File "/home/christoph/Downloads/depthai/calibrate.py", line 416, in create_pipeline cam_node.setResolution(camToMonoRes[cam_info['sensorName']]) KeyError: 'OV9782'

erik · Jun 8, 2023

Hi christoph ,
The FW itself can't know the difference between OV9282 and OV9782 - they are exactly the same sensor, the only difference is the bayer pattern filter. Which cameras are you using on which ports?

Cchristoph · Jun 15, 2023

Hi Erik,

We designed a PCB for the OAK SOM PRO. On that one we connect OV9282 to input RGB/CAM_A, CAM_B and CAM_C. Is there any chance we get a Firmware Version that we can flash which would recognize camera sensor OV9282 correctly on RGB/CAM_A?

erik · Jun 15, 2023

Hi christoph ,
You can just create MonoCamera instead of ColorCamera node for the CAM_A to tell it its monochrome. For actual calibration, you'd need to specify that in your board json; https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration.html#modular-cameras-calibration

Cchristoph · Jul 23, 2023

Hi Eric,
thanks for the help. I got them calibrated and tested. Works very well now.

I have a new Question. When I run three depth streams and put the Median Filter to KERNEL_3x3 or higher I get

[1944301051D3E31200] [5.1] [2.728] [StereoDepth(3)] [critical] 'Median+Disparity to depth' out of system resources: '246'! You can increase the available memory pool by setting 'sippBufferSize' and/or 'sippDmaBufferSize' using 'setSippBufferSize' and/or 'setSippDmaBufferSize' API.
[1944301051D3E31200] [5.1] [2.728] [StereoDepth(3)] [critical] Invalid StereoDepth config. Error code: '382'. Halting.

I see the instruction that I can increase the Memory Pool with sippBufferSize or sippDmaBufferSize, but I am not sure how. Is that something I can do or does that have to be in the Firmware?

Best,
Christoph

jakaskerl · Jul 24, 2023

Hi christoph
I believe this might help you:
https://docs.luxonis.com/projects/api/en/latest/references/python/#depthai.GlobalProperties

Thanks,
Jaka