OAK-FFC-3P Calibration not great

robomick

Hi Oak Community,

I've bought an OAK-FFC-3P and two OAK-FFC OV9782 global shutter cameras listed on the "cameras are compatible with this OAK FFC baseboard". It's running on a Jetson Orin Nano with 8GB RAM.

Can anyone see where I am going wrong? I think I have followed the steps correctly?

There is no CAM_B or CAM_C on the OAK-FFC-3P like noted in the calibration tutorial, but I assume that the CAM_L is CAM_B and CAM_R is CAM_C. It notes R3M0E3 on the OAK-FFC-3P so I have noted that as the revision.

The cameras are spaced 6cm apart.

This is the calibration json file (I've just replaced the hfov and the x translation like in the example):
{

"board_config": {

"name": "OAK-FCC-3P",

"revision": "R3M0E3",

"cameras": {

"CAM_C": {

"name": "right",

"hfov": 75,

"type": "mono",

"extrinsics": {

"to_cam": "CAM_B",

"specTranslation": {

"x": 6,

"y": 0,

"z": 0

},

"rotation": {

"r": 0,

"p": 0,

"y": 0

}

},

"CAM_B": {

"name": "left",

"hfov": 75,

"type": "mono"

}

},

"stereo_config": {

"left_cam": "CAM_B",

"right_cam": "CAM_C"

}

I run it like this (with no RGB camera):
python3 calibrate.py -s 3.4 --board OAK-FFC-3P.json -nx 11 -ny 8 -dsb rgb

The EEPROM flashes successfully. These are the calibration dots and some example images saved in the dataset folder:

This is the resulting depth results running depth_preview.py.

And this is the depth_preview_sr.py results:

Trying depth_preview_lr.py just errors:

[14442C10619FCAD600] [1.2.4] [1.838] [ColorCamera(1)] [error] Camera not detected on socket: 0

[14442C10619FCAD600] [1.2.4] [1.838] [ColorCamera(0)] [warning] Unsupported resolution set for detected camera OV9782, needs 800_P or 720_P. Defaulting to 800_P

[14442C10619FCAD600] [1.2.4] [1.839] [ColorCamera(2)] [warning] Unsupported resolution set for detected camera OV9782, needs 800_P or 720_P. Defaulting to 800_P

[14442C10619FCAD600] [1.2.4] [2.023] [StereoDepth(4)] [error] Left input image width ('854') must be multiple of 16. Skipping frame!

[14442C10619FCAD600] [1.2.4] [2.039] [StereoDepth(4)] [error] Left input image width ('854') must be multiple of 16. Skipping frame!

[14442C10619FCAD600] [1.2.4] [2.072] [StereoDepth(4)] [error] Left input image width ('854') must be multiple of 16. Skipping frame!

The /depthai-experiments/gen2-camera-demo seems to get slightly better results, but still not close the demo quality?

The point cloud generated using open3D:

This is sad quality compared to examples demos and won't be viable 😰 Not sure how to improve this?

erik

Hi @robomick ,
Are you sure left and right aren't flipped, perhaps during calibration, or in the example itself? Note that "depth_preview_lr" isn't the correct demo, as it assumes you are using AR0234 (2.1MP sensor, so it does downscaling), while you are using OV9282 (1MP sensor).

robomick

The example notes CAM_C as right so I have it connected to CAM_R, and CAM_B as left so I have it connected to CAM_L? The guide notes a positive translation from C to B so I entered positive 6? I think that is all correct?

The camera text is upside down on the camera modules, but the cameras appear up the correct way in the testing, so I assume that's not critical?

erik

Hi @robomick ,
Could you also provide the dataset folder inside depthai repo? It contains images that were used for the calibration. Also, how firm is the holder for the stereo cameras? seems like everything is made out of plastic, even screws, which might be prone to bending. And even a tiny bending will cause significant depth quality loss.

erik

@robomick ,
Could you please try switching inpurts to StereoDepth node? So link left mono cam to depth.right, and right mono cam to depth.left input:

monoLeft.out.link(depth.right)
monoRight.out.link(depth.left)

robomick

erik

So the original code without swapping:

And with the swapped:
monoLeft.out.link(depth.right)

monoRight.out.link(depth.left)

This is the code I am running to view the stereo:

#!/usr/bin/env python3

import cv2

import depthai as dai

import numpy as np

# Closer-in minimum depth, disparity range is doubled (from 95 to 190):

extended_disparity = False

# Better accuracy for longer distance, fractional disparity 32-levels:

subpixel = False

# Better handling for occlusions:

lr_check = True

# Create pipeline

pipeline = dai.Pipeline()

# Define sources and outputs

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

depth = pipeline.create(dai.node.StereoDepth)

# Output nodes for disparity, left, and right images

xoutDisparity = pipeline.create(dai.node.XLinkOut)

xoutLeft = pipeline.create(dai.node.XLinkOut)

xoutRight = pipeline.create(dai.node.XLinkOut)

xoutDisparity.setStreamName("disparity")

xoutLeft.setStreamName("left")

xoutRight.setStreamName("right")

# Properties

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT) # Use setBoardSocket with enum

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)

monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT) # Use setBoardSocket with enum

# Configure StereoDepth node

depth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)

depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)

depth.setLeftRightCheck(lr_check)

depth.setExtendedDisparity(extended_disparity)

depth.setSubpixel(subpixel)

# Linking

monoLeft.out.link(depth.left)

monoRight.out.link(depth.right)

depth.disparity.link(xoutDisparity.input)

monoLeft.out.link(xoutLeft.input)

monoRight.out.link(xoutRight.input)

# Connect to device and start pipeline

with dai.Device(pipeline) as device:

# Output queues for disparity, left, and right frames

qDisparity = device.getOutputQueue(name="disparity", maxSize=4, blocking=False)

qLeft = device.getOutputQueue(name="left", maxSize=4, blocking=False)

qRight = device.getOutputQueue(name="right", maxSize=4, blocking=False)

while True:

inDisparity = qDisparity.get() # blocking call, will wait until a new data has arrived

disparityFrame = inDisparity.getFrame()

# Normalization for better visualization of disparity

disparityFrame = (disparityFrame * (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)

# Applying color map to disparity

disparityColor = cv2.applyColorMap(disparityFrame, cv2.COLORMAP_JET)

# Retrieve left and right camera frames

inLeft = qLeft.tryGet()

inRight = qRight.tryGet()

if inLeft:

leftFrame = inLeft.getCvFrame()

cv2.imshow("left", leftFrame)

if inRight:

rightFrame = inRight.getCvFrame()

cv2.imshow("right", rightFrame)

cv2.imshow("disparity", disparityFrame)

cv2.imshow("disparity_color", disparityColor)

if cv2.waitKey(1) == ord('q'):

break

Here are the OAK calibration images, the OAK-FCC-3P json file, and the stereo_test.py file. Let me know if there's any issues accessing.
Dropbox to images

robomick

Also thanks for the fast responses. I am on a field trip a bit of the coast of Australia, so the timezones, and internet access are a bit wacky.

We're going to try use the stereo, and just bag the raw mono output from the OAK so we can at least get some data. Hopefully can fix the depth data output after the mission.

jakaskerl

Hi @robomick
Thanks, we'll get back to you on that.

robomick

jakaskerl Just checking in to see if you are getting similar results when calibrating, or if I am not following the guide correctly, or have a faulty board? Not sure if I should continue down this path?

jakaskerl

Hi @robomick
We are still trying to repro the issue, this time with your exact setup and calibration. So far no luck, but I highly doubt you have a faulty board.
Does pointing a device at a textured object improve the fill rate and accuracy?

Thanks,
Jaka

jakaskerl

Hi @robomick
Looks like you are using 9782 which is a color camera. Please make the calibration json accordingly.

Thanks,
Jaka

robomick

jakaskerl

Setting the JSON file to color throws errors running the calibration script? Are the OV9782 cameras supported for OAK-FFC-3P? Maybe they have only been tested on the RGB camera when calibrating? The both show up in the file as mono and RGB?

This is the error:

python3 calibrate.py -s 4.30 --board OAK-FFC-3P.json -nx 13 -ny 7 -dsb rgb

Cam: left and focus: False

Cam: right and focus: False

Saving dataset to: dataset

Sensor name for right is OV9282

Traceback (most recent call last):

File "calibrate.py", line 1171, in <module>

Main().run()

File "calibrate.py", line 1160, in run

self.startPipeline()

File "calibrate.py", line 460, in startPipeline

pipeline = self.create_pipeline()

File "calibrate.py", line 551, in create_pipeline

cam_node.setResolution(camToRgbRes[cam_info['sensorName'].upper()])

KeyError: 'OV9282'

python3 calibrate.py -s 4.30 --board OAK-FFC-3P.json -nx 13 -ny 7 -dsb rgb

Cam: left and focus: False

Cam: right and focus: False

Saving dataset to: dataset

Sensor name for right is OV9282

Traceback (most recent call last):

File "calibrate.py", line 1171, in <module>

Main().run()

File "calibrate.py", line 1160, in run

self.startPipeline()

File "calibrate.py", line 460, in startPipeline

pipeline = self.create_pipeline()

File "calibrate.py", line 551, in create_pipeline

cam_node.setResolution(camToRgbRes[cam_info['sensorName'].upper()])

KeyError: 'OV9282'

For this JSON file:

{

"board_config": {

"name": "OAK-FCC-3P",

"revision": "R3M0E3",

"cameras": {

"CAM_C": {

"name": "right",

"hfov": 75,

"type": "color",

"extrinsics": {

"to_cam": "CAM_B",

"specTranslation": {

"x": 6,

"y": 0,

"z": 0

},

"rotation": {

"r": 0,

"p": 0,

"y": 0

}

},

"CAM_B": {

"name": "left",

"hfov": 75,

"type": "color"

}

},

"stereo_config": {

"left_cam": "CAM_B",

"right_cam": "CAM_C"

}

jakaskerl

Hi @robomick
I see. Then there is no other way atm, than to change the calibrate.py script.

Add this to the line below 356:

cameraProperties[1].sensorName = "OV9782"
cameraProperties[1].supportedTypes = [dai.CameraSensorType.COLOR]
cameraProperties[2].sensorName = "OV9782"
cameraProperties[2].supportedTypes = [dai.CameraSensorType.COLOR]

We have to force the properties since they aren't read correctly by the FW.

Thanks,
Jaka

robomick

That doesn't seem to work? Your calibration file might be different to mine?

Here is line 356 from the calibration file:

A colleague suggested hard coding it in here like this instead which allowed me to run calibration (edited from line 553).

                cam_node.setBoardSocket(stringToCam[cam_id])

                # sketchy hard code to check please remove

                #sensorName = cam_info['sensorName']

                cam_info['sensorName'] = "OV9782"

Calibration ran.

Results are pretty sad still I think? I'm not sure what to do at the moment?

Running this example code:

#!/usr/bin/env python3

import cv2

import depthai as dai

import numpy as np

# Closer-in minimum depth, disparity range is doubled (from 95 to 190):

extended_disparity = False

# Better accuracy for longer distance, fractional disparity 32-levels:

subpixel = False

# Better handling for occlusions:

lr_check = True

# Create pipeline

pipeline = dai.Pipeline()

# Define sources and outputs

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

depth = pipeline.create(dai.node.StereoDepth)

# Output nodes for disparity, left, and right images

xoutDisparity = pipeline.create(dai.node.XLinkOut)

xoutLeft = pipeline.create(dai.node.XLinkOut)

xoutRight = pipeline.create(dai.node.XLinkOut)

xoutDisparity.setStreamName("disparity")

xoutLeft.setStreamName("left")

xoutRight.setStreamName("right")

# Properties

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

monoLeft.setBoardSocket(dai.CameraBoardSocket.LEFT) # Use setBoardSocket with enum

monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_720_P)

monoRight.setBoardSocket(dai.CameraBoardSocket.RIGHT) # Use setBoardSocket with enum

# Configure StereoDepth node

depth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)

depth.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)

depth.setLeftRightCheck(lr_check)

depth.setExtendedDisparity(extended_disparity)

depth.setSubpixel(subpixel)

# Linking

monoLeft.out.link(depth.left)

monoRight.out.link(depth.right)

depth.disparity.link(xoutDisparity.input)

monoLeft.out.link(xoutLeft.input)

monoRight.out.link(xoutRight.input)

# Connect to device and start pipeline

with dai.Device(pipeline) as device:

# Output queues for disparity, left, and right frames

qDisparity = device.getOutputQueue(name="disparity", maxSize=4, blocking=False)

qLeft = device.getOutputQueue(name="left", maxSize=4, blocking=False)

qRight = device.getOutputQueue(name="right", maxSize=4, blocking=False)

while True:

    inDisparity = qDisparity.get()  # blocking call, will wait until a new data has arrived

    disparityFrame = inDisparity.getFrame()

    # Normalization for better visualization of disparity

    disparityFrame = (disparityFrame \* (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)

    

    # Applying color map to disparity

    disparityColor = cv2.applyColorMap(disparityFrame, cv2.COLORMAP_JET)

    # Retrieve left and right camera frames

    inLeft = qLeft.tryGet()

    inRight = qRight.tryGet()

    if inLeft:

        leftFrame = inLeft.getCvFrame()

        cv2.imshow("left", leftFrame)

    if inRight:

        rightFrame = inRight.getCvFrame()

        cv2.imshow("right", rightFrame)

    cv2.imshow("disparity", disparityFrame)

    cv2.imshow("disparity_color", disparityColor)

    if cv2.waitKey(1) == ord('q'):

        break

Updated the dataset images for review: https://www.dropbox.com/scl/fo/h9nav3q3y448q6mqkjrjf/h?rlkey=nikezpxqkgh68u1q3gd6x9b36&dl=0

jakaskerl

Hi @robomick
Yeah, the results are bad. But you are also running an older script it seems. git pull --recurse-submodules. Lmk if this changes anything.

Thanks,
Jaka

robomick

jakaskerl
After running git pull --recurse-submodules

Running:
depthai-python/examples/StereoDepth$ python3 depth_preview.py

depthai-python/examples/StereoDepth$ python3 depth_preview_sr.py

jakaskerl

Hi @robomick
The idea is to redo the calibration with the updated script. There might have been a change that will fix the depth you are getting when recalibrating. Could you check that please. Also make sure the calibrate.py file has changed during git pull.
Let us know so we can fix the issue.

Thanks,
Jaka

robomick

jakaskerl If you want to post your step-by-step process calibrating 2 x OV9782 cameras using a OAK-FFC-3P I would appreciate to see where I have gone wrong? I have done a fresh install following the guide steps (except for the step where I have to edit the broken calibration file). Is there something else I need to edit so the OV9782 cameras work with the OAK-FCC-3P?

robomick

I thought I would try installing DepthAI on my PC so I could use the depthAI viewer as it doesn't run on out AUV jetson.

sudo wget -qO- https://docs.luxonis.com/install_depthai.sh | bashcd depthaigit submodule update --init --recursivepython3 install_requirements.py

#Edited calibrate with:

#sensorName = cam_info['sensorName']

#print(f'Sensor name for {cam_info["name"]} is {sensorName}')

cam_info['sensorName'] = "OV9782"

# JSON file

{

"board_config":

{

"name": "OAK-FCC-3P",

"revision": "R1M0E1",

"cameras":{

"CAM_C": {

"name": "right",

"hfov": 75,

"type": "color",

"extrinsics": {

"to_cam": "CAM_B",

"specTranslation": {

"x": 7,

"y": 0,

"z": 0

},

"rotation":{

"r": 0,

"p": 0,

"y": 0

}

},

"CAM_B": {

"name": "left",

"hfov": 75,

"type": "color"

}

},

"stereo_config":{

"left_cam": "CAM_B",

"right_cam": "CAM_C"

}

python3 calibrate.py -s 4.30 --board OAK-FFC-3P.json -nx 13 -ny 7 -dsb rgb

I setup a little jig with the cameras 70mm apart.

After calibrating with the script, this was the depthAI output. I am using OAK-FFC-OV9782-M12, not OV9282. This looks like something isn't handling the OV9782 images correctly.

robomick

Is this the expected result I should be seeing using the python3 -m depthai_viewer? Are the squares caused by it converting the RGB image to mono?