Custom baseboard not detecting camera in CAM_D slot

MernokAdriaan · 13 Feb

Hi,

I have a custom baseboard for the OAK-SoM-Pro that should support 2x 2-lane cameras and 2x 4-lane cameras.
I've based the design on the CM4 and the 4P-FFC baseboards, with some additional redundant M.2 storage among a few other items.

I have the FSIN4-lane connected to connector B pin 72 (IO42) and CAMD_nRST to connector B pin 75 (IO42). And then of course the usual CAMD MIPI lines + I2C3 + shared A and D cam clocks.

Routing and pinouts seem good with no visible flaws the circuit or the board (tried two different boards to try and rule out manufacturing defects). It is however not detecting the module, giving the usual "Camera not detected on socket: 3" error.

I've noticed that the NG2093 design uses pins 70 and 72 as opposed to 72 and 74, respectively.

Is it possible that the firmware on the unit is following the former convention as opposed to the latter, or am I better off looking for a flaw in my design?

Unfortunately, I cannot share the designs publicly, but I am more than willing to share them will a Luxonis engineer for private review.

jakaskerl · 16 Feb

MernokAdriaan
NG2099 and BW2099 (which commonly comes on FFC-4P) should both work the same. Can you detect any signal on CAM-D to make sure the connections are OK and that correct voltage is supplied to the socket.

What happens if you run a script with DEPTHAI_DEBUG=1 env variable.

Thanks,
Jaka

MernokAdriaan · 17 Feb

@jakaskerl

Slight correction on my original message: I have the FSIN and nRST CAMD connected to pins 72 & 74, respectively (172 & 174 in my case since it is created as a single part in our libraries):

And then I2C3 of the SoM is used for control:

Performing a single measurement on CAMD's SDA & SCL (I2C3) line upon application startup, it seems that, what I can only imagine is some sort of discovery/initialization command, is sent to it - this signal is the same for all modules on startup from what I can tell:

After this, there is no more activity on the I2C bus (which is to be expected).

Both CAMA and CAMD share reference clocks as recommended bt the datasheet:

Here is the measured waveform at the pins of the FFC going to the module:

CAMA performs as expected, while CAMD is not detected at all.

Setting the environment variable does not seem to have any impact on my application.
Note that this is a headless devices and I am displaying the video feed via a flask webserver, but the pipeline is pretty simple, a camera node for each and an xout stream for each. Using CAMA through CAMB sockets.

Measuring CAMD_nRST does however indicate that the pin is never set high on startup, whereas all the other modules' reset is pulled high to 1.8V.

jakaskerl · 17 Feb

MernokAdriaan
cc @Luxonis-Alex

MernokAdriaan · 18 Feb

Update: I've manually pulled the RST signal high by means of a pullup resistor on the CCM carrier board itself, but no luck.
I've also probed the MIPI_CLK_N line in an attempt to see if there is at something going on at all, and by means of a FFT, I can see the expected 400MHz peak for CAMs A, B, and C, but nothing for CAMD - this might be expected though since I suspect if the initialization over I2C is unsuccessful, no data is sent over the MIPI bus.

Any thoughts? I've run out of ideas at this point…

MernokAdriaan · 3 Mar

@jakaskerl / @Luxonis-Alex where is the best place to ask for support regarding this?
Due to time constraints, I am now updating the board, but leaving the design surrounding this the same as I cannot see any flaw in what I've done. Thus, I am banking on it being some sort of firmware/configuration bug/fault - if not, the product will only support 3x camera sensors instead of 4 - an inconvenience for sure, but not critical.

jakaskerl · 3 Mar

MernokAdriaan
Hey, sorry, @Luxonis-Alex is swamped right now and I don't know enough to say for certain.

Just looking at the FW code, the CAM-D PWR was moved to GPIO43 in rev7, older ones have it at 42. GPIO42 is now used for 4lane FSIN connection enable.

What does your calibration file look like?

Thanks,
Jaka

MernokAdriaan · 4 Mar

jakaskerl

No problem, swamped myself.

See calibration below (note, not calibrated with CAMD yet, but that shouldn't matter when just trying to get a preview of all cameras, no?):

{
    "batchName": "",
    "batchTime": 0,
    "boardConf": "",
    "boardCustom": "",
    "boardName": "OCULUS100",
    "boardOptions": 0,
    "boardRev": "REV1",
    "cameraData": [
        [
            0,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -8.551840782165527,
                    53.17557144165039,
                    -0.0006793601205572486,
                    0.005195034667849541,
                    -81.15084838867188,
                    -8.631146430969238,
                    53.50436782836914,
                    -81.47747039794922,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ],
                        [
                            0.0,
                            0.0,
                            0.0
                        ]
                    ],
                    "specTranslation": {
                        "x": -0.0,
                        "y": -0.0,
                        "z": -0.0
                    },
                    "toCameraSocket": -1,
                    "translation": {
                        "x": 0.0,
                        "y": 0.0,
                        "z": 0.0
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        819.1251831054688,
                        0.0,
                        691.3241577148438
                    ],
                    [
                        0.0,
                        821.2589111328125,
                        418.86907958984375
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 132,
                "specHfovDeg": 68.7938003540039,
                "width": 1280
            }
        ],
        [
            1,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -12.539551734924316,
                    119.6891860961914,
                    0.0010478135664016008,
                    0.005322601180523634,
                    -147.6470947265625,
                    -12.58570384979248,
                    119.41400146484375,
                    -146.74142456054688,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9995344877243042,
                            -0.02753101848065853,
                            0.01314602605998516
                        ],
                        [
                            0.02729462832212448,
                            0.9994683265686035,
                            0.017834849655628204
                        ],
                        [
                            -0.01363004744052887,
                            -0.01746773160994053,
                            0.9997545480728149
                        ]
                    ],
                    "specTranslation": {
                        "x": -15.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 2,
                    "translation": {
                        "x": -14.97307300567627,
                        "y": -0.018007850274443626,
                        "z": 0.10245031863451004
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        800.89111328125,
                        0.0,
                        665.762939453125
                    ],
                    [
                        0.0,
                        804.4785766601563,
                        388.82196044921875
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 71.86000061035156,
                "width": 1280
            }
        ],
        [
            2,
            {
                "cameraType": 0,
                "distortionCoeff": [
                    -8.244771957397461,
                    64.31938171386719,
                    -0.00015859985433053225,
                    0.002660101279616356,
                    -90.57433319091797,
                    -8.331904411315918,
                    64.49740600585938,
                    -90.5763931274414,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                "extrinsics": {
                    "rotationMatrix": [
                        [
                            0.9998677372932434,
                            0.007823566906154156,
                            -0.01425823662430048
                        ],
                        [
                            -0.007816142402589321,
                            0.9999693036079407,
                            0.000576336111407727
                        ],
                        [
                            0.014262308366596699,
                            -0.00046481547178700566,
                            0.9998981952667236
                        ]
                    ],
                    "specTranslation": {
                        "x": 5.0,
                        "y": 0.0,
                        "z": 0.0
                    },
                    "toCameraSocket": 0,
                    "translation": {
                        "x": 4.983494758605957,
                        "y": -0.04398958012461662,
                        "z": 0.037372641265392303
                    }
                },
                "height": 800,
                "intrinsicMatrix": [
                    [
                        806.313232421875,
                        0.0,
                        628.2617797851563
                    ],
                    [
                        0.0,
                        808.0611572265625,
                        383.96063232421875
                    ],
                    [
                        0.0,
                        0.0,
                        1.0
                    ]
                ],
                "lensPosition": 0,
                "specHfovDeg": 71.86000061035156,
                "width": 1280
            }
        ]
    ],
    "deviceName": "",
    "hardwareConf": "",
    "housingExtrinsics": {
        "rotationMatrix": [],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "imuExtrinsics": {
        "rotationMatrix": [
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ]
        ],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },
    "miscellaneousData": [],
    "productName": "",
    "stereoEnableDistortionCorrection": false,
    "stereoRectificationData": {
        "leftCameraSocket": 1,
        "rectifiedRotationLeft": [
            [
                0.9996364712715149,
                -0.026208823546767235,
                0.006326698698103428
            ],
            [
                0.026151668280363083,
                0.999617874622345,
                0.00895375944674015
            ],
            [
                -0.006558948662132025,
                -0.008785051293671131,
                0.9999399185180664
            ]
        ],
        "rectifiedRotationRight": [
            [
                0.999975860118866,
                0.0012026532785966992,
                -0.006842138711363077
            ],
            [
                -0.0012632976286113262,
                0.9999598860740662,
                -0.008865943178534508
            ],
            [
                0.00683120172470808,
                0.008874373510479927,
                0.9999372959136963
            ]
        ],
        "rightCameraSocket": 2
    },
    "stereoUseSpecTranslation": true,
    "version": 7,
    "verticalCameraSocket": -1
}

jakaskerl · 5 Mar

MernokAdriaan
I'm slightly confused as to how you got it working so far since your calibration file board conf and name don't match any of luxonis' configs. -- perhaps there is some other setting you are applying?

When the firmware loads, it checks the entries for boardName and revision and matches it to a predefined FW config and sets the GPIOs and (possibly) other settings.

Example from FFC-4P rev7:

{
  "batchName": "",
  "batchTime": 1685015064,
  "boardConf": "IR-C00M00-00",
  "boardCustom": "",
  "boardName": "DD2090",
  "boardOptions": 0,
  "boardRev": "R7M1E7",
  "cameraData": [
    [
      1,
      {
        "cameraType": 0,
        "distortionCoeff": [
          2.355395555496216,
          25.66667938232422,
          -0.00154523225
 ....

I would suggest you set these the same way so the FW thinks your carrier board is a FFC4P.

Sample from FW which sets the GPIOS just so you have an idea of how it works:

BoardEntry oak_ffc_4p_r7m1e7 = {
    "OAK-FFC-4P (R7M1E7)",
    {
        "DD2090", "R7M1E7", "", ""
    },
    [](const BoardMatchingKey&){
        dai::BoardConfig cfg;

        // Modify GPIO43 to be output (CAM-D PWR), default for this function was 42 on older revisions
        using GPIO = dai::BoardConfig::GPIO;
        cfg.gpio[43] = GPIO(GPIO::OUTPUT, GPIO::LOW);
        // 42 is used now as FSIN_4LANE (on older revisions it was 40), make sure to change to input
        cfg.gpio[42] = GPIO(GPIO::INPUT, GPIO::LOW, GPIO::PULL_DOWN);

Thanks,
Jaka

MernokAdriaan · 5 Mar

jakaskerl

I'll give this a spin by modifying and updating the calibration on the unit manually since my calibration process is a bit broken at this point (modified version of the official one, streaming video to a webserver since it is a headless device and hence cannot use the cv.imshow functions).

I changed the config since it is a custom baseboard, but I might have misunderstood the instructions. Nevertheless, I did not believe that the calibration would affect whether or not the sensor is detected or not (validated by @erik's response in this discussion. The current pipeline is a simple preview of all cameras.

I'll report back as soon as I have the calibration updated to match the FFC-4P version.

MernokAdriaan · 5 Mar

Trying to flash the modified calibration results in the following error:

Failed flashing calibration: No PROTECTED permissions to override protected EEPROM fields

I suspect that even if I fix my calibration pipeline, I will have the same issue since I need the camera module to connect in the first place in order to get the images, calibrate, and eventually update the calibration (including the board configuration).

Is there any way I can just force update the board config to make it think it is a 4P so I can get the video feed going? I can then worry about getting everything calibrated nicely.

jakaskerl · 6 Mar

MernokAdriaan
How did you manage to flash the json in the first place?

You need to set the magic environment variable: DEPTHAI_ALLOW_FACTORY_FLASHING=868632271 when running the flash script (I will delete this tomorrow as we don't wan't people bricking their device if the don't know what they are doing)

Thanks,
Jaka

MernokAdriaan · 6 Mar

jakaskerl
I used the custom board config and did the calibration procedure.
Let me give this a spin..

MernokAdriaan · 6 Mar

@jakaskerl, it worked! Thanks so much.

I am worried that if I do the calibration procedure again with my custom board config, that things will break again since I am not "fooling" the firmware into believing that I am using this specific board, thus it will not do the pin change.

Is this something I can set/override somewhere?

Here is the board config I used for calibration on the device:

{
    "description": "Oculus 100, 2x OV9282 (NFOV), 1x OV9782(NFOV)",
    "board_config":
    {
        "name": "OCULUS100",
        "revision": "REV1",
        "cameras":{
            "CAM_A": {
                "name": "rgb",
                "hfov": 68.7938,
                "type": "color"
            },
            "CAM_B": {
                "name": "left",
                "hfov": 71.86,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_C",
                    "specTranslation": {
                        "x": -15.0,
                        "y": 0,
                        "z": 0
                    },
                    "rotation":{
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            },
            "CAM_C": {
                "name": "right",
                "hfov": 71.86,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_A",
                    "specTranslation": {
                        "x": 5,
                        "y": 0,
                        "z": 0
                    },
                    "rotation":{
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            }
        },
        "stereo_config":{
            "left_cam": "CAM_B",
            "right_cam": "CAM_C"
        }
    }
}

Note that this was done for the first revision of the baseboard that only had 3 cameras, but I did not expect this to effect whether or not I can get previews from the cameras.

I will create a new board config for the current baseboard, but I'm wondering if I need to set the "name" and "revision" properties to "DD2090" and "R7M1E7", respectively for things to work?

PS: How would this board config look if I were to use both B&C and A&D as stereo pairs, respectively? There does not seem to be such and example in the luxonis/depthai-boards.

jakaskerl · 6 Mar

MernokAdriaan

MernokAdriaan I will create a new board config for the current baseboard, but I'm wondering if I need to set the "name" and "revision" properties to "DD2090" and "R7M1E7", respectively for things to work?

Yes. Otherwise FW won't know which device it is. I wouldn't play around with it too much. Keep it as I posted above otherwise you might brick the device.

MernokAdriaan PS: How would this board config look if I were to use both B&C and A&D as stereo pairs, respectively? There does not seem to be such and example in the luxonis/depthai-boards.

I think just adding two sockets as -1 (origins) and then stereo pairs (eg. camD and CamC) toCam to that camera (camA and camB respectively).

Thanks,
Jaka

MernokAdriaan · 7 Mar

jakaskerl

Just to confirm, would the board config look like this, considering my cameras are arranged as C--D--A--B with 5cm spacing?

{
    "description": "Oculus 100, 2x OV9282 (NFOV), 2x OV9782(NFOV)",
    "board_config":
    {
        "name": "DD2090",
        "revision": "R7M1E7",
        "cameras":{
            "CAM_A": {
                "name": "rgb_right",
                "hfov": 68.7938,
                "type": "color"
            },
            "CAM_B": {
                "name": "mono_right",
                "hfov": 71.86,
                "type": "mono"
            },
            "CAM_C": {
                "name": "mono_left",
                "hfov": 71.86,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_B",
                    "specTranslation": {
                        "x": -15,
                        "y": 0,
                        "z": 0
                    },
                    "rotation":{
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            },
            "CAM_D": {
                "name": "rgb_left",
                "hfov": 68.7938,
                "type": "color",
                "extrinsics": {
                    "to_cam": "CAM_A",
                    "specTranslation": {
                        "x": -5,
                        "y": 0,
                        "z": 0
                    },
                    "rotation":{
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            }
        },
        "stereo_config":{
            "left_cam": "CAM_B",
            "right_cam": "CAM_C"
        }
    }
}

Seems unintuitive since we are still only defining the stereo config as B and C, even though we are using A & D in the same way. Also, this does not fully constrain the positioning of the sensors. I would have expected something like this since it fully constrains all sensors and configures both stereo pairs:

{
    "description": "Oculus 100, 2x OV9282 (NFOV), 2x OV9782(NFOV)",
    "board_config": {
        "name": "DD2090",
        "revision": "R7M1E7",
        "cameras": {
            "CAM_A": {
                "name": "rgb_right",
                "hfov": 68.7938,
                "type": "color"
            },
            "CAM_B": {
                "name": "mono_right",
                "hfov": 71.86,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_A",
                    "specTranslation": {
                        "x": 5,
                        "y": 0,
                        "z": 0
                    },
                    "rotation": {
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            },
            "CAM_C": {
                "name": "mono_left",
                "hfov": 71.86,
                "type": "mono",
                "extrinsics": {
                    "to_cam": "CAM_A",
                    "specTranslation": {
                        "x": -10,
                        "y": 0,
                        "z": 0
                    },
                    "rotation": {
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            },
            "CAM_D": {
                "name": "rgb_left",
                "hfov": 68.7938,
                "type": "color",
                "extrinsics": {
                    "to_cam": "CAM_A",
                    "specTranslation": {
                        "x": -5,
                        "y": 0,
                        "z": 0
                    },
                    "rotation": {
                        "r": 0,
                        "p": 0,
                        "y": 0
                    }
                }
            }
        },
        "stereo_config": [
            {
                "left_cam": "CAM_B",
                "right_cam": "CAM_C"
            },
            {
                "left_cam": "CAM_A",
                "right_cam": "CAM_D"
            }
        ]
    }
}

jakaskerl · 7 Mar

MernokAdriaan
Let's just leave this for now and make sure the setting does actually work - Does

"name": "DD2090",
"revision": "R7M1E7",

now activate CAM_D as well?

Next steps would be to define the .json file to use in https://docs.luxonis.com/hardware/platform/depth/calibration/.
You will need to do this for both stereo pair separately. The configs should append in the end calibration file.

MernokAdriaan "stereo_config": [
{
"left_cam": "CAM_B",
"right_cam": "CAM_C"
},

this is probably ignored.

ALSO
in your .json file, make sure extrinsics are correctly set to the right camera. CAM_C to B or the other way around, and CAM_D to A or swapped.

Thanks,
Jaka

MernokAdriaan · 9 Mar

jakaskerl

Yes, CAM_D is active as well. For now, this is good enough since I needed to validate the hardware design.

I'll update the calibration pipeline within the next week or so and revert back on a separate discussion if I encounter any issues.

Thanks for your support!