HiepNguyen

HiepNguyen · Dec 13, 2023

To make it clear, I want to align the depth output of the stereo pair with the other camera. It's similar to OAK-D LR camera use case, I believe so.

For example : with 3 RGB AR0234 sensors, I expect to have 3 baselines : 5cm, 10cm and 15cm from the camera, and the depth output from stereo pair is aligned with the other RGB camera. Current status :

The setup as normal OAK-D camera with stereo pair in 2 side, RGB camera in the middle, then the depth output from stereo pair can be aligned with RBG camera.
With RGB camera in the side, 2 sensors left is stereo pair, then the depth output from stereo pair can't be aligned with RGB camera, with error logs:

[14442C10D11AE7D600] [1.1] [3.203] [StereoDepth(3)] [error] DepthAlign is only implemented for RGB camera and LEFT and RIGHT stereo inputs!
terminate called after throwing an instance of 'std::runtime_error'
  what():  Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'rgb' (X_LINK_ERROR)'
Aborted (core dumped)

Could you please help to confirm that the OAK-D LR works as my expectation, and do the OAK-FFC-4P work same as OAK-D LR multiple baseline functionality ?

Many thanks in advance !

Best regards,
Hiep Nguyen

HiepNguyen · Dec 7, 2023

Hi Luxonis experts,

I'm using the OAK-FFC-4P with 3 AR0234 sensors, and configure the the OAK-FFC-4P for multiple baseline : 5cm, 10cm and 15cm.
- With baseline 15cm, the camera left is CAM_B, the camera right is CAM_C and the RGB camera is CAM_A. In this case, the stereo depth alignment to CAM_A works normally.

However, with baseline 10cm, the camera left is CAM_A, the camera right is CAM_C and the RGB camera is CAM_B, then the stereo depth alignment to CAM_B error with logs below. It's same for baseline 5cm, where left camera is CAM_B, right camera is CAM_A and the RGB camera is CAM_C.

[14442C10D11AE7D600] [1.1] [3.203] [StereoDepth(3)] [error] DepthAlign is only implemented for RGB camera and LEFT and RIGHT stereo inputs!
terminate called after throwing an instance of 'std::runtime_error'
  what():  Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'rgb' (X_LINK_ERROR)'
Aborted (core dumped)

Look like the firmware have not supported the DepthAlign to other socket then RGB camera (CAM_A) yet. Am I correct ?
Could you please guide me how to fix this issue ? Many thank in advance !

Best regards,
Hiep Nguyen

HiepNguyen · Dec 6, 2023

Hi @jakaskerl ,

I'm facing the same issue, when using then OAK-D camera with Stella SLAM. With the default OAK-D camera intrinsics, the output of Stella SLAM is wrong. Then I used OpenCV calibration tools to calibrate the OAK-D camera and use that output for Stella SLAM, the output is pretty correct.

With Intel Realsense camera intrinsics, the Stella SLAM works normally.

Therefore, I don't know what is different between default OAK-D camera intrinsics and OpenCV calibration tools, because the OAK-D calibration tools use the same functions as OpenCV calibration tools.

Can you explain the scale and how can I re-scale these OAK-D camera intrinsics, to get the same result as OpenCV calibration tools ?

Many thanks in advance !

HiepNguyen · Nov 2, 2023

Hi Luxonis experts,

I'm using the OAK-FFC-4P with the BN0086 IMU inside. I'm finding the IMU document about BN0086 noises, as they are necessary for inputs for Kalibr Camera-IMU calibration. I found BN0086 specifications but none of them reveal anything about the IMU noises.

Noise parameters needed for Kalibr Camera-IMU calibration are :

#Accelerometers
accelerometer_noise_density:   #Noise density (continuous-time)
accelerometer_random_walk:     #Bias random walk
#Gyroscopes
gyroscope_noise_density:       #Noise density (continuous-time)
gyroscope_random_walk:         #Bias random walk

I'm integrating the OAK-FFC-4P with 3 AR0234 to Nvidia Isaac ROS Visual SLAM, it also needs above parameters to launch the Visual SLAM application.

I'm looking forward to seeing your response. Many thanks in advance !

Best regards,
Hiep Nguyen

HiepNguyen · Oct 18, 2023

Hi jakaskerl ,

Thank you very much, that makes sense to me.

Best regards,
Hiep Nguyen

HiepNguyen · Oct 18, 2023

Hi jakaskerl ,

Thank you for your reply. I'm a bit unclear with your response.

So let's take cam0's T_cam_imu, corresponding to OAK-D Cam left, as an example.

The cam0's T_cam_imu format are :

cam0:
  T_cam_imu:
    [-0.001827859233177992, -0.999909119974655, 0.013357047657805154, 0.030298330137844887,
    0.0039387461853423456, -0.013364165204938133, -0.9999029379729119, 0.028682372533570984,
    0.9999905725601166, -0.0017750717969459706, 0.003962816044752765, -0.014747523214162677,
    0.0, 0.0, 0.0, 1.0]

The default imuExtrinsics matrixes format are :

    "imuExtrinsics": {
        "rotationMatrix": [
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ],
            [
                0.0,
                0.0,
                0.0
            ]
        ],
        "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "toCameraSocket": -1,
        "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        }
    },

The cam0's T_cam_imu is in cam->imu, hence I need to inverse it, to have matrix imu->cam. Am I correct ?

cam0_T_cam_imu = np.array(T_cam_imu)
inverse = np.linalg.inv(cam0_T_cam_imu)

Asume I have the inverse matrix, it is in 4x4 matrix, so I take 3 first columns and fill to rotationMatrix of imuExtrinsics ? Am I correct ? And of course, the toCameraSocket=-1 is change to toCameraSocket=1 to corresponding to OAK-D left camera number.

Please correct me if I'm wrong. Thanks in advance !

Best regards,
Hiep Nguyen

HiepNguyen · Oct 17, 2023

Hi Luxonis experts,

I'm using Kalibr to do the IMU to Camera calibration, and have the result file camchain-imu cam.yaml

cam0:
  T_cam_imu:
    [-0.001827859233177992, -0.999909119974655, 0.013357047657805154, 0.030298330137844887,
    0.0039387461853423456, -0.013364165204938133, -0.9999029379729119, 0.028682372533570984,
    0.9999905725601166, -0.0017750717969459706, 0.003962816044752765, -0.014747523214162677,
    0.0, 0.0, 0.0, 1.0]
...
cam1:
  T_cam_imu:
    [-0.008382154257505742, -0.9999532504652721, 0.00482041221684304, -0.08958472628850654,
    0.006913259577842834, -0.004878415733911984, -0.9999642033102668, 0.028854934277130317,
    0.99994097142382, -0.00834852944320319, 0.006953828019388014, -0.015662499900516994,
    0.0, 0.0, 0.0, 1.0]
  T_cn_cnm1:
    [0.9999420825622907, 0.008510580920464224, -0.006587984015209744, -0.12022256172433968,
    -0.008491233601698718, 0.9999595702862754, 0.0029591797526304676, 0.0004746321347084293,
    0.006612902003644948, -0.0029030682532840716, 0.9999739205208336, -0.001032454297023307,
    0.0, 0.0, 0.0, 1.0]
...

My question is: how do I convert the T_cam_imu matrixes from the Kalibr results to the imuExtrinsics of DepthAI calibration format , and how to flash the result to *EEPROM.
*
I'm looking forward to seeing your response soon. Many thanks in advance!

Best regards,
Hiep Nguyen

HiepNguyen · Oct 17, 2023

Hi erik ,

Thank you for your comment. I wrote a feature request to depthai-ros to support for timestamp syncing.

HiepNguyen · Oct 16, 2023

Hi,
I'm using the OAK-FFC-4P + AR0234 as well, and I have the issue with Rectified Left and Right get sequence number mismatch. I suspect that relates to HW sync, therefore, I comment on my issue here, instead of creating a new ticket.

Testing on OAK-FFC-4P + AR0234 stereo pair with DepthAI v2.23.0.0, the Rectified Left frame is faster than Rectified Right frame 1 sequence number, but the timestamp is synchronized very well.

frameRectLeft seq: 1, ts: 5:46:22.442598
frameRectRight seq: 0, ts: 5:46:22.442610
------------------- 
frameRectLeft seq: 2, ts: 5:46:22.475933
frameRectRight seq: 1, ts: 5:46:22.475954
------------------- 
frameRectLeft seq: 3, ts: 5:46:22.509263
frameRectRight seq: 2, ts: 5:46:22.509279

Testing on OAK-D Pro, the Rectified Left frame is matching with Rectified Right frame.

frameRectLeft seq: 0, ts: 5:57:52.572746
frameRectRight seq: 0, ts: 5:57:52.572757
------------------- 
frameRectLeft seq: 1, ts: 5:57:52.606080
frameRectRight seq: 1, ts: 5:57:52.606092
------------------- 
frameRectLeft seq: 2, ts: 5:57:52.639413
frameRectRight seq: 2, ts: 5:57:52.639436

Because the depthai-ros uses getSequenceNum to check whether left and right synchronized frames are synchronized or not, so this is a bottleneck to my VIO/SLAM development. Hope to have a hands-on experience with HW Synchronization for the OAK-FFC-4P + AR0234 stereo pair soon.

Best regards,
Hiep Nguyen

HiepNguyen · Aug 16, 2023

Hi admin,

I used 12V-3A power supply to connect to OAK-D version 1, look like it's overheating and have some smell.
Now I connected the OAK-D with 5V-3A power supply, the camera can't get power source.
Could you please help to guide me how to save the OAK-D camera ?

Many thanks in advance !

P/s : check the dmesg message, it shows:
[ 218.264316] usb usb2-port3: over-current condition

[ 218.264316] usb usb2-port3: over-current condition

Best regards,
Hiep Nguyen