Please share the status of HW sync for OAK-FFC-4P + AR0234 based stereo pair

l4es · Oct 10, 2023

Dear Community,

This is the follow up of this ticket : https://discuss.luxonis.com/d/2008-hw-sync-trigger-mode-for-oak-ffc-4p-with-2-x-oak-ffc-ar0234cs-or-more

It has been a while that we started developing our own stereo camera based on OAK-FFC-4P + 3xOAK-FFC-AR0234. And the most expecting feature has always been the hardware synchronization of the AR0234 stereo pair due to the nature of the sensor : It works either in standalone or slave/external trigger mode.

We first looked at the https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames/#sensor-fsync-support

Then we already tested the following example as suggested by @erik but it did not work, then we realized that there was someone having the same issue : luxonis/depthai-core776

We also follow the bugs and issues reported in different repositories of Depthai, and from the most relevant ticket luxonis/depthai-python883, it seems promising that there's been firmware update with the improvement in the operation/stability in fsync input mode for AR0234 :

Yes, the firmware was just updated earlier today (develop branch, and we'll release soon 2.23.0.0), it improves the operation/stability in fsync input mode for AR0234, and other sensors as well.

Previously this error happened if the fsync signal was not being provided within some time (approx 1 second) after enabling the sensor, it should be no longer an issue now:
[system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'
(technical detail: MSS CPU was stuck handling continuously triggered MIPI DPHY error interrupts coming from the sensor, and couldn't communicate with the main CSS CPU, but didn't actually crash)

AR0234 sensors can be configured like:
camNodeA.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
...
However, if only AR0234 sensors are present on the device, they don't have the hardware capability to generate a sync signal (while OV9782/OV9282 have this option). So either the signal has to be provided externally, or RVC2 GPIOs can be configured to output it. We don't have this option by default in FW yet, but can be done through DeviceConfig, or a Script node for more options. I'll follow up with an example.

We are now waiting for the example of using one of RVC2's GPIOs for triggering the AR0234 externally for a better hardware synchronization, especially when it comes to use the i_publish_synced_rect_pair in the configuration of the camera's ROS driver for visual SLAM.

Thanks in advance and best regards,
Khang

erik · Oct 10, 2023

Hi l4es ,
Can you share some schematics/images of the wiring? Also, do you require it in snapshot mode or continuous mode (at fixed FPS)?

l4es · Oct 11, 2023

Hi erik ,
Below is our current setup of OAK FFC 4P + 3xAR0234 with the exception that we removed the OAK FFC IMX477 due to unsupported mix sensor/isp configuration :

As you can see, the additional yellow wires are for purpose of connecting the TRIG signal on the camera module side to the FSIN pad on the FFC cable side.
Beside that, we added another wire to connect the FSIN/TRIG of CAM_A to FSIN/TRIG of CAM_D (which is not present in the above picture as we took the photo before adding it) in order to have HW sync of all the 3 AR0234 cameras (streo + RGB).

Also, do you require it in snapshot mode or continuous mode (at fixed FPS)?

I would be ideal to use the continuous mode, but it seems not the case of AR0234 sensor, therefore snapshot more is the only option. Am I right?

Update :
From the following discussion https://discuss.luxonis.com/d/1327-best-suggested-choice-for-fsin-gpio-pin-as-of-now, the new revision of OAK-D LR (BC2087 R1M0E1) uses PWM-capable MXIO46 as COM_AUX_IO2 to drive the FSIN pins (or TRIGGER pins of the AR0234CS).

In the case of OAK FFC 4P (DD2090 4CAM FFC R7M1E7), it uses the MXIO13 as COM_AUX_IO2 to drive the FSIN pins (i.e. TRIGGER pins of the AR0234CS if manually soldered). My questions :
a. Is MXIO13 PWM-capable ?
b. How to use it to drive the FSIN/TRIGGER pins of all the AR0234 ? By replacing the GPIO_PIN=41 # Trigger to GPIO_PIN=13 # Trigger in your example ? I remember we did replace but it did not work.
c. Do the following lines put the AR0234 into slave mode in order to be driven by FSIN?

monoLeftorRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
monoLeftorRight.initialControl.setExternalTrigger(4,3)

This question is also important because as a work-around, we plan to use a PWM capable GPIO from the host CPU board (Nvidia Jetson) to generate the pules with required frequency for the external triggering. I also suppose that the TRIGGER signal of the OAK FFC AR0234-M12 is 1.8v logic level. Can you confirm this as well ?

Best Regards,
Khang

erik · Oct 11, 2023

Hi l4es ,
On the OAK-FFC-4P you'd also need to put GPIO38 on high (for FFC-4P R6 or higher), as per schematics below (and docs here). Looking at SOM pin specs, neither of these (mxio 40/41/42) are PWM-capable, while mxio 46 is, but on the FFC-4P it's connected to the IMU BOOTN line. That's correct, it's 1V8 voltage lvl.

(DD2090 4CAM FFC R7M1E7), it uses the MXIO13 as COM_AUX_IO2

I don't think that's correct, it's correct for SOM-PRO-S3 (RVC3/Keembay).

l4es · Oct 11, 2023

Hi erik ,

It looks like you were focusing on the FSIN_MODE_SELECT, which is use to connect all FSIN signals together by driving the NC7SZ66M5X. And this FSIN_MODE_SELECT is NOT used for generating the pulse for external triggering of the AR0234.
The one that is used to generate the external triggering pulse in my opinion is COM_AUX_IO2, which is MXIO13 on the OAK FFC 4P (DD2090 4CAM FFC R7M1E7). The reason is that, if one does not need to connect all FSIN signals together but only needs to generate the external triggering pulse for the stereo pair (CAM_B and CAM_C) only, then we would only need to do that with COM_AUX_IO2.

On the OAK-FFC-4P you'd also need to put GPIO38 on high (for FFC-4P R6 or higher), as per schematics below (and docs here)

Also, the schematic in the link to the docs seems not up to date with the revision R7M1E7 that I have : at first glance, they differ in the naming of the signals (MXIO_xx vs IOxx) and FSIN_MODE_SELECT is on IO42/ETH_PHY_CTL_TX for the revision R7M1E7 instead of MXIO_38 in your version (which matches the R2M1E2 I would guess), unless the DD2090FFC_DepthAI_4-camera_FFC is NOT the OAK FFC 4P.

And I haven't found the mapping of COM_AUX_IO2 to any MXIO_xx, but only IO13/CAMFS_2L in the R7M1E7.

Best Regards,
K.

erik · Oct 11, 2023

Hi l4es ,

FSIN_MODE_SELECT is NOT used for generating the pulse

Correct. But since you have 3x AR0234 sensors, you'd likely want to connect both 4lane and 2lane MIPI FSIN together, so all 3 AR0234 are triggered when a signal is applied.

The one that is used to generate the external triggering pulse in my opinion is COM_AUX_IO2, which is MXIO13 on the OAK FFC 4P

External trigger.. as in supplied by the SOC, or completely external? Note that IO13 in the schematics linked is for Keembay (see the name of the connector on the schematics, it's ... SOM-KB_A), not Myriad X, so it's actually still MXIO41 (2lane fsync pin).

erik · Oct 11, 2023

And I haven't found the mapping of COM_AUX_IO2 to any MXIO_xx, but only IO13/CAMFS_2L in the R7M1E7.

(checking internally if we can share SOM IO descriptions/mappings as well)

l4es · Oct 11, 2023

Hi erik,

But since you have 3x AR0234 sensors, you'd likely want to connect both 4lane and 2lane MIPI FSIN together, so all 3 AR0234 are triggered when a signal is applied.

Got this. But I am focusing on the hardware sync of the stereo pair by generating the triggering pulse on COM_AUX_IO2 in the first place.

External trigger.. as in supplied by the SOC, or completely external?

It would be ideal if we could use the external triggering for the AR0234 from the COM_AUX_IO2 of RVC2. We are looking for doing this, with your support.
But if it does not work due to some dependency to the firmware of the RVC2 within the OAK FFC 4P for example (seeing the relevant bugs relevant to hardware sync), we would have to trigger the all the AR0234 more externally from a GPIO pin of a separate host (in our case the Nvidia Jetson that the OAK FFC 4P connects to). Is it possible according to you?

Note that IO13 in the schematics linked is for Keembay (see the name of the connector on the schematics, it's ... SOM-KB_A), not Myriad X, so it's actually still MXIO41 (2lane fsync pin).

I also noticed that both IO13/CAMFS_2L pin in R7M1E7 and COM_AUX_IO2 in R2M1E2 are on the same connector's pin 42. Maybe the carrier board of OAK FFC 4P can also be used with the RVC3 SoM (SoM Pro S3) then the schematic uses signals name of Keembay. But it is a little bit confusing for us using the default RVC2 Myriad X on the top to the carrier board.

(checking internally if we can share SOM IO descriptions/mappings as well)

Please help! You already shared the mapping of the FSIN_MODE_SELECT. I would also need the mapping of COM_AUX_IO2 in order to generate the trigger signal/pulse from it to the AR0234 sensor (by replacing the GPIO_PIN=41 # Trigger in your example).

Best Regards,
Khang

erik · Oct 11, 2023

Hi l4es ,
COM_AUX_IO2 is mapped to GPIO 41, you can scope the signal to verify. I am not sure why it wouldn't trigger, perhaps there's something with the pulse length (too long?), I'm checking with the Firmware team on how we accomplished FSYNC signal generating on OAK-D-LR (where we have 3x AR0234).

l4es · Oct 12, 2023

Hi erik ,

COM_AUX_IO2 is mapped to GPIO 41, you can scope the signal to verify.

So you confirmed that COM_AUX_IO2 is MXIO_41, and I can use your example as-is?

I am not sure why it wouldn't trigger, perhaps there's something with the pulse length (too long?)

I'd modified it to GPIO_PIN=13 # Trigger based on the signal name IO13/CAMFS_2L appearing in R7M1E7 (which is for Keembay RVC3 according to you) and that might be the reason.

Beside that, we added another wire to connect the FSIN/TRIG of CAM_A to FSIN/TRIG of CAM_D (which is not present in the above picture as we took the photo before adding it) in order to have HW sync of all the 3 AR0234 cameras (streo + RGB).

We hard-wired the the FSIN/TRIG of CAM_A and FSIN/TRIG of CAM_D together (like this) so there will not be the need of driving FSIN_MODE_SELECT (MXIO_38 or IO42/ETH_PHY_CTL_TX in R7M1E7). Can you confirm?

erik · Oct 12, 2023

Hi l4es ,

and I can use your example as-is?

No, I haven't tried it with AR0234, so I can't confirm that.

so there will not be the need of driving FSIN_MODE_SELECT

I see, then you are correct.

We will be testing with 3x AR0234 and GPIO triggering this/next week and let you know.

l4es · Oct 13, 2023

Hi erik,

We will be testing with 3x AR0234 and GPIO triggering this/next week and let you know.

Looking forward to having your update about this subject.

In case we expect to drive the trigger signal from a host other than the RVC2 for a quick work-around while waiting for your test (and potentially some update in the device's firmware), we would need to ensure that the sensors are in Trigger Mode .
Therefore, I would like to know what DepthAI's CameraControl APIs or their combination that allow(s) the AR0234CS entering :

The (Pulse/Automatic) Trigger Mode, or
The Multi−sensor Sync Mode

Please note that both modes seem relevant to the register 0x301A (bits 2, 8, 11) :

and additionally relevant to the register 0x30CE (bit 8) :

Update : We re-tested your example (by keeping GPIO_PIN=41 # Trigger) and did see the triggering pulses on the TRIG pin of 3xAR0234CS sensors (in the attached video, I only probed the TRIG pins of the stereo pair) :

ar0234cs-trigger.mp4

3MB

But :

With depthai v2.22.0 : Only left camera (CAM_B) showed live images/stream, other cameras did not show anything (no other window popup),
With depthai v2.23.0 : Left camera (CAM_B) showed first sequence(s) of image(s) but then froze, right camera (CAM_C) showed live images/stream :

got queue for left
got queue for right
got frame left, seq 0, ts 23951.337242
got frame right, seq 0, ts 23952.350503
got frame right, seq 1, ts 23952.448778
got frame right, seq 2, ts 23952.515428
got frame right, seq 3, ts 23952.582079

Best Regards,
Khang

HiepNguyen · Oct 16, 2023

Hi,
I'm using the OAK-FFC-4P + AR0234 as well, and I have the issue with Rectified Left and Right get sequence number mismatch. I suspect that relates to HW sync, therefore, I comment on my issue here, instead of creating a new ticket.

Testing on OAK-FFC-4P + AR0234 stereo pair with DepthAI v2.23.0.0, the Rectified Left frame is faster than Rectified Right frame 1 sequence number, but the timestamp is synchronized very well.

frameRectLeft seq: 1, ts: 5:46:22.442598
frameRectRight seq: 0, ts: 5:46:22.442610
------------------- 
frameRectLeft seq: 2, ts: 5:46:22.475933
frameRectRight seq: 1, ts: 5:46:22.475954
------------------- 
frameRectLeft seq: 3, ts: 5:46:22.509263
frameRectRight seq: 2, ts: 5:46:22.509279

Testing on OAK-D Pro, the Rectified Left frame is matching with Rectified Right frame.

frameRectLeft seq: 0, ts: 5:57:52.572746
frameRectRight seq: 0, ts: 5:57:52.572757
------------------- 
frameRectLeft seq: 1, ts: 5:57:52.606080
frameRectRight seq: 1, ts: 5:57:52.606092
------------------- 
frameRectLeft seq: 2, ts: 5:57:52.639413
frameRectRight seq: 2, ts: 5:57:52.639436

Because the depthai-ros uses getSequenceNum to check whether left and right synchronized frames are synchronized or not, so this is a bottleneck to my VIO/SLAM development. Hope to have a hands-on experience with HW Synchronization for the OAK-FFC-4P + AR0234 stereo pair soon.

Best regards,
Hiep Nguyen

erik · Oct 16, 2023

Hi HiepNguyen ,
These are actually perfectly synced, there's only sequence number mismatch, which is expected. I'd suggest writing an Issue on depthai-ros so we add support for timestamp syncing as well (not only sequenceNum-based syncing).

HiepNguyen · Oct 17, 2023

Hi erik ,

Thank you for your comment. I wrote a feature request to depthai-ros to support for timestamp syncing.

l4es · Oct 20, 2023

Dear erik ,

We will be testing with 3x AR0234 and GPIO triggering this/next week and let you know.

Would you have any update to share on this, please ?

Best Regards,
Khang

LLuxonis-Alex · Oct 20, 2023

@l4es I added an example here: luxonis/depthai-python883, also pasted below

Modified cam_test.py:
https://github.com/luxonis/depthai-python/compare/main...alex-luxonis:depthai-python:fsync_gpio_ffc_4p
(alex-luxonis/depthai-pythonedc5666)

Mainly added a new Script node with the content:

import time
import GPIO

# Script static arguments
fps = %f

calib = Device.readCalibration2().getEepromData()
prodName  = calib.productName
boardName = calib.boardName
boardRev  = calib.boardRev

node.warn(f'Product name  : {prodName}') 
node.warn(f'Board name    : {boardName}') 
node.warn(f'Board revision: {boardRev}')

revision = -1
# Very basic parsing here, TODO improve
if len(boardRev) >= 2 and boardRev[0] == 'R':
    revision = int(boardRev[1])
node.warn(f'Parsed revision number: {revision}')

# Defaults for OAK-FFC-4P older revisions (<= R5)
GPIO_FSIN_2LANE = 41  # COM_AUX_IO2
GPIO_FSIN_4LANE = 40
GPIO_FSIN_MODE_SELECT = 6  # Drive 1 to tie together FSIN_2LANE and FSIN_4LANE

if revision >= 6:
    GPIO_FSIN_2LANE = 41  # still COM_AUX_IO2, no PWM capable
    GPIO_FSIN_4LANE = 42  # also not PWM capable
    GPIO_FSIN_MODE_SELECT = 38  # Drive 1 to tie together FSIN_2LANE and FSIN_4LANE
# Note: on R7 GPIO_FSIN_MODE_SELECT is pulled up, driving high isn't necessary (but fine to do)

# GPIO initialization
GPIO.setup(GPIO_FSIN_2LANE, GPIO.OUT)
GPIO.write(GPIO_FSIN_2LANE, 0)

GPIO.setup(GPIO_FSIN_4LANE, GPIO.IN)

GPIO.setup(GPIO_FSIN_MODE_SELECT, GPIO.OUT)
GPIO.write(GPIO_FSIN_MODE_SELECT, 1)

period = 1 / fps
active = 0.001

node.warn(f'FPS: {fps}  Period: {period}')

withInterrupts = False
if withInterrupts:
    node.critical(f'[TODO] FSYNC with timer interrupts (more precise) not implemented')
else:
    overhead = 0.003  # Empirical, TODO add thread priority option!
    while True:
        GPIO.write(GPIO_FSIN_2LANE, 1)
        time.sleep(active)
        GPIO.write(GPIO_FSIN_2LANE, 0)
        time.sleep(period - active - overhead)

And configured the enabled cameras in sync mode:

# Num frames to capture on trigger, with first to be discarded (due to degraded quality)
# Note: Only OV9282/9782 supports dropping frames. For AR0234 the arguments passed here have no effect
# Note2: in this mode it's best to set sensor FPS as max supported (e.g 60) to avoid missed triggers,
# as exposure starts immediately after trigger and is not overlapped with previous frame MIPI readout
#cam[c].initialControl.setExternalTrigger(1, 0)

# Note: setFrameSyncMode takes priority over setExternalTrigger (if both are set)
cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

Getting with 3 AR0234 attached on A,B,C ports and cable FSIN soldered to TRIGGER pin of Arducam module
python3 utilities/cam_test.py -cams cama,c camb,c camc,c -cres 1200 -ds 2 -fps 30 -show

[mxid] [3.1] [4.141] [Script(8)] [warning] Product name  : OAK-FFC 4P
[mxid] [3.1] [4.141] [Script(8)] [warning] Board name    : DD2090
[mxid] [3.1] [4.141] [Script(8)] [warning] Board revision: R5M1E5
[mxid] [3.1] [4.141] [Script(8)] [warning] Parsed revision number: 5
[mxid] [3.1] [4.142] [Script(8)] [warning] FPS: 30.0  Period: 0.03333333333333333
...
FPS:  28.10| 28.30  28.30| 28.30  28.29| 28.30 
[cama ,  100, 1159028.071043] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5386 K
[camb ,  100, 1159028.071054] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,   99, 1159028.071063] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.43| 28.38  28.43| 28.38  28.43| 28.38 
[cama ,  101, 1159028.104990] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5380 K
[camb ,  101, 1159028.105001] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,  100, 1159028.105012] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.35| 28.38  28.40| 28.38  28.48| 28.38 
[cama ,  102, 1159028.139839] Exp: 30.000 ms, ISO:  198, Lens pos:  -1, Color temp: 5374 K
[camb ,  102, 1159028.139851] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
[camc ,  101, 1159028.139862] Exp: 30.000 ms, ISO:  177, Lens pos:  -1, Color temp: 5351 K
FPS:  28.54| 28.38  28.36| 28.38  28.37| 28.38