FFC-4p hardware synchronization

LLuxonis-Alex · Oct 12, 2022

The B & C (or Left & Right) ports are synced by sharing the same I2C bus. For the ports A & D (RGB and CamD) this is not the case, and even though their FSIN pins are tied together, hardware sync has to be explicitly enabled. Can be done for example by configuring in the pipeline:

cam['rgb']  .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
cam['camd'] .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

And to get all 4 sensors synchronized, need to configure also FSIN for L/R sensors:

cam['left'] .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
cam['right'].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

AND importantly to tie the FSIN signals of A+D and B+C pairs, by setting a GPIO:

# OAK-FFC-4P requires driving GPIO6 high (FSIN_MODE_SELECT) to connect together
# the A+D FSIN group (4-lane pair) with the B+C group (2-lane pair)
config = dai.Device.Config()
config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT,
                                            dai.BoardConfig.GPIO.Level.HIGH)

with dai.Device(config) as device:
    device.startPipeline(pipeline)

Can use this commit as a reference for the changes to be made:
https://github.com/alex-luxonis/depthai-python/commit/7db4ef58d3ca8b9b84ea0dca6a80e3e382fc2d1a

RRBJin · Oct 12, 2022

The delay is still greater than 1ms. I used the2.17.3.1 version of depthai. Do I need to switch to a specific version?

import depthai as dai
import time
import cv2
import collections

set_fps = 30

class FPS:
    def __init__(self, window_size=30):
        self.dq = collections.deque(maxlen=window_size)
        self.fps = 0

    def update(self, timestamp=None):
        if timestamp == None: timestamp = time.monotonic()
        count = len(self.dq)
        if count > 0: self.fps = count / (timestamp - self.dq[0])
        self.dq.append(timestamp)

    def get(self):
        return self.fps


# cam_list = ['left', 'right']
# cam_socket_opts = {
#     'left' : dai.CameraBoardSocket.LEFT,  # Or CAM_B
#     'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C
# }

cam_list = ['rgb', 'left', 'right', 'camd']
cam_socket_opts = {
    'rgb'  : dai.CameraBoardSocket.RGB,   # Or CAM_A
    'left' : dai.CameraBoardSocket.LEFT,  # Or CAM_B
    'right': dai.CameraBoardSocket.RIGHT, # Or CAM_C
    'camd' : dai.CameraBoardSocket.CAM_D,
}

pipeline = dai.Pipeline()
cam = {}
xout = {}
for c in cam_list:
    cam[c] = pipeline.create(dai.node.MonoCamera)
    cam[c].setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
    if c == 'rgb':
        cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
    else:
        cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
    cam[c].setBoardSocket(cam_socket_opts[c])
    xout[c] = pipeline.create(dai.node.XLinkOut)
    xout[c].setStreamName(c)
    cam[c].out.link(xout[c].input)


config = dai.Device.Config()
config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT,
                                            dai.BoardConfig.GPIO.Level.HIGH)

with dai.Device(config) as device:
    device.startPipeline(pipeline)
    q = {}
    fps_host = {}  # FPS computed based on the time we receive frames in app
    fps_capt = {}  # FPS computed based on capture timestamps from device
    for c in cam_list:
        q[c] = device.getOutputQueue(name=c, maxSize=1, blocking=False)
        cv2.namedWindow(c, cv2.WINDOW_NORMAL)
        cv2.resizeWindow(c, (640, 480))
        fps_host[c] = FPS()
        fps_capt[c] = FPS()

    while True:
        frame_list = []
        for c in cam_list:
            pkt = q[c].tryGet()
            if pkt is not None:
                fps_host[c].update()
                fps_capt[c].update(pkt.getTimestamp().total_seconds())
                print(c+":",pkt.getTimestampDevice())
                frame = pkt.getCvFrame()
                cv2.imshow(c, frame)
        print("-------------------------------")
        # print("\rFPS:",
        #       *["{:6.2f}|{:6.2f}".format(fps_host[c].get(), fps_capt[c].get()) for c in cam_list],
        #       end='', flush=True)

        key = cv2.waitKey(1)
        if key == ord('q'):
            break

LLuxonis-Alex · Oct 12, 2022

@RBJin Are your camera modules produced by Luxonis, or by Arducam? For the latter case, if they are used with Arducam UC-796 FFC cables, the FSIN pin isn't actually reaching the camera, it ends on the FFC cable as a testpoint.
You could also post a picture with the setup, we'll confirm and see about potential workarounds (may need soldering wires to interconnect the sync pins).

RRBJin · Oct 13, 2022

@Luxonis-Alex The camera module I use is the Arducam OV9282 with UC-796 FFC cable.

Do I just need to connect the FSIN testpoint on each cable to the FSIN pin on the corresponding camera board with a jump wire? As shown in the red circle. Is there anything else I need to do?

LLuxonis-Alex · Oct 13, 2022

Yes, that would be a way to fix. You can use some thin wire, to reduce the risk of tearing the testpoint off the FFC cable.

Or for a simpler and less riskier way, as the signal is generated by one camera and doesn't need to reach the OAK SoM, you can just interconnect the FSIN pins from all the cameras together, like in the picture below (can also solder header pins on the camera PCB and use jumper cables). In this case, no need to configure GPIO 6, but just sensor FSIN pins to be in OUTPUT or INPUT mode.

RRBJin · Oct 14, 2022

@Luxonis-Alex I chose the less risker way. Now the delay is less than 1ms successfully. Thank you so much!

Asa · Apr 20, 2023

Is the timing between the camera triggers and the triggering of reads from internal imu etc fixed in this case? What manages the triggering of the first camera(which drives sync to all the other camers) relative to imu reads?

erik · Apr 21, 2023

Hi Asa , I don't think IMU/frames are hardware synced. But having IMU at higher frequency (eg 5x higher), even timestamp syncing could be good enough. More information here. Thoughts?

Asa · Apr 24, 2023

State estimation systems like to have nice hardware synchronization. If you can trigger the imu at multiples of the images this would help build these kind of tightly coupled systems.

thebambuproject · Apr 25, 2023

How to control GPIO with C++ API? I´m talking about this part:

config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT, dai.BoardConfig.GPIO.Level.HIGH)

Thanks!

erik · Apr 25, 2023

thebambuproject Would it not be the same? C++/Python API should be 1:1, as Python just has bindings over.

thebambuproject · Apr 25, 2023

Sorry but its not trivial to me

dai::Device device(pipeline);
dai::Device::Config c = pipeline.getDeviceConfig();
c.board.gpio[6] = dai::BoardConfig::GPIO(dai::BoardConfig::GPIO::OUTPUT, dai::BoardConfig::GPIO::Level::HIGH);

Can somebody point me, its ok? config seems to be private… Thanks

jakaskerl · Apr 25, 2023

Hi thebambuproject

I'd change it to:

dai::Device device(pipeline);
dai::Device::Config c = pipeline.getDeviceConfig();
c.board.gpio[6] = dai::BoardConfig::GPIO(dai::BoardConfig::GPIO::Direction::OUTPUT, dai::BoardConfig::GPIO::Level::HIGH);

Otherwise it looks good (i'm not familiar with c++).

Hope it works,
Jaka

Aaiyangsky · Mar 16, 2024

@Luxonis-Alex Hi, I also tried to sync 4 cameras in FFC-4P, but only camera B and camera C synced successfully anyway.The camera uses a UC-788 module with Arducam OV9282 and the FSIN signal is Connect correctly.

Here's my code

#include "depthai/depthai.hpp"
#include <chrono>
#include <opencv2/opencv.hpp>
#include <time.h>
#include <iostream>
#include <thread>

int main()
{
    // Create pipeline
    dai::Pipeline pipeline;

    // Define sources and outputs
    // cam_a
    auto camA = pipeline.create<dai::node::MonoCamera>();
    camA->setBoardSocket(dai::CameraBoardSocket::CAM_A);
    camA->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    camA->setFps(30);
    camA->initialControl.setFrameSyncMode(dai::CameraControl::FrameSyncMode::OUTPUT);

    auto AOut = pipeline.create<dai::node::XLinkOut>();
    AOut->setStreamName("CAM_A");
    camA->out.link(AOut->input);

    // cam_d
    auto camD = pipeline.create<dai::node::MonoCamera>();
    camD->setBoardSocket(dai::CameraBoardSocket::CAM_D);
    camD->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    camD->setFps(30);
    camD->initialControl.setFrameSyncMode(dai::CameraControl::FrameSyncMode::INPUT);

    auto DOut = pipeline.create<dai::node::XLinkOut>();
    DOut->setStreamName("CAM_D");
    camD->out.link(DOut->input);

    // cam_b
    auto camB = pipeline.create<dai::node::MonoCamera>();
    camB->setBoardSocket(dai::CameraBoardSocket::CAM_B);
    camB->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    camB->setFps(30);
    camB->initialControl.setFrameSyncMode(dai::CameraControl::FrameSyncMode::INPUT);

    auto BOut = pipeline.create<dai::node::XLinkOut>();
    BOut->setStreamName("CAM_B");
    camB->out.link(BOut->input);

    // cam_c
    // auto camC = pipeline.create<dai::node::MonoCamera>();
    // camC->setBoardSocket(dai::CameraBoardSocket::CAM_C);
    // camC->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
    // camC->initialControl.setFrameSyncMode(dai::CameraControl::FrameSyncMode::INPUT);

    // auto COut = pipeline.create<dai::node::XLinkOut>();
    // COut->setStreamName("CAM_C");
    // camC->out.link(COut->input);

    dai::BoardConfig Config = pipeline.getBoardConfig();
    Config.gpio[6] = dai::BoardConfig::GPIO(dai::BoardConfig::GPIO::OUTPUT, dai::BoardConfig::GPIO::Level::HIGH);
    pipeline.setBoardConfig(Config);

    dai::Device device(pipeline);

    auto lastTime = std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::system_clock::now().time_since_epoch()).count();
    auto currTime = std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::system_clock::now().time_since_epoch()).count();

    cv::Mat AMat, BMat, CMat, DMat;
    cv::Mat ADMat, BCMat;

    auto imageA = device.getOutputQueue("CAM_A", 1, false);
    auto imageB = device.getOutputQueue("CAM_B", 1, false);
    // auto imageC = device.getOutputQueue("CAM_C", 1, false);
    auto imageD = device.getOutputQueue("CAM_D", 1, false);

    while (true)
    {
        auto A = imageA->get<dai::ImgFrame>();
        auto B = imageB->get<dai::ImgFrame>();
        // auto C = imageC->get<dai::ImgFrame>();
        auto D = imageD->get<dai::ImgFrame>();

        AMat = A->getCvFrame();
        BMat = B->getCvFrame();
        // CMat = C->getCvFrame();
        DMat = D->getCvFrame();

        auto tsA = A->getTimestampDevice();
        auto tsB = B->getTimestampDevice();
        // auto tsC = C->getTimestampDevice();
        auto tsD = D->getTimestampDevice();

        auto timeDifference = tsA - tsB;
        auto timeDifferenceUs = std::chrono::duration_cast<std::chrono::microseconds>(timeDifference).count();
        std::cout << "Time difference between messages is: " << std::abs(timeDifferenceUs / 1000.0) << " ms" << std::endl;

        // timeDifference = tsC - tsB;
        // timeDifferenceUs = std::chrono::duration_cast<std::chrono::microseconds>(timeDifference).count();
        // std::cout << "Time difference between messages is: " << std::abs(timeDifferenceUs / 1000.0) << " ms" << std::endl;

        timeDifference = tsD - tsB;
        timeDifferenceUs = std::chrono::duration_cast<std::chrono::microseconds>(timeDifference).count();
        std::cout << "Time difference between messages is: " << std::abs(timeDifferenceUs / 1000.0) << " ms" << std::endl;

        currTime = std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::system_clock::now().time_since_epoch()).count();
        auto Ts = currTime - lastTime;
        std::cout << "read 1 count;Ts = " << Ts / (1000.0f) << "us" << std::endl;
        lastTime = currTime;

        int key = cv::waitKey(1);
        if (key == 'q')
        {
            break;
        }
    }
    return 0;
}

The result is as follows

Time difference between messages is: 15.496 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 31927.5us
Time difference between messages is: 15.681 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33455.4us
Time difference between messages is: 15.866 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33237us
Time difference between messages is: 16.052 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33648.6us
Time difference between messages is: 16.237 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 32952.1us
Time difference between messages is: 16.422 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33447.5us
Time difference between messages is: 16.608 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33302.4us
Time difference between messages is: 16.793 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 24855.3us
Time difference between messages is: 16.978 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 41743.3us
Time difference between messages is: 17.163 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 24911.8us
Time difference between messages is: 17.349 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33330.5us
Time difference between messages is: 17.534 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 31817.8us
Time difference between messages is: 17.719 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33400.9us
Time difference between messages is: 17.905 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33321.5us
Time difference between messages is: 18.09 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33280.4us
Time difference between messages is: 18.275 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33403.9us
Time difference between messages is: 18.461 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33301.3us
Time difference between messages is: 18.646 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33319us
Time difference between messages is: 18.831 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33308.3us
Time difference between messages is: 19.017 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33329.3us
Time difference between messages is: 19.202 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33298.9us
Time difference between messages is: 19.387 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33434.2us
Time difference between messages is: 19.572 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33381.4us
Time difference between messages is: 19.758 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33206.2us
Time difference between messages is: 19.943 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33371.4us
Time difference between messages is: 20.128 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33273.5us
Time difference between messages is: 20.314 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33521.1us
Time difference between messages is: 20.499 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33146.4us
Time difference between messages is: 20.684 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33411.6us
Time difference between messages is: 20.87 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33324us
Time difference between messages is: 21.055 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33303.9us
Time difference between messages is: 21.24 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33418us
Time difference between messages is: 21.426 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33243.8us
Time difference between messages is: 21.611 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33410.5us
Time difference between messages is: 21.796 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33277.6us
Time difference between messages is: 21.982 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33358.2us
Time difference between messages is: 22.167 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33361us
Time difference between messages is: 22.352 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33288.1us
Time difference between messages is: 22.538 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33337.2us
Time difference between messages is: 22.723 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33390.5us
Time difference between messages is: 22.908 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33347.6us
Time difference between messages is: 23.093 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33334.9us
Time difference between messages is: 23.279 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33339.1us
Time difference between messages is: 23.464 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33334.2us
Time difference between messages is: 23.65 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33329.7us
Time difference between messages is: 23.835 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33344.9us
Time difference between messages is: 24.02 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33313.7us
Time difference between messages is: 24.205 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33297.9us
Time difference between messages is: 24.391 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33415.8us
Time difference between messages is: 24.576 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33307.8us
Time difference between messages is: 24.761 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33351.2us
Time difference between messages is: 24.947 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33301.3us
Time difference between messages is: 25.132 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33319.4us
Time difference between messages is: 25.317 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33356.4us
Time difference between messages is: 25.503 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33454us
Time difference between messages is: 25.688 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33218.8us
Time difference between messages is: 25.873 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33299.1us
Time difference between messages is: 26.058 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33346.6us
Time difference between messages is: 26.244 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33337.1us
Time difference between messages is: 26.429 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33527.1us
Time difference between messages is: 26.614 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33177us
Time difference between messages is: 26.8 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33389us
Time difference between messages is: 26.985 ms
Time difference between messages is: 22.579 ms
read 1 count;Ts = 33325.2us

erik · Mar 16, 2024

@aiyangsky could you share some photo of the wiring as well?

Aaiyangsky · Mar 18, 2024

erik

Of course, a schematic diagram of the line connection (orange cable from FSIN to XVS):

Below is the FSIN signal (blue to CAM_A, yellow is CAM_B

jakaskerl · Mar 18, 2024

Hi @aiyangsky
Try setting the OUTPUT mode to the CAM_B and the rest to INPUT.

Change gpio6 to gpio42 on R7.

dai::Device device(pipeline);
dai::Device::Config c = pipeline.getDeviceConfig();
c.board.gpio[42] = dai::BoardConfig::GPIO(dai::BoardConfig::GPIO::Direction::OUTPUT, dai::BoardConfig::GPIO::Level::HIGH);

Thanks,
Jaka

Aaiyangsky · Mar 19, 2024

jakaskerl Thanks for the suggestion. But I tried this way and he didn't improve.

erik · Mar 20, 2024

@aiyangsky the scoped signal doesn't look correct - it starts at 1V (should be 0v?) and goes to 1.8V. Is this a measurement error?

Aaiyangsky · Mar 21, 2024

erik I double-checked today that this is indeed not a measurement error.