WHOI-Stewart-Jamieson

Hi, on the AR0234 page you show a motorized focus variant -- is this available from one of the usual vendors (arducam, uctronics, or the Luxonis shop)? If not, is there a way we could order them directly through you?

I'm referring to the "M12-Mount AF (PY056) - Motorized Focus" on the page https://docs.luxonis.com/projects/hardware/en/latest/pages/articles/sensors/ar0234/#sensor-ccms

Thanks for your help!

Cheers,
Stewart Jamieson

After a fair bit of testing on our end it seems like this issue may be due to a loose or damaged FFC connection somewhere. Would this match up with the definition of the error code?

We have also attempted this after building the v2.7.3 library from source but it hasn't solved any of our issues.

HI,

We are now getting the following error from one of our OAK-FFC-4P boards:

[1944301001BCE31200] [1.2.3] [4.449] [system] [critical] Fatal error. Please report to developers. Log: 'PlgSrcMipi' '985'

The code to reproduce is the same given in https://discuss.luxonis.com/d/1683-setting-cam-a-to-input-sync-mode-on-oak-ffc-4p/4

Please let us know if you have any solutions.

Best,
Stewart

Oh, and we are using the Ubuntu 20.04 apt package ros-noetic-depthai-ros

As an update on this, we can only seem to successfully put the camera on the CAM_D port into INPUT mode. If we put the same camera on any other port (A-C) it ignores the XVS/sync pin and seems to follow the software trigger, regardless of whether we set the frame sync mode to INPUT/OUTPUT/OFF.

We are using R4M1E4 OAK-FFC-4P boards, in case that is relevant.

Cheers,
Stewart

I didn't have a ton of time to make the MRE as minimal as possible but here is something that should compile -- it expects color cameras (IMX477s) on CAM A&D and a mono (OV9282) on CAM_C

#include <ros/ros.h>
#include <ros/console.h>
#include <sensor_msgs/Image.h>

#include <cstdio>
#include <functional>
#include <iostream>
#include <tuple>
#include <optional>
#include <bits/stdc++.h>

// Inludes common necessary includes for development using depthai library
#include <depthai/depthai.hpp>
#include <depthai_bridge/BridgePublisher.hpp>
#include <depthai_bridge/ImageConverter.hpp>
#include <depthai_bridge/depthaiUtility.hpp>

using namespace dai::node;

dai::DeviceInfo findDevice(std::string const &mxId = "") {
    std::vector<dai::DeviceInfo> const availableDevices = dai::Device::getAllAvailableDevices();
    dai::DeviceInfo device;
    bool found = false;
    ROS_INFO("Listing available devices...");
    for (auto const &deviceInfo: availableDevices) {
        if (found) continue;
        if (mxId.empty()) {
            if (deviceInfo.state == X_LINK_UNBOOTED || deviceInfo.state == X_LINK_BOOTLOADER) {
                device = deviceInfo;
                found = true;
            }
        } else if (deviceInfo.getMxId() == mxId) {
            if (deviceInfo.state == X_LINK_UNBOOTED || deviceInfo.state == X_LINK_BOOTLOADER) {
                device = deviceInfo;
                found = true;
            } else {
                throw std::runtime_error(
                        "DepthAI Device with MxId  \"" + mxId + "\" is already booted on different process.");
            }
        }
    }
    if (!found)
        throw std::runtime_error((mxId.empty()) ? ("Failed to find any available DepthAI device.") : (
                "Failed to find DepthAI device with MxId \"" + mxId + "\"."));
    return device;
}

class Modular4PCamera {
    template<class T>
    using SPtr = std::shared_ptr<T>;

    dai::Pipeline pipeline;
    int rgbWidth = 0, rgbHeight = 0, rawWidth = 0, rawHeight = 0;
    SPtr<ColorCamera> leftCam, rightCam;
    SPtr<MonoCamera> monoCam;
    SPtr<VideoEncoder> leftEncoder, rightEncoder;
    SPtr<ImageManip> leftResizer, rightResizer;
    SPtr<StereoDepth> stereoDepth;
    SPtr<XLinkIn> leftControlIn, rightControlIn, stereoControlIn, monoControlIn;
    SPtr<XLinkOut> rawOut, leftOut, rightOut, leftStereoOut, rightStereoOut, compressedLeftOut, compressedRightOut;
    SPtr<XLinkOut> depthOut, stereoControlOut;

public:
    Modular4PCamera(dai::CameraBoardSocket const left_camera,
                    dai::CameraBoardSocket const right_camera,
                    bool const enableRight,
                    bool const enableUncompressed,
                    bool const enableCompressed,
                    int const compressionQuality,
                    int const compressionKBitrate,
                    float const fps,
                    int const rgbScaleNumerator,
                    int const rgbScaleDenominator) {
        pipeline.setXLinkChunkSize(0);

        // RGB image
        leftCam = pipeline.create<dai::node::ColorCamera>();
        leftCam->setBoardSocket(left_camera);
        leftCam->setFps(fps);

        dai::node::ColorCamera::Properties::SensorResolution rawResolution;
        rawResolution = dai::node::ColorCamera::Properties::SensorResolution::THE_1080_P;
        rawWidth = 1920;
        rawHeight = 1080;

        leftCam->setResolution(rawResolution);
        leftCam->setIspScale(rgbScaleNumerator, rgbScaleDenominator);
        std::tie(rgbWidth, rgbHeight) = leftCam->getIspSize();
        leftCam->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);
        leftCam->setInterleaved(false);

        leftControlIn = pipeline.create<dai::node::XLinkIn>();
        leftControlIn->setStreamName("leftControl");
        leftControlIn->out.link(leftCam->inputControl);

        monoCam = pipeline.create<dai::node::MonoCamera>();
        monoCam->setBoardSocket(dai::CameraBoardSocket::CAM_C);
        monoCam->setFps(24);

        monoControlIn = pipeline.create<dai::node::XLinkIn>();
        monoControlIn->setStreamName("monoControl");
        monoControlIn->out.link(monoCam->inputControl);

        if (enableUncompressed) {
            ROS_INFO("Configuring uncompressed video stream at %dx%d", rgbWidth, rgbHeight);
            leftOut = pipeline.create<dai::node::XLinkOut>();
            leftOut->setStreamName("left");
            leftCam->isp.link(leftOut->input);
        }

        if (enableCompressed) {
            ROS_INFO("Configuring compressed video stream at %dx%d", rgbWidth, rgbHeight);

            leftEncoder = pipeline.create<dai::node::VideoEncoder>();
            leftEncoder->setDefaultProfilePreset(fps, dai::VideoEncoderProperties::Profile::H264_MAIN);
            leftEncoder->setQuality(compressionQuality);
            leftEncoder->setBitrateKbps(compressionKBitrate);
            leftEncoder->setLossless(false);
            compressedLeftOut = pipeline.create<dai::node::XLinkOut>();
            compressedLeftOut->setStreamName("left_stream");

            leftCam->video.link(leftEncoder->input);
            leftEncoder->bitstream.link(compressedLeftOut->input);
        }

        if (enableRight) {
            // RGB image
            rightCam = pipeline.create<dai::node::ColorCamera>();
            rightCam->setBoardSocket(right_camera);
            rightCam->setFps(fps);

            rightCam->setResolution(rawResolution);
            rightCam->setIspScale(rgbScaleNumerator, rgbScaleDenominator);
            int rightRgbWidth, rightRgbHeight;
            std::tie(rightRgbWidth, rightRgbHeight) = rightCam->getIspSize();
            rightCam->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);
            rightCam->setInterleaved(false);

            rightControlIn = pipeline.create<dai::node::XLinkIn>();
            rightControlIn->setStreamName("rightControl");
            rightControlIn->out.link(rightCam->inputControl);

            if (enableUncompressed) {
                ROS_INFO("Configuring uncompressed right video stream at %dx%d", rgbWidth, rgbHeight);
                rightOut = pipeline.create<dai::node::XLinkOut>();
                rightOut->setStreamName("right");
                rightCam->isp.link(rightOut->input);
            }

            if (enableCompressed) {
                ROS_INFO("Configuring compressed right video stream at %dx%d", rgbWidth, rgbHeight);

                rightEncoder = pipeline.create<dai::node::VideoEncoder>();
                rightEncoder->setDefaultProfilePreset(fps, dai::VideoEncoderProperties::Profile::H264_MAIN);
                rightEncoder->setQuality(compressionQuality);
                rightEncoder->setBitrateKbps(compressionKBitrate);
                rightEncoder->setLossless(false);
                compressedRightOut = pipeline.create<dai::node::XLinkOut>();
                compressedRightOut->setStreamName("right_stream");

                rightCam->video.link(rightEncoder->input);
                rightEncoder->bitstream.link(compressedRightOut->input);
            }
        }
    }

    dai::Pipeline *getPipeline() {
        return &pipeline;
    }

    std::pair<int, int> getRgbSize() const {
        return {rgbWidth, rgbHeight};
    }

    std::pair<int, int> getRawRgbSize() const {
        return {rawWidth, rawHeight};
    }
};

sensor_msgs::CameraInfo
tryLoadCameraInfo(dai::rosBridge::ImageConverter &converter, dai::CalibrationHandler &calibHandler,
                  dai::CameraBoardSocket const &cameraId, int const &width, int const &height) {
    try {
        return converter.calibrationToCameraInfo(calibHandler, cameraId, width, height);
    } catch (std::runtime_error const &e) {
        sensor_msgs::CameraInfo info;
        info.width = width;
        info.height = height;
        return info;
    }
}

int main(int argc, char **argv) {
    ros::init(argc, argv, "oak_ffc_4p_node");
    ros::NodeHandle pnh("~");

    std::string tfPrefix, mxId, robotName;
    std::string rgbResolution = "1080p";
    uint64_t frame;
    int badParams = 0;
    float fps, hfov;
    int rgbScaleNumerator, rgbScaleDenominator, compressionQuality, compressionKBitrate;
    bool swapLeftRight;
    bool enableRight, enableUncompressed, enableCompressed;

    badParams += !pnh.param<std::string>("mxid", mxId, "");
    badParams += !pnh.param<std::string>("robot_name", robotName, "");
    badParams += !pnh.param<bool>("swap_left_right", swapLeftRight, false);

    badParams += !pnh.param<std::string>("tf_prefix", tfPrefix, "");
    badParams += !pnh.param("enable_right", enableRight, true);
    badParams += !pnh.param("enable_uncompressed", enableUncompressed, true);
    badParams += !pnh.param("enable_compressed", enableCompressed, true);
    badParams += !pnh.param("compression_quality", compressionQuality, 100);
    badParams += !pnh.param("compression_kbitrate", compressionKBitrate, 0);

    badParams += !pnh.param<float>("fps", fps, 15);
    badParams += !pnh.param<float>("hfov", hfov, 120);

    badParams += !pnh.param("rgbScaleNumerator", rgbScaleNumerator, 1);
    badParams += !pnh.param("rgbScaleDenominator", rgbScaleDenominator, 1);

    if (badParams) ROS_WARN("Couldn't find %d parameters", badParams);

    ROS_DEBUG("Building DepthAI pipeline");

    int rgbWidth, rgbHeight, rawWidth, rawHeight;
    auto const left_camera = (swapLeftRight) ? dai::CameraBoardSocket::CAM_D : dai::CameraBoardSocket::CAM_A;
    auto const right_camera = (swapLeftRight) ? dai::CameraBoardSocket::CAM_A : dai::CameraBoardSocket::CAM_D;

    Modular4PCamera oakFfc4p(left_camera,
                             right_camera,
                             enableRight,
                             enableUncompressed,
                             enableCompressed,
                             compressionQuality,
                             compressionKBitrate,
                             fps,
                             rgbScaleNumerator,
                             rgbScaleDenominator);

    std::shared_ptr<dai::Device> device = std::make_shared<dai::Device>(*oakFfc4p.getPipeline(), findDevice(mxId));

    std::tie(rgbWidth, rgbHeight) = oakFfc4p.getRgbSize();
    std::tie(rawWidth, rawHeight) = oakFfc4p.getRawRgbSize();

    std::shared_ptr<dai::DataOutputQueue> leftQueue, rightQueue, leftStreamQueue, rightStreamQueue;
    std::shared_ptr<dai::DataInputQueue> leftControlQueue, rightControlQueue, monoControlQueue;
    std::shared_ptr<dai::rosBridge::BridgePublisher<sensor_msgs::Image, dai::ImgFrame>> leftPublish, rightPublish;
    std::shared_ptr<dai::rosBridge::BridgePublisher<sensor_msgs::CompressedImage, dai::ImgFrame>> leftStreamPublish, rightStreamPublish;
    std::shared_ptr<dai::rosBridge::ImageConverter> leftConverter, rightConverter;
    std::optional<ros::Publisher> leftInfoPublisher, rightInfoPublisher;
    sensor_msgs::CameraInfo leftInfo, rightInfo;

    if (!device)
        throw std::runtime_error(
                "ros::NodeHandle() from Node \"" + pnh.getNamespace() + "\" DepthAI Device with MxId  \"" + mxId +
                "\" not found.  \"");

    ROS_INFO("Reading calibrations");

    auto calib = device->readCalibration();

    auto const time_ref = std::make_pair(ros::Time::now(), std::chrono::steady_clock::now());

    ROS_INFO("Loading converters");
    leftConverter = std::make_shared<dai::rosBridge::ImageConverter>(tfPrefix + "_left_camera_optical_frame", true);
    leftInfo = tryLoadCameraInfo(*leftConverter, calib, left_camera, rgbWidth, rgbHeight);
    leftInfo.header.frame_id = tfPrefix + "_left_camera_optical_frame";

    ROS_INFO("Loading queues");
    leftControlQueue = device->getInputQueue("leftControl");
    rightControlQueue = device->getInputQueue("rightControl");
    monoControlQueue = device->getInputQueue("monoControl");

    auto initialControlMsg = dai::CameraControl();
    initialControlMsg.setFrameSyncMode(dai::CameraControl::FrameSyncMode::INPUT);
    leftControlQueue->send(initialControlMsg);
    rightControlQueue->send(initialControlMsg);

    auto monoControlMsg = dai::CameraControl();
    monoControlMsg.setFrameSyncMode(dai::CameraControl::FrameSyncMode::OUTPUT);
    monoControlQueue->send(monoControlMsg);

    ROS_INFO("Configuring callbacks");
    if (enableUncompressed) {
        leftQueue = device->getOutputQueue("left", 1, false);
        leftPublish = std::make_shared<dai::rosBridge::BridgePublisher<sensor_msgs::Image, dai::ImgFrame>>(
                leftQueue,
                pnh,
                "left/image_color",
                [&leftConverter](std::shared_ptr<dai::ImgFrame> const &inData,
                                                            auto &outMsg) {
                    leftConverter->toRosMsg(inData, outMsg);
                },
                3,
                leftInfo,
                "left");
        leftPublish->addPublisherCallback();
    } else {
        leftInfoPublisher = pnh.advertise<sensor_msgs::CameraInfo>("left/camera_info", 1, true);
    }

    if (enableCompressed) {
        ROS_INFO("Advertising compressed video stream topic");
        leftStreamQueue = device->getOutputQueue("left_stream", 1, false);
        auto const profile = dai::VideoEncoderProperties::Profile::H264_MAIN;
        std::string const topic = "left/image_stream/h264", format = "h264";
        leftStreamPublish = std::make_shared<dai::rosBridge::BridgePublisher<sensor_msgs::CompressedImage, dai::ImgFrame>>(
                leftStreamQueue,
                pnh,
                topic,
                [&enableUncompressed, &leftInfoPublisher, &leftInfo, time_ref, format](
                        std::shared_ptr<dai::ImgFrame> const &inData,
                        auto &outMsg) {
                    using namespace std::chrono;
                    auto const currentTime = std::chrono::steady_clock::now();
                    auto const dataTime = inData->getTimestamp();
                    sensor_msgs::CompressedImage msg;
                    msg.data = inData->getData();
                    msg.format = format;
                    msg.header = leftInfo.header;
                    // msg.header.stamp = ros::Time(duration<double, std::ratio<1>>(dataTime.time_since_epoch()).count());
                    msg.header.stamp = dai::ros::getFrameTime(time_ref.first, time_ref.second, dataTime);
                    if (!enableUncompressed) {
                        leftInfo.header = msg.header;
                        leftInfoPublisher->publish(leftInfo);
                    }
                    outMsg.push_back(std::move(msg));
                    ROS_INFO_THROTTLE(2, "Compressed RGB Latency: %lu ms",
                                      std::chrono::duration_cast<std::chrono::milliseconds>(
                                              currentTime - dataTime).count());
                },
                3,
                leftInfo,
                "left");
        leftStreamPublish->addPublisherCallback();
    }

    if (enableRight) {
        rightConverter = std::make_shared<dai::rosBridge::ImageConverter>(tfPrefix + "_right_camera_optical_frame",
                                                                          true);
        rightInfo = tryLoadCameraInfo(*rightConverter, calib, right_camera,
                                      rgbWidth,
                                      rgbHeight);
        rightInfo.header.frame_id = tfPrefix + "_right_camera_optical_frame";

        if (enableUncompressed) {
            rightQueue = device->getOutputQueue("right", 1, false);
            rightPublish = std::make_shared<dai::rosBridge::BridgePublisher<sensor_msgs::Image, dai::ImgFrame>>(
                    rightQueue,
                    pnh,
                    "right/image_color",
                    [&rightConverter](auto const &inData, auto &outMsg) {
                        rightConverter->toRosMsg(inData, outMsg);
                    },
                    3,
                    rightInfo,
                    "right");
            rightPublish->addPublisherCallback();
        } else {
            rightInfoPublisher = pnh.advertise<sensor_msgs::CameraInfo>("right/camera_info", 1, true);
        }

        if (enableCompressed) {
            ROS_INFO("Advertising compressed right camera video stream topic");
            rightStreamQueue = device->getOutputQueue("right_stream", 1, false);
            std::string const topic = "right/image_stream/h264", format = "h264";
            rightStreamPublish = std::make_shared<dai::rosBridge::BridgePublisher<sensor_msgs::CompressedImage, dai::ImgFrame>>(
                    rightStreamQueue,
                    pnh,
                    topic,
                    [&enableUncompressed, &rightInfoPublisher, &rightInfo, time_ref, format](
                            std::shared_ptr<dai::ImgFrame> const &inData,
                            auto &outMsg) {
                        using namespace std::chrono;
                        auto const currentTime = std::chrono::steady_clock::now();
                        auto const dataTime = inData->getTimestamp();
                        sensor_msgs::CompressedImage msg;
                        msg.data = inData->getData();
                        msg.format = format;
                        msg.header = rightInfo.header;
                        msg.header.stamp = dai::ros::getFrameTime(time_ref.first, time_ref.second, dataTime);
                        if (!enableUncompressed) {
                            rightInfo.header = msg.header;
                            rightInfoPublisher->publish(rightInfo);
                        }
                        outMsg.push_back(std::move(msg));
                        ROS_INFO_THROTTLE(2, "Compressed RGB Latency: %lu ms",
                                          std::chrono::duration_cast<std::chrono::milliseconds>(
                                                  currentTime - dataTime).count());
                    },
                    3,
                    rightInfo,
                    "right");
            rightStreamPublish->addPublisherCallback();
        }
    }

    ROS_INFO("Ready! Spinning...");
    ros::spin();
    return 0;
}

Hi Eric,

I'll reduce our code into a more minimal reproducible example and post it here shortly. We are not linking the A&D and B&C ports - to my understanding that would have no effect since the arducam boards we are using have the fsync lines terminated on the 26-22 pin connector. We have directly linked the XVS ports of the three cameras without connecting anything to the fsync terminals on the cables.

Cheers,

Stewart

Hi,

We are trying to run 2x IMX477s (Cam A and Cam D) and 2x OV9282 together, synchronized. We are currently testing with just 1x OV9282 on CAM_C.

We have configured the A&D ports to be inputs, and port C to be an output. I can see the clock signal on the OV9282 at the appropriate 24Hz using an oscilloscope. As expected, the IMX477 on CAM_D is synchronized with the OV9282, and will not trigger if the sync cable is disconnected.

Problem 1: Even with the sync cables disconnected, the camera on CAM_A is still triggering as though its frame sync mode were set to OFF. Problem 2: With the cable connected, we have found that it still isn't synchronized.

Both problems seem to be caused by the IMX477 on CAM_A ignoring the initial control msg to input mode. Is this a bug, or are we potentially missing a step? As far as I can tell, our software and electrical configurations for A&D are identical, so we are very confused.

Thanks,
Stewart

Hi,

My pipeline crashed with the following error:

[184430102153310F00] [1.10] [49.001] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

Is there any additional info I could provide that may help you folks investigate this? It's an OAK-FFC-4P board with IMX477 motorized focus cameras running on CAMA_4L and CAMD_4L

Cheers,
Stewart Jamieson

Adding the following fixed it! Thank you @jakaskerl
stereoConfigOutput->setBlocking(false);

I found some more tips on that here and here

Best,
Stewart

Hi, I am using an OAK-FFC-4P setup with a stereo camera configuration, and I am trying to configure some of the post processing. I'd like to read the configuration from the StereoDepth outConfig stream, and then adjust it and write it back to inputConfig.

Writing to inputConfig works fine, but when I add the following to my pipeline, the entire device (all streams) hang indefinitely:

stereoControlOut = pipeline.create<dai::node::XLinkOut>();
stereoControlOut->setStreamName("stereoConfigOutput");
stereoDepth->outConfig.link(stereoControlOut->input);

And elsewhere:

stereoConfigOutput = device->getOutputQueue("stereoConfigOutput");
stereoConfigOutput->addCallback({
std::cout << "New stereo config output" << std::endl;
return;
});

I see New stereo config output several times in the output before the process stops responding. If I remove the above lines, I am able to view h264 encoded video from the camera streams, as well as the stereo depth, just fine. Am I doing something wrong here? Thanks for your help.

Best,
Stewart Jamieson

Ah brilliant, thanks GergelySzabolcs!

Cheers,
Stewart Jamieson

Thanks GergelySzabolcs -- so in the example I gave, it will get "warped" up to 4K, then downscaled to 720p when I call setOutputSize(1280, 720)?

Somewhat related, I've also observed that setDepthAlign seems to have the unexpected side-effect of setting the output depth image size to match the target socket. If this call is followed by a call to setOutputSize, does it end up internally scaling the image twice?

E.g. suppose that with a 4K RGB config and 720p stereo, I call setDepthAlign(dai::CameraBoardSocket::RGB) , resulting in 4K stereo depth output, but then call setOutputSize(1280, 720) to get back the "original" depth computed from the 720p stereo pair. Will it internally upscale the disparity to 4K and then downscale back to 720p, or just skip scaling altogether?

Hi,

Our downstream processing expects that depth images (e.g., coming from a StereoDepth node) are aligned with the left camera stream, but it looks like DepthAI nodes (at least StereoDepth) default to the right.

The two functions which seem to determine left vs. right alignment are setRectifyMirrorFrame and setDepthAlign. Is it sufficient to simply call setDepthAlign(dai::CameraBoardSocket::LEFT), or does aligning with the left camera stream also require changing the mirror settings?

As an aside, I'm using rosBridge, and it looks clear that I should then use a left aligned image converter and camera info for the BridgePublisher.

Cheers,
Stewart Jamieson

HI @erik,

Thank you, the warp support seems perfectly suited for rectifying the RGB stream with a new camera model -- it looks like I just need to compute a mesh based on the double sphere model, and the ImageManip node will handle the rest.

To apply the same type of calibration to the stereo images before disparity computation, it looks like I can similarly use the loadMeshData/loadMeshFiles methods of the StereoDepth node, to get a depth image is properly aligned with the rectified RGB stream?

Cheers,
Stewart