`dai::ImgFrame::getCvFrame()` throws `OutOfMemoryError`

xperroni

I wrote a custom ROS2 node to connect to my OAK camera and publish RGB images as ROS messages. It basically works by subscribing the callback below to a queue:

void Camera::callback(std::shared_ptr<dai::ADatatype> message) {
  auto *data = dynamic_cast<dai::ImgFrame*>(message.get());
  if (data == nullptr)
    return;

  cv::Mat frame = data->getCvFrame();
  if (frame.empty())
    return;

  header_.stamp = toTime(data->getTimestamp());

  auto image = cv_bridge::CvImage(header_, encoding_, frame).toImageMsg();
  publisher_.publish(image);
}

This works fine when compiled in an X86-64 environment, but when compiled for ARM-64 the code throws an OutOfMemoryError on the call to getCvFrame().

What could be the source of this exception? I'm building my node against ros-humble-depthai 2.31.1-1 on both architectures.

The Python code below (running against depthai 2.32.0.0) works fine on both architectures:

#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
xoutVideo = pipeline.create(dai.node.XLinkOut)

xoutVideo.setStreamName("video")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setIspScale(1, 3)

xoutVideo.input.setBlocking(False)
xoutVideo.input.setQueueSize(1)

# Linking
camRgb.video.link(xoutVideo.input)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:
    cv2.namedWindow("video", cv2.WINDOW_NORMAL | cv2.WINDOW_KEEPRATIO | cv2.WINDOW_GUI_NORMAL)
    video = device.getOutputQueue(name="video", maxSize=1, blocking=False)
    while True:
        videoIn = video.get()
        frame = videoIn.getCvFrame()
        cv2.imshow("video", frame)
        if cv2.waitKey(1) == ord('q'):
            break

xperroni

Update: I was able to get the code running on ARM-64 by manually converting the dai::ImgFrame to a cv::Mat:

void Camera::callback(std::shared_ptr<dai::ADatatype> message) {
  auto *data = dynamic_cast<dai::ImgFrame*>(message.get());
  if (data == nullptr) {
    return;
  }

  int width = data->getWidth();
  int height = data->getHeight();
  auto &buffer = data->getData();

  cv::Mat frame(cv::Size(width, height * 3 / 2), CV_8UC1, buffer.data());
  cv::cvtColor(frame, frame, cv::ColorConversionCodes::COLOR_YUV2BGR_NV12);

  header_.stamp = toTime(data->getTimestamp());
  auto image = cv_bridge::CvImage(header_, encoding_, frame).toImageMsg();
  publisher_.publish(image);
}

So the problem seems to be specifically with the getCvFrame() logic, since both retrieving the raw data from dai::ImgFrame and converting it to cv::Mat work when done outside of that function.

OskarSonc

Thanks for reporting this!