How to XLinkIn a frame to a VideoEncoder Node

IanChu

I'm trying to push a frame (OpenCV, numpy array) to a VideoEncoder Node. I'm getting a fatal error when I try to send it.

[14442C10A1A661D700] [1.1] [1.135] [system] [critical] Fatal error. Please report to developers. Log: 'eConfigs' '29'
[14442C10A1A661D700] [1.1] [1708021325.224] [host] [warning] Device crashed, but no crash dump could be extracted.

My ultimate goal is to draw a Yolo detection on the frame and push it back to the camera's video encoder. I could also encode the video stream locally, but I'm trying to avoid that if possible. Ideally the whole thing would run on the OAK camera, but I couldn't figure out how to extract a frame inside a Script node (I get lbp.ImgFrame has no attribute 'get')

Minimum Reproducible Example:

import cv2
import numpy as np

import depthai as dai

# init pipeline
pipeline = dai.Pipeline()

# init camera
FPS = 30
colorCam = pipeline.create(dai.node.ColorCamera)
colorCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
colorCam.setInterleaved(False)
colorCam.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
colorCam.setFps(FPS)
camOut = pipeline.create(dai.node.XLinkOut)
camOut.setStreamName("cam")

# init video encoder
videnc = pipeline.create(dai.node.VideoEncoder)
videnc.setDefaultProfilePreset(FPS, dai.VideoEncoderProperties.Profile.MJPEG)

# output link
imageOut = pipeline.create(dai.node.XLinkOut)
imageOut.setStreamName("cameraToImage")
colorCam.video.link(imageOut.input)

# input link
imageIn = pipeline.create(dai.node.XLinkIn)
imageIn.setStreamName("imageToVideo")
imageIn.out.link(videnc.input)

# pipe out of video encoder
veOut = pipeline.create(dai.node.XLinkOut)
veOut.setStreamName("encoded")
videnc.bitstream.link(veOut.input)

# start running
MAX_RESOLUTION = (1750, 984) # XlinkIn has a max size
device_info = dai.Device.getAllAvailableDevices()[0]
with dai.Device(pipeline, device_info) as device:
    # pull the image
    qPreview = device.getOutputQueue(name="cameraToImage", maxSize=4)
    qImageInput = device.getInputQueue(name="imageToVideo")

    # send the data to the rtsp stream
    encoded = device.getOutputQueue("encoded", maxSize=30, blocking=True)
    while True:
        # pull image from camera
        frame = qPreview.get().getCvFrame()

        # pre-process
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        frame = cv2.resize(frame, MAX_RESOLUTION)
        # print(frame.shape)

        # show
        cv2.imshow("Frame", frame)
        key = cv2.waitKey(1)
        if key == ord('q'):
            break

        # create ImgFrame and push to video
        daiImage = dai.ImgFrame()
        daiImage.setFrame(frame)
        daiImage.setType(dai.RawImgFrame.Type.GRAY8)
        # print(daiImage.getType())
        qImageInput.send(daiImage)

        # pull from video and send to server
        # data = encoded.get().getData()

Full Stacktrace of Error:

[14442C10A1A661D700] [1.1] [1.135] [system] [critical] Fatal error. Please report to developers. Log: 'eConfigs' '29'
[14442C10A1A661D700] [1.1] [1708021325.224] [host] [warning] Device crashed, but no crash dump could be extracted.
terminate called without an active exception
Stack trace (most recent call last):
#29   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in 
#28   Object "python", at 0x55d44cff6f24, in _start
#27   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc610029e3f, in __libc_start_main
#26   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc610029d8f, in 
#25   Object "python", at 0x55d44cff702c, in Py_BytesMain
#24   Object "python", at 0x55d44d020a5d, in Py_RunMain
#23   Object "python", at 0x55d44d02df82, in _PyRun_AnyFileObject
#22   Object "python", at 0x55d44d02e337, in _PyRun_SimpleFileObject
#21   Object "python", at 0x55d44d02ee54, in 
#20   Object "python", at 0x55d44d0289ca, in 
#19   Object "python", at 0x55d44d02f107, in 
#18   Object "python", at 0x55d44d004255, in PyEval_EvalCode
#17   Object "python", at 0x55d44cf0e9c5, in 
#16   Object "python", at 0x55d44cf16938, in _PyEval_EvalFrameDefault
#15   Object "python", at 0x55d44cf02468, in 
#14   Object "python", at 0x55d44cf37aca, in 
#13   Object "python", at 0x55d44cf1fa7a, in _PyObject_MakeTpCall
#12   Object "python", at 0x55d44cf2910d, in 
#11   Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e415763f, in 
#10   Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e4190532, in 
#9    Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e43bcd14, in dai::DeviceBase::close()
#8    Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e43b6214, in dai::Device::closeImpl()
#7    Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e43d09be, in dai::DeviceBase::closeImpl()
#6    Object "/home/ichu/envs/oak-rtsp/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7fc5e43bd0b3, in dai::DeviceBase::~DeviceBase()
#5    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fc5e64ae276, in std::terminate()
#4    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fc5e64ae20b, in 
#3    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fc5e64a2b9d, in 
#2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc6100287f2, in abort
#1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc610042475, in raise
#0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc6100969fc, in pthread_kill
Aborted (Signal sent by tkill() 41580 1000)
Aborted (core dumped)

jakaskerl

Hi @IanChu
I think you need to set the dai.ImgFrame size as well. Otherwise it will be a 1D array.

Thanks,
Jaka

IanChu

Yep, that was it, thanks!
(and for future searchers, the width needs to be a multiple of 32 and the height needs to be a multiple of 8)

IanChu

Additional issue. When I send a frame to the video encoder I get some sort of color offset. Does anyone know how to convert a BGR frame to a format that the VideoEncoder node will accept (is NV12 some variant of YUV)? Currently I'm using COLOR_BGR2YUV_YV12 to do the conversion and that seems to be closest to correct so far.

jakaskerl

Hi @IanChu
Could be a mismatch between planar and interleaved formats.

IanChu

I'll have to table working on this for now. I'm doing the video encoding outside of the OAK camera instead to get around this. Your reply about the planar and interleaved formats sounds promising, but I don't know enough about this sort of thing to try it and I don't have time to research it. If I get some more time to work on this and figure it out I'll post the results here.

IanChu

I haven't had time to test, but a promising looking line for doing everything within the pipeline would be to write a custom network blob with Kornia. It can draw the Yolo boxes and do the color conversion.