Integrating custom YOLO RVC4 model into PoE TCP streaming pipeline

BadriPratti

Hi all,

I’m working off the gen3/stream-manipulation/poe-tcp-streaming example in depthai-experiments (link below). The demo itself runs perfectly, but when I swap in my custom YOLO model (a .rvc4.tar.xz archive) I’m running into a few problems:

No detections even though my script launches and frames stream.

A minimal code snippet showing how to load/run a .rvc4.tar.xz YOLO archive in this pipeline would be helpful.

Tips on extracting and using the archive’s class labels and input shape would also be helpful

The TCP‐streaming Script node sometimes hangs when I call .get() on the NN output.

jakob

Hi @BadriPratti ,

No detections even though my script launches and frames stream.
A minimal code snippet showing how to load/run a .rvc4.tar.xz YOLO archive in this pipeline would be helpful.

Please consult the neural-networks/generic-example for a simple illustration of the neural network inference pipeline (you can also use it as a sanity check that your model is able to produce predictions).

Tips on extracting and using the archive’s class labels and input shape would also be helpful

You can use dai.NNArchive object to load the NN Archive. The required info can be obtained from the object itself (e..g by calling .getInputWidth(), .getInputHeight(), etc. on it)

The TCP‐streaming Script node sometimes hangs when I call .get() on the NN output.

What exactly do you mean by "hangs"? Does it freeze completely or just take some time to get the output?

BadriPratti

Hello,
Thank You for your fast response.
I have tried to apply the recommndation that you gave to the code but I am getting this error:

Also this is my current code that I modifed from the POE-TCP-Streaming. I tried to edit the oak.py into accpeting the .tar.xz. Also the tar.xz file that I have does work fine. Although the device isnt using any other application I am getting this error message.

'#!/usr/bin/env python3
from pathlib import Path
import depthai as dai
from utils.annotation_node import AnnotationNode
from utils.oak_arguments import initialize_argparser
from utils.scripts import get_client_script, get_server_script

_, args = initialize_argparser()
device = dai.Device(dai.DeviceInfo(args.device)) if args.device else dai.Device()
platform = device.getPlatform().name

MODEL_PATH = "/app/utils/yolov6n-r2-288x512.rvc4.tar.xz"
nn_archive = dai.NNArchive(MODEL_PATH)
W, H = nn_archive.getInputWidth(), nn_archive.getInputHeight()
CLASSES = nn_archive.getConfig().model.heads[0].metadata.classes

pipeline = dai.Pipeline()

cam = pipeline.create(dai.node.ColorCamera)
cam.setBoardSocket(dai.CameraBoardSocket.CAM_A)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
cam.setPreviewSize(W, H)
cam.setInterleaved(False)
cam.setFps(args.fps_limit)

left  = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
right = pipeline.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)
stereo = pipeline.create(dai.node.StereoDepth).build(
    left=left.requestOutput((W, H)),
    right=right.requestOutput((W, H)),
    presetMode=dai.node.StereoDepth.PresetMode.HIGH_DETAIL
)
stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
stereo.setLeftRightCheck(True)
stereo.setRectification(True)
if platform == "RVC2":
    stereo.setOutputSize(W, H)

nn = pipeline.create(dai.node.SpatialDetectionNetwork).build(
    input     = cam,       # Camera node itself
    stereo    = stereo,    # Or None if you only want 2D
    nnArchive = nn_archive,
    fps       = args.fps_limit,
)
nn.setConfidenceThreshold(0.5)
nn.setBoundingBoxScaleFactor(0.7)

annot = pipeline.create(AnnotationNode).build(
    input_detections = nn.out,
    labels            = CLASSES,
)
nn.passthrough.link(annot.inputs["frame"])

video_enc = pipeline.create(dai.node.VideoEncoder)
video_enc.setDefaultProfilePreset(args.fps_limit, dai.VideoEncoderProperties.Profile.MJPEG)
annot.out_annotations.link(video_enc.input)

script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

video_enc.bitstream.link(script.inputs["frame"])
nn.out.link          (script.inputs["detection"])

for port in ("frame", "detection"):
    script.inputs[port].setBlocking(False)
    script.inputs[port].setMaxSize(1)

if args.mode.lower() == "client":
    script.setScript(get_client_script(args.address))
else:
    script.setScript(get_server_script())

script.outputs["control"].link(cam.inputControl)

with dai.Device(pipeline) as dev:
    print("Pipeline started on PoE TCP with custom YOLO archive.")
    dev.waitUntilPipelineRunning()
    # The host.py client will connect on port 5000 and parse:
    #   32-byte header + detection string + JPEG bytes
    while True:
        pass

jakob

Hi @BadriPratti,

not sure I understand exactly what you try to achieve. If trying to edit the oak.py to also run your custom model, I'd suggest you first tightly follow the original structure of the app and only change the pipeline at places where required (i.e. to insert a nn node with your custom model). The error you are describing occurs because the pipeline is not created properly (you seem to call dai.Pipeline() instead of dai.Pipeline(device) as is the case for the original app).

BadriPratti

Hi @jakob ,
What I’m trying to achieve is running a custom object detection model using the poe-tcp-streaming pipeline instead of the WebSocket-based approach. The main reason for this is to avoid relying on a special URL or link that needs to be manually pasted each time.

I’ve already successfully integrated the .tar.xz model into a different standalone script. Now, I’m working on merging that functionality into the poe-tcp-streaming app so the model runs directly through the PoE interface with TCP-based streaming. I hope this clarifies the goal I’m working towards.

Thanks!

jakob

Hi,

makes sense. Good luck with the merging and feel free to ask for more help if needed!