RTSP Streaming of Both Video And YOLO Detections

KevinRomero

Hello,

First time poster here. I recently got an OAK-D W and am trying to stream via RTSP both video and YOLO detections. I have successfully been able to run an experiment that streams video over RTSP (luxonis/depthai-experiments/tree/master/gen2-rtsp-streaming) and a separate experiment that runs YOLO detections on the camera video (https://docs.luxonis.com/projects/sdk/en/latest/samples/NNComponent/sdk_yolo/), so now I'd like to combine these two capabilities.

One thing I noticed is that there was a dependency issue, at least between these two out-of-box examples that believe were using different versions of depthAI - I could not run both experiments on the same virtual environment, but got them to work in separate ones.

Any help sincerely appreciated!

jakaskerl

Hi KevinRomero
RTSP should work on the latest depthai/depthai_sdk version. The problem will likely be that the server expects encoded data from the device. If you use sdk yolo example, you will get a standard image frame. So in order to pass it to the server, you need to either encode it on the host, or send it back to the device to encode.

Another problem might be that RTSP server is running in different thread. The SDK is very annoying about anything not running in its thread and might not send any data at all.

What about going the API approach?

Thanks,
Jaka

KevinRomero

jakaskerl Hey Jaka,

Thanks for the response. Could you elaborate on the API approach (i.e. general steps outline ?), or post some relevant documentation? I am currently looking at https://docs.luxonis.com/projects/api/en/latest/tutorials/hello_world/#hello-world

jakaskerl

Hi KevinRomero
I'd start with this example. Then you paste the RTSP server class inside. You will need to encode the frame your receive from the queue as .h265 and send it using server.send().

Then you should be able to access it on the other side.

Thanks,
Jaka

KevinRomero

Hey! So I have went with your approach and am attempting to merge the encoder example and the tiny yolo example (code below). My current error is detectionNetwork.out.link(videnc2.input). Could really use some help!

RuntimeError: Cannot link 'DetectionNetwork.out' to 'VideoEncoder.in'

i basically tried creating a new stream:

FPS =30

videnc2 = pipeline.create(dai.node.VideoEncoder)

videnc2.setDefaultProfilePreset(FPS, dai.VideoEncoderProperties.Profile.H265_MAIN)

detectionNetwork.out.link(videnc2.input) #this is also done above

videnc2.bitstream.link(nnOut.input)

nnout = pipeline.create(dai.node.XLinkOut)

nnout.setStreamName("encoded2")

videnc2.bitstream.link(nnout.input)

Then I create a second queue and attempt to send it as a second paramater using .send_data:

with dai.Device(pipeline, device_info) as device:

encoded = device.getOutputQueue("encoded", maxSize=30, blocking=True)

encoded2 = device.getOutputQueue("encoded2", maxSize=30, blocking=True) #TEST

print("Setup finished, RTSP stream available under \"rtsp://localhost:8554/preview\"")

while True:

data = encoded.get().getData()

data2 = encoded2.get().getData() #TEST

server.send_data(data, data2)