Hi there dear Luxonis Community! š
Iām transferring a video stream via TCP from one PC to another. Iām using gstreamer 1.24
. The Stream gets encoded using to h265/hevc by the camera (hw) itself Rgb Encoding which makes it quite easy on compute-resources.
My goal is to decode that stream AS FAST as possible with the LOWEST possible latency.
Here is my decoding pipeline
tcpclientsrc host=robot-car port=8554 timeout=10
! application/x-rtp-stream
! rtpstreamdepay
! application/x-rtp,media=video,clock-rate=90000,encoding-name=H265
! rtph265depay
! queue
! h265parse config-interval=-1
! video/x-h265,stream-format=hvc1,alignment=au
! queue
! vah265dec qos=false
! video/x-raw(memory:DMABuf),drm-format=NV12
! queue max-size-buffers=1 leaky=1
! glimagesink sync=false qos=false
I analysed the gstreamer latency using the environment variables
GST_TRACERS="latency(flags=pipeline+element+reported)"
GST_DEBUG_FILE=trace.log
GST_DEBUG="1,GST_TRACER:7"
I'm able to detect that vah265dec
is reporting and actually using 100ms
for a framerate of 30/1
. Which indicates to me, that the 100ms
/3 frames-caching
is happening by design.
Time: 0:00:09.549951361
Latency Statistics:
0x55c61a9124d0.tcpclientsrc0.src|0x55c61a9023b0.glimagesinkbin0.sink: mean=0:00:00.100234675 min=0:00:00.050495205 max=0:00:00.121120488
0x55c61a9124d0.tcpclientsrc0.src|0x55c61aaf81d0.sink.sink: mean=0:00:00.100317872 min=0:00:00.050545080 max=0:00:00.121196562
Element Latency Statistics:
0x55c61aab37f0.capsfilter0.src: mean=0:00:00.000006225 min=0:00:00.000003737 max=0:00:00.000566180
0x55c61aa24230.rtpstreamdepay0.src: mean=0:00:00.000018388 min=0:00:00.000011602 max=0:00:00.000102474
0x55c61ab05ea0.capsfilter1.src: mean=0:00:00.000009136 min=0:00:00.000002785 max=0:00:00.000039395
0x55c61a80ebf0.rtph265depay0.src: mean=0:00:00.000035421 min=0:00:00.000019036 max=0:00:00.001843301
0x55c61a90bbd0.queue0.src: mean=0:00:00.000013551 min=0:00:00.000008907 max=0:00:00.000028313
0x55c61aa337f0.h265parse0.src: mean=0:00:00.000118532 min=0:00:00.000057499 max=0:00:00.001058128
0x55c61ab02100.capsfilter2.src: mean=0:00:00.000013271 min=0:00:00.000008676 max=0:00:00.000036559
0x55c61a880640.queue1.src: mean=0:00:00.000020159 min=0:00:00.000008676 max=0:00:00.001672268
0x55c61ab7d2e0.vah265dec0.src: mean=0:00:00.099842792 min=0:00:00.049800864 max=0:00:00.120715323
0x55c61a6ff020.capsfilter3.src: mean=0:00:00.000017780 min=0:00:00.000007975 max=0:00:00.000036318
0x55c61ab6b040.queue2.src: mean=0:00:00.000084689 min=0:00:00.000015750 max=0:00:00.000448497
0x55c61a910a00.gluploadelement0.src: mean=0:00:00.000059767 min=0:00:00.000031881 max=0:00:00.000378325
0x55c61a884b60.glcolorconvertelement0.src: mean=0:00:00.000011727 min=0:00:00.000008746 max=0:00:00.000036179
0x55c61aae5390.glcolorbalance0.src: mean=0:00:00.000011701 min=0:00:00.000008977 max=0:00:00.000027973
Element Reported Latency:
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451577889
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451589761
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451596213
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451604469
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.451613756
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.451619778
0x55c61aa337f0.h265parse0: min=0:00:00.033333333 max=0:00:00.000000000 ts=0:00:00.451625208
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452703745
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452719985
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452726798
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452732198
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.452737779
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452743360
0x55c61aa337f0.h265parse0: min=0:00:00.033333333 max=0:00:00.000000000 ts=0:00:00.452748509
0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.452753499
0x55c61a880640.queue1: min=5124095:34:33.676218283 max=0:00:00.000000000 ts=0:00:00.452758789
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453004593
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453018700
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453027326
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453034800
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.453055419
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453063274
0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.453078442
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454061529
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454071748
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454077459
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454082459
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.454087879
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454097597
0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454106734
0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454113537
0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.454120901
0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.454126211
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525850890
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525866730
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525873834
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525879454
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.525885065
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525894312
0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525904942
0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525912256
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525902348
0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525918067
0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.525935610
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525935650
0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525947172
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525956400
0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.525961940
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525968623
0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525975436
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.525983551
0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525988240
0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525995904
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.525999291
0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526016032
0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526003128
0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526024538
0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526037162
0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.526045558
0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526053062
0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.526060767
0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526068592
0x55c61a9124d0.tcpclientsrc0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526069313
0x55c61aab37f0.capsfilter0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526082778
0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526076697
0x55c61aa24230.rtpstreamdepay0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526089962
0x55c61ab05ea0.capsfilter1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526096905
0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526099730
0x55c61a80ebf0.rtph265depay0: min=0:00:00.000000000 max=99:99:99.999999999 ts=0:00:00.526102175
0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526108878
0x55c61a90bbd0.queue0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526117704
0x55c61aa337f0.h265parse0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526127473
0x55c61ab02100.capsfilter2: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526132993
0x55c61a880640.queue1: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526138263
0x55c61ab7d2e0.vah265dec0: min=0:00:00.100000000 max=0:00:00.000000000 ts=0:00:00.526143343
0x55c61a6ff020.capsfilter3: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526148242
0x55c61ab6b040.queue2: min=5124095:34:33.609551616 max=0:00:00.000000000 ts=0:00:00.526153342
0x55c61a9023b0.glimagesinkbin0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526158511
0x55c61a910a00.gluploadelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526164553
0x55c61a884b60.glcolorconvertelement0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526171486
0x55c61aae5390.glcolorbalance0: min=0:00:00.000000000 max=0:00:00.000000000 ts=0:00:00.526177688
I'm currently trying very hard to shake that latency of 100ms
introduced by the decoder down to ~0ms
Has anyone any idea WHY vah265dec
is doing so. I also tried:
- avdec_h265
- libde265dec
- vaapih265dec
all with the same result.
I also tried a all-in-one pipeline, with no DepthAi involved
#!/usr/bin/env bash
set -e
(
export GST_DEBUG="GST_TRACER:7"
export GST_TRACERS='latency(flags=pipeline+element)'
# export GST_TRACERS='latency(flags=pipeline+element)'
export GST_DEBUG_FILE=trace.log
# export GST_DEBUG="3"
export GST_DEBUG_DUMP_DOT_DIR="$(pwd)"
rm trace.log || true
gst-launch-1.0 \
videotestsrc num-buffers=30 is-live=true \
! video/x-raw,width=1280,height=720,framerate=10/1 \
! vah265enc b-frames=0 \
! h265parse \
! rtph265pay aggregate-mode=zero-latency config-interval=-1 pt=96 \
! rtpstreampay \
! application/x-rtp-stream \
! rtpstreamdepay \
! application/x-rtp,media=video,clock-rate=90000,encoding-name=H265 \
! rtph265depay \
! h265parse config-interval=1 \
! video/x-h265,stream-format=hvc1,alignment=au \
! vah265dec qos=false \
! fakesink sync=false qos=false
)
gst-stats-1.0 trace.log
This does NOT introduce any latencies.
So because ALL decoders introduce that latency of 3 frames, and other encoders DO NOT introduce that latency, I'm beginning to blame the h265 stream š
Here's my DepthAI code:
pipeline = dai.Pipeline()
# Define sources and output
camRgb = pipeline.create(dai.node.ColorCamera)
videoEnc = pipeline.create(dai.node.VideoEncoder)
xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("h265")
# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_720_P)
camRgb.setFps(30)
videoEnc.setDefaultProfilePreset(
camRgb.getFps(), dai.VideoEncoderProperties.Profile.H265_MAIN
)
videoEnc.setNumBFrames(0)
videoEnc.setBitrateKbps(10000) # 10 Mbps
videoEnc.setKeyframeFrequency(240)
# Linking
camRgb.video.link(videoEnc.input)
videoEnc.bitstream.link(xout.input)
I disabled the BFrames, which is the only idea I had, why there might be a latency.
Anyone any other idea, on why the decoder needs to cache 3 frames/100ms to decode the DepthAI h265 stream?
The camera is a OAK-D S2
Any help is greatly appreciated, thanks for your time! š