I'm trying to generate output(GPIO) synced with specific frame based on input(GPIO) status. The output and frame should sync in 1ms. I assume script node feature could achieve this goal by following way: 1. Get images from color camera node and send frames to a script node. 2. Get frame and input(GPIO) in the script node 3. Based on the inputs, set output(GPIO) and send frame to host. In second step, I think `frame = node.io['frames'].get()` is the way to access frame from script node. But how long does it take to send frame and how much is the deviation? I use OAK-D Pro PoE. Any help will be appreciated. Thanks,

Thank you @"erik"#p6191 ! To measure the time lag, I'm trying with following script. ``` import time while True: frame = node.io['frames'].get() node.warn(f"[{frame.getSequenceNum():08}] Time lag: {time.monotonic() - frame.getTimestamp().total_seconds()}") ``` Result in my environment are: 1. `time.monotonic() - frame.getTimestamp().total_seconds()` returns around 0.35-0.37 for 60 fps for (1080, 720). And the sequence number was not continuous. 2. `time.monotonic() - frame.getTimestamp().total_seconds()` returns around 0.313 for 30 fps for (1080, 720). And the sequence number seems to be continuous. I'm not sure these timestamp comparison make sense. So please tell me where should I confirm? Thanks,

Hi @"izumi"#p6378 , My guess is non-blocking behaviour and and too many messages (as device can't send frames fast enough, they buffer up), some [docs here](https://docs.luxonis.com/projects/api/en/latest/components/device/). But one would need to test this (changing blocking type / queue sizes of different nodes) to be sure. Thanks, Erik

Hi @"izumi"#p6401 , Other type of queues are also [device-side queues](https://docs.luxonis.com/projects/api/en/latest/components/nodes/), which is then probably the culprit (if it's not host-side queue that's dropping frames). Thanks, Erik

Hi @"izumi"#p6414 , Please provide [full MRE](https://docs.luxonis.com/en/latest/pages/support/) with pipeline graph - as msg drop can happen at any node, not only at Script node. Example: xLinkOut.

How much is the time lag to get frame in script node?

izumi

I'm trying to generate output(GPIO) synced with specific frame based on input(GPIO) status.
The output and frame should sync in 1ms.
I assume script node feature could achieve this goal by following way:

Get images from color camera node and send frames to a script node.
Get frame and input(GPIO) in the script node
Based on the inputs, set output(GPIO) and send frame to host.

In second step, I think frame = node.io['frames'].get() is the way to access frame from script node. But how long does it take to send frame and how much is the deviation?

I use OAK-D Pro PoE.

Any help will be appreciated.
Thanks,

erik

Hi izumi ,
Yes, the .get() will block until the message arrives. You can send it via node.op['name'].send(depthai_msg).I don't think we have done any benchmarks and I'm also not sure how we could measure that, but it should be below 1ms if system isn't overloaded.
Thoughts?
Thanks, Erik

izumi

Thank you erik !
To measure the time lag, I'm trying with following script.

import time

while True:
    frame = node.io['frames'].get()
    node.warn(f"[{frame.getSequenceNum():08}] Time lag: {time.monotonic() - frame.getTimestamp().total_seconds()}")

Result in my environment are:

time.monotonic() - frame.getTimestamp().total_seconds() returns around 0.35-0.37 for 60 fps for (1080, 720). And the sequence number was not continuous.
time.monotonic() - frame.getTimestamp().total_seconds() returns around 0.313 for 30 fps for (1080, 720). And the sequence number seems to be continuous.

I'm not sure these timestamp comparison make sense. So please tell me where should I confirm?
Thanks,

izumi

Hi @erik ,
I try following script to check frame drop in script node. This script prepare color stream node and the script node check received sequence number. If there is frame drop, then the node warn it. This script doesn't provide frame drop more than 10 min. But if I uncomment 35 and 40 to enable color stream to host, Frame drops are warned in script node. If the node.io['name'].get() in script node running in device, why frame is dropped?

Thank you,

import depthai as dai

fps = 10
pipeline = dai.Pipeline()

# Color camera
colorCam = pipeline.createColorCamera()
colorCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
colorCam.setFps(fps)
colorCam.setBoardSocket(dai.CameraBoardSocket.RGB)
colorCam.setInterleaved(False)

# Script node
io_node = pipeline.create(dai.node.Script)
script = """
import time
preSeqNum = 0
while True:
    cf = node.io['c'].get()
    seqNum = cf.getSequenceNum()
    if seqNum - preSeqNum > 1:
        node.warn(f"Drop {seqNum - preSeqNum} frames at {seqNum}")
    preSeqNum = seqNum
"""
io_node.setScript(script)

# Linking
xout_color = pipeline.createXLinkOut()
xout_color.setStreamName("color")

xout_color_stream = pipeline.createXLinkOut()
xout_color_stream.setStreamName("color_stream")

colorCam.isp.link(io_node.inputs['c'])
colorCam.isp.link(xout_color_stream.input)
io_node.outputs["color"].link(xout_color.input)

with dai.Device(pipeline) as device:
    q_color = device.getOutputQueue("color")
    q_color_stream = device.getOutputQueue("color_stream")
    print("Started")
    while True:
        color_frame = q_color.get()

erik

Hi izumi ,
I tried the code without modification (so uncommented) and it works as expected, no frames dropped, using USB3 connection. Perhaps with lower bandwidth problems could arise, as link isn't able to send all the data through.
Thanks, Erik

izumi

Hi @erik ,
Thanks for sharing your result. I tried with OAK-D Pro PoE. So as you pointed out, the bandwidth might effect result. In this script, I added one xlink by uncomment but I didn't change any script on the device. Then frame drop happens on the device. I wonder why the bandwidth between host and camera effects frame drop on the device? Please share your insight.

Thank you,

erik

Hi izumi ,
My guess is non-blocking behaviour and and too many messages (as device can't send frames fast enough, they buffer up), some docs here. But one would need to test this (changing blocking type / queue sizes of different nodes) to be sure.
Thanks, Erik

izumi

Hi erik ,
Thanks for sharing your insight.
I've tried following 2 parameters. But the frame drop frequency seems to be same...
q_color_stream = device.getOutputQueue("color_stream", maxSize=5, blocking=False)
q_color_stream = device.getOutputQueue("color_stream", maxSize=30, blocking=True)
Still I'm not sure why the frame drop at the queue between the device with the host effects frame drop at the queue between the sensor(the device) with the script node(the device).
I assume the parameters(blocking type / queue sizes) you pointed out are for the queues on the host computer (in RAM). These parameters may change behavior on host computer side but I don't understand how does it effect message handling on the device side. The frame drop for host might be make sense to me because of network stuff, but I have no idea why there is the frame drop in script node. I want to get color image stream and snapshot image at an event on script node. It is okay to have frame drop for stream, but for snapshot it shouldn't have frame drop. Any advice would be helpful. Thank you.

erik

Hi izumi ,
Other type of queues are also device-side queues, which is then probably the culprit (if it's not host-side queue that's dropping frames).
Thanks, Erik

izumi

Hi erik ,
Thanks for your input.
I tried following settings for device stream but it doesn't effect frame drop frequency in script queue.
io_node.inputs['c'].setQueueSize(10)
io_node.inputs['c'].setBlocking(True)
Any thoughts?
Thank you.

erik

Hi izumi ,
Please provide full MRE with pipeline graph - as msg drop can happen at any node, not only at Script node. Example: xLinkOut.

izumi

Hi erik ,
Please try following code with OAK-D Pro PoE and please compare printed out results by running with the code uncommented line 41 and 46. And I attached a pipeline graph, please tell me if I'm wrong. In this graph, I still not sure why adding XLinkOut effects frame drop in script node. It is nice to help me to understand what's going on. How can I get color image stream(frame droppable) and snapshot image(without frame drop) at an event on script node? Any advice would be helpful. Thank you!

import depthai as dai

fps = 30
pipeline = dai.Pipeline()

# Color camera
colorCam = pipeline.createColorCamera()
colorCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
colorCam.setFps(fps)
colorCam.setBoardSocket(dai.CameraBoardSocket.RGB)
colorCam.setInterleaved(False)

# Script node
io_node = pipeline.create(dai.node.Script)
script = """
import time
preSeqNum = 0
while True:
    cf = node.io['c'].get()
    seqNum = cf.getSequenceNum()
    if seqNum - preSeqNum > 1:
        node.warn(f"Drop {seqNum - preSeqNum} frames at {seqNum}")
    if seqNum % 10 == 0:
        node.io['color'].send(cf)
    preSeqNum = seqNum
"""
io_node.setScript(script)
io_node.inputs['c'].setQueueSize(10)
io_node.inputs['c'].setBlocking(True)

# Linking
xout_color = pipeline.createXLinkOut()
xout_color.setStreamName("color")

xout_color_stream = pipeline.createXLinkOut()
xout_color_stream.setStreamName("color_stream")
xout_color_stream.input.setBlocking(False)
xout_color_stream.input.setQueueSize(10)

colorCam.isp.link(io_node.inputs['c'])
# colorCam.isp.link(xout_color_stream.input)
io_node.outputs["color"].link(xout_color.input)

with dai.Device(pipeline) as device:
    q_color = device.getOutputQueue("color")
    # q_color_stream = device.getOutputQueue("color_stream", maxSize=30, blocking=False)
    print("Started")
    while True:
        color_frame = q_color.get()
        print(f"Received sequence number: {color_frame.getSequenceNum()}")

This is output in my environment.

(Original)
Started
[-] [-] [5.556] [Script(1)] [warning] Drop 3 frames at 7
Received sequence number: 0
Received sequence number: 10
Received sequence number: 20
Received sequence number: 30
Received sequence number: 40
Received sequence number: 50
...
(With uncomment)
Started
Received sequence number: 0
[-] [-] [5.614] [Script(1)] [warning] Drop 3 frames at 7
[-] [-] [5.616] [Script(1)] [warning] Drop 2 frames at 9
Received sequence number: 10
Received sequence number: 20
[-] [-] [6.162] [Script(1)] [warning] Drop 2 frames at 24
Received sequence number: 30
[-] [-] [6.501] [Script(1)] [warning] Drop 2 frames at 34
...
Received sequence number: 150
[-] [-] [10.502] [Script(1)] [warning] Drop 2 frames at 154
Received sequence number: 160
[-] [-] [10.766] [Script(1)] [warning] Drop 2 frames at 162
[-] [-] [11.011] [Script(1)] [warning] Drop 2 frames at 170
Received sequence number: 170
[-] [-] [11.279] [Script(1)] [warning] Drop 2 frames at 178
...

erik

Hi izumi ,
So just streaming 1080P frames at 30FPS likely won't be possible - total 750mbps, while GigE on OAK, from out tests, can do up to 700mpbs (due to some overhead).

The reason for dropping is the pool size of ColorCamera (see pool/output queue docs here). As you had a large input queue to XLinkOut, all frames were buffered there until OAK was able to send another frame to host, and after that, pointer was returned back to the pool and new image was taken (before that it wasn't, as pool was saturated).

Fixed code here:

Thanks, Erik

izumi

Hi erik ,
Sorry for my late reply and thank you very much for your help. Your code works without frame drop!
Please let me confirm my understanding. So in order to avoid frame drop, the pool size should be larger than sum of queue size(in my case sum of size of xout_color_stream.input and io_node.inputs['c']) at least. Am I collect? If I miss something please tell me.
Thank you!!!

erik

Hi izumi ,
Yep correct, and the input queue should be blocking.
Thanks, Erik