Request for assistance with OAK-D POE CM4 and Jetson TX2 integration

hello jakaskerl

I followed this setup guide (https://github.com/luxonis/depthai-experiments/tree/422abacfab83fe8fa2035b80c46f9afdc7a1c374/gen2-deepsort-tracking) for DeepSort, but I cannot determine with certainty whether the latency is coming from the OAK module or the Raspberry Pi's processing, as they are tightly integrated and interdependent.

jakaskerl

I am using an Oak-D PoE CM4 camera, so all the processing is done directly on the Raspberry Pi board. To measure the latency, I view the camera feed on the Raspberry Pi by connecting via SSH from my computer.

Hi jakaskerl

But how to launch the program without connecting via SSH? By opening the camera and plugging in an HDMI cable? I tried that, but I'm not receiving any image on my screen.

jakaskerl

Here are the results I obtained by running the code on my Raspberry Pi via an SSH connection:

• Latency: 108.63 ms, Average latency: 103.88 ms, Standard deviation: 10.33

• Latency: 112.14 ms, Average latency: 103.90 ms, Standard deviation: 10.32

• Latency: 115.76 ms, Average latency: 103.92 ms, Standard deviation: 10.33

• Latency: 99.22 ms, Average latency: 103.91 ms, Standard deviation: 10.32

Hi jakaskerl

Thank you for your previous insights. I want to clarify that the latency measurements I shared with you earlier were taken without showing the preview (I had commented out `cv2.imshow('frame', imgFrame.getCvFrame())`).

After including the preview display in the computation, here are the new values I obtained:

Latency: 481.45 ms, Average latency: 527.23 ms, Std: 36.31

Latency: 492.62 ms, Average latency: 527.20 ms, Std: 36.31

Latency: 488.96 ms, Average latency: 527.18 ms, Std: 36.31

Latency: 486.37 ms, Average latency: 527.15 ms, Std: 36.31

Latency: 496.27 ms, Average latency: 527.13 ms, Std: 36.31

Latency: 492.14 ms, Average latency: 527.10 ms, Std: 36.31

Latency: 503.84 ms, Average latency: 527.09 ms, Std: 36.30

Latency: 515.83 ms, Average latency: 527.08 ms, Std: 36.29

Latency: 507.38 ms, Average latency: 527.07 ms, Std: 36.28

Latency: 507.18 ms, Average latency: 527.05 ms, Std: 36.27

Latency: 498.62 ms, Average latency: 527.03 ms, Std: 36.27

Latency: 515.08 ms, Average latency: 527.02 ms, Std: 36.25

As you can see, adding the preview display significantly increases the latency.

Best,

Babacar

jakaskerl

Hi Jaka,

Apologies for the delay in response. I want to confirm whether this is the correct modification to the code that you requested:

import depthai as dai

import numpy as np

import time

# Create pipeline

pipeline = dai.Pipeline()

pipeline.setXLinkChunkSize(0)

# Define source and output

camRgb = pipeline.create(dai.node.ColorCamera)

camRgb.setFps(60)

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

xout = pipeline.create(dai.node.XLinkOut)

xout.setStreamName("out")

camRgb.isp.link(xout.input)

# Connect to device and start pipeline

with dai.Device(pipeline) as device:

print(device.getUsbSpeed())

q = device.getOutputQueue(name="out")

diffs = np.array([])

while True:

start_time = time.time() # Record start time of the loop

imgFrame = q.get()

latencyMs = (dai.Clock.now() - imgFrame.getTimestamp()).total_seconds() * 1000

diffs = np.append(diffs, latencyMs)

print('Latency: {:.2f} ms, Average latency: {:.2f} ms, Std: {:.2f}'.format(latencyMs, np.average(diffs), np.std(diffs)))

end_time = time.time() # Record end time of the loop

loop_time = (end_time - start_time) * 1000 # Calculate loop time in ms

print('Loop time: {:.2f} ms'.format(loop_time))

Please let me know if this is correct, or if there are any further changes that I should make.

Thanks,
Babacar

Hi jakaskerl

Here's the code that I implemented:

import time

# ...

while True:

# Begin timing

start_time = time.time()

for name, q in queues.items():

    # Add all msgs (color frames, object detections and recognitions) to the Sync class.

    if q.has():

        sync.add_msg(q.get(), name)

msgs = sync.get_msgs()

if msgs is not None:

    frame = msgs["color"].getCvFrame()

    detections = msgs["detection"].detections

    embeddings = msgs["embedding"]

    # Write raw frame to the raw_output video

    raw_out.write(frame)

    # Update the tracker

    object_tracks = tracker_iter(detections, embeddings, tracker, frame)

    # For each tracking object

    for track in object_tracks:

        #... All existing code 

    # Write the frame with annotations to the output video

    out.write(frame)

# End timing and print elapsed time

end_time = time.time()

elapsed_time = end_time - start_time

print(f"Elapsed time for iteration: {elapsed_time} seconds")

raw_out.release()

out.release()

These are the results I got:

Elapsed time for iteration: 0.13381719589233398 seconds

Elapsed time for iteration: 0.1333160400390625 seconds

Elapsed time for iteration: 0.13191676139831543 seconds

...

...

Elapsed time for iteration: 0.13199663162231445 seconds

Thanks, Jaka, for your input so far.I would appreciate any further suggestions you might have to fix this issue.

jakaskerl

Following your advice, I've made some further modifications to my code and have also removed the video writing part. The changes have resulted in considerable improvements in the performance. However, the time taken per iteration now varies widely. Here's a subset of the results:

Elapsed time for iteration: 2.3365020751953125e-05 seconds

Elapsed time for iteration: 2.3603439331054688e-05 seconds

...

...

Elapsed time for iteration: 2.6702880859375e-05 seconds

Elapsed time for iteration: 2.3603439331054688e-05 seconds

Elapsed time for iteration: 3.361701965332031e-05 seconds

Elapsed time for iteration: 2.4080276489257812e-05 seconds

...

...

Elapsed time for iteration: 0.00014281272888183594 seconds

Elapsed time for iteration: 0.06066274642944336 seconds

Elapsed time for iteration: 0.05930662155151367 seconds

Elapsed time for iteration: 0.05977463722229004 seconds

Elapsed time for iteration: 0.06491947174072266 seconds

jakaskerl

Now I am going to try training my own model by following this tutorial:

https://github.com/luxonis/depthai-ml-training/blob/master/colab-notebooks/YoloV8_training.ipynb

But after, does DeepSORT support YOLOv8 and can YOLOv8 be imported in JSON format like YOLOv6 from the DeepSORT_Tracking GitHub repository?

Thank you for your help.

Hi Jaka,

I think there might have been a misunderstanding in our last exchange. I intend to train my YOLOv8 model using this code: https://github.com/luxonis/depthai-ml-training/blob/master/colab-notebooks/YoloV8_training.ipynb, and then import it in JSON format, as indicated in the tutorial.

I plan on using this specific DeepSORT repository from Luxonis

and I would like to, instead of launching it with yolov6.json, do it with a yolov8 that I have trained on my own database.

Furthermore, I'm not quite sure about the "deepsort/deepsort/detection" directory you mentioned. I don't see the yolov4.cfg and yolov4.weights files.

Could you provide more clarification on this?

Best regards,

Babacar

Hi erik

Thank you, I am currently training my model, afterwards I will try with the Deep SORT demo.