Depthai Viewer

michaelsalerno

I continue to appreciate all of the support your team has provided. While running the depthai viewer I was adjusting the viewing windows to full screen to optimize the viewer, but when I do that the system begins to lag and then closes abruptly. The following error is output. Can I not view full-screen while in the viewer?

thread 'ThreadId(1)' panicked at 'Can't maximize this space view', re_viewer/src/ui/auto_layout.rs:523

0: wgpu_render_pass_execute_bundles

1: wgpu_render_pass_execute_bundles

2: PyInit_depthai_viewer_bindings

3: PyInit_depthai_viewer_bindings

4: wgpu_render_pass_execute_bundles

5: wgpu_render_pass_execute_bundles

6: wgpu_render_pass_execute_bundles

7: wgpu_render_pass_execute_bundles

8: wgpu_render_pass_execute_bundles

9: PyInit_depthai_viewer_bindings

10: PyInit_depthai_viewer_bindings

11: PyInit_depthai_viewer_bindings

12: PyInit_depthai_viewer_bindings

13: PyInit_depthai_viewer_bindings

14: PyInit_depthai_viewer_bindings

15: PyInit_depthai_viewer_bindings

16: PyInit_depthai_viewer_bindings

17: PyInit_depthai_viewer_bindings

18: PyInit_depthai_viewer_bindings

19: PyInit_depthai_viewer_bindings

20: PyInit_depthai_viewer_bindings

21: PyInit_depthai_viewer_bindings

22: PyInit_depthai_viewer_bindings

23: PyInit_depthai_viewer_bindings

24: PyInit_depthai_viewer_bindings

25: PyInit_depthai_viewer_bindings

26: DispatchMessageW

27: CallWindowProcW

28: wglSwapBuffers

29: DispatchMessageW

30: DispatchMessageW

31: GetClassLongW

32: KiUserCallbackDispatcher

33: NtUserDispatchMessage

34: DispatchMessageW

35: PyInit_depthai_viewer_bindings

36: PyInit_depthai_viewer_bindings

37: PyInit_depthai_viewer_bindings

38: PyInit_depthai_viewer_bindings

39: wgpu_render_bundle_insert_debug_marker

40: PyInit_depthai_viewer_bindings

41: wgpu_render_bundle_insert_debug_marker

42: wgpu_render_bundle_insert_debug_marker

43: wgpu_render_bundle_insert_debug_marker

44: wgpu_render_bundle_insert_debug_marker

45: PyArg_ParseTuple_SizeT

46: PyEval_EvalFrameDefault

47: PyEval_EvalFrameDefault

48: PyObject_GC_Del

49: PyEval_EvalCodeWithName

50: PyEval_EvalCodeEx

51: PyEval_EvalCode

52: PyEval_GetBuiltins

53: PyEval_GetBuiltins

54: PyEval_EvalFrameDefault

55: PyObject_GC_Del

56: PyEval_EvalFrameDefault

57: PyObject_GC_Del

58: PyFunction_Vectorcall

59: PyVectorcall_Call

60: PyObject_Call

61: Py_Exit

62: Py_RunMain

63: Py_RunMain

64: Py_Main

65: <unknown>

66: BaseThreadInitThunk

67: RtlUserThreadStart

Troubleshooting Rerun: https://www.rerun.io/docs/getting-started/troubleshooting

[2024-05-07T18:27:55Z WARN re_sdk_comms::buffered_client] Failed to send message: Failed to send to Rerun server at [127.0.0.1:9876]: An existing connection was forcibly closed by the remote host. (os error 10054)

Resetting...

Closing device...

PS C:\Users\sale4088\AppData\Local\Programs\DepthAI> Done

[2024-05-07 11:27:57] INFO [websockets.server.handler:264] connection closed

jakaskerl

michaelsalerno
Huh, I've tried this on MacOS and have no issues.

Could you describe this in more detail along with viewer version / OS?

I'll pass it on to the devs so we can have it fixed asap.

Thanks,
Jaka

michaelsalerno

@jakaskerl ,

I get this issue when I also run the Demo that is provided in the apps folder in the Depthai Experiments Github page.

I am using a windows 10 laptop, and when I started the interface I began to resize both the RGB viewer and the Depth viewer, to get a better understanding of the picture. Once I adjusted the sizes the system began to slow very quickly before freezing and closing. The error message is then put out. I am not entirely sure what is happening. I have also had to start turning off the Point cloud generation feature because it causes additional stress when I am running the application. Please let me know if you need anymore information.

michaelsalerno

@jakaskerl ,

Hey, I wanted to let you know. I was working with the viewing window again today and after only about 45 seconds of operation, the program begins to "freak out". Basically, the image freezes, then after it starts to close and try to reload every window multiple times causing the screen to be unfunctional while they are moving around. Then it stops and reloads everything as if I just opened it, then it continues to go back to freezing before closing. This has made it nearly impossible to function with this interface. Can you please help? I am attempting to finish my work since I am expected to present at an international conference and would really like to show your equipment and technology to the highest degree. Please let me know any suggestions, or if you need any information from me.

filipjeretina

Hi @michaelsalerno , I find it weird that the even the depthai-experiment app (I am assuming this one: apps/conference-demos/rgb-depth-detections/main.py) crashes. Could you expand a bit on how that app crashes? Ideally if you could send over the logs or maybe a video recording of the crash.
It seems like the issues are linked with each other and probably have a common causality, but it will be easier to debug the depthai-experiment app.
We have started the app internally on a windows laptop where a similar depthai-viewer issue has been observed in the past (viewer windows "freaking out" after some time), we will see if we get a crash.

michaelsalerno

filipjeretina @jakaskerl

The file I am uploading is saying that it is to large, I can take a couple different videos of what is happening between both the Depthai Viewer and the App demo.

Please let me know if I can send it to an email or someone at your company.

In regards to the crashing with the viewer, the application will turn on, and begin to start projecting in the viewing window. Initially, it begins to lag, but that is just from the start up I am assuming. After that is done, the viewing windows begin to function for a little while (45 seconds - 1 minute) approximately before crashing.

If I manipulate the size of any of the viewing windows in the viewer, the application begins to lag, but that is again most likely due to the re-sizing of the image. But then, after, it begins to freeze more before changing the location of all the viewing windows. It seems to try and re-set them as if I started the application over again, but since I didn't it keeps rapidly changing the location and size of the windows. I know its them because the titles still remain while it is doing this.

In regards to the application, I first started it in my office. Once it starts it begins to work again for about 30 seconds, before slowing down. Sometimes during this slow down, the AI segmentation/identifier that is used begins to spam identify one or two particular objects which can crash the viewing windows. In cases where this didn't happen the windows slowed to a point where it froze the screen which caused a force close of the application.

Please let me know if there is anything else I can provide. Again I would love to send you a couple of test examples but I can not upload them to the website due to the file sizes being to large. I can try and record it off my laptop from my phone but that might reduce the image quality.

filipjeretina

@MichaelBerger hi,
Please do send the videos over / upload them to drive or something like that. My email: filip.jeretina@luxonis.com

I am thinking this could be a problem with data throughput - since the viewer has a timeout of 200ms between messages and if no new messages are received in this time, the viewport is cleared. This could explain the viewport going crazy behaviour. Then the app crashes because I think you found an edge case in the code (when there are no views in the viewport and we try to lay them out) … : )

Could you try lowering the camera fps to something like 5 fps just as a test to see if the issue dissappears?

michaelsalerno

@filipjeretina ,

Sounds great I can try that now, I will also send you some videos to show what is happening.

michaelsalerno

@filipjeretina

I just sent you the first email. I will continue to send emails with the additional videos. they are to large to send all at once, but all the videos will be in the same email chain.

michaelsalerno

@filipjeretina ,

Running it at 5 FPS seems to prevent it from crashing. I attempted to replicate what I did earlier but I could not replicate the viewing window spam sadly.

filipjeretina

Nice! @michaelsalerno thanks for trying it out.. Now - I think you should be able to push a few more FPS something like 10-15 perhaps… Hope this works for you. The same can be applied to the demo app, or better yet - if you are using a usb device, you could limit the USB speed to HIGH (USB2) in the device constructor.

michaelsalerno

@filipjeretina ,

How would I adjust the speed in the app code? And since this is a known problem from my device, can I adjust the FPS for all the experimental codes?

filipjeretina

michaelsalerno

The simplest would be to limit the usb speed.
For example here:
luxonis/depthai-experimentsblob/e0bd127018f375847bec5b3e8a434f842796796e/apps/conference-demos/rgb-depth-detections/main.py#L177

You could add a bool to the dai.Device like this:
ẁith dai.Device(pipeline=pipeline, usb2Mode=True) as device:

michaelsalerno

@jakaskerl @filipjeretina ,

Is there a way to get real world measurements from the Depthai Viewing interface? I am curious to know if I would be able to directly get, or be able to calculate based on the information from the viewer. Please let me know if this is available.

filipjeretina

michaelsalerno
I am not entirely sure what you are referring to in regards to real world measurements. Could you expand?
You can see the depth measurement for a point when you hover over the depth image, if this is what you meant?

michaelsalerno

@filipjeretina ,

Thank you for your assistance for the previous question, it will be really helpful and useful.

In regards to measurements, although the depth measurement is useful, I was curious if it was able to determine what the size of something is at depth, for instance, knowing that someone who is 6 meters away is 6 feet tall, rather than just 2 inches tall according to pixel values. Please let me know if this makes sense.

jakaskerl

michaelsalerno
I guess you could achieve this through https://docs.luxonis.com/projects/api/en/latest/components/nodes/yolo_spatial_detection_network/ where you could take the bbox coordinates and fuse them with the spatials to get a rough estimate of the height of the detected object. Of course the bbox would have to accurately align to the objects silhouette.

Thanks,
Jaka