AmmarKothari
I'll forward this to the dev team so we can fix it asap.
What hardware are you using as the "computer"? Sees like the network interface crashes...
Thanks,
Jaka
Computer Freezes Seemingly related to Oak Camera
Hi Jaka,
I appreciate the response. I am using this computer: https://premioinc.com/products/rco-1000-ehl-10. Ah that is interesting that the networking interface is crashing. If there is any additional logging I can enable or places to look for logs that would be helpful, please let me know.
Thank you!
- try different ETH port
- Check dmesg for any logs prior to the crash
- update network driver
- Try with a non-lowlatency kernel
Thanks,
Jaka
I have tried a different ETH port and had the same result. Sometimes the message I copied into the first message appears. Are there any clues there to where else I should be looking?
Is there a recommended or approved network driver list I can reference?
I can try with a non-lowlatency kernel. Is there a list of approved kernels that luxonis has tested? I'd love to be able to choose one of those to reduce the number of variations from a tested setup by Luxonis.
Thank you!
- Edited
AmmarKothari
If you can, easiest way would be to test on an ubuntu OS with the default kernel... Regarding the driver, I just mentioned it in case it was outdated and the patch is already available..
Though it could just be a power issue as well.. Is there a POE switch in the middle?
Thanks,
Jaka
Yes there is a POE switch in the middle. We have used this POE switch successfully in other deployments. I can experiment with another POE device as well.
Ah ok i'll try out the default kernel for 22.04.4 . I'll see if there are any network driver updates, but i am pretty sure its all up to date.
Was there any information from the stack trace in my first post that helps indicate where the issue is?
AmmarKothari
Perhaps but it is not directly visible. Seems to be related to the network interface.
Thanks,
Jaka
Hi Jaka,
I did some more debugging and reinstalled python. I am now seeing this error. This seems to be coming from the depth ai library. It has not caused the computer to freeze yet, but seems concerning. Is this a known bug?
Nov 12 18:28:31 SCN-009 python[2147]: Stack trace (most recent call last) in thread 2196:
Nov 12 18:28:31 SCN-009 python[2147]: #9 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
Nov 12 18:28:31 SCN-009 python[2147]: #8 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f235dcbc84f, in
Nov 12 18:28:31 SCN-009 python[2147]: #7 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f235dc2aac2, in
Nov 12 18:28:31 SCN-009 python[2147]: #6 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f2337bc161f, in
Nov 12 18:28:31 SCN-009 python[2147]: #5 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f233777aae1, in
Nov 12 18:28:31 SCN-009 python[2147]: #4 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f23377789f5, in
Nov 12 18:28:31 SCN-009 python[2147]: #3 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f2337502e03, in
Nov 12 18:28:31 SCN-009 python[2147]: #2 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f2337475098, in
Nov 12 18:28:31 SCN-009 python[2147]: #1 Object "/home/scan/.local/lib/python3.10/site-packages/depthai.cpython-310-x86_64-linux-gnu.so", at 0x7f2337475084, in
Nov 12 18:28:31 SCN-009 python[2147]: #0 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f235dfce6a8, in _PyThreadState_DeleteCurrent
Nov 12 18:28:31 SCN-009 python[2147]: Segmentation fault (Address not mapped to object [0x7])
Here is another suspect stack trace. This one doesn't reference the depthai library specifically, but the calls seem to be from the library based on the naming.
Nov 13 08:51:44 SCN-009 python[2189]: #31 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #30 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f06450017ea, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #29 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #28 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f06450017ea, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #27 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064505dd03, in
Nov 13 08:51:44 SCN-009 python[2189]: #26 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #25 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f0645004eca, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #24 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #23 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f0645007cc6, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #22 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064505dc67, in
Nov 13 08:51:44 SCN-009 python[2189]: #21 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #20 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f0645007ed3, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #19 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064505ae8b, in _PyObject_MakeTpCall
Nov 13 08:51:44 SCN-009 python[2189]: #18 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f06450c0c1d, in
Nov 13 08:51:44 SCN-009 python[2189]: #17 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f06450c906d, in
Nov 13 08:51:44 SCN-009 python[2189]: #16 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064505b33f, in _PyObject_Call_Prepend
Nov 13 08:51:44 SCN-009 python[2189]: #15 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064505b00a, in _PyObject_FastCallDictTstate
Nov 13 08:51:44 SCN-009 python[2189]: #14 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f064514dc43, in
Nov 13 08:51:44 SCN-009 python[2189]: #13 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f0645007cc6, in _PyEval_EvalFrameDefault
Nov 13 08:51:44 SCN-009 python[2189]: #12 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f06450a4b98, in
Nov 13 08:51:44 SCN-009 python[2189]: #11 Object "/home/scan/.local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so", at 0x7f0642e64374, in
Nov 13 08:51:44 SCN-009 python[2189]: #10 Object "/home/scan/.local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so", at 0x7f0642dc7186, in
Nov 13 08:51:44 SCN-009 python[2189]: #9 Object "/home/scan/.local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so", at 0x7f0642dc6ba8, in
Nov 13 08:51:44 SCN-009 python[2189]: #8 Object "/home/scan/.local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so", at 0x7f0642da8040, in
Nov 13 08:51:44 SCN-009 python[2189]: #7 Object "/home/scan/.local/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so", at 0x7f0642da6e7e, in
Nov 13 08:51:44 SCN-009 python[2189]: #6 Object "/home/scan/.pyenv/versions/3.10.15/lib/libpython3.10.so.1.0", at 0x7f0645096220, in PyDict_GetItem
Nov 13 08:51:44 SCN-009 python[2189]: #5 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644e8f565, in __stack_chk_fail
Nov 13 08:51:44 SCN-009 python[2189]: #4 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644e8f599, in __fortify_fail
Nov 13 08:51:44 SCN-009 python[2189]: #3 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644de2675, in
Nov 13 08:51:44 SCN-009 python[2189]: #2 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644d817f2, in abort
Nov 13 08:51:44 SCN-009 python[2189]: #1 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644d9b475, in raise
Nov 13 08:51:44 SCN-009 python[2189]: #0 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f0644def9fc, in pthread_kill
Nov 13 08:51:44 SCN-009 python[2189]: Aborted (Signal sent by tkill() 2189 1000)
- Edited
@AmmarKothari We may be seeing something similar here with the OAK-D Pro PoE, though we're still struggling to reproduce the issue consistently.
Does your whole computer freeze or just the windowing server? In our case, most freezes are momentary (3-5 seconds) then the GPU resets. In rarer cases, the freeze is indefinite and we need to SSH in and reboot.
@paulmurphy I appreciate you sharing that. Its helpful to know its more general than just a few of our units. Hopefully we can get to the root cause!
We are using the Pro POE. We have issues where it seems like the computer completely freezes (logs abruptly stop in journal) and some where python crashes, but the computer is fine and I can ssh in and restart the script. I don't think we see issues where the freeze is momentary although, i am not sure we would notice that in our use case.
The computer we are using doesn't have a GPU if that is a helpful data point.
Is there a short term fix that I can try? We are seeing this on some deployed units. I tried rolling back to version 2.26, but saw a similar freezing issue.
AmmarKothari
Does it help if you use some simple example (like rgb_preview or even simpler)? The last error you have sent could point to a depthai issue, but I am unsure whether it is caused by the pipeline or there is something else going on.
Thanks,
Jaka
I can try that. It often takes hours for the issue to appear so the debug iteration is quite slow.