There's a warning that came up while running the camera
The warning advises to upgrade the firmware of the camera, but there isn't any docs about how to upgrade the camera firmware from 0.0.26 ~ 0.0.28. Any advices?
Here's more details about the stack used
Hi @syg , we'd suggest following this guide:
https://docs.luxonis.com/software/depthai-components/bootloader/#Danger%20Zone
Hello, following up on this.
Other than the camera firmware not being updated, are there any factors that could’ve caused the ping to miss for the cameras?
For context, we’ve 24 OAK-D Pro PoE cameras connected to a switch that supports 24 ethernet ports enabling us to connect all the cameras. Each of the ports is dual speed 100/1000Mbps SFP slots, and they can deliver 30W of power as the switch is configured to have IEEE 802.3at standard PoE+.
The service we’re deploying only activates 13 cameras and it’s booted in sequence. However, the issue starts when we wait for the cameras to boot. Out of the 13 cameras, up to 2~3 cameras will randomly fail to boot. So far here’s our remedy:
Kill the service
Wait for a while (30s)
Boot up the service
Hope that all the cameras have successfully booted w/o any pinging issues
We repeat steps 1~4 until all the cameras are booted. This is quite time-consuming and problematic as the connection might drop while the cameras run without warning.
syg
It's generally the bootloader issue so updating it is best you can do.
Also as another safety check:
# Check every second for new devices
while True:
device_infos = dai.Device.getAllAvailableDevices()
for device_info in device_infos:
if device_info.state != dai.XLinkDeviceState.X_LINK_BOOTLOADER:
continue
# Start thread here
sleep(1)Thanks,
Jaka
We've updated the camera boot loader version to the 0.0.28 which is the latest version.
However, we're facing another issue where an error is being thrown from one camera which eventually caused all cameras to stop their ping and our service to stopped. The attached image contains the error.
Specifically, F: [global] [ 606417] [Scheduler02Thr] dispatcherResponseServe:928 [19443010C172801300] [192.168.1.240] [1726580608.248] [host] [warning] Monitor thread (device: 19443010C172801300 [192.168.1.240]) - ping was missed, closing the device connection
What's the likely cause of this & is there anyway to catch this global error that's being thrown by the camera?
Each of our camera bandwidth consumption is less than 10mbps which result in a total usage of 13 * 10 = 130 Mbps which is well within our network maximum bandwidth of 1000mbps.
Is our only choice left with auto reconnect and being in standalone mode? Or are there any debugging options we can take?
If we're in standalone mode, is it possible to stream the frames from the camera to our server? If so are there any tutorials on it?
Hi Syg. Could you share how you updated the firmware. I followed the instructions in the link provided by jakaskeri
(https://docs.luxonis.com/software/depthai-components/bootloader/#Danger%20Zone)
but I can't get past 0.0.26 or depthai version 2.24.0.0
I've deleted the depthai and depthai-python foldesr then cloned it from the current repo master, I've followed the docs again, as though starting with a clean install, installed depthai, installed all requirements. The utility flashed the camera but still at 0.0.26
I'm at a loss to understand why my depthai version won't get past 2.24.0.0 when using the latest repo clone.
syg
Can you send me a link to the POE switch you are using? Could be that the power requirements exceed the specs (despite saying each port can output 30W). The devices die in a quick succession which could point to a power issue.
For standalone mode streaming:
luxonis/depthai-experimentstree/master/gen2-mjpeg-streaming
luxonis/depthai-experimentstree/master/gen2-poe-tcp-streaming
mikegardner
Pulling latest repo is not enough. The depthai version has to be latest. The version is the carrier of the bootloader.
Thanks,
Jaka
Thanks Jaka. I've deleted all of the folders and re-cloned them at the latest level and re-installed depthai.
Please consider my query on this thread closed now.
Regarding the drop-out I'm experiencing- I'll continue on my original thread.
Hey @jakaskerl, thanks for being patient with this
To give you a better overview of how our camera / architecture is setup, here is a diagram
Our server is connected to the cameras via two switches, one switch has all 1000mbps ports fully utilized by the Oak Cameras, it extends to another switch via 1000Base SFP module, the other switch is also rated for 1000Mbps
See details of both switches here:
Main network switch: https://www.volktek.com/productdetail_en.php?id=110
Unmanaged switch: https://www.volktek.com/productdetail_en.php?id=82
The power should be sufficient according to the calculations that we did.
As mentioned we are only actually using 13 cameras actively but the rest will still be consuming power, just not in use as it is not connected to. ( we may try to disconnect them and see if it improves the situation)
This issue only appears on boot and seems to happen more when we connect to 10 or more cameras simultaneously. Usually one or two cameras will have their ping missed.
However, in the off chance that all cameras successfully connect and boot the pipeline, the cameras are able to run for few days non-stop without interruption.
Here are some crash logs. I'd love to update files, but somehow the forum doesn't allow a json not zip file attachment
Crash Dump 1
{"crashReports":[{"crashedThreadId":184614928,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2201975248,"trapName":"illegal instruction","trapNumber":2}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2205053928,"calledTarget":2205069644,"context":"","framePointer":2213011968},{"callSite":2205046092,"calledTarget":2205053672,"context":"","framePointer":2213012064},{"callSite":2200950800,"calledTarget":2205046004,"context":"","framePointer":2213012200},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213012312},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213012408},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213012504}],"instructionPointer":2205053928,"stackBottom":2212881528,"stackPointer":2213011968,"stackTop":2213012599,"threadId":167837697,"threadName":"CBTH","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2213668384},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2213668488},{"callSite":2201029996,"calledTarget":2205044620,"context":"","framePointer":2213668608},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213668712},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213668808},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213668904}],"instructionPointer":2205044912,"stackBottom":2213537928,"stackPointer":2213668384,"stackTop":2213668999,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214062520},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214062624},{"callSite":2205052780,"calledTarget":2205049704,"context":"","framePointer":2214062744},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214062848},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214062944},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214063040}],"instructionPointer":2205044912,"stackBottom":2213932064,"stackPointer":2214062520,"stackTop":2214063135,"threadId":167837699,"threadName":"TIME","threadStatus":"READY"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214193744},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214193848},{"callSite":2204968012,"calledTarget":2205049704,"context":"","framePointer":2214193968},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214194072},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214194168},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214194264}],"instructionPointer":2205044912,"stackBottom":2214063288,"stackPointer":2214193744,"stackTop":2214194359,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214324968},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214325072},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214325192},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214325296},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214325392},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214325488}],"instructionPointer":2205044912,"stackBottom":2214194512,"stackPointer":2214324968,"stackTop":2214325583,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214456200},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214456304},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2214456400},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214456520},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214456616},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214456712}],"instructionPointer":2203927968,"stackBottom":2214325736,"stackPointer":2214456200,"stackTop":2214456807,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214587424},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214587528},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214587624},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214587744},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214587840},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214587936}],"instructionPointer":2203927968,"stackBottom":2214456960,"stackPointer":2214587424,"stackTop":2214588031,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214718640},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214718744},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214718864},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214718968},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214719064},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214719160}],"instructionPointer":2205044912,"stackBottom":2214588184,"stackPointer":2214718640,"stackTop":2214719255,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214849872},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214849976},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214850072},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214850192},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214850288},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214850384}],"instructionPointer":2203927968,"stackBottom":2214719408,"stackPointer":2214849872,"stackTop":2214850479,"threadId":167837705,"threadName":"thread","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214981088},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214981192},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214981312},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214981416},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214981512},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214981608}],"instructionPointer":2205044912,"stackBottom":2214850632,"stackPointer":2214981088,"stackTop":2214981703,"threadId":167837706,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215122560},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2215122664},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2215122784},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215122888},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215122984},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215123080}],"instructionPointer":2205044912,"stackBottom":2214992104,"stackPointer":2215122560,"stackTop":2215123175,"threadId":167837707,"threadName":"swi1","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215254408},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2215254512},{"callSite":2200930100,"calledTarget":2205049704,"context":"","framePointer":2215254632},{"callSite":2200932044,"calledTarget":2200929852,"context":"","framePointer":2215254736},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215254840},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215254936},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215255032}],"instructionPointer":2205044912,"stackBottom":2215124056,"stackPointer":2215254408,"stackTop":2215255127,"threadId":167837708,"threadName":"MMCD","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215385744},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215385848},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2215385944},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215386064},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215386160},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215386256}],"instructionPointer":2203927968,"stackBottom":2215255280,"stackPointer":2215385744,"stackTop":2215386351,"threadId":167837709,"threadName":"softirq","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213272312},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213272480},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2213272576},{"callSite":2198285908,"calledTarget":2198107960,"context":"","framePointer":2213272688},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213274608},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213274704},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213274800}],"instructionPointer":2205038500,"stackBottom":2213012752,"stackPointer":2213272312,"stackTop":2213274895,"threadId":184614913,"threadName":"main","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2213536648},{"callSite":2199378436,"calledTarget":2205048116,"context":"","framePointer":2213536784},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213536904},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213537000},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213537096}],"instructionPointer":2205048268,"stackBottom":2213275048,"stackPointer":2213536648,"stackTop":2213537191,"threadId":184614914,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213930424},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213930592},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2213930688},{"callSite":2198328272,"calledTarget":2197952628,"context":"","framePointer":2213930800},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2213930912},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213931008},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213931104},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213931200}],"instructionPointer":2205038500,"stackBottom":2213669152,"stackPointer":2213930424,"stackTop":2213931295,"threadId":184614915,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215647272},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215647376},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2215647472},{"callSite":2203982928,"calledTarget":2204641024,"context":"","framePointer":2215647592},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2215647696},{"callSite":2203990764,"calledTarget":2203973052,"context":"","framePointer":2215647792},{"callSite":2203991264,"calledTarget":2203990560,"context":"","framePointer":2215647936},{"callSite":2197953048,"calledTarget":2203991164,"context":"","framePointer":2215648072},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2215648264},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215648360},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215648456},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215648552}],"instructionPointer":2203927968,"stackBottom":2215386504,"stackPointer":2215647272,"stackTop":2215648647,"threadId":184614916,"threadName":"","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2215917088},{"callSite":2200379648,"calledTarget":2205048116,"context":"","framePointer":2215917224},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215917320},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215917416},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215917512}],"instructionPointer":2205048268,"stackBottom":2215655464,"stackPointer":2215917088,"stackTop":2215917607,"threadId":184614917,"threadName":"RmtBarrThrd","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2216179856},{"callSite":2201042580,"calledTarget":2205048116,"context":"","framePointer":2216179992},{"callSite":2200380308,"calledTarget":0,"context":"Thread handler","framePointer":2216180176},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216180272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216180368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216180464}],"instructionPointer":2205048268,"stackBottom":2215918416,"stackPointer":2216179856,"stackTop":2216180559,"threadId":184614918,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2216441456},{"callSite":2205080484,"calledTarget":2205080224,"context":"","framePointer":2216441552},{"callSite":2197840640,"calledTarget":2205080456,"context":"","framePointer":2216441688},{"callSite":2197841728,"calledTarget":2197840504,"context":"","framePointer":2216441816},{"callSite":2197842060,"calledTarget":2197841608,"context":"","framePointer":2216441936},{"callSite":2197842224,"calledTarget":2197841932,"context":"","framePointer":2216442376},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2216442472},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216442568},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216442664},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216442760}],"instructionPointer":2205080272,"stackBottom":2216180712,"stackPointer":2216441456,"stackTop":2216442855,"threadId":184614919,"threadName":"","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2216704040},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2216704176},{"callSite":2200352600,"calledTarget":2205044052,"context":"","framePointer":2216704272},{"callSite":2200354568,"calledTarget":2200352540,"context":"","framePointer":2216704368},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216704864},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216704960},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216705056}],"instructionPointer":2205081468,"stackBottom":2216443008,"stackPointer":2216704040,"stackTop":2216705151,"threadId":184614920,"threadName":"Scheduler00Thr","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2216974080},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2216974184},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2216974280},{"callSite":2203980240,"calledTarget":2204641024,"context":"","framePointer":2216974400},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2216974536},{"callSite":2204635336,"calledTarget":2203973052,"context":"","framePointer":2216974632},{"callSite":2204998500,"calledTarget":0,"context":"Thread handler","framePointer":2216974768},{"callSite":2200345908,"calledTarget":2204998388,"context":"","framePointer":2216974864},{"callSite":2200360620,"calledTarget":2200345824,"context":"","framePointer":2216974960},{"callSite":2200358084,"calledTarget":0,"context":"Thread handler","framePointer":2216975136},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216975448},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216975544},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216975640}],"instructionPointer":2203927968,"stackBottom":2216713592,"stackPointer":2216974080,"stackTop":2216975735,"threadId":184614921,"threadName":"EventRead00Thr","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2217237736},{"callSite":2205080432,"calledTarget":2205080224,"context":"","framePointer":2217237832},{"callSite":2205528068,"calledTarget":2205080408,"context":"","framePointer":2217237968},{"callSite":2197960232,"calledTarget":2205528060,"context":"","framePointer":2217238064},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217238232},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217238328},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217238424},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217238520}],"instructionPointer":2205080272,"stackBottom":2216976472,"stackPointer":2217237736,"stackTop":2217238615,"threadId":184614922,"threadName":"Logg","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2217499448},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2217499584},{"callSite":2200358496,"calledTarget":2205044052,"context":"","framePointer":2217499680},{"callSite":2200346936,"calledTarget":2200358388,"context":"","framePointer":2217499776},{"callSite":2200347044,"calledTarget":2200346660,"context":"","framePointer":2217499888},{"callSite":2200348636,"calledTarget":2200347020,"context":"","framePointer":2217500024},{"callSite":2200365560,"calledTarget":2200348440,"context":"","framePointer":2217500320},{"callSite":2198104656,"calledTarget":2200365552,"context":"","framePointer":2217500416},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217500528},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217500624},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217500720},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217500816}],"instructionPointer":2205081468,"stackBottom":2217238768,"stackPointer":2217499448,"stackTop":2217500911,"threadId":184614923,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2217762344},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2217762512},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2217762608},{"callSite":2198108156,"calledTarget":2198107960,"context":"","framePointer":2217762720},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217762824},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217762920},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217763016},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217763112}],"instructionPointer":2205038500,"stackBottom":2217501064,"stackPointer":2217762344,"stackTop":2217763207,"threadId":184614924,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218024496},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218024664},{"callSite":2198169076,"calledTarget":2205038904,"context":"","framePointer":2218024760},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218025120},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218025216},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218025312},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218025408}],"instructionPointer":2205038500,"stackBottom":2217763360,"stackPointer":2218024496,"stackTop":2218025503,"threadId":184614925,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218286936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218287104},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2218287200},{"callSite":2198104924,"calledTarget":2197952628,"context":"","framePointer":2218287312},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218287416},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218287512},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218287608},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218287704}],"instructionPointer":2205038500,"stackBottom":2218025656,"stackPointer":2218286936,"stackTop":2218287799,"threadId":184614926,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218549144},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218549312},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2218549408},{"callSite":2198169684,"calledTarget":2198107960,"context":"","framePointer":2218549520},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218549712},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218549808},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218549904},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218550000}],"instructionPointer":2205038500,"stackBottom":2218287952,"stackPointer":2218549144,"stackTop":2218550095,"threadId":184614927,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2200372616,"calledTarget":2200369092,"context":"","framePointer":2211250392},{"callSite":2205076112,"calledTarget":0,"context":"Thread handler","framePointer":2211250488},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211250584},{"callSite":2205076264,"calledTarget":1881154720,"context":"","framePointer":2211250704},{"callSite":2204962712,"calledTarget":2205076244,"context":"","framePointer":2211250808},{"callSite":2205908836,"calledTarget":0,"context":"Thread handler","framePointer":2211250912},{"callSite":2202060388,"calledTarget":2202056820,"context":"","framePointer":2218807000},{"callSite":2201975476,"calledTarget":2201975148,"context":"","framePointer":2218807280},{"callSite":2201978260,"calledTarget":2201975492,"context":"","framePointer":2218807384},{"callSite":2201912664,"calledTarget":2201978120,"context":"","framePointer":2218807480},{"callSite":2201915636,"calledTarget":0,"context":"Thread handler","framePointer":2218807664},{"callSite":2201332116,"calledTarget":2201913020,"context":"","framePointer":2218807792},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218807928},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808032},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808136},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808320},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218808416},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808520},{"callSite":2201908968,"calledTarget":2201886824,"context":"Thread handler","framePointer":2218808624},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808808},{"callSite":184614929,"calledTarget":0,"context":"Thread exit","framePointer":0}],"instructionPointer":2200372616,"stackBottom":2218550248,"stackPointer":2211250392,"stackTop":2218812391,"threadId":184614928,"threadName":"XLin","threadStatus":"READY"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219082936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219083104},{"callSite":2199473912,"calledTarget":2205038904,"context":"","framePointer":2219083200},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219083328},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219083424},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219083520},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219083616}],"instructionPointer":2205038500,"stackBottom":2218821568,"stackPointer":2219082936,"stackTop":2219083711,"threadId":184614929,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219345440},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219345608},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2219345704},{"callSite":2200324484,"calledTarget":2197952628,"context":"","framePointer":2219345816},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219346208},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219346304},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219346400},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219346496}],"instructionPointer":2205038500,"stackBottom":2219084448,"stackPointer":2219345440,"stackTop":2219346591,"threadId":184614931,"threadName":"","threadStatus":"UNKNOWN"}]}],"depthaiCommitHash":"9ed7c9ae4c232ff93a3500a585a6b1c00650e22c","deviceId":"184430109126E1F400"}Crash Dump 2
{"crashReports":[{"crashedThreadId":184614928,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2201976176,"trapName":"illegal instruction","trapNumber":2}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2205053928,"calledTarget":2205069644,"context":"","framePointer":2213011968},{"callSite":2205046092,"calledTarget":2205053672,"context":"","framePointer":2213012064},{"callSite":2200950800,"calledTarget":2205046004,"context":"","framePointer":2213012200},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213012312},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213012408},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213012504}],"instructionPointer":2205053928,"stackBottom":2212881528,"stackPointer":2213011968,"stackTop":2213012599,"threadId":167837697,"threadName":"CBTH","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2213668384},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2213668488},{"callSite":2201029996,"calledTarget":2205044620,"context":"","framePointer":2213668608},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213668712},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213668808},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213668904}],"instructionPointer":2205044912,"stackBottom":2213537928,"stackPointer":2213668384,"stackTop":2213668999,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214062520},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214062624},{"callSite":2205052780,"calledTarget":2205049704,"context":"","framePointer":2214062744},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214062848},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214062944},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214063040}],"instructionPointer":2205044912,"stackBottom":2213932064,"stackPointer":2214062520,"stackTop":2214063135,"threadId":167837699,"threadName":"TIME","threadStatus":"READY"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214193744},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214193848},{"callSite":2204968012,"calledTarget":2205049704,"context":"","framePointer":2214193968},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214194072},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214194168},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214194264}],"instructionPointer":2205044912,"stackBottom":2214063288,"stackPointer":2214193744,"stackTop":2214194359,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214324968},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214325072},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214325192},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214325296},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214325392},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214325488}],"instructionPointer":2205044912,"stackBottom":2214194512,"stackPointer":2214324968,"stackTop":2214325583,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214456200},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214456304},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2214456400},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214456520},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214456616},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214456712}],"instructionPointer":2203927968,"stackBottom":2214325736,"stackPointer":2214456200,"stackTop":2214456807,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214587424},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214587528},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214587624},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214587744},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214587840},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214587936}],"instructionPointer":2203927968,"stackBottom":2214456960,"stackPointer":2214587424,"stackTop":2214588031,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214718640},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214718744},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214718864},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214718968},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214719064},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214719160}],"instructionPointer":2205044912,"stackBottom":2214588184,"stackPointer":2214718640,"stackTop":2214719255,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214849872},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214849976},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214850072},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214850192},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214850288},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214850384}],"instructionPointer":2203927968,"stackBottom":2214719408,"stackPointer":2214849872,"stackTop":2214850479,"threadId":167837705,"threadName":"thread","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214981088},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214981192},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214981312},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214981416},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214981512},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214981608}],"instructionPointer":2205044912,"stackBottom":2214850632,"stackPointer":2214981088,"stackTop":2214981703,"threadId":167837706,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215122560},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2215122664},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2215122784},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215122888},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215122984},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215123080}],"instructionPointer":2205044912,"stackBottom":2214992104,"stackPointer":2215122560,"stackTop":2215123175,"threadId":167837707,"threadName":"swi1","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215254408},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2215254512},{"callSite":2200930100,"calledTarget":2205049704,"context":"","framePointer":2215254632},{"callSite":2200932044,"calledTarget":2200929852,"context":"","framePointer":2215254736},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215254840},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215254936},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215255032}],"instructionPointer":2205044912,"stackBottom":2215124056,"stackPointer":2215254408,"stackTop":2215255127,"threadId":167837708,"threadName":"MMCD","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215385744},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215385848},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2215385944},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215386064},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215386160},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215386256}],"instructionPointer":2203927968,"stackBottom":2215255280,"stackPointer":2215385744,"stackTop":2215386351,"threadId":167837709,"threadName":"softirq","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213272312},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213272480},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2213272576},{"callSite":2198285908,"calledTarget":2198107960,"context":"","framePointer":2213272688},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213274608},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213274704},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213274800}],"instructionPointer":2205038500,"stackBottom":2213012752,"stackPointer":2213272312,"stackTop":2213274895,"threadId":184614913,"threadName":"main","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2213536648},{"callSite":2199378436,"calledTarget":2205048116,"context":"","framePointer":2213536784},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213536904},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213537000},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213537096}],"instructionPointer":2205048268,"stackBottom":2213275048,"stackPointer":2213536648,"stackTop":2213537191,"threadId":184614914,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213930424},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213930592},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2213930688},{"callSite":2198328272,"calledTarget":2197952628,"context":"","framePointer":2213930800},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2213930912},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213931008},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213931104},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213931200}],"instructionPointer":2205038500,"stackBottom":2213669152,"stackPointer":2213930424,"stackTop":2213931295,"threadId":184614915,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215647272},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215647376},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2215647472},{"callSite":2203982928,"calledTarget":2204641024,"context":"","framePointer":2215647592},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2215647696},{"callSite":2203990764,"calledTarget":2203973052,"context":"","framePointer":2215647792},{"callSite":2203991264,"calledTarget":2203990560,"context":"","framePointer":2215647936},{"callSite":2197953048,"calledTarget":2203991164,"context":"","framePointer":2215648072},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2215648264},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215648360},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215648456},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215648552}],"instructionPointer":2203927968,"stackBottom":2215386504,"stackPointer":2215647272,"stackTop":2215648647,"threadId":184614916,"threadName":"","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2215917088},{"callSite":2200379648,"calledTarget":2205048116,"context":"","framePointer":2215917224},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215917320},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215917416},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215917512}],"instructionPointer":2205048268,"stackBottom":2215655464,"stackPointer":2215917088,"stackTop":2215917607,"threadId":184614917,"threadName":"RmtBarrThrd","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2216179856},{"callSite":2201042580,"calledTarget":2205048116,"context":"","framePointer":2216179992},{"callSite":2200380308,"calledTarget":0,"context":"Thread handler","framePointer":2216180176},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216180272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216180368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216180464}],"instructionPointer":2205048268,"stackBottom":2215918416,"stackPointer":2216179856,"stackTop":2216180559,"threadId":184614918,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2216441456},{"callSite":2205080484,"calledTarget":2205080224,"context":"","framePointer":2216441552},{"callSite":2197840640,"calledTarget":2205080456,"context":"","framePointer":2216441688},{"callSite":2197841728,"calledTarget":2197840504,"context":"","framePointer":2216441816},{"callSite":2197842060,"calledTarget":2197841608,"context":"","framePointer":2216441936},{"callSite":2197842224,"calledTarget":2197841932,"context":"","framePointer":2216442376},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2216442472},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216442568},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216442664},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216442760}],"instructionPointer":2205080272,"stackBottom":2216180712,"stackPointer":2216441456,"stackTop":2216442855,"threadId":184614919,"threadName":"","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2216704040},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2216704176},{"callSite":2200352600,"calledTarget":2205044052,"context":"","framePointer":2216704272},{"callSite":2200354568,"calledTarget":2200352540,"context":"","framePointer":2216704368},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216704864},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216704960},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216705056}],"instructionPointer":2205081468,"stackBottom":2216443008,"stackPointer":2216704040,"stackTop":2216705151,"threadId":184614920,"threadName":"Scheduler00Thr","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2216974080},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2216974184},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2216974280},{"callSite":2203980240,"calledTarget":2204641024,"context":"","framePointer":2216974400},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2216974536},{"callSite":2204635336,"calledTarget":2203973052,"context":"","framePointer":2216974632},{"callSite":2204998500,"calledTarget":0,"context":"Thread handler","framePointer":2216974768},{"callSite":2200345908,"calledTarget":2204998388,"context":"","framePointer":2216974864},{"callSite":2200360620,"calledTarget":2200345824,"context":"","framePointer":2216974960},{"callSite":2200358084,"calledTarget":0,"context":"Thread handler","framePointer":2216975136},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216975448},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216975544},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216975640}],"instructionPointer":2203927968,"stackBottom":2216713592,"stackPointer":2216974080,"stackTop":2216975735,"threadId":184614921,"threadName":"EventRead00Thr","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2217237736},{"callSite":2205080432,"calledTarget":2205080224,"context":"","framePointer":2217237832},{"callSite":2205528068,"calledTarget":2205080408,"context":"","framePointer":2217237968},{"callSite":2197960232,"calledTarget":2205528060,"context":"","framePointer":2217238064},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217238232},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217238328},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217238424},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217238520}],"instructionPointer":2205080272,"stackBottom":2216976472,"stackPointer":2217237736,"stackTop":2217238615,"threadId":184614922,"threadName":"Logg","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2217499448},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2217499584},{"callSite":2200358496,"calledTarget":2205044052,"context":"","framePointer":2217499680},{"callSite":2200346936,"calledTarget":2200358388,"context":"","framePointer":2217499776},{"callSite":2200347044,"calledTarget":2200346660,"context":"","framePointer":2217499888},{"callSite":2200348636,"calledTarget":2200347020,"context":"","framePointer":2217500024},{"callSite":2200365560,"calledTarget":2200348440,"context":"","framePointer":2217500320},{"callSite":2198104656,"calledTarget":2200365552,"context":"","framePointer":2217500416},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217500528},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217500624},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217500720},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217500816}],"instructionPointer":2205081468,"stackBottom":2217238768,"stackPointer":2217499448,"stackTop":2217500911,"threadId":184614923,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2217762344},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2217762512},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2217762608},{"callSite":2198108156,"calledTarget":2198107960,"context":"","framePointer":2217762720},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217762824},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217762920},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217763016},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217763112}],"instructionPointer":2205038500,"stackBottom":2217501064,"stackPointer":2217762344,"stackTop":2217763207,"threadId":184614924,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218024496},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218024664},{"callSite":2198169076,"calledTarget":2205038904,"context":"","framePointer":2218024760},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218025120},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218025216},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218025312},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218025408}],"instructionPointer":2205038500,"stackBottom":2217763360,"stackPointer":2218024496,"stackTop":2218025503,"threadId":184614925,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218286936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218287104},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2218287200},{"callSite":2198104924,"calledTarget":2197952628,"context":"","framePointer":2218287312},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218287416},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218287512},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218287608},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218287704}],"instructionPointer":2205038500,"stackBottom":2218025656,"stackPointer":2218286936,"stackTop":2218287799,"threadId":184614926,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218549144},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218549312},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2218549408},{"callSite":2198169684,"calledTarget":2198107960,"context":"","framePointer":2218549520},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218549712},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218549808},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218549904},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218550000}],"instructionPointer":2205038500,"stackBottom":2218287952,"stackPointer":2218549144,"stackTop":2218550095,"threadId":184614927,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2200372616,"calledTarget":2200369092,"context":"","framePointer":2211250392},{"callSite":2205076112,"calledTarget":0,"context":"Thread handler","framePointer":2211250488},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211250584},{"callSite":2205076264,"calledTarget":1881154720,"context":"","framePointer":2211250704},{"callSite":2204962712,"calledTarget":2205076244,"context":"","framePointer":2211250808},{"callSite":2205908836,"calledTarget":0,"context":"Thread handler","framePointer":2211250912},{"callSite":2201912644,"calledTarget":2201976164,"context":"","framePointer":2218807304},{"callSite":2201915636,"calledTarget":0,"context":"Thread handler","framePointer":2218807664},{"callSite":2201332116,"calledTarget":2201913020,"context":"","framePointer":2218807792},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218807928},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808032},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808136},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808320},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218808416},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808520},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808624},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808808},{"callSite":2201333056,"calledTarget":0,"context":"Thread handler","framePointer":2218808904},{"callSite":2201334288,"calledTarget":2201332896,"context":"Thread handler","framePointer":2218809000},{"callSite":2201334704,"calledTarget":2201334496,"context":"Thread handler","framePointer":2218809120},{"callSite":184614929,"calledTarget":0,"context":"Thread exit","framePointer":0}],"instructionPointer":2200372616,"stackBottom":2218550248,"stackPointer":2211250392,"stackTop":2218812391,"threadId":184614928,"threadName":"XLin","threadStatus":"READY"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219082936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219083104},{"callSite":2199473912,"calledTarget":2205038904,"context":"","framePointer":2219083200},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219083328},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219083424},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219083520},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219083616}],"instructionPointer":2205038500,"stackBottom":2218821568,"stackPointer":2219082936,"stackTop":2219083711,"threadId":184614929,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219345440},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219345608},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2219345704},{"callSite":2200324484,"calledTarget":2197952628,"context":"","framePointer":2219345816},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219346208},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219346304},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219346400},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219346496}],"instructionPointer":2205038500,"stackBottom":2219084448,"stackPointer":2219345440,"stackTop":2219346591,"threadId":184614931,"threadName":"","threadStatus":"UNKNOWN"}]}],"depthaiCommitHash":"9ed7c9ae4c232ff93a3500a585a6b1c00650e22c","deviceId":"184430100102E8F400"}syg
Hey can you check the bandwidth requirements?
I feel like the there would be a bottleneck in bandwidth from 5 port switch to the GPU server if many devices are streaming at once.
https://docs.luxonis.com/software/depthai/optimizing/#Bandwidth
Thanks,
Jaka
syg's colleague here, the gpu server has a 1Gbps rated port, everything else sharing the port also should not be using much bandwidth as confirmed by iftop.
we'll do another test by connecting the gpu server directly to the 24 port switch by disconnecting some of the backup cameras.
On an unrelated question, we've followed the depthai guide on syncing frames from multiple streams, however we discovered some skipping in videos recorded (people and objects moving across frames much faster than 5fps). Could bandwidth issues be causing this too?
Hi hotd
Yes, that could be the culprit.
The reason why is when the bandwidth is saturated, the frames from the cameras get dropped when not read in time.
The would mean there are less frames than the set framerate expects for proper playback, so the end videos look like they have been sped up.
ps. when recording, make sure the CPU and disk is fast enough for real time writing. If host side loop is blocked for too long, the same thing could happen. In this case the PC would be the bottleneck.
Thanks,
Jaka
Is there a more systematic way to tell if there are any power or bandwidth or frame dropping issues without having to observe through the side effects?
Currently our server is quite powerful and handling this should not actually be an issue, the server is currently not doing anything other than streaming frames from the camera and recording them into video files
An update, we’re still facing some issues with the cameras having connection issues, and dropped frames from the videos.
Here are some of the things we’ve tried to reduce the bandwidth saturation:
Disconnecting the backup cameras (13) from the main switch (24 port), this is ensure that the connected backup cameras aren’t saturating the bandwidth from the active cameras (11)
Have a direct connection between the GPU server & the main switch (24 port) instead of going through the 5 port switch. This is to reduce the bandwidth saturation on the 5 port switch as it’s used to connect between the main switch & our file server
However, upon booting our application the camera ping isn’t resolved. Out of the 13 cameras, only 7 ~ 10 cameras managed to successfully start. So for now we’ve updated our configuration on only support 10 and it seems to work but we’d still need to restart a few times before the cameras manages to connect.
We’d like to schedule a call with @jakaskerl & @erik to share more details in order to better troubleshoot we’re facing, let us know what’s required to make this happen.
Hi @syg , could you also share the MRE that leads to failed connects after you connect >10 devices to POE switch, so we can test it locally? I'd also suggest Standalone mode as an alternative, as there's no XLink disconnects (because there's no XLink). In general, live debugging over call isn't very fruitful.
Hey @erik, thanks for the prompt reply.
I want to clarify does MRE mean Minimum Reproducible Example? If so, I'd have to discuss with my colleague on how to come up with one.
Additionally, the call I've mentioned isn't to perform a live debugging on our system rather it's us providing you guys a more comprehensive overview of our architecture and specs of our server, network switches etc.
We're hoping that with a much more complete overview it's much easier to point out any potential setup or flaws in our system
