• DepthAI-v2
  • How to update outdated OAK-D Pro PoE firmware

syg
It's generally the bootloader issue so updating it is best you can do.

Also as another safety check:

 # Check every second for new devices
while True:
device_infos = dai.Device.getAllAvailableDevices()
for device_info in device_infos:
if device_info.state != dai.XLinkDeviceState.X_LINK_BOOTLOADER:
continue
# Start thread here
sleep(1)

Thanks,
Jaka

We've updated the camera boot loader version to the 0.0.28 which is the latest version.

However, we're facing another issue where an error is being thrown from one camera which eventually caused all cameras to stop their ping and our service to stopped. The attached image contains the error.

Specifically, F: [global] [ 606417] [Scheduler02Thr] dispatcherResponseServe:928 [19443010C172801300] [192.168.1.240] [1726580608.248] [host] [warning] Monitor thread (device: 19443010C172801300 [192.168.1.240]) - ping was missed, closing the device connection

What's the likely cause of this & is there anyway to catch this global error that's being thrown by the camera?

    syg
    Explained here. Could be high usage. I'd also check the connections. Even a small disturbance in the TCP connection (a drop of packets) could cause this issue.

    Thanks,
    Jaka

    Each of our camera bandwidth consumption is less than 10mbps which result in a total usage of 13 * 10 = 130 Mbps which is well within our network maximum bandwidth of 1000mbps.

    Is our only choice left with auto reconnect and being in standalone mode? Or are there any debugging options we can take?

    If we're in standalone mode, is it possible to stream the frames from the camera to our server? If so are there any tutorials on it?

      syg

      Hi Syg. Could you share how you updated the firmware. I followed the instructions in the link provided by jakaskeri

      (https://docs.luxonis.com/software/depthai-components/bootloader/#Danger%20Zone)

      but I can't get past 0.0.26 or depthai version 2.24.0.0

      I've deleted the depthai and depthai-python foldesr then cloned it from the current repo master, I've followed the docs again, as though starting with a clean install, installed depthai, installed all requirements. The utility flashed the camera but still at 0.0.26

      I'm at a loss to understand why my depthai version won't get past 2.24.0.0 when using the latest repo clone.

        syg
        Can you send me a link to the POE switch you are using? Could be that the power requirements exceed the specs (despite saying each port can output 30W). The devices die in a quick succession which could point to a power issue.

        For standalone mode streaming:
        luxonis/depthai-experimentstree/master/gen2-mjpeg-streaming
        luxonis/depthai-experimentstree/master/gen2-poe-tcp-streaming

        mikegardner
        Pulling latest repo is not enough. The depthai version has to be latest. The version is the carrier of the bootloader.

        Thanks,
        Jaka

        Thanks Jaka. I've deleted all of the folders and re-cloned them at the latest level and re-installed depthai.
        Please consider my query on this thread closed now.
        Regarding the drop-out I'm experiencing- I'll continue on my original thread.

        Hey @jakaskerl, thanks for being patient with this

        To give you a better overview of how our camera / architecture is setup, here is a diagram

        Our server is connected to the cameras via two switches, one switch has all 1000mbps ports fully utilized by the Oak Cameras, it extends to another switch via 1000Base SFP module, the other switch is also rated for 1000Mbps

        See details of both switches here:

        The power should be sufficient according to the calculations that we did.

        As mentioned we are only actually using 13 cameras actively but the rest will still be consuming power, just not in use as it is not connected to. ( we may try to disconnect them and see if it improves the situation)

        This issue only appears on boot and seems to happen more when we connect to 10 or more cameras simultaneously. Usually one or two cameras will have their ping missed.

        However, in the off chance that all cameras successfully connect and boot the pipeline, the cameras are able to run for few days non-stop without interruption.

        Here are some crash logs. I'd love to update files, but somehow the forum doesn't allow a json not zip file attachment

        Crash Dump 1

        {"crashReports":[{"crashedThreadId":184614928,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2201975248,"trapName":"illegal instruction","trapNumber":2}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2205053928,"calledTarget":2205069644,"context":"","framePointer":2213011968},{"callSite":2205046092,"calledTarget":2205053672,"context":"","framePointer":2213012064},{"callSite":2200950800,"calledTarget":2205046004,"context":"","framePointer":2213012200},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213012312},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213012408},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213012504}],"instructionPointer":2205053928,"stackBottom":2212881528,"stackPointer":2213011968,"stackTop":2213012599,"threadId":167837697,"threadName":"CBTH","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2213668384},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2213668488},{"callSite":2201029996,"calledTarget":2205044620,"context":"","framePointer":2213668608},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213668712},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213668808},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213668904}],"instructionPointer":2205044912,"stackBottom":2213537928,"stackPointer":2213668384,"stackTop":2213668999,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214062520},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214062624},{"callSite":2205052780,"calledTarget":2205049704,"context":"","framePointer":2214062744},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214062848},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214062944},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214063040}],"instructionPointer":2205044912,"stackBottom":2213932064,"stackPointer":2214062520,"stackTop":2214063135,"threadId":167837699,"threadName":"TIME","threadStatus":"READY"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214193744},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214193848},{"callSite":2204968012,"calledTarget":2205049704,"context":"","framePointer":2214193968},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214194072},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214194168},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214194264}],"instructionPointer":2205044912,"stackBottom":2214063288,"stackPointer":2214193744,"stackTop":2214194359,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214324968},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214325072},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214325192},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214325296},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214325392},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214325488}],"instructionPointer":2205044912,"stackBottom":2214194512,"stackPointer":2214324968,"stackTop":2214325583,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214456200},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214456304},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2214456400},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214456520},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214456616},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214456712}],"instructionPointer":2203927968,"stackBottom":2214325736,"stackPointer":2214456200,"stackTop":2214456807,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214587424},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214587528},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214587624},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214587744},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214587840},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214587936}],"instructionPointer":2203927968,"stackBottom":2214456960,"stackPointer":2214587424,"stackTop":2214588031,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214718640},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214718744},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214718864},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214718968},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214719064},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214719160}],"instructionPointer":2205044912,"stackBottom":2214588184,"stackPointer":2214718640,"stackTop":2214719255,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214849872},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214849976},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214850072},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214850192},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214850288},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214850384}],"instructionPointer":2203927968,"stackBottom":2214719408,"stackPointer":2214849872,"stackTop":2214850479,"threadId":167837705,"threadName":"thread","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214981088},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214981192},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214981312},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214981416},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214981512},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214981608}],"instructionPointer":2205044912,"stackBottom":2214850632,"stackPointer":2214981088,"stackTop":2214981703,"threadId":167837706,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215122560},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2215122664},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2215122784},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215122888},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215122984},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215123080}],"instructionPointer":2205044912,"stackBottom":2214992104,"stackPointer":2215122560,"stackTop":2215123175,"threadId":167837707,"threadName":"swi1","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215254408},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2215254512},{"callSite":2200930100,"calledTarget":2205049704,"context":"","framePointer":2215254632},{"callSite":2200932044,"calledTarget":2200929852,"context":"","framePointer":2215254736},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215254840},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215254936},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215255032}],"instructionPointer":2205044912,"stackBottom":2215124056,"stackPointer":2215254408,"stackTop":2215255127,"threadId":167837708,"threadName":"MMCD","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215385744},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215385848},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2215385944},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215386064},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215386160},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215386256}],"instructionPointer":2203927968,"stackBottom":2215255280,"stackPointer":2215385744,"stackTop":2215386351,"threadId":167837709,"threadName":"softirq","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213272312},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213272480},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2213272576},{"callSite":2198285908,"calledTarget":2198107960,"context":"","framePointer":2213272688},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213274608},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213274704},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213274800}],"instructionPointer":2205038500,"stackBottom":2213012752,"stackPointer":2213272312,"stackTop":2213274895,"threadId":184614913,"threadName":"main","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2213536648},{"callSite":2199378436,"calledTarget":2205048116,"context":"","framePointer":2213536784},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213536904},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213537000},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213537096}],"instructionPointer":2205048268,"stackBottom":2213275048,"stackPointer":2213536648,"stackTop":2213537191,"threadId":184614914,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213930424},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213930592},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2213930688},{"callSite":2198328272,"calledTarget":2197952628,"context":"","framePointer":2213930800},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2213930912},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213931008},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213931104},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213931200}],"instructionPointer":2205038500,"stackBottom":2213669152,"stackPointer":2213930424,"stackTop":2213931295,"threadId":184614915,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215647272},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215647376},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2215647472},{"callSite":2203982928,"calledTarget":2204641024,"context":"","framePointer":2215647592},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2215647696},{"callSite":2203990764,"calledTarget":2203973052,"context":"","framePointer":2215647792},{"callSite":2203991264,"calledTarget":2203990560,"context":"","framePointer":2215647936},{"callSite":2197953048,"calledTarget":2203991164,"context":"","framePointer":2215648072},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2215648264},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215648360},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215648456},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215648552}],"instructionPointer":2203927968,"stackBottom":2215386504,"stackPointer":2215647272,"stackTop":2215648647,"threadId":184614916,"threadName":"","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2215917088},{"callSite":2200379648,"calledTarget":2205048116,"context":"","framePointer":2215917224},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215917320},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215917416},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215917512}],"instructionPointer":2205048268,"stackBottom":2215655464,"stackPointer":2215917088,"stackTop":2215917607,"threadId":184614917,"threadName":"RmtBarrThrd","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2216179856},{"callSite":2201042580,"calledTarget":2205048116,"context":"","framePointer":2216179992},{"callSite":2200380308,"calledTarget":0,"context":"Thread handler","framePointer":2216180176},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216180272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216180368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216180464}],"instructionPointer":2205048268,"stackBottom":2215918416,"stackPointer":2216179856,"stackTop":2216180559,"threadId":184614918,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2216441456},{"callSite":2205080484,"calledTarget":2205080224,"context":"","framePointer":2216441552},{"callSite":2197840640,"calledTarget":2205080456,"context":"","framePointer":2216441688},{"callSite":2197841728,"calledTarget":2197840504,"context":"","framePointer":2216441816},{"callSite":2197842060,"calledTarget":2197841608,"context":"","framePointer":2216441936},{"callSite":2197842224,"calledTarget":2197841932,"context":"","framePointer":2216442376},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2216442472},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216442568},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216442664},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216442760}],"instructionPointer":2205080272,"stackBottom":2216180712,"stackPointer":2216441456,"stackTop":2216442855,"threadId":184614919,"threadName":"","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2216704040},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2216704176},{"callSite":2200352600,"calledTarget":2205044052,"context":"","framePointer":2216704272},{"callSite":2200354568,"calledTarget":2200352540,"context":"","framePointer":2216704368},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216704864},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216704960},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216705056}],"instructionPointer":2205081468,"stackBottom":2216443008,"stackPointer":2216704040,"stackTop":2216705151,"threadId":184614920,"threadName":"Scheduler00Thr","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2216974080},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2216974184},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2216974280},{"callSite":2203980240,"calledTarget":2204641024,"context":"","framePointer":2216974400},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2216974536},{"callSite":2204635336,"calledTarget":2203973052,"context":"","framePointer":2216974632},{"callSite":2204998500,"calledTarget":0,"context":"Thread handler","framePointer":2216974768},{"callSite":2200345908,"calledTarget":2204998388,"context":"","framePointer":2216974864},{"callSite":2200360620,"calledTarget":2200345824,"context":"","framePointer":2216974960},{"callSite":2200358084,"calledTarget":0,"context":"Thread handler","framePointer":2216975136},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216975448},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216975544},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216975640}],"instructionPointer":2203927968,"stackBottom":2216713592,"stackPointer":2216974080,"stackTop":2216975735,"threadId":184614921,"threadName":"EventRead00Thr","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2217237736},{"callSite":2205080432,"calledTarget":2205080224,"context":"","framePointer":2217237832},{"callSite":2205528068,"calledTarget":2205080408,"context":"","framePointer":2217237968},{"callSite":2197960232,"calledTarget":2205528060,"context":"","framePointer":2217238064},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217238232},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217238328},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217238424},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217238520}],"instructionPointer":2205080272,"stackBottom":2216976472,"stackPointer":2217237736,"stackTop":2217238615,"threadId":184614922,"threadName":"Logg","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2217499448},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2217499584},{"callSite":2200358496,"calledTarget":2205044052,"context":"","framePointer":2217499680},{"callSite":2200346936,"calledTarget":2200358388,"context":"","framePointer":2217499776},{"callSite":2200347044,"calledTarget":2200346660,"context":"","framePointer":2217499888},{"callSite":2200348636,"calledTarget":2200347020,"context":"","framePointer":2217500024},{"callSite":2200365560,"calledTarget":2200348440,"context":"","framePointer":2217500320},{"callSite":2198104656,"calledTarget":2200365552,"context":"","framePointer":2217500416},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217500528},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217500624},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217500720},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217500816}],"instructionPointer":2205081468,"stackBottom":2217238768,"stackPointer":2217499448,"stackTop":2217500911,"threadId":184614923,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2217762344},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2217762512},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2217762608},{"callSite":2198108156,"calledTarget":2198107960,"context":"","framePointer":2217762720},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217762824},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217762920},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217763016},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217763112}],"instructionPointer":2205038500,"stackBottom":2217501064,"stackPointer":2217762344,"stackTop":2217763207,"threadId":184614924,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218024496},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218024664},{"callSite":2198169076,"calledTarget":2205038904,"context":"","framePointer":2218024760},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218025120},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218025216},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218025312},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218025408}],"instructionPointer":2205038500,"stackBottom":2217763360,"stackPointer":2218024496,"stackTop":2218025503,"threadId":184614925,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218286936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218287104},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2218287200},{"callSite":2198104924,"calledTarget":2197952628,"context":"","framePointer":2218287312},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218287416},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218287512},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218287608},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218287704}],"instructionPointer":2205038500,"stackBottom":2218025656,"stackPointer":2218286936,"stackTop":2218287799,"threadId":184614926,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218549144},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218549312},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2218549408},{"callSite":2198169684,"calledTarget":2198107960,"context":"","framePointer":2218549520},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218549712},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218549808},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218549904},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218550000}],"instructionPointer":2205038500,"stackBottom":2218287952,"stackPointer":2218549144,"stackTop":2218550095,"threadId":184614927,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2200372616,"calledTarget":2200369092,"context":"","framePointer":2211250392},{"callSite":2205076112,"calledTarget":0,"context":"Thread handler","framePointer":2211250488},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211250584},{"callSite":2205076264,"calledTarget":1881154720,"context":"","framePointer":2211250704},{"callSite":2204962712,"calledTarget":2205076244,"context":"","framePointer":2211250808},{"callSite":2205908836,"calledTarget":0,"context":"Thread handler","framePointer":2211250912},{"callSite":2202060388,"calledTarget":2202056820,"context":"","framePointer":2218807000},{"callSite":2201975476,"calledTarget":2201975148,"context":"","framePointer":2218807280},{"callSite":2201978260,"calledTarget":2201975492,"context":"","framePointer":2218807384},{"callSite":2201912664,"calledTarget":2201978120,"context":"","framePointer":2218807480},{"callSite":2201915636,"calledTarget":0,"context":"Thread handler","framePointer":2218807664},{"callSite":2201332116,"calledTarget":2201913020,"context":"","framePointer":2218807792},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218807928},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808032},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808136},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808320},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218808416},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808520},{"callSite":2201908968,"calledTarget":2201886824,"context":"Thread handler","framePointer":2218808624},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808808},{"callSite":184614929,"calledTarget":0,"context":"Thread exit","framePointer":0}],"instructionPointer":2200372616,"stackBottom":2218550248,"stackPointer":2211250392,"stackTop":2218812391,"threadId":184614928,"threadName":"XLin","threadStatus":"READY"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219082936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219083104},{"callSite":2199473912,"calledTarget":2205038904,"context":"","framePointer":2219083200},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219083328},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219083424},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219083520},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219083616}],"instructionPointer":2205038500,"stackBottom":2218821568,"stackPointer":2219082936,"stackTop":2219083711,"threadId":184614929,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219345440},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219345608},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2219345704},{"callSite":2200324484,"calledTarget":2197952628,"context":"","framePointer":2219345816},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219346208},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219346304},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219346400},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219346496}],"instructionPointer":2205038500,"stackBottom":2219084448,"stackPointer":2219345440,"stackTop":2219346591,"threadId":184614931,"threadName":"","threadStatus":"UNKNOWN"}]}],"depthaiCommitHash":"9ed7c9ae4c232ff93a3500a585a6b1c00650e22c","deviceId":"184430109126E1F400"}

        Crash Dump 2

        {"crashReports":[{"crashedThreadId":184614928,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2201976176,"trapName":"illegal instruction","trapNumber":2}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2205053928,"calledTarget":2205069644,"context":"","framePointer":2213011968},{"callSite":2205046092,"calledTarget":2205053672,"context":"","framePointer":2213012064},{"callSite":2200950800,"calledTarget":2205046004,"context":"","framePointer":2213012200},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213012312},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213012408},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213012504}],"instructionPointer":2205053928,"stackBottom":2212881528,"stackPointer":2213011968,"stackTop":2213012599,"threadId":167837697,"threadName":"CBTH","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2213668384},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2213668488},{"callSite":2201029996,"calledTarget":2205044620,"context":"","framePointer":2213668608},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2213668712},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213668808},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213668904}],"instructionPointer":2205044912,"stackBottom":2213537928,"stackPointer":2213668384,"stackTop":2213668999,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214062520},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214062624},{"callSite":2205052780,"calledTarget":2205049704,"context":"","framePointer":2214062744},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214062848},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214062944},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214063040}],"instructionPointer":2205044912,"stackBottom":2213932064,"stackPointer":2214062520,"stackTop":2214063135,"threadId":167837699,"threadName":"TIME","threadStatus":"READY"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214193744},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2214193848},{"callSite":2204968012,"calledTarget":2205049704,"context":"","framePointer":2214193968},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214194072},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214194168},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214194264}],"instructionPointer":2205044912,"stackBottom":2214063288,"stackPointer":2214193744,"stackTop":2214194359,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214324968},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214325072},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214325192},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214325296},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214325392},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214325488}],"instructionPointer":2205044912,"stackBottom":2214194512,"stackPointer":2214324968,"stackTop":2214325583,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214456200},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214456304},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2214456400},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214456520},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214456616},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214456712}],"instructionPointer":2203927968,"stackBottom":2214325736,"stackPointer":2214456200,"stackTop":2214456807,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214587424},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214587528},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214587624},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214587744},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214587840},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214587936}],"instructionPointer":2203927968,"stackBottom":2214456960,"stackPointer":2214587424,"stackTop":2214588031,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214718640},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214718744},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214718864},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214718968},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214719064},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214719160}],"instructionPointer":2205044912,"stackBottom":2214588184,"stackPointer":2214718640,"stackTop":2214719255,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2214849872},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2214849976},{"callSite":2203928968,"calledTarget":2203870504,"context":"","framePointer":2214850072},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214850192},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214850288},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214850384}],"instructionPointer":2203927968,"stackBottom":2214719408,"stackPointer":2214849872,"stackTop":2214850479,"threadId":167837705,"threadName":"thread","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2214981088},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2214981192},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2214981312},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2214981416},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214981512},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214981608}],"instructionPointer":2205044912,"stackBottom":2214850632,"stackPointer":2214981088,"stackTop":2214981703,"threadId":167837706,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215122560},{"callSite":2205044724,"calledTarget":2205044748,"context":"","framePointer":2215122664},{"callSite":2204625012,"calledTarget":2205044620,"context":"","framePointer":2215122784},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215122888},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215122984},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215123080}],"instructionPointer":2205044912,"stackBottom":2214992104,"stackPointer":2215122560,"stackTop":2215123175,"threadId":167837707,"threadName":"swi1","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205044912,"calledTarget":1881154236,"context":"","framePointer":2215254408},{"callSite":2205049812,"calledTarget":2205044748,"context":"","framePointer":2215254512},{"callSite":2200930100,"calledTarget":2205049704,"context":"","framePointer":2215254632},{"callSite":2200932044,"calledTarget":2200929852,"context":"","framePointer":2215254736},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215254840},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215254936},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215255032}],"instructionPointer":2205044912,"stackBottom":2215124056,"stackPointer":2215254408,"stackTop":2215255127,"threadId":167837708,"threadName":"MMCD","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215385744},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215385848},{"callSite":2204632796,"calledTarget":2203870504,"context":"","framePointer":2215385944},{"callSite":2205068756,"calledTarget":0,"context":"Thread handler","framePointer":2215386064},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215386160},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215386256}],"instructionPointer":2203927968,"stackBottom":2215255280,"stackPointer":2215385744,"stackTop":2215386351,"threadId":167837709,"threadName":"softirq","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213272312},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213272480},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2213272576},{"callSite":2198285908,"calledTarget":2198107960,"context":"","framePointer":2213272688},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213274608},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213274704},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213274800}],"instructionPointer":2205038500,"stackBottom":2213012752,"stackPointer":2213272312,"stackTop":2213274895,"threadId":184614913,"threadName":"main","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2213536648},{"callSite":2199378436,"calledTarget":2205048116,"context":"","framePointer":2213536784},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213536904},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213537000},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213537096}],"instructionPointer":2205048268,"stackBottom":2213275048,"stackPointer":2213536648,"stackTop":2213537191,"threadId":184614914,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2213930424},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2213930592},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2213930688},{"callSite":2198328272,"calledTarget":2197952628,"context":"","framePointer":2213930800},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2213930912},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2213931008},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213931104},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213931200}],"instructionPointer":2205038500,"stackBottom":2213669152,"stackPointer":2213930424,"stackTop":2213931295,"threadId":184614915,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2215647272},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2215647376},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2215647472},{"callSite":2203982928,"calledTarget":2204641024,"context":"","framePointer":2215647592},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2215647696},{"callSite":2203990764,"calledTarget":2203973052,"context":"","framePointer":2215647792},{"callSite":2203991264,"calledTarget":2203990560,"context":"","framePointer":2215647936},{"callSite":2197953048,"calledTarget":2203991164,"context":"","framePointer":2215648072},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2215648264},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215648360},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215648456},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215648552}],"instructionPointer":2203927968,"stackBottom":2215386504,"stackPointer":2215647272,"stackTop":2215648647,"threadId":184614916,"threadName":"","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2215917088},{"callSite":2200379648,"calledTarget":2205048116,"context":"","framePointer":2215917224},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2215917320},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2215917416},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2215917512}],"instructionPointer":2205048268,"stackBottom":2215655464,"stackPointer":2215917088,"stackTop":2215917607,"threadId":184614917,"threadName":"RmtBarrThrd","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205048268,"calledTarget":2205069644,"context":"","framePointer":2216179856},{"callSite":2201042580,"calledTarget":2205048116,"context":"","framePointer":2216179992},{"callSite":2200380308,"calledTarget":0,"context":"Thread handler","framePointer":2216180176},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216180272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216180368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216180464}],"instructionPointer":2205048268,"stackBottom":2215918416,"stackPointer":2216179856,"stackTop":2216180559,"threadId":184614918,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2216441456},{"callSite":2205080484,"calledTarget":2205080224,"context":"","framePointer":2216441552},{"callSite":2197840640,"calledTarget":2205080456,"context":"","framePointer":2216441688},{"callSite":2197841728,"calledTarget":2197840504,"context":"","framePointer":2216441816},{"callSite":2197842060,"calledTarget":2197841608,"context":"","framePointer":2216441936},{"callSite":2197842224,"calledTarget":2197841932,"context":"","framePointer":2216442376},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2216442472},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216442568},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216442664},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216442760}],"instructionPointer":2205080272,"stackBottom":2216180712,"stackPointer":2216441456,"stackTop":2216442855,"threadId":184614919,"threadName":"","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2216704040},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2216704176},{"callSite":2200352600,"calledTarget":2205044052,"context":"","framePointer":2216704272},{"callSite":2200354568,"calledTarget":2200352540,"context":"","framePointer":2216704368},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216704864},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216704960},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216705056}],"instructionPointer":2205081468,"stackBottom":2216443008,"stackPointer":2216704040,"stackTop":2216705151,"threadId":184614920,"threadName":"Scheduler00Thr","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2203927968,"calledTarget":1881154236,"context":"","framePointer":2216974080},{"callSite":2203870736,"calledTarget":2203928452,"context":"","framePointer":2216974184},{"callSite":2204641092,"calledTarget":2203870504,"context":"","framePointer":2216974280},{"callSite":2203980240,"calledTarget":2204641024,"context":"","framePointer":2216974400},{"callSite":2203973100,"calledTarget":0,"context":"Thread handler","framePointer":2216974536},{"callSite":2204635336,"calledTarget":2203973052,"context":"","framePointer":2216974632},{"callSite":2204998500,"calledTarget":0,"context":"Thread handler","framePointer":2216974768},{"callSite":2200345908,"calledTarget":2204998388,"context":"","framePointer":2216974864},{"callSite":2200360620,"calledTarget":2200345824,"context":"","framePointer":2216974960},{"callSite":2200358084,"calledTarget":0,"context":"Thread handler","framePointer":2216975136},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2216975448},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2216975544},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2216975640}],"instructionPointer":2203927968,"stackBottom":2216713592,"stackPointer":2216974080,"stackTop":2216975735,"threadId":184614921,"threadName":"EventRead00Thr","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205080272,"calledTarget":2205069644,"context":"","framePointer":2217237736},{"callSite":2205080432,"calledTarget":2205080224,"context":"","framePointer":2217237832},{"callSite":2205528068,"calledTarget":2205080408,"context":"","framePointer":2217237968},{"callSite":2197960232,"calledTarget":2205528060,"context":"","framePointer":2217238064},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217238232},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217238328},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217238424},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217238520}],"instructionPointer":2205080272,"stackBottom":2216976472,"stackPointer":2217237736,"stackTop":2217238615,"threadId":184614922,"threadName":"Logg","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2205081468,"calledTarget":2205069644,"context":"","framePointer":2217499448},{"callSite":2205044092,"calledTarget":2205081384,"context":"","framePointer":2217499584},{"callSite":2200358496,"calledTarget":2205044052,"context":"","framePointer":2217499680},{"callSite":2200346936,"calledTarget":2200358388,"context":"","framePointer":2217499776},{"callSite":2200347044,"calledTarget":2200346660,"context":"","framePointer":2217499888},{"callSite":2200348636,"calledTarget":2200347020,"context":"","framePointer":2217500024},{"callSite":2200365560,"calledTarget":2200348440,"context":"","framePointer":2217500320},{"callSite":2198104656,"calledTarget":2200365552,"context":"","framePointer":2217500416},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217500528},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217500624},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217500720},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217500816}],"instructionPointer":2205081468,"stackBottom":2217238768,"stackPointer":2217499448,"stackTop":2217500911,"threadId":184614923,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2217762344},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2217762512},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2217762608},{"callSite":2198108156,"calledTarget":2198107960,"context":"","framePointer":2217762720},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2217762824},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2217762920},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2217763016},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2217763112}],"instructionPointer":2205038500,"stackBottom":2217501064,"stackPointer":2217762344,"stackTop":2217763207,"threadId":184614924,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218024496},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218024664},{"callSite":2198169076,"calledTarget":2205038904,"context":"","framePointer":2218024760},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218025120},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218025216},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218025312},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218025408}],"instructionPointer":2205038500,"stackBottom":2217763360,"stackPointer":2218024496,"stackTop":2218025503,"threadId":184614925,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218286936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218287104},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2218287200},{"callSite":2198104924,"calledTarget":2197952628,"context":"","framePointer":2218287312},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218287416},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218287512},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218287608},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218287704}],"instructionPointer":2205038500,"stackBottom":2218025656,"stackPointer":2218286936,"stackTop":2218287799,"threadId":184614926,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2218549144},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2218549312},{"callSite":2198108016,"calledTarget":2205038904,"context":"","framePointer":2218549408},{"callSite":2198169684,"calledTarget":2198107960,"context":"","framePointer":2218549520},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2218549712},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2218549808},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2218549904},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2218550000}],"instructionPointer":2205038500,"stackBottom":2218287952,"stackPointer":2218549144,"stackTop":2218550095,"threadId":184614927,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2200372616,"calledTarget":2200369092,"context":"","framePointer":2211250392},{"callSite":2205076112,"calledTarget":0,"context":"Thread handler","framePointer":2211250488},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211250584},{"callSite":2205076264,"calledTarget":1881154720,"context":"","framePointer":2211250704},{"callSite":2204962712,"calledTarget":2205076244,"context":"","framePointer":2211250808},{"callSite":2205908836,"calledTarget":0,"context":"Thread handler","framePointer":2211250912},{"callSite":2201912644,"calledTarget":2201976164,"context":"","framePointer":2218807304},{"callSite":2201915636,"calledTarget":0,"context":"Thread handler","framePointer":2218807664},{"callSite":2201332116,"calledTarget":2201913020,"context":"","framePointer":2218807792},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218807928},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808032},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808136},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808320},{"callSite":2201886280,"calledTarget":0,"context":"Thread handler","framePointer":2218808416},{"callSite":2201887236,"calledTarget":2201886328,"context":"","framePointer":2218808520},{"callSite":2201908968,"calledTarget":2201886824,"context":"","framePointer":2218808624},{"callSite":2201330124,"calledTarget":0,"context":"Thread handler","framePointer":2218808808},{"callSite":2201333056,"calledTarget":0,"context":"Thread handler","framePointer":2218808904},{"callSite":2201334288,"calledTarget":2201332896,"context":"Thread handler","framePointer":2218809000},{"callSite":2201334704,"calledTarget":2201334496,"context":"Thread handler","framePointer":2218809120},{"callSite":184614929,"calledTarget":0,"context":"Thread exit","framePointer":0}],"instructionPointer":2200372616,"stackBottom":2218550248,"stackPointer":2211250392,"stackTop":2218812391,"threadId":184614928,"threadName":"XLin","threadStatus":"READY"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219082936},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219083104},{"callSite":2199473912,"calledTarget":2205038904,"context":"","framePointer":2219083200},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219083328},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219083424},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219083520},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219083616}],"instructionPointer":2205038500,"stackBottom":2218821568,"stackPointer":2219082936,"stackTop":2219083711,"threadId":184614929,"threadName":"","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2205038500,"calledTarget":2205069644,"context":"","framePointer":2219345440},{"callSite":2205038920,"calledTarget":2205038412,"context":"","framePointer":2219345608},{"callSite":2197952728,"calledTarget":2205038904,"context":"","framePointer":2219345704},{"callSite":2200324484,"calledTarget":2197952628,"context":"","framePointer":2219345816},{"callSite":2205742888,"calledTarget":0,"context":"Thread handler","framePointer":2219346208},{"callSite":2205068780,"calledTarget":0,"context":"Thread handler","framePointer":2219346304},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2219346400},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2219346496}],"instructionPointer":2205038500,"stackBottom":2219084448,"stackPointer":2219345440,"stackTop":2219346591,"threadId":184614931,"threadName":"","threadStatus":"UNKNOWN"}]}],"depthaiCommitHash":"9ed7c9ae4c232ff93a3500a585a6b1c00650e22c","deviceId":"184430100102E8F400"}

          jakaskerl

          syg's colleague here, the gpu server has a 1Gbps rated port, everything else sharing the port also should not be using much bandwidth as confirmed by iftop.

          we'll do another test by connecting the gpu server directly to the 24 port switch by disconnecting some of the backup cameras.

          On an unrelated question, we've followed the depthai guide on syncing frames from multiple streams, however we discovered some skipping in videos recorded (people and objects moving across frames much faster than 5fps). Could bandwidth issues be causing this too?

            Hi hotd
            Yes, that could be the culprit.

            The reason why is when the bandwidth is saturated, the frames from the cameras get dropped when not read in time.
            The would mean there are less frames than the set framerate expects for proper playback, so the end videos look like they have been sped up.

            ps. when recording, make sure the CPU and disk is fast enough for real time writing. If host side loop is blocked for too long, the same thing could happen. In this case the PC would be the bottleneck.

            Thanks,
            Jaka

            • hotd replied to this.

              jakaskerl

              Is there a more systematic way to tell if there are any power or bandwidth or frame dropping issues without having to observe through the side effects?

              Currently our server is quite powerful and handling this should not actually be an issue, the server is currently not doing anything other than streaming frames from the camera and recording them into video files

                hotd
                You could check the sequence numbers of messages when they arrive to the host side. If the numbers don't increment, it's likely some frames are dropped.

                An update, we’re still facing some issues with the cameras having connection issues, and dropped frames from the videos. 

                Here are some of the things we’ve tried to reduce the bandwidth saturation:

                1. Disconnecting the backup cameras (13) from the main switch (24 port), this is ensure that the connected backup cameras aren’t saturating the bandwidth from the active cameras (11)

                2. Have a direct connection between the GPU server & the main switch (24 port) instead of going through the 5 port switch. This is to reduce the bandwidth saturation on the 5 port switch as it’s used to connect between the main switch & our file server

                However, upon booting our application the camera ping isn’t resolved. Out of the 13 cameras, only 7 ~ 10 cameras managed to successfully start. So for now we’ve updated our configuration on only support 10 and it seems to work but we’d still need to restart a few times before the cameras manages to connect.

                We’d like to schedule a call with  @jakaskerl & @erik to share more details in order to better troubleshoot we’re facing, let us know what’s required to make this happen.

                Hi @syg , could you also share the MRE that leads to failed connects after you connect >10 devices to POE switch, so we can test it locally? I'd also suggest Standalone mode as an alternative, as there's no XLink disconnects (because there's no XLink). In general, live debugging over call isn't very fruitful.

                Hey @erik, thanks for the prompt reply.

                I want to clarify does MRE mean Minimum Reproducible Example? If so, I'd have to discuss with my colleague on how to come up with one.

                Additionally, the call I've mentioned isn't to perform a live debugging on our system rather it's us providing you guys a more comprehensive overview of our architecture and specs of our server, network switches etc.

                We're hoping that with a much more complete overview it's much easier to point out any potential setup or flaws in our system

                  syg I want to clarify does MRE mean Minimum Reproducible Example? If so, I'd have to discuss with my colleague on how to come up with one.

                  Yes.

                  17 days later

                  Hey @erik @jakaskerl ,

                  Some updates on our end.

                  Currently, we are actively using 13 cameras and we’re still facing the issue of ping was missed at two points of our docker application:

                  1. When we are starting our docker application.

                  2. In the middle of the week

                    1. As our solution is deployed on prem, we check on it every week. During our checks we discovered that a few cameras had ping was missed which cause our docker application to hang

                    2. As an example, we started our docker service at 07/10/2024 and the ping was missed showed up at 09/10/2024 causing our service to hang

                  Here are some troubleshooting steps we’ve tried:

                  We’ve used a few scripts from your repository for us to obtain some metrics, this enabled us to benchmark against the desired result shown below:

                  Benchmark

                  Scripts

                  Poe Link test result

                  Here’s a sample result of a few cameras

                  # Cam 1
                  Connecting to  10.0.0.11 ...
                  mxid: 184430101110E8F400 (OK)
                  speed: 1000 (OK)
                  full duplex: 1 (OK)
                  boot mode: 3 (OK)
                  
                  # Cam 3
                  Connecting to  10.0.0.13 ...
                  mxid: 18443010F16EE6F400 (OK)
                  speed: 1000 (OK)
                  full duplex: 1 (OK)
                  boot mode: 3 (OK)
                  
                  # Cam 6
                  Connecting to  10.0.0.16 ...
                  mxid: 18443010517BE7F400 (OK)
                  speed: 1000 (OK)
                  full duplex: 1 (OK)
                  boot mode: 3 (OK)

                  As seen, our link test managed to hit the benchmark which requires a speed of 1000 Mbps

                  OAK bandwidth test result

                  Here’s a sample result of a few camera:

                  # Cam 1
                  Downlink 883.4 mbps
                  Uplink 219.4 mbps
                  
                  # Cam 3
                  Downlink 890.0 mbps
                  Uplink 218.3 mbps
                  
                  # Cam 6
                  Downlink 890.6 mbps
                  Uplink 218.4 mbps

                  As seen, our downlink and uplink are within the acceptable range of 800 Mbps uplink & 200 Mbps downlink

                  OAK Latency test

                  Here’s a sample result of a few cameras:

                  # Cam 1
                  Average latency 2.48 ms, Std: 0.4
                  
                  # Cam 3
                  Average latency 2.73 ms, Std: 0.3
                  
                  # Cam 6
                  Average latency 3.06 ms, Std: 0.3

                  As seen, our average latency across the camera is less than 10ms which is required to pass the benchmark

                  After running the test, we can conclude that we’re not limited by our network equipment as the resulting latency, bandwidth and link speed are within the benchmark range stated above.

                  Additionally, we can confirm that enough power is supplied to the cameras based on the following

                  As seen from the image, each port can deliver up to 30W to the cameras and each of it is only consuming up to 4~5W which is sufficient for the cameras to function

                  CPU Usage of cameras

                  To ensure that our pipeline isn’t too complex which can cause a high CPU usage of the LeonOS and potentially lead to a ping was missed error, we’ve enabled logging on our setup through DEPTHAI_LEVEL=info to observe the CPU usage.

                  The result shows that our pipeline is only consuming up to 12% of the CPU . Proving that our pipeline isn’t complex enough to cause a high CPU usage.

                  Allowing longer boot time for the cameras

                  On the debugging guide, a section indicates that some network equipment might not work well with the default timeout they’ve set.

                  To ensure that there’s enough time for the camera to boot & the watchdog doesn’t disconnect too early, we’ve introduced two environment variables:

                  • DEPTHAI_BOOTUP_TIMEOUT

                  • DEPTHAI_WATCHDOG_INITIAL_DELAY

                  And set it’s value to 60000 (60s), allowing more time for camera to ping the server.

                  However, this didn’t work as ping was missed error still occured upon boot

                  Additional Context

                  Based on our above troubleshooting steps, we’re still facing the ping was missed issue. Hence, we’re providing our hardware architecture to provide a better overview of our setup

                  Lastly, here’s a Minimal Reproducible Example (MRE) that you guys can run on your end to simulate our environment:

                  import os
                  import re
                  import cv2
                  import math
                  import time
                  import yaml
                  import numpy as np
                  import socket
                  #os.environ["DEPTHAI_LEVEL"] = "debug"
                  import depthai as dai
                  from collections import deque
                  from threading import Thread
                  from datetime import datetime, timedelta
                  from loguru import logger
                  from typing import List, Any
                  from pydantic import BaseModel
                  
                  
                  
                  class Frame(BaseModel):
                      has_rgb: bool
                      has_depth: bool
                      rgb_frame: Any = None
                      depth_frame: Any = None
                      timestamp: timedelta 
                  
                  class OakDProPOE(Thread):
                  
                      def __init__(
                          self,
                          buffer: deque,
                          mxid: str,
                          server_ip: str,
                          server_port: int,
                          fps: int = 30,
                      ):
                          super().__init__()
                          self.fps = fps
                          self.buffer = buffer
                          self.recording = False
                          self.ip = server_ip
                          self.port = server_port
                          self.mxid = mxid
                          self.info = self.lookup_camera(mxid)
                          self.recording = False
                          self.connection: socket.socket = None
                  
                      def lookup_camera(self, mxid: str):
                  
                          for i in range(3):
                              ret, info = dai.Device.getDeviceByMxId(mxid)
                              if ret:
                                  self.mxid = mxid
                                  return info
                          else:
                              raise Exception(f"{mxid} not found!")
                          
                      def start_camera(self):
                  
                          self.recording = True
                  
                      def stop_camera(self):
                  
                          if self.connection:
                              self.connection.close()
                          self.recording = False
                  
                      def poe_pipeline(self):
                  
                          logger.info(f"{self.mxid} pipeline initiated")
                          pipeline = dai.Pipeline()
                  
                          camRgb = pipeline.createColorCamera()
                          camRgb.setIspScale(2,3)
                  
                          videoEnc = pipeline.create(dai.node.VideoEncoder)
                          videoEnc.setDefaultProfilePreset(30, dai.VideoEncoderProperties.Profile.MJPEG)
                          camRgb.video.link(videoEnc.input)
                  
                          script = pipeline.create(dai.node.Script)
                          script.setProcessor(dai.ProcessorType.LEON_CSS)
                          videoEnc.bitstream.link(script.inputs['frame'])
                  
                          script.setScript(
                              f"""
                  HOST_IP = '{self.ip}'
                  HOST_PORT = {self.port}
                                   
                  import socket
                  import time
                  
                  node.warn(f'>Going to connect to {{HOST_IP}}:{{HOST_PORT}}<')
                  sock = socket.socket()
                  sock.connect((HOST_IP, HOST_PORT))
                  
                  while True:
                  
                      pck = node.io["frame"].get()
                      data = pck.getData()
                      ts = pck.getTimestamp()
                      header = f"ABCDE " + str(ts.total_seconds()).ljust(18) + str(len(data)).ljust(8)
                      sock.send(bytes(header, encoding='ascii'))
                      sock.send(data)
                  """
                          )
                          logger.info(self.mxid)
                          device_info = dai.DeviceInfo(self.mxid)
                          logger.info(device_info)
                  
                          try:
                  
                              with dai.Device(pipeline, device_info) as device:
                                  logger.info(f"Pipeline running on {self.mxid}")
                                  while True:
                                      time.sleep(1)
                  
                          except Exception as e:
                              logger.error(e)
                              raise
                  
                      def run(self):
                  
                          def get_frame(socket, size):
                              bytes = socket.recv(4096)
                              while True:
                                  read = 4096
                                  if size - len(bytes) < read:
                                      read = size - len(bytes)
                                  bytes += socket.recv(read)
                                  if size == len(bytes):
                                      return bytes
                  
                          server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                          server.bind(("0.0.0.0", self.port))
                  
                          cam_thread = Thread(target=self.poe_pipeline)
                          cam_thread.daemon = True
                          logger.info("Pipeline started")
                  
                          server.listen()
                  
                          logger.info("Starting cam thread")
                          cam_thread.start()
                          self.connection, client = server.accept()
                          try:
                              logger.info(f"{self.mxid} connected")
                              while self.recording:
                                  header = str(self.connection.recv(32), encoding="ascii")
                                  chunks = re.split(" +", header)
                                  if chunks[0] == "ABCDE":
                                      # print(f">{header}<")
                                      ts = float(chunks[1])
                                      imgSize = int(chunks[2])
                                      img = get_frame(self.connection, imgSize)
                                      buf = np.frombuffer(img, dtype=np.byte)
                                      # print(buf.shape, buf.size)
                                      frame = cv2.imdecode(buf, cv2.IMREAD_COLOR)
                  
                                      frame = Frame(
                                          has_rgb=True,
                                          has_depth=True,
                                          rgb_frame=cv2.cvtColor(frame, cv2.COLOR_BGR2RGB),
                                          depth_frame=None,
                                          timestamp=timedelta(seconds=ts),
                                      )
                  
                                      self.buffer.append(frame)
                  
                          except Exception as e:
                              # TODO: Handle Server Error
                              logger.error(f"{self.mxid}: " + str(e))
                              raise
                  
                          server.close()
                  
                  
                  class Controller:
                  
                  
                      def __init__(self, yaml_file: str, fps: int, server_ip: str):
                  
                          self.fps = fps
                          self.sync_threshold = timedelta(milliseconds=math.ceil(5000 / self.fps))
                          self.server_ip = server_ip
                          self.load_camera_mapping_file(yaml_file)
                      
                      def load_camera_mapping_file(self, yaml_file):
                  
                          logger.info(f"Controller reading yaml file: {yaml_file}")
                          with open(yaml_file) as f:
                              self.camera_mapping = yaml.load(f, Loader=yaml.FullLoader)
                  
                          # Display the yaml config
                          logger.info(
                              yaml.dump(self.camera_mapping, default_flow_style=False, sort_keys=False)
                          )
                  
                          camera_type = self.camera_mapping["CameraType"]
                  
                          self.active_cameras = {}
                          for cam in self.camera_mapping["ActiveCameras"]:
                              # Create a buffer for the camera
                              buffer = deque([], maxlen=self.fps * 50) # buffer holds up to 10 seconds worth of frames
                              main_camera_key = cam["main_camera"]["camera_id"]
                  
                              if camera_type == "OakDProPoE":
                                  main_camera = OakDProPOE(
                                      fps=self.fps,
                                      buffer=buffer,
                                      mxid=cam["main_camera"]["mxid"],
                                      server_ip=self.server_ip,
                                      server_port=cam["main_camera"]["server_port"],
                                  )
                  
                  
                              else:
                                  raise Exception(f"Camera type {camera_type} is invalid!")
                              
                              self.active_cameras[main_camera_key] = {
                                  "buffer": buffer,
                                  "synced_buffer": deque([], maxlen=500),
                                  "main_camera": main_camera,
                              }
                  
                      def get_active_cameras(self):
                  
                          return self.active_cameras
                  
                      def start_cameras(self):
                  
                          for cam_idx in self.active_cameras:
                              camera = self.active_cameras[cam_idx]["main_camera"]
                              camera.daemon = True
                              camera.start_camera()
                              camera.start()
                  
                  
                          logger.info("Cameras started")
                  
                      def stop_cameras(self):
                  
                          logger.info("Controller stopping cameras")
                  
                          for cam_idx in self.active_cameras:
                              camera = self.active_cameras[cam_idx]["main_camera"]
                  
                              camera.stop_camera()
                              camera.join()
                  
                          logger.info("Cameras stopped")
                  
                      def check_sync(self, timestamp: timedelta):
                  
                          matching_frame_indexes = []
                          # Try to find matching frame in each queue
                          for active_cam in self.active_cameras.values():
                              for i, frame in enumerate(active_cam["synced_buffer"]):
                                  time_diff = abs(frame.timestamp - timestamp)
                                  if time_diff <= self.sync_threshold:
                                      # We now have the synced frame index for this particular camera
                                      matching_frame_indexes.append(i)
                                      break
                  
                          # When synced frames are found, clear all unused/out-of-sync frames
                          if len(matching_frame_indexes) == len(self.active_cameras):
                  
                              for i, q in enumerate(self.active_cameras.values()):
                                  for j in range(0, matching_frame_indexes[i]):
                                      q["synced_buffer"].popleft()
                  
                              return True
                  
                          else:
                  
                              return False
                  
                  
                      def get_synced_frames(self) -> dict[int, Frame]:
                  
                          # Iterate to try and get new frame from any buffer
                          start = time.perf_counter()
                          for cam in self.active_cameras:
                  
                              if self.active_cameras[cam]["buffer"]:
                                  # Get the frame from camera's buffer to controller's sync buffer
                                  # this ensures we can synchronously process the frames in sync buffer without worrying about thread safety from the camera buffer
                                  frame = self.active_cameras[cam]["buffer"].popleft()
                                  self.active_cameras[cam]["synced_buffer"].append(frame)
                                  # Check sync to see if we have a group of synchronized frames across all sync buffers
                                  if self.check_sync(frame.timestamp):
                  
                                      data = {}
                                      for cam in self.active_cameras:
                                          data[cam] = self.active_cameras[cam]["synced_buffer"].popleft()
                                      logger.debug(f"Sync took {time.perf_counter() - start} seconds")
                                      start = time.perf_counter()
                                      return data
                                  
                  
                  if __name__ == "__main__":
                      
                      controller = Controller(
                          yaml_file="configs/camera_mapping_office.yaml",
                          fps=5,
                          server_ip="192.168.1.236"
                      )
                  
                      controller.start_cameras()
                  
                      while True:
                  
                          synced_frame = controller.get_synced_frames()

                  To run the MRE you’d need to install the requirements in from the import statement of the file and have a camera-config.yaml file which reflects the configuration of the cameras.

                  Here’s how a sample camera camera-config.yaml looks like:

                  CameraType: OakDProPoE
                  ActiveCameras:
                    - main_camera: 
                        camera_id: 25
                        mxid: 18443010117CE9F400
                        server_port: 10025
                      backup:
                    - main_camera: 
                        camera_id: 26
                        mxid: 18443010D1F3E7F400
                        server_port: 10026
                      backup:
                    - main_camera: 
                        camera_id: 27
                        mxid: 18443010C1D6E1F400
                        server_port: 10027
                      backup:
                    - main_camera: 
                        camera_id: 28
                        mxid: 19443010C172801300
                        server_port: 10028
                      backup:

                  Some notes on the camera-config.yaml

                  • On the actual file, it should contain the configuration for 13 cameras

                  • camera_id & backup: is not required

                  • Each mxid should contain a unique camera mxid

                  It’d be filled in at line 303 of mre.py:

                  Here’s how a potential directory structure can look like:

                  .
                  └── (some-folder)/
                      ├── mre.py
                      ├── camera-config.yaml
                      └── requirements.txt

                    syg
                    Thanks for the thorough investigation. This seems to have checked all probable points of failure. The issue is depthai's overall stability which we can not fix as the issues stem deeper in Intel's FW.

                    So I suggest writing a reconnect script that quickly restarts the pipeline in case of missed pings. Simple try-catch would work, but you can make it more sophisticated.
                    We have implemented reconnect functionality in V3 as well if you wish to use that when it comes out.

                    Thanks,
                    Jaka