Capture color and depth only on event

vkmNuller

Hi,

I'm aiming to capture the next five color frames along with their corresponding depth maps triggered by a user-driven event. Outside of these five frames, I want to minimize processing on the device, especially avoiding depth/disparity calculations.

My idea is to configure the pipeline on the device to drop frames immediately after the cameras until the event occurs. Once triggered, I'd like to allow five synchronized frames through before resuming frame dropping until the next event.

Currently, I'm considering using a script node right after the cameras to pass frames on a trigger.
Another option is to dynamically link or unlink nodes on the device and/or the host.

Some things i wonder about:

Is linking/unlinking sensitive as in gstreamer or I can do it runtime?
How could I time and sync the exact number of frames?

I'd appreciate any suggestions, examples, APIs, or any words of wisdom and caution you may have.
Thanks in advance!

jakaskerl

Hi @vkmNuller
I'd suggest doing something similar to this: https://docs.luxonis.com/projects/api/en/latest/samples/Script/script_change_pipeline_flow/.
The pipeline is static once uploaded to the device, so you cannot unlink/relink the nodes.

You can use the msg.getTimestampDevice() to know exactly when the frame was taken (device time), so you can discard the rest.

For syncing: https://docs.luxonis.com/projects/api/en/latest/components/nodes/sync_node/

Thanks,
Jaka

vkmNuller

Hi jakaskerl

Thank you for sharing the sources; they've been really helpful.

However, I'm currently encountering an odd problem while implementing the sync node. When I place the MessageDemux node after the sync node, the application seems fairly stable. But when I move it in front of the script, it crashes after 10 to 15 seconds (the crashes are not consistent as well).

The error:

[1844301021E4691200] [1.3] [50.010] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 805AC7DC' '0'
Traceback (most recent call last):
  File "/data/vkm/code/weed/weedinator/weedinator2/test1.py", line 448, in <module>
    colorFrame = camera.colorQueue.tryGet()
RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'color' (X_LINK_ERROR)'
[1844301021E4691200] [1.3] [1715279736.172] [host] [warning] Device crashed. Crash dump saved to /tmp/depthai_C2KjGW/1844301021E4691200-depthai_crash_dump.json

What /tmp/depthai_C2KjGW/1844301021E4691200-depthai_crash_dump.json contains:

{"crashReports":[{"crashedThreadId":184614960,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2153433052,"trapName":"Bad trap","trapNumber":9}},"processor":1,"threadCallstack":[{"callStack":[{"callSite":2153490964,"calledTarget":2153506808,"context":"","framePointer":2155377344},{"callSite":2153484260,"calledTarget":2153490708,"context":"","framePointer":2155377440},{"callSite":2150440796,"calledTarget":2153484172,"context":"","framePointer":2155377576},{"callSite":2153505936,"calledTarget":0,"context":"Thread handler","framePointer":2155377688},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155377784},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155377880}],"instructionPointer":2153490964,"stackBottom":2155361592,"stackPointer":2155377344,"stackTop":2155377975,"threadId":167837697,"threadName":"CBTH","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2153477260,"calledTarget":2153506808,"context":"","framePointer":2155409784},{"callSite":2153477680,"calledTarget":2153477172,"context":"","framePointer":2155409952},{"callSite":2152616216,"calledTarget":2153477664,"context":"","framePointer":2155410048},{"callSite":2147607616,"calledTarget":2152616196,"context":"","framePointer":2155410160},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155410608},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155410704},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155410800}],"instructionPointer":2153477260,"stackBottom":2155378128,"stackPointer":2155409784,"stackTop":2155410895,"threadId":184614913,"threadName":"main","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155443272},{"callSite":2149111340,"calledTarget":2153486788,"context":"","framePointer":2155443408},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155443528},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155443624},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155443720}],"instructionPointer":2153486940,"stackBottom":2155411048,"stackPointer":2155443272,"stackTop":2155443815,"threadId":184614914,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155476800},{"callSite":2150134404,"calledTarget":2153486788,"context":"","framePointer":2155476936},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155477032},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155477128},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155477224}],"instructionPointer":2153486940,"stackBottom":2155444552,"stackPointer":2155476800,"stackTop":2155477319,"threadId":184614915,"threadName":"RmtBarrThrd","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155509536},{"callSite":2150819572,"calledTarget":2153486788,"context":"","framePointer":2155509672},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155509856},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155509952},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155510048},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155510144}],"instructionPointer":2153486940,"stackBottom":2155477472,"stackPointer":2155509536,"stackTop":2155510239,"threadId":184614916,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155542168},{"callSite":2149752384,"calledTarget":2149270580,"context":"","framePointer":2155542304},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155542776},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155542872},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155542968},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155543064}],"instructionPointer":2153486940,"stackBottom":2155510392,"stackPointer":2155542168,"stackTop":2155543159,"threadId":184614917,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153517436,"calledTarget":2153506808,"context":"","framePointer":2155574680},{"callSite":2153517648,"calledTarget":2153517388,"context":"","framePointer":2155574776},{"callSite":2147698020,"calledTarget":2153517620,"context":"","framePointer":2155574912},{"callSite":2147699420,"calledTarget":2147697884,"context":"","framePointer":2155575040},{"callSite":2147699752,"calledTarget":2147699300,"context":"","framePointer":2155575160},{"callSite":2147699916,"calledTarget":2147699624,"context":"","framePointer":2155575600},{"callSite":2153232304,"calledTarget":0,"context":"Thread handler","framePointer":2155575696},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155575792},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155575888},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155575984}],"instructionPointer":2153517436,"stackBottom":2155543312,"stackPointer":2155574680,"stackTop":2155576079,"threadId":184614918,"threadName":"","threadStatus":"WAITING_FOR_CONDITION_VARIABLE"},{"callStack":[{"callSite":2153490964,"calledTarget":2153506808,"context":"","framePointer":2155611960},{"callSite":2153474768,"calledTarget":2153490708,"context":"","framePointer":2155612056},{"callSite":2150239244,"calledTarget":2153474616,"context":"","framePointer":2155612192},{"callSite":2150300684,"calledTarget":2150239228,"context":"","framePointer":2155612296},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155612408},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155612504},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155612600}],"instructionPointer":2153490964,"stackBottom":2155579928,"stackPointer":2155611960,"stackTop":2155612695,"threadId":184614919,"threadName":"SIPP","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155654768},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2155654904},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2155655032},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2155655136},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2155655248},{"callSite":2149053348,"calledTarget":2148012996,"context":"","framePointer":2155655344},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155655624},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155655720},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155655816},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155655912}],"instructionPointer":2153486940,"stackBottom":2155623240,"stackPointer":2155654768,"stackTop":2155656007,"threadId":184614920,"threadName":"PlgStereoSync","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153480644,"calledTarget":2153504836,"context":"","framePointer":2155696416},{"callSite":2150179004,"calledTarget":2153480476,"context":"","framePointer":2155696552},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155696744},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155696840},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155696936},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155697032}],"instructionPointer":2153480644,"stackBottom":2155664360,"stackPointer":2155696416,"stackTop":2155697127,"threadId":184614921,"threadName":"PlgStereo","threadStatus":"UNKNOWN"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155729280},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2155729416},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2155729544},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2155729648},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2155729760},{"callSite":2149029072,"calledTarget":2148012996,"context":"","framePointer":2155729856},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155730248},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155730344},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155730440},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155730536}],"instructionPointer":2153486940,"stackBottom":2155697864,"stackPointer":2155729280,"stackTop":2155730631,"threadId":184614922,"threadName":"PlgStereoLoader","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155763008},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2155763144},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155763264},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155763360},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155763456}],"instructionPointer":2153486940,"stackBottom":2155730784,"stackPointer":2155763008,"stackTop":2155763551,"threadId":184614923,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155795128},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2155795264},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2155795392},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2155795496},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2155795608},{"callSite":2150113720,"calledTarget":2148012996,"context":"","framePointer":2155795704},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155796088},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155796184},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155796280},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155796376}],"instructionPointer":2153486940,"stackBottom":2155763704,"stackPointer":2155795128,"stackTop":2155796471,"threadId":184614924,"threadName":"PlgStereoRectif","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155828848},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2155828984},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155829104},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155829200},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155829296}],"instructionPointer":2153486940,"stackBottom":2155796624,"stackPointer":2155828848,"stackTop":2155829391,"threadId":184614925,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155861768},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2155861904},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155862024},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155862120},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155862216}],"instructionPointer":2153486940,"stackBottom":2155829544,"stackPointer":2155861768,"stackTop":2155862311,"threadId":184614926,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155894032},{"callSite":2148004236,"calledTarget":2153486788,"context":"","framePointer":2155894168},{"callSite":2149072628,"calledTarget":2148004152,"context":"","framePointer":2155894296},{"callSite":2149071588,"calledTarget":0,"context":"Thread handler","framePointer":2155894400},{"callSite":2149056260,"calledTarget":0,"context":"Thread handler","framePointer":2155894512},{"callSite":2150215396,"calledTarget":2149056228,"context":"","framePointer":2155894608},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155894848},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155894944},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155895040},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155895136}],"instructionPointer":2153486940,"stackBottom":2155862464,"stackPointer":2155894032,"stackTop":2155895231,"threadId":184614927,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155926504},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2155926640},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2155926768},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2155926872},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2155926984},{"callSite":2149055344,"calledTarget":2148012996,"context":"","framePointer":2155927080},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2155927768},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155927864},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155927960},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155928056}],"instructionPointer":2153486940,"stackBottom":2155895384,"stackPointer":2155926504,"stackTop":2155928151,"threadId":184614928,"threadName":"","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2155969840},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2155969976},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2155970096},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2155970192},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2155970288}],"instructionPointer":2153486940,"stackBottom":2155937616,"stackPointer":2155969840,"stackTop":2155970383,"threadId":184614929,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156002560},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156002696},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156002824},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156002928},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156003040},{"callSite":2150175584,"calledTarget":2148027800,"context":"","framePointer":2156003136},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156003360},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156003456},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156003552},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156003648}],"instructionPointer":2153486940,"stackBottom":2155970976,"stackPointer":2156002560,"stackTop":2156003743,"threadId":184614930,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156035976},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156036112},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156036232},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156036328},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156036424}],"instructionPointer":2153486940,"stackBottom":2156003752,"stackPointer":2156035976,"stackTop":2156036519,"threadId":184614931,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156068184},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156068320},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156068448},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156068552},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156068664},{"callSite":2150158492,"calledTarget":2148012996,"context":"","framePointer":2156068760},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156068912},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156069008},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156069104},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156069200}],"instructionPointer":2153486940,"stackBottom":2156036528,"stackPointer":2156068184,"stackTop":2156069295,"threadId":184614932,"threadName":"PlgIspCtrl","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156101040},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156101176},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156101304},{"callSite":2148047752,"calledTarget":0,"context":"Thread handler","framePointer":2156101408},{"callSite":2150159804,"calledTarget":0,"context":"Thread handler","framePointer":2156101504},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156101832},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156101928},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156102024},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156102120}],"instructionPointer":2153486940,"stackBottom":2156069448,"stackPointer":2156101040,"stackTop":2156102215,"threadId":184614933,"threadName":"PlgMonoIsp","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156134592},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156134728},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156134848},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156134944},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156135040}],"instructionPointer":2153486940,"stackBottom":2156102368,"stackPointer":2156134592,"stackTop":2156135135,"threadId":184614934,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156167320},{"callSite":2148066696,"calledTarget":2153486788,"context":"","framePointer":2156167456},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156167672},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156167768},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156167864},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156167960}],"instructionPointer":2153486940,"stackBottom":2156135288,"stackPointer":2156167320,"stackTop":2156168055,"threadId":184614935,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156199672},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156199808},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156199936},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156200040},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156200152},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156200248},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156200592},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156200688},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156200784},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156200880}],"instructionPointer":2153486940,"stackBottom":2156168208,"stackPointer":2156199672,"stackTop":2156200975,"threadId":184614936,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156240952},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156241088},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156241216},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156241320},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156241432},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156241528},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156241872},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156241968},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156242064},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156242160}],"instructionPointer":2153486940,"stackBottom":2156209488,"stackPointer":2156240952,"stackTop":2156242255,"threadId":184614937,"threadName":"MRmO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156274576},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156274712},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156274840},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156274944},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156275056},{"callSite":2150175584,"calledTarget":2148027800,"context":"","framePointer":2156275152},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156275376},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156275472},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156275568},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156275664}],"instructionPointer":2153486940,"stackBottom":2156242992,"stackPointer":2156274576,"stackTop":2156275759,"threadId":184614938,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156308136},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156308272},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156308392},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156308488},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156308584}],"instructionPointer":2153486940,"stackBottom":2156275912,"stackPointer":2156308136,"stackTop":2156308679,"threadId":184614939,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156340488},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156340624},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156340752},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156340856},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156340968},{"callSite":2150158492,"calledTarget":2148012996,"context":"","framePointer":2156341064},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156341216},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156341312},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156341408},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156341504}],"instructionPointer":2153486940,"stackBottom":2156308832,"stackPointer":2156340488,"stackTop":2156341599,"threadId":184614940,"threadName":"PlgIspCtrl","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156373344},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156373480},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156373608},{"callSite":2148047752,"calledTarget":0,"context":"Thread handler","framePointer":2156373712},{"callSite":2150159804,"calledTarget":0,"context":"Thread handler","framePointer":2156373808},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156374136},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156374232},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156374328},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156374424}],"instructionPointer":2153486940,"stackBottom":2156341752,"stackPointer":2156373344,"stackTop":2156374519,"threadId":184614941,"threadName":"PlgMonoIsp","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156406896},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156407032},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156407152},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156407248},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156407344}],"instructionPointer":2153486940,"stackBottom":2156374672,"stackPointer":2156406896,"stackTop":2156407439,"threadId":184614942,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156439624},{"callSite":2148066696,"calledTarget":2153486788,"context":"","framePointer":2156439760},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156439976},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156440072},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156440168},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156440264}],"instructionPointer":2153486940,"stackBottom":2156407592,"stackPointer":2156439624,"stackTop":2156440359,"threadId":184614943,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156471976},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156472112},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156472240},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156472344},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156472456},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156472552},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156472896},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156472992},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156473088},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156473184}],"instructionPointer":2153486940,"stackBottom":2156440512,"stackPointer":2156471976,"stackTop":2156473279,"threadId":184614944,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156513296},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156513432},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156513560},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156513664},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156513776},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156513872},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156514216},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156514312},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156514408},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156514504}],"instructionPointer":2153486940,"stackBottom":2156481832,"stackPointer":2156513296,"stackTop":2156514599,"threadId":184614945,"threadName":"MRmO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156546776},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156546912},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156547040},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156547144},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156547256},{"callSite":2150175584,"calledTarget":2148027800,"context":"","framePointer":2156547352},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156547576},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156547672},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156547768},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156547864}],"instructionPointer":2153486940,"stackBottom":2156515192,"stackPointer":2156546776,"stackTop":2156547959,"threadId":184614946,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156580336},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156580472},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156580592},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156580688},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156580784}],"instructionPointer":2153486940,"stackBottom":2156548112,"stackPointer":2156580336,"stackTop":2156580879,"threadId":184614947,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156612688},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156612824},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156612952},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156613056},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156613168},{"callSite":2150158492,"calledTarget":2148012996,"context":"","framePointer":2156613264},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156613416},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156613512},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156613608},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156613704}],"instructionPointer":2153486940,"stackBottom":2156581032,"stackPointer":2156612688,"stackTop":2156613799,"threadId":184614948,"threadName":"PlgIspCtrl","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156645432},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156645568},{"callSite":2148065872,"calledTarget":2148001352,"context":"","framePointer":2156645696},{"callSite":2148043288,"calledTarget":0,"context":"Thread handler","framePointer":2156645808},{"callSite":2148027552,"calledTarget":0,"context":"Thread handler","framePointer":2156645920},{"callSite":2150155188,"calledTarget":2148027520,"context":"","framePointer":2156646016},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156646336},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156646432},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156646528},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156646624}],"instructionPointer":2153486940,"stackBottom":2156613952,"stackPointer":2156645432,"stackTop":2156646719,"threadId":184614949,"threadName":"PlgColorIsp1xPo","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156679096},{"callSite":2148047376,"calledTarget":2153486788,"context":"","framePointer":2156679232},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156679352},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156679448},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156679544}],"instructionPointer":2153486940,"stackBottom":2156646872,"stackPointer":2156679096,"stackTop":2156679639,"threadId":184614950,"threadName":"rmtPool","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156711824},{"callSite":2148066696,"calledTarget":2153486788,"context":"","framePointer":2156711960},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156712176},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156712272},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156712368},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156712464}],"instructionPointer":2153486940,"stackBottom":2156679792,"stackPointer":2156711824,"stackTop":2156712559,"threadId":184614951,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156744176},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156744312},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156744440},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156744544},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156744656},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156744752},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156745096},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156745192},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156745288},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156745384}],"instructionPointer":2153486940,"stackBottom":2156712712,"stackPointer":2156744176,"stackTop":2156745479,"threadId":184614952,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156785536},{"callSite":2148001436,"calledTarget":2153486788,"context":"","framePointer":2156785672},{"callSite":2148028628,"calledTarget":2148001352,"context":"","framePointer":2156785800},{"callSite":2148043552,"calledTarget":0,"context":"Thread handler","framePointer":2156785904},{"callSite":2148027832,"calledTarget":0,"context":"Thread handler","framePointer":2156786016},{"callSite":2148033976,"calledTarget":2148027800,"context":"","framePointer":2156786112},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156786456},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156786552},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156786648},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156786744}],"instructionPointer":2153486940,"stackBottom":2156754072,"stackPointer":2156785536,"stackTop":2156786839,"threadId":184614953,"threadName":"CRmO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156817272},{"callSite":2148001636,"calledTarget":2153486788,"context":"","framePointer":2156817408},{"callSite":2148091208,"calledTarget":2148001552,"context":"","framePointer":2156817536},{"callSite":2148088872,"calledTarget":0,"context":"Thread handler","framePointer":2156817640},{"callSite":2148081724,"calledTarget":0,"context":"Thread handler","framePointer":2156817752},{"callSite":2148733124,"calledTarget":0,"context":"Thread handler","framePointer":2156817848},{"callSite":2148358244,"calledTarget":0,"context":"Thread handler","framePointer":2156818024},{"callSite":2151275000,"calledTarget":0,"context":"Thread handler","framePointer":2156818352},{"callSite":2151084708,"calledTarget":0,"context":"Thread handler","framePointer":2156818448},{"callSite":2151092848,"calledTarget":2151084948,"context":"","framePointer":2156818544},{"callSite":2151085548,"calledTarget":0,"context":"Thread handler","framePointer":2156818664},{"callSite":2151605076,"calledTarget":2151604168,"context":"","framePointer":2156818768},{"callSite":2151627852,"calledTarget":2151604664,"context":"","framePointer":2156818872},{"callSite":2151633188,"calledTarget":0,"context":"Thread handler","framePointer":2156819056},{"callSite":2151633340,"calledTarget":2151630572,"context":"","framePointer":2156819184},{"callSite":2151633456,"calledTarget":2151633236,"context":"","framePointer":2156819320},{"callSite":2151633516,"calledTarget":2151633356,"context":"","framePointer":2156819456},{"callSite":2148394768,"calledTarget":2151633472,"context":"","framePointer":2156819576},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156819816},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156819912},{"callSite":184614955,"calledTarget":1333030226,"context":"Thread exit","framePointer":0}],"instructionPointer":2153486940,"stackBottom":2156787432,"stackPointer":2156817272,"stackTop":2156820199,"threadId":184614954,"threadName":"PlgCPython","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156851848},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156851984},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156852112},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156852216},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156852328},{"callSite":2149977384,"calledTarget":2148012996,"context":"","framePointer":2156852424},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156852736},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156852832},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156852928},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156853024}],"instructionPointer":2153486940,"stackBottom":2156820352,"stackPointer":2156851848,"stackTop":2156853119,"threadId":184614955,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156884768},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156884904},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156885032},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156885136},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156885248},{"callSite":2149977384,"calledTarget":2148012996,"context":"","framePointer":2156885344},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156885656},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156885752},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156885848},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156885944}],"instructionPointer":2153486940,"stackBottom":2156853272,"stackPointer":2156884768,"stackTop":2156886039,"threadId":184614956,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156917688},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156917824},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156917952},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156918056},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156918168},{"callSite":2149977384,"calledTarget":2148012996,"context":"","framePointer":2156918264},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156918576},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156918672},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156918768},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156918864}],"instructionPointer":2153486940,"stackBottom":2156886192,"stackPointer":2156917688,"stackTop":2156918959,"threadId":184614957,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156951176},{"callSite":2149977960,"calledTarget":2149270580,"context":"","framePointer":2156951312},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156951496},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156951592},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156951688},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156951784}],"instructionPointer":2153486940,"stackBottom":2156919112,"stackPointer":2156951176,"stackTop":2156951879,"threadId":184614958,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2156983528},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2156983664},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2156983792},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2156983896},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2156984008},{"callSite":2149977384,"calledTarget":2148012996,"context":"","framePointer":2156984104},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2156984416},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2156984512},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2156984608},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2156984704}],"instructionPointer":2153486940,"stackBottom":2156952032,"stackPointer":2156983528,"stackTop":2156984799,"threadId":184614959,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2150078744,"calledTarget":2150072708,"context":"","framePointer":2154797464},{"callSite":2153513276,"calledTarget":0,"context":"Thread handler","framePointer":2154797560},{"callSite":1881162980,"calledTarget":0,"context":"Thread handler","framePointer":2154797656},{"callSite":2153513428,"calledTarget":1881162912,"context":"","framePointer":2154797776},{"callSite":2153431488,"calledTarget":2153513408,"context":"","framePointer":2154797880},{"callSite":2153532612,"calledTarget":0,"context":"Thread handler","framePointer":2154797984},{"callSite":2153433084,"calledTarget":2153432708,"context":"","framePointer":2157016688},{"callSite":2150238692,"calledTarget":2153432972,"context":"","framePointer":2157016960},{"callSite":2147994988,"calledTarget":2150238684,"context":"","framePointer":2157017056},{"callSite":2149962184,"calledTarget":2147994940,"context":"","framePointer":2157017152},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2157017336},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157017432},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157017528},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157017624}],"instructionPointer":2150078744,"stackBottom":2156984952,"stackPointer":2154797464,"stackTop":2157017719,"threadId":184614960,"threadName":"RmtI","threadStatus":"READY"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157058416},{"callSite":2149977960,"calledTarget":2149270580,"context":"","framePointer":2157058552},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2157058736},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157058832},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157058928},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157059024}],"instructionPointer":2153486940,"stackBottom":2157026352,"stackPointer":2157058416,"stackTop":2157059119,"threadId":184614961,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157091776},{"callSite":2149977960,"calledTarget":2149270580,"context":"","framePointer":2157091912},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2157092096},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157092192},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157092288},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157092384}],"instructionPointer":2153486940,"stackBottom":2157059712,"stackPointer":2157091776,"stackTop":2157092479,"threadId":184614962,"threadName":"RmtI","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157124128},{"callSite":2148001636,"calledTarget":2153486788,"context":"","framePointer":2157124264},{"callSite":2148091208,"calledTarget":2148001552,"context":"","framePointer":2157124392},{"callSite":2148088872,"calledTarget":0,"context":"Thread handler","framePointer":2157124496},{"callSite":2148081724,"calledTarget":0,"context":"Thread handler","framePointer":2157124608},{"callSite":2149977680,"calledTarget":2148081692,"context":"","framePointer":2157124704},{"callSite":2150135224,"calledTarget":0,"context":"Thread handler","framePointer":2157125016},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157125112},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157125208},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157125304}],"instructionPointer":2153486940,"stackBottom":2157092632,"stackPointer":2157124128,"stackTop":2157125399,"threadId":184614963,"threadName":"RmtO","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153517944,"calledTarget":2153506808,"context":"","framePointer":2157155328},{"callSite":2148202572,"calledTarget":2153517852,"context":"","framePointer":2157155464},{"callSite":2150182000,"calledTarget":2148202560,"context":"","framePointer":2157155560},{"callSite":2150187336,"calledTarget":2150181612,"context":"","framePointer":2157157936},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157158032},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157158128},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157158224}],"instructionPointer":2153517944,"stackBottom":2157125552,"stackPointer":2157155328,"stackTop":2157158319,"threadId":184614964,"threadName":"PlgStereo","threadStatus":"WAITING_FOR_MUTEX"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157187904},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2157188040},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2157188168},{"callSite":2148020104,"calledTarget":0,"context":"Thread handler","framePointer":2157188272},{"callSite":2148013028,"calledTarget":0,"context":"Thread handler","framePointer":2157188384},{"callSite":2150182016,"calledTarget":2148012996,"context":"","framePointer":2157188480},{"callSite":2150187336,"calledTarget":2150181612,"context":"","framePointer":2157190856},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157190952},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157191048},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157191144}],"instructionPointer":2153486940,"stackBottom":2157158472,"stackPointer":2157187904,"stackTop":2157191239,"threadId":184614965,"threadName":"lgStereo","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153483080,"calledTarget":1881162428,"context":"","framePointer":2157225064},{"callSite":2153482892,"calledTarget":2153482916,"context":"","framePointer":2157225168},{"callSite":2150168696,"calledTarget":2153482788,"context":"","framePointer":2157225288},{"callSite":2150169200,"calledTarget":2150168584,"context":"","framePointer":2157225448},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157225544},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157225640},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157225736}],"instructionPointer":2153483080,"stackBottom":2157193064,"stackPointer":2157225064,"stackTop":2157225831,"threadId":184614966,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2153490964,"calledTarget":2153506808,"context":"","framePointer":2157257912},{"callSite":2153474768,"calledTarget":2153490708,"context":"","framePointer":2157258008},{"callSite":2150170320,"calledTarget":2153474616,"context":"","framePointer":2157258144},{"callSite":2150172084,"calledTarget":2150170192,"context":"","framePointer":2157258368},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157258464},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157258560},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157258656}],"instructionPointer":2153490964,"stackBottom":2157225984,"stackPointer":2157257912,"stackTop":2157258751,"threadId":184614967,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157293488},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2157293624},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2157293752},{"callSite":2148047752,"calledTarget":0,"context":"Thread handler","framePointer":2157293856},{"callSite":2150168960,"calledTarget":0,"context":"Thread handler","framePointer":2157293952},{"callSite":2150169200,"calledTarget":2150168584,"context":"","framePointer":2157294112},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157294208},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157294304},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157294400}],"instructionPointer":2153486940,"stackBottom":2157261728,"stackPointer":2157293488,"stackTop":2157294495,"threadId":184614968,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153490964,"calledTarget":2153506808,"context":"","framePointer":2157335088},{"callSite":2153474768,"calledTarget":2153490708,"context":"","framePointer":2157335184},{"callSite":2150170320,"calledTarget":2153474616,"context":"","framePointer":2157335320},{"callSite":2150172084,"calledTarget":2150170192,"context":"","framePointer":2157335544},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157335640},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157335736},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157335832}],"instructionPointer":2153490964,"stackBottom":2157303160,"stackPointer":2157335088,"stackTop":2157335927,"threadId":184614969,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_MESSAGE"},{"callStack":[{"callSite":2153486940,"calledTarget":2153506808,"context":"","framePointer":2157369952},{"callSite":2148001336,"calledTarget":2153486788,"context":"","framePointer":2157370088},{"callSite":2148026464,"calledTarget":2148001252,"context":"","framePointer":2157370216},{"callSite":2148047752,"calledTarget":0,"context":"Thread handler","framePointer":2157370320},{"callSite":2150168960,"calledTarget":0,"context":"Thread handler","framePointer":2157370416},{"callSite":2150169200,"calledTarget":2150168584,"context":"","framePointer":2157370576},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157370672},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157370768},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157370864}],"instructionPointer":2153486940,"stackBottom":2157338192,"stackPointer":2157369952,"stackTop":2157370959,"threadId":184614970,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_SEMAPHORE"},{"callStack":[{"callSite":2153490964,"calledTarget":2153506808,"context":"","framePointer":2157403040},{"callSite":2153474768,"calledTarget":2153490708,"context":"","framePointer":2157403136},{"callSite":2150170320,"calledTarget":2153474616,"context":"","framePointer":2157403272},{"callSite":2150172084,"calledTarget":2150170192,"context":"","framePointer":2157403496},{"callSite":2153505960,"calledTarget":0,"context":"Thread handler","framePointer":2157403592},{"callSite":1881162860,"calledTarget":0,"context":"Thread handler","framePointer":2157403688},{"callSite":1881162764,"calledTarget":0,"context":"Thread exit","framePointer":2157403784}],"instructionPointer":2153490964,"stackBottom":2157371112,"stackPointer":2157403040,"stackTop":2157403879,"threadId":184614971,"threadName":"PlgSrcMipi","threadStatus":"WAITING_FOR_MESSAGE"}]}],"depthaiCommitHash":"7665aed8712bb941958660a0103868908726eb56","deviceId":"1844301021E4691200"}

#1 Pipeline link:

Script.syncFramesOut (12) -> MessageDemux.input (1)
StereoDepth.depth (29) -> XLinkOut.in (13)
MessageDemux.rightSync (2) -> StereoDepth.right (14)
MessageDemux.leftSync (3) -> StereoDepth.left (15)
MessageDemux.colorSync (4) -> XLinkOut.in (38)
MonoCamera.out (33) -> Sync.rightSync (5)
MonoCamera.out (37) -> Sync.leftSync (6)
ColorCamera.isp (43) -> Sync.colorSync (7)
Sync.out (8) -> Script.syncFramesIn (10)
XLinkIn.out (9) -> Script.trigger (11)

#2 Pipeline link (working):

Sync.out (8) -> MessageDemux.input (1)
StereoDepth.depth (33) -> XLinkOut.in (17)
Script.rightScript (14) -> StereoDepth.right (18)
Script.leftScript (15) -> StereoDepth.left (19)
Script.colorScript (16) -> XLinkOut.in (42)
MonoCamera.out (37) -> Sync.rightSync (5)
MonoCamera.out (41) -> Sync.leftSync (6)
ColorCamera.isp (47) -> Sync.colorSync (7)
MessageDemux.rightSync (2) -> Script.rightSync (10)
MessageDemux.leftSync (3) -> Script.leftSync (11)
MessageDemux.colorSync (4) -> Script.colorSync (12)
XLinkIn.out (9) -> Script.trigger (13)

jakaskerl

Hi @vkmNuller
What's the script logic? Seems like the messageGroup is not handled properly.

Thanks,
Jaka

vkmNuller

Hi jakaskerl

This is the code i'm running:

import cv2
import numpy as np
import depthai as dai

from typing import Tuple
from datetime import timedelta, datetime
from enum import Enum

class Resolution(Enum):
    NONE    = (-1, -1)
    FULLHD  = (1920, 1080)
    UHD     = (3840, 2160)
    MP12    = (4056, 3040)


class LogType(Enum):
    INFO    = "INFO"
    CAMERA  = "CAMERA INFO"
    DEBUG   = "DEBUG"
    WARNING = "WARNING"
    ERROR   = "ERROR"

class OutputType(Enum):
    NONE   = "None"
    RAW    = "RAW"
    OUTPUT = "OUTPUT"

class TestCamera:
    def __init__(
        self,
        resolution : Resolution,
        FPS : int = 60,
        logLevel : LogType = LogType.INFO
    ):
        self.logLevel = logLevel
        self.FPS      = FPS

        self.bufferSize = -1

        self.logMessages = []

        self.streamNameColor  = "color"
        self.streamNameDepth  = "depth"
        self.streamNameScript = "script"
        self.streamNameSync   = "sync"

        self.nameAccumulationScript = "Stream Accumulation Valve"
        self.namePassScript         = "Stream Pass Valve"

        self.COLOR_CAMERA_SOCKET = dai.CameraBoardSocket.RGB
        self.LEFT_CAMERA_SOCKET  = dai.CameraBoardSocket.LEFT
        self.RIGHT_CAMERA_SOCKET = dai.CameraBoardSocket.RIGHT

        # In miliseconds
        self.syncThreshold = 15

        # Pixel size of the largest Color & Left/Right
        self.maxPixelsSizeColor  = 21000000
        self.maxPixelsSizeStereo = 6220800

        self.pipeline = dai.Pipeline()

        self._setReolutions(resolution)

        self._initStereoDepthProperties()


    def _setReolutions(self, resolution):
        if resolution == Resolution.FULLHD:
            self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_1080_P
        elif resolution == Resolution.UHD:
            self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_4_K
        elif resolution == Resolution.MP12:
            self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_12_MP
        else:
            self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_4_K

        self.MONO_RESOLUTION  = dai.MonoCameraProperties.SensorResolution.THE_400_P

        self.imageWidth  = resolution.value[0]
        self.imageHeight = resolution.value[1]


    def _initStereoDepthProperties(self):
        self.medianFilter        = dai.MedianFilter.MEDIAN_OFF
        self.profilePreset       = dai.node.StereoDepth.PresetMode.HIGH_ACCURACY
        self.lrCheck             = True
        self.subPixel            = False
        self.extendedDisparity   = False
        self.confidenceThreshold = 128

        self.speckleFilterEnable              = False
        self.speckleFilterSpeckleRange        = 5
        self.temporalFilterEnable             = True
        self.temporalFilterPersistencyMode    = dai.RawStereoDepthConfig.PostProcessing.TemporalFilter.PersistencyMode.VALID_1_IN_LAST_8
        self.spatialFilterEnable              = True
        self.spatialFilterHoleFillingRadius   = 2
        self.spatialFilterHumIterations       = 1
        self.thresholdFilterMinRange          = 200
        self.thresholdFilterMaxRange          = 12000
        self.decimationFilterDecimationFactor = 1
        self.decimationFilterDecimationMode   = dai.RawStereoDepthConfig.PostProcessing.DecimationFilter.DecimationMode.NON_ZERO_MEAN


    def startCapture(self):
        self._initPipeline()

        self._setProperties()

        self._linkPipeline()

        self._initQueues()

        self._logMessage(f"Start Capturing", LogType.INFO)


    def stopCapture(self):
        if self.device:
            self.device.close()

        self._logMessage(f"Stop Capturing", LogType.INFO)


    def triggerFramePass(self):
        if self.scriptQueue:
            triggerTS = dai.Clock.now().total_seconds()

            buf = dai.Buffer()
            buf.setData(True)

            self.scriptQueue.send(buf)

            self._logMessage(f"Trigger at {triggerTS}", LogType.INFO)


    def getBuffers(self):
        colorBuffer     = None
        leftBuffer      = None
        rightBuffer     = None
        depthBuffer     = None
        disparityBuffer = None

        colorBuffer = self.colorQueue.tryGet()
        depthBuffer = self.depthQueue.tryGet()

        # if colorBuffer or leftBuffer or rightBuffer or depthBuffer or disparityBuffer:
        #     self._logMessage(
        #         f"{'Frame' if colorBuffer else 'None'} | {'Frame' if leftBuffer else 'None'} | {'Frame' if rightBuffer else 'None'} | {'Frame' if depthBuffer else 'None'} | {'Frame' if disparityBuffer else 'None'}",
        #         LogType.INFO
        #     )

        return (
            colorBuffer,
            leftBuffer,
            rightBuffer,
            depthBuffer,
            disparityBuffer
        )


    def getFrames(self):
        colorFrame     = None
        leftFrame      = None
        rightFrame     = None
        depthFrame     = None
        disparityFrame = None

        colorBuffer, leftBuffer, rightBuffer, depthBuffer, disparityBuffer = self.getBuffers()

        colorFrame     = colorBuffer.getCvFrame()     if colorBuffer     is not None else None
        depthFrame     = depthBuffer.getCvFrame()     if depthBuffer     is not None else None
        leftFrame      = leftBuffer.getCvFrame()      if leftBuffer      is not None else None
        rightFrame     = rightBuffer.getCvFrame()     if rightBuffer     is not None else None
        disparityFrame = disparityBuffer.getCvFrame() if disparityBuffer is not None else None

        return (
            colorFrame,
            leftFrame,
            rightFrame,
            depthFrame,
            disparityFrame
        )


    def _initPipeline(self):
        # Improves latency: https://discuss.luxonis.com/d/731-how-to-increase-oakd-lite-s-stereo-pair-frame-rate
        self.pipeline.setXLinkChunkSize(0)

        self._initColorStream(
            streamNameColor = self.streamNameColor
        )
        self._initMonoStream(
        )
        self._initStereoDepthStream(
            streamNameDepth = self.streamNameDepth
        )
        self._initScripts(
            streamNameScript = self.streamNameScript
        )
        self._initSync()


    def _setProperties(self):
        self._setColorCameraProperties(
            socket     = self.COLOR_CAMERA_SOCKET,
            resolution = self.COLOR_RESOLUTION,
            FPS        = self.FPS
        )
        self._setMonoCameraProperties(
            socketLeft  = self.LEFT_CAMERA_SOCKET,
            socketRight = self.RIGHT_CAMERA_SOCKET,
            resolution  = self.MONO_RESOLUTION,
            FPS         = self.FPS
        )
        self._setStereoDepthProperties(
            depthAlignSocket    = self.COLOR_CAMERA_SOCKET,
            medianFilter        = self.medianFilter,
            profilePreset       = self.profilePreset,
            lrCheck             = self.lrCheck,
            subPixel            = self.subPixel,
            extendedDisparity   = self.extendedDisparity,
            confidenceThreshold = self.confidenceThreshold
        )
        self._setScriptProperties(
            scriptAccumulationName = self.nameAccumulationScript
        )
        self._setSyncProperties(
            syncThreshold = self.syncThreshold
        )


    def _linkPipeline(self):
        # Link Color/Left/Right stream based on replay mode
        self.color.isp.link(self.sync.inputs["colorSync"])
        self.monoLeft.out.link(self.sync.inputs["leftSync"])
        self.monoRight.out.link(self.sync.inputs["rightSync"])

        # Link Sync output to Accumulation Script input
        self.sync.out.link(self.accumulationScript.inputs["inFrames"])

        # Link Accumulation Script trigger input
        self.xinScript.out.link(self.accumulationScript.inputs['trigger'])

        # Link Accumulation Script outputs based on configuration
        self.accumulationScript.outputs["outFrames"].link(self.demux.input)

        self.demux.outputs["colorSync"].link(self.xoutColor.input)
        self.demux.outputs["leftSync"].link(self.stereoDepth.left)
        self.demux.outputs["rightSync"].link(self.stereoDepth.right)

        self.stereoDepth.depth.link(self.xoutDepth.input)



    def _initQueues(self):
        self.device = dai.Device(self.pipeline)

        self.device.setTimesync(timedelta(seconds=5), 10, True)

        # self.device.setLogLevel(dai.LogLevel.ERR)
        # self.device.setLogOutputLevel(dai.LogLevel.ERR)

        outputFrames = 1

        self.colorQueue  = self.device.getOutputQueue(name=self.streamNameColor, maxSize=outputFrames, blocking=True)
        self.depthQueue  = self.device.getOutputQueue(name=self.streamNameDepth,  maxSize=outputFrames, blocking=False)
        self.scriptQueue = self.device.getInputQueue(name=self.streamNameScript, maxSize=outputFrames, blocking=True)


    def _initColorStream(
        self,
        streamNameColor : str
    ):
        self.color     = self.pipeline.create(dai.node.ColorCamera)
        self.xoutColor = self.pipeline.createXLinkOut()
        self.xoutColor.setStreamName(streamNameColor)


    def _initMonoStream(
        self
    ):
        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)


    def _initStereoDepthStream(
        self,
        streamNameDepth : str
    ):
        self.stereoDepth = self.pipeline.create(dai.node.StereoDepth)
        self.xoutDepth   = self.pipeline.create(dai.node.XLinkOut)
        self.xoutDepth.setStreamName(streamNameDepth)


    def _initScripts(
        self,
        streamNameScript : str
    ):
        self.accumulationScript = self.pipeline.create(dai.node.Script)

        self.xinScript = self.pipeline.create(dai.node.XLinkIn)
        self.xinScript.setStreamName(streamNameScript)


    def _initSync(self):
        self.sync  = self.pipeline.create(dai.node.Sync)
        self.demux = self.pipeline.create(dai.node.MessageDemux)


    def _setColorCameraProperties(
        self,
        socket : dai.CameraBoardSocket,
        resolution : dai.ColorCameraProperties.SensorResolution,
        FPS : int
    ):
        self.color.setBoardSocket(socket)
        self.color.setResolution(resolution)
        self.color.setFps(FPS)


    def _setMonoCameraProperties(
        self,
        socketLeft : dai.CameraBoardSocket,
        socketRight : dai.CameraBoardSocket,
        resolution : dai.ColorCameraProperties.SensorResolution,
        FPS : int
    ):
        self.monoLeft.setBoardSocket(socketLeft)
        self.monoLeft.setFps(FPS)
        self.monoLeft.setResolution(resolution)

        self.monoRight.setBoardSocket(socketRight)
        self.monoRight.setFps(FPS)
        self.monoRight.setResolution(resolution)

    def _setStereoDepthProperties(
            self,
            depthAlignSocket : dai.CameraBoardSocket,
            medianFilter : dai.MedianFilter,
            profilePreset : dai.node.StereoDepth.PresetMode,
            lrCheck : bool,
            subPixel : bool,
            extendedDisparity : bool,
            confidenceThreshold : int
    ):
        self.stereoDepth.setDepthAlign(depthAlignSocket)

        self.stereoDepth.initialConfig.setMedianFilter(medianFilter)
        self.stereoDepth.initialConfig.setConfidenceThreshold(confidenceThreshold)

        self.stereoDepth.setDefaultProfilePreset(profilePreset)
        self.stereoDepth.setLeftRightCheck(lrCheck)
        self.stereoDepth.setExtendedDisparity(extendedDisparity)
        self.stereoDepth.setSubpixel(subPixel)

        config = self.stereoDepth.initialConfig.get()
        config.postProcessing.speckleFilter.enable              = self.speckleFilterEnable
        config.postProcessing.speckleFilter.speckleRange        = self.speckleFilterSpeckleRange

        config.postProcessing.temporalFilter.enable             = self.temporalFilterEnable
        config.postProcessing.temporalFilter.persistencyMode    = self.temporalFilterPersistencyMode

        config.postProcessing.spatialFilter.enable              = self.spatialFilterEnable
        config.postProcessing.spatialFilter.holeFillingRadius   = self.spatialFilterHoleFillingRadius
        config.postProcessing.spatialFilter.numIterations       = self.spatialFilterHumIterations

        config.postProcessing.thresholdFilter.minRange          = self.thresholdFilterMinRange
        config.postProcessing.thresholdFilter.maxRange          = self.thresholdFilterMaxRange

        config.postProcessing.decimationFilter.decimationFactor = self.decimationFilterDecimationFactor
        config.postProcessing.decimationFilter.decimationMode   = self.decimationFilterDecimationMode

        self.stereoDepth.initialConfig.set(config)


    def _setScriptProperties(
        self,
        scriptAccumulationName : str
    ):
        accumulationScript = """
        while True:
            # Drop all color/left/right frames except the last one
            frames = node.io['inFrames'].get()

            msg = node.io['trigger'].tryGet()

            if msg is not None:
                node.warn(f"Trigger accumulation")

                node.io['outFrames'].send(frames)
        """

        self.accumulationScript.setScript(
            script = accumulationScript,
            name   = scriptAccumulationName
        )

    def _setSyncProperties(
        self,
        syncThreshold : int
    ):
        self.sync.setSyncThreshold(timedelta(milliseconds=syncThreshold))


    def _logMessage(
        self,
        message : str,
        logType : LogType
    ):
        timestamp = datetime.now().strftime("%Y.%m.%d-%H:%M:%S")
        message = f"[{timestamp}] [CAMERA CORE] [{logType.value}] {message}"

        self.logMessages.append(message)

        if self.logLevel == logType:
            print(message)



if __name__ == "__main__":
    camera = TestCamera(
        resolution = Resolution.FULLHD
    )

    windowSize      = (1280, 980)
    windowSizeSmall = (400, 200)

    cv2.namedWindow("color", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("color", windowSize)

    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("depth", windowSize)

    cv2.namedWindow("Current TS", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("Current TS", windowSizeSmall)

    exitNow = False

    font     = cv2.FONT_HERSHEY_DUPLEX
    color    = (0, 0, 255)
    fontsize = 1
    position = (10, 100)

    snapTS = None

    camera.startCapture()

    while not exitNow:
        key = cv2.waitKey(1)

        if key == ord("q"):
            print("Exit")
            exitNow = True

        if key == ord("s"):
            camera.triggerFramePass()

        blankFrame = np.zeros((200, 400, 3), dtype=np.uint8)
        blankFrame.fill(255)
        cv2.putText(blankFrame, f"TS: {dai.Clock.now().total_seconds()}", (0, 100), font, fontsize, (0, 0, 0), 3)
        cv2.imshow("Current TS", blankFrame)

        colorFrame, leftFrame, rightFrame, depthFrame, disparityFrame = camera.getBuffers()

        if colorFrame:
            ts = colorFrame.getTimestamp().total_seconds()
            cvColorFrame = colorFrame.getCvFrame()

            print(f"Color TS: {ts}")

            cv2.putText(cvColorFrame, f"TS: {ts} FRAME", position, font, fontsize, color)
            cv2.putText(cvColorFrame, f"TS: {dai.Clock.now().total_seconds()} DAI", (10, 130), font, fontsize, color)
            cv2.imshow("color", cvColorFrame)

        if depthFrame:
            ts = depthFrame.getTimestamp().total_seconds()
            cvDepthFrame = depthFrame.getCvFrame()

            cvDepthFrame = (cvDepthFrame / 10).astype(np.uint8)

            print(f"Depth TS: {ts}")

            cv2.putText(cvDepthFrame, f"TS: {ts} FRAME", position, font, fontsize, color)
            cv2.putText(cvDepthFrame, f"TS: {dai.Clock.now().total_seconds()} DAI", (10, 130), font, fontsize, color)
            cv2.imshow("depth", cvDepthFrame)

    camera.stopCapture()

Something interesting that I found while testing:

When you pass every frame, it takes a bit longer for the program to crash.
Script example for passing every frame

        accumulationScript = """
        process = 0
        while True:
            # Drop all color/left/right frames except the last one
            frames = node.io['inFrames'].get()
            process = process + 1
            
            msg = node.io['trigger'].tryGet()

            if msg is not None or True:
                node.warn(f"Trigger accumulation")

                node.io['outFrames'].send(frames)
        """

When I passed every 3rd frame, it took less time for it to crash, and for every larger offset, less and less time, but at the same time, sometimes it crashed instantly.

The message remains consistent:

[1844301021E4691200] [1.3] [10.059] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 805AC7DC' '0'

jakaskerl

Hi @vkmNuller
Please make a MRE, this is impossible to read.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl

It looks like I fixed the problem with the sync & demux pipeline crashing. However, the pipeline appears to be stable only when the FPS is set to 20 or below. I tested it in a long run and it seems to be stable at 30 FPS, but I haven't tested it for an extended period.

My question is, why does the pipeline freeze at random points when the FPS is above 20 (or 30)?

I hope this is clear enough:

import json
import time
import random

import cv2
import numpy as np

import depthai as dai

from typing import Tuple
from datetime import timedelta


class CameraInterface:
    def __init__(
        self
    ):
        self.framesToAccumulation = 5
        self.FPS                  = 30

        self.streamNameColor     = "color"
        self.streamNameLeft      = "left"
        self.streamNameRight     = "right"
        self.streamNameColorIn   = "colorIn"
        self.streamNameLeftIn    = "leftIn"
        self.streamNameRightIn   = "rightIn"
        self.streamNameDisparity = "disparity"
        self.streamNameDepth     = "depth"
        self.streamNameScript    = "script"
        self.streamStatus        = "status"

        self.nameAccumulationScript = "Stream Accumulation Valve"
        self.namePassScript         = "Stream Pass Valve"

        self.COLOR_CAMERA_SOCKET = dai.CameraBoardSocket.RGB
        self.LEFT_CAMERA_SOCKET  = dai.CameraBoardSocket.LEFT
        self.RIGHT_CAMERA_SOCKET = dai.CameraBoardSocket.RIGHT

        # In miliseconds
        self.syncThreshold = 1000 / self.FPS

        self.pipeline = dai.Pipeline()

        self.calibration = self.pipeline.getCalibrationData()

        self.pipeline = None

        self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_4_K
        self.MONO_RESOLUTION  = dai.MonoCameraProperties.SensorResolution.THE_480_P

        # 4K
        self.imageWidth  = 3840
        self.imageHeight = 2160


    def startCapture(self):
        self._initPipeline()

        self._setProperties()

        self._linkPipeline()

        self._initQueues()


    def stopCapture(self):
        if self.device:
            self.device.close()

        self.device   = None
        self.pipeline = None


    def triggerFramePass(self):
        if self.scriptQueue:
            print(f"Trigger at: {dai.Clock.now().total_seconds()}")

            buf = dai.Buffer()
            buf.setData(True)

            self.scriptQueue.send(buf)
        else:
            print("Pipeline not inited")


    def getBuffers(self):
        colorBuffer     = None
        depthBuffer     = None

        EOS = False

        while True:
            statusData = self.statusQueue.tryGet()

            if statusData:
                statusText = str(statusData.getData(), 'utf-8')
                data = json.loads(statusText)

                if data["status"] == "invalid":
                    EOS = True
                    break

                if data["status"] == "valid":
                    colorBuffer = self.colorQueue.get()
                    depthBuffer = self.depthQueue.get()
                    break

        return (
            EOS,
            colorBuffer,
            depthBuffer
        )


    def _initPipeline(self):
        self.pipeline = dai.Pipeline()

        # Improves latency: https://discuss.luxonis.com/d/731-how-to-increase-oakd-lite-s-stereo-pair-frame-rate
        self.pipeline.setXLinkChunkSize(0)

        self.pipeline.setCalibrationData(self.calibration)

        self.color     = self.pipeline.create(dai.node.ColorCamera)
        self.xoutColor = self.pipeline.createXLinkOut()
        self.xoutColor.setStreamName(self.streamNameColor)



        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)


        self.stereoDepth = self.pipeline.create(dai.node.StereoDepth)
        self.xoutDepth   = self.pipeline.create(dai.node.XLinkOut)
        self.xoutDepth.setStreamName(self.streamNameDepth)


        self.accumulationScript = self.pipeline.create(dai.node.Script)
        self.passScript         = self.pipeline.create(dai.node.Script)

        self.xinScript = self.pipeline.create(dai.node.XLinkIn)
        self.xinScript.setStreamName(self.streamNameScript)


        self.xoutStatus = self.pipeline.create(dai.node.XLinkOut)
        self.xoutStatus.setStreamName(self.streamStatus)


        self.sync  = self.pipeline.create(dai.node.Sync)
        self.demux = self.pipeline.create(dai.node.MessageDemux)


        mesh, meshWidth, meshHeight = self._getMesh(
            self.calibration,
            self.COLOR_CAMERA_SOCKET,
            (self.imageWidth, self.imageHeight)
        )

        self.manip = self.pipeline.create(dai.node.ImageManip)
        self.manip.setWarpMesh(mesh, meshWidth, meshHeight)
        self.manip.setMaxOutputFrameSize(self.imageWidth * self.imageHeight * 3)


    def _setProperties(self):
        self.color.setBoardSocket(self.COLOR_CAMERA_SOCKET)
        self.color.setResolution(self.COLOR_RESOLUTION)
        self.color.setFps(self.FPS)


        self.monoLeft.setBoardSocket(self.LEFT_CAMERA_SOCKET)
        self.monoLeft.setFps(self.FPS)
        self.monoLeft.setResolution(self.MONO_RESOLUTION)

        self.monoRight.setBoardSocket(self.RIGHT_CAMERA_SOCKET)
        self.monoRight.setFps(self.FPS)
        self.monoRight.setResolution(self.MONO_RESOLUTION)


        self.stereoDepth.setDepthAlign(self.COLOR_CAMERA_SOCKET)
        self.stereoDepth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_ACCURACY)
        self.stereoDepth.setLeftRightCheck(True)


        accumulationScript = f"""
        passed = 0

        while True:
            # Drop all color/left/right frames except the last one
            colorFrame = node.io['inColor'].get()
            leftFrame  = node.io['inLeft'].get()
            rightFrame = node.io['inRight'].get()

            trigger = node.io['inTrigger'].tryGet()

            if trigger:
                passed = {self.framesToAccumulation}

            if passed > 0:
                node.warn("Accumulation: " + str(passed))
                passed -= 1

                # Send only the last color frame
                if passed == 0:
                    node.io['outColor'].send(colorFrame)

                    monoSN = leftFrame.getSequenceNum()
                    colorFrame.setSequenceNum(monoSN)

                node.io['outLeft'].send(leftFrame)
                node.io['outRight'].send(rightFrame)

                node.warn("Mono SN: " + str(leftFrame.getSequenceNum()))
        """

        self.accumulationScript.setScript(
            script = accumulationScript,
            name   = self.nameAccumulationScript
        )

        passScript = f"""
            import time
            import json

            colorFrame = None
            depthFrame = None

            startTime   = None
            timeoutTime = 0.200 # in seconds
            timeoutPass = False

            statusBuffer = Buffer(30)

            while True:
                # Try to get color frame
                currentColorFrame = node.io['inColor'].tryGet()

                if currentColorFrame:
                    colorFrame = currentColorFrame
                    startTime = Clock.now().total_seconds()

                    colorSN = colorFrame.getSequenceNum()
                    node.warn("color: " + str(colorSN))

                if colorFrame:
                    currentTime = Clock.now().total_seconds()
                    delta = currentTime - startTime

                    if delta >= timeoutTime:
                        if depthFrame:
                            status = {{"status": "valid"}}
                            statusBuffer.setData(json.dumps(status).encode('utf-8'))

                            node.io['outColor'].send(colorFrame)
                            node.io['outDepth'].send(depthFrame)
                            node.io['outStatus'].send(statusBuffer)

                            colorFrame = None
                            depthFrame = None
                            timeoutPass = False

                            node.warn("Delta: " + str(delta))
                            node.warn("Pass (Timeout)")

                        elif timeoutPass:
                            status = {{"status": "invalid"}}
                            statusBuffer.setData(json.dumps(status).encode('utf-8'))

                            node.io['outStatus'].send(statusBuffer)

                            colorFrame = None
                            depthFrame = None
                            timeoutPass = False

                            node.warn("Delta: " + str(delta))
                            node.warn("Timeout Hard Pass")

                        else:
                            timeoutPass = True
                            node.warn("Timeout Pass ON")

                # Drop all depth frames except the matching one
                currentDepthFrame = node.io['inDepth'].tryGet()

                if currentDepthFrame:
                    depthFrame = currentDepthFrame

                    depthSN = depthFrame.getSequenceNum()
                    node.warn("depth: " + str(depthSN))

                if colorFrame and depthFrame:
                    colorSN = colorFrame.getSequenceNum()
                    depthSN = depthFrame.getSequenceNum()

                    if colorSN == depthSN or timeoutPass:
                        status = {{"status": "valid"}}
                        statusBuffer.setData(json.dumps(status).encode('utf-8'))

                        node.io['outColor'].send(colorFrame)
                        node.io['outDepth'].send(depthFrame)

                        node.io['outStatus'].send(statusBuffer)

                        colorFrame = None
                        depthFrame = None
                        timeoutPass = False

                        node.warn(f"Pass")
                time.sleep(0.005)
        """

        self.passScript.setScript(
            script = passScript,
            name   = self.namePassScript
        )

        self.sync.setSyncThreshold(timedelta(milliseconds=self.syncThreshold))
        self.sync.setSyncAttempts(-1)


    def _linkPipeline(self):
        # Link from Color/Mono components to Sync inputs
        self.color.isp.link(self.sync.inputs["colorSync"])
        self.monoLeft.out.link(self.sync.inputs["leftSync"])
        self.monoRight.out.link(self.sync.inputs["rightSync"])

        # Link Sync output to Demux input
        self.sync.out.link(self.demux.input)

        # # Link Demux outputs to Accumulation Script inputs
        self.demux.outputs["colorSync"].link(self.accumulationScript.inputs["inColor"])
        self.demux.outputs["leftSync"].link(self.accumulationScript.inputs["inLeft"])
        self.demux.outputs["rightSync"].link(self.accumulationScript.inputs["inRight"])

        # Link Accumulation Script trigger input
        self.xinScript.out.link(self.accumulationScript.inputs['inTrigger'])

        # Link Accumulation Script outputs based on configuration
        self.accumulationScript.outputs["outColor"].link(self.manip.inputImage)
        self.accumulationScript.outputs["outLeft"].link(self.stereoDepth.left)
        self.accumulationScript.outputs["outRight"].link(self.stereoDepth.right)

        # Link StereoDepth output to Pass Script input
        self.stereoDepth.depth.link(self.passScript.inputs["inDepth"])

        # Link Undistortion output to Pass Script input if in use
        self.manip.out.link(self.passScript.inputs["inColor"])

        # Link Pass Script outputs based on configuration
        self.passScript.outputs["outColor"].link(self.xoutColor.input)
        self.passScript.outputs["outDepth"].link(self.xoutDepth.input)

        self.passScript.outputs["outStatus"].link(self.xoutStatus.input)


    def _initQueues(self):
        self.device = dai.Device(pipeline=self.pipeline)

        self.device.setTimesync(timedelta(seconds=5), 10, True)

        self.device.setLogLevel(dai.LogLevel.DEBUG)
        self.device.setLogOutputLevel(dai.LogLevel.DEBUG)

        outputFrames = 1

        self.colorQueue = self.device.getOutputQueue(name=self.streamNameColor, maxSize=outputFrames, blocking=True)
        self.depthQueue = self.device.getOutputQueue(name=self.streamNameDepth, maxSize=outputFrames, blocking=False)

        self.scriptQueue = self.device.getInputQueue(name=self.streamNameScript, maxSize=outputFrames, blocking=True)
        self.statusQueue = self.device.getOutputQueue(name=self.streamStatus,    maxSize=outputFrames, blocking=False)


    def _getMesh(
        self,
        calibrationData : dai.CalibrationHandler,
        cameraSocket : dai.CameraBoardSocket,
        ispSize : Tuple[int, int]
    ):
        M1 = np.array(calibrationData.getCameraIntrinsics(cameraSocket, ispSize[0], ispSize[1]))
        d1 = np.array(calibrationData.getDistortionCoefficients(cameraSocket))
        R1 = np.identity(3)

        mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

        meshCellSize = 16
        mesh0 = []
        # Creates subsampled mesh which will be loaded on to device to undistort the image
        for y in range(mapX.shape[0] + 1): # iterating over height of the image
            if y % meshCellSize == 0:
                rowLeft = []
                for x in range(mapX.shape[1]): # iterating over width of the image
                    if x % meshCellSize == 0:
                        if y == mapX.shape[0] and x == mapX.shape[1]:
                            rowLeft.append(mapX[y - 1, x - 1])
                            rowLeft.append(mapY[y - 1, x - 1])
                        elif y == mapX.shape[0]:
                            rowLeft.append(mapX[y - 1, x])
                            rowLeft.append(mapY[y - 1, x])
                        elif x == mapX.shape[1]:
                            rowLeft.append(mapX[y, x - 1])
                            rowLeft.append(mapY[y, x - 1])
                        else:
                            rowLeft.append(mapX[y, x])
                            rowLeft.append(mapY[y, x])
                if (mapX.shape[1] % meshCellSize) % 2 != 0:
                    rowLeft.append(0)
                    rowLeft.append(0)

                mesh0.append(rowLeft)

        mesh0 = np.array(mesh0)
        meshWidth = mesh0.shape[1] // 2
        meshHeight = mesh0.shape[0]
        mesh0.resize(meshWidth * meshHeight, 2)

        mesh = list(map(tuple, mesh0))

        return mesh, meshWidth, meshHeight

if __name__ == "__main__":
    camera = CameraInterface()

    windowSize = (1280, 980)

    cv2.namedWindow("color", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("color", windowSize)

    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("depth", windowSize)

    exitNow = False

    for idxOut in range(37):
        camera.startCapture()

        delay = 0.000

        if idxOut % 5 == 0:
            delay = random.uniform(0.0, 3.0)

        for idxIn in range(1337):
            print(f"============== Run: {idxOut}:{idxIn} | Delay: {delay:.3} ==============")
            time.sleep(delay)

            camera.triggerFramePass()
            EOS, colorBuffer, depthBuffer = camera.getBuffers()

            if EOS:
                print("Hard Timeout EOS!")
                break

            colorFrame = colorBuffer.getCvFrame()
            depthFrame = depthBuffer.getCvFrame()

            depthFrame = (depthFrame / 10).astype(np.uint8)

            cv2.waitKey(1)
            cv2.imshow("color", colorFrame)
            cv2.imshow("depth", depthFrame)

        camera.stopCapture()

jakaskerl

Hi @vkmNuller
Script doesn't start.

self.calibration = self.pipeline.getCalibrationData()

vkmNuller

Hi @jakaskerl
Sorry for the incorrect code; I was probably testing something.

Anyway, here is the correct version:

import json
import time
import random

import cv2
import numpy as np

import depthai as dai

from typing import Tuple
from datetime import timedelta


class CameraInterface:
    def __init__(
        self
    ):
        self.framesToAccumulation = 5
        self.FPS                  = 30

        self.streamNameColor     = "color"
        self.streamNameLeft      = "left"
        self.streamNameRight     = "right"
        self.streamNameColorIn   = "colorIn"
        self.streamNameLeftIn    = "leftIn"
        self.streamNameRightIn   = "rightIn"
        self.streamNameDisparity = "disparity"
        self.streamNameDepth     = "depth"
        self.streamNameScript    = "script"
        self.streamStatus        = "status"

        self.nameAccumulationScript = "Stream Accumulation Valve"
        self.namePassScript         = "Stream Pass Valve"

        self.COLOR_CAMERA_SOCKET = dai.CameraBoardSocket.RGB
        self.LEFT_CAMERA_SOCKET  = dai.CameraBoardSocket.LEFT
        self.RIGHT_CAMERA_SOCKET = dai.CameraBoardSocket.RIGHT

        # In miliseconds
        self.syncThreshold = 1000 / self.FPS

        self.device = dai.Device()

        self.device.setTimesync(timedelta(seconds=5), 10, True)

        self.device.setLogLevel(dai.LogLevel.DEBUG)
        self.device.setLogOutputLevel(dai.LogLevel.DEBUG)

        self.calibration = self.device.readCalibration()

        self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_4_K
        self.MONO_RESOLUTION  = dai.MonoCameraProperties.SensorResolution.THE_480_P

        # 4K
        self.imageWidth  = 3840
        self.imageHeight = 2160


    def startCapture(self):
        self._initPipeline()

        self._setProperties()

        self._linkPipeline()

        self.device.startPipeline(self.pipeline)

        self._initQueues()


    def stopCapture(self):
        if self.device:
            self.device.close()

        self.device   = None
        self.pipeline = None


    def triggerFramePass(self):
        if self.scriptQueue:
            print(f"Trigger at: {dai.Clock.now().total_seconds()}")

            buf = dai.Buffer()
            buf.setData(True)

            self.scriptQueue.send(buf)
        else:
            print("Pipeline not inited")


    def getBuffers(self):
        colorBuffer     = None
        depthBuffer     = None

        EOS = False

        while True:
            statusData = self.statusQueue.tryGet()

            if statusData:
                statusText = str(statusData.getData(), 'utf-8')
                data = json.loads(statusText)

                if data["status"] == "invalid":
                    EOS = True
                    break

                if data["status"] == "valid":
                    colorBuffer = self.colorQueue.get()
                    depthBuffer = self.depthQueue.get()
                    break

        return (
            EOS,
            colorBuffer,
            depthBuffer
        )


    def _initPipeline(self):
        self.pipeline = dai.Pipeline()

        # Improves latency: https://discuss.luxonis.com/d/731-how-to-increase-oakd-lite-s-stereo-pair-frame-rate
        self.pipeline.setXLinkChunkSize(0)

        self.pipeline.setCalibrationData(self.calibration)

        self.color     = self.pipeline.create(dai.node.ColorCamera)
        self.xoutColor = self.pipeline.createXLinkOut()
        self.xoutColor.setStreamName(self.streamNameColor)



        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)


        self.stereoDepth = self.pipeline.create(dai.node.StereoDepth)
        self.xoutDepth   = self.pipeline.create(dai.node.XLinkOut)
        self.xoutDepth.setStreamName(self.streamNameDepth)


        self.accumulationScript = self.pipeline.create(dai.node.Script)
        self.passScript         = self.pipeline.create(dai.node.Script)

        self.xinScript = self.pipeline.create(dai.node.XLinkIn)
        self.xinScript.setStreamName(self.streamNameScript)


        self.xoutStatus = self.pipeline.create(dai.node.XLinkOut)
        self.xoutStatus.setStreamName(self.streamStatus)


        self.sync  = self.pipeline.create(dai.node.Sync)
        self.demux = self.pipeline.create(dai.node.MessageDemux)


        mesh, meshWidth, meshHeight = self._getMesh(
            self.calibration,
            self.COLOR_CAMERA_SOCKET,
            (self.imageWidth, self.imageHeight)
        )

        self.manip = self.pipeline.create(dai.node.ImageManip)
        self.manip.setWarpMesh(mesh, meshWidth, meshHeight)
        self.manip.setMaxOutputFrameSize(self.imageWidth * self.imageHeight * 3)


    def _setProperties(self):
        self.color.setBoardSocket(self.COLOR_CAMERA_SOCKET)
        self.color.setResolution(self.COLOR_RESOLUTION)
        self.color.setFps(self.FPS)


        self.monoLeft.setBoardSocket(self.LEFT_CAMERA_SOCKET)
        self.monoLeft.setFps(self.FPS)
        self.monoLeft.setResolution(self.MONO_RESOLUTION)

        self.monoRight.setBoardSocket(self.RIGHT_CAMERA_SOCKET)
        self.monoRight.setFps(self.FPS)
        self.monoRight.setResolution(self.MONO_RESOLUTION)


        self.stereoDepth.setDepthAlign(self.COLOR_CAMERA_SOCKET)
        self.stereoDepth.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_ACCURACY)
        self.stereoDepth.setLeftRightCheck(True)


        accumulationScript = f"""
        passed = 0

        while True:
            # Drop all color/left/right frames except the last one
            colorFrame = node.io['inColor'].get()
            leftFrame  = node.io['inLeft'].get()
            rightFrame = node.io['inRight'].get()

            trigger = node.io['inTrigger'].tryGet()

            if trigger:
                passed = {self.framesToAccumulation}

            if passed > 0:
                node.warn("Accumulation: " + str(passed))
                passed -= 1

                # Send only the last color frame
                if passed == 0:
                    node.io['outColor'].send(colorFrame)

                    monoSN = leftFrame.getSequenceNum()
                    colorFrame.setSequenceNum(monoSN)

                node.io['outLeft'].send(leftFrame)
                node.io['outRight'].send(rightFrame)

                node.warn("Mono SN: " + str(leftFrame.getSequenceNum()))
        """

        self.accumulationScript.setScript(
            script = accumulationScript,
            name   = self.nameAccumulationScript
        )

        passScript = f"""
            import time
            import json

            colorFrame = None
            depthFrame = None

            startTime   = None
            timeoutTime = 0.200 # in seconds
            timeoutPass = False

            statusBuffer = Buffer(30)

            while True:
                # Try to get color frame
                currentColorFrame = node.io['inColor'].tryGet()

                if currentColorFrame:
                    colorFrame = currentColorFrame
                    startTime = Clock.now().total_seconds()

                    colorSN = colorFrame.getSequenceNum()
                    node.warn("color: " + str(colorSN))

                if colorFrame:
                    currentTime = Clock.now().total_seconds()
                    delta = currentTime - startTime

                    if delta >= timeoutTime:
                        if depthFrame:
                            status = {{"status": "valid"}}
                            statusBuffer.setData(json.dumps(status).encode('utf-8'))

                            node.io['outColor'].send(colorFrame)
                            node.io['outDepth'].send(depthFrame)
                            node.io['outStatus'].send(statusBuffer)

                            colorFrame = None
                            depthFrame = None
                            timeoutPass = False

                            node.warn("Delta: " + str(delta))
                            node.warn("Pass (Timeout)")

                        elif timeoutPass:
                            status = {{"status": "invalid"}}
                            statusBuffer.setData(json.dumps(status).encode('utf-8'))

                            node.io['outStatus'].send(statusBuffer)

                            colorFrame = None
                            depthFrame = None
                            timeoutPass = False

                            node.warn("Delta: " + str(delta))
                            node.warn("Timeout Hard Pass")

                        else:
                            timeoutPass = True
                            node.warn("Timeout Pass ON")

                # Drop all depth frames except the matching one
                currentDepthFrame = node.io['inDepth'].tryGet()

                if currentDepthFrame:
                    depthFrame = currentDepthFrame

                    depthSN = depthFrame.getSequenceNum()
                    node.warn("depth: " + str(depthSN))

                if colorFrame and depthFrame:
                    colorSN = colorFrame.getSequenceNum()
                    depthSN = depthFrame.getSequenceNum()

                    if colorSN == depthSN or timeoutPass:
                        status = {{"status": "valid"}}
                        statusBuffer.setData(json.dumps(status).encode('utf-8'))

                        node.io['outColor'].send(colorFrame)
                        node.io['outDepth'].send(depthFrame)

                        node.io['outStatus'].send(statusBuffer)

                        colorFrame = None
                        depthFrame = None
                        timeoutPass = False

                        node.warn(f"Pass")
                time.sleep(0.005)
        """

        self.passScript.setScript(
            script = passScript,
            name   = self.namePassScript
        )

        self.sync.setSyncThreshold(timedelta(milliseconds=self.syncThreshold))
        self.sync.setSyncAttempts(-1)


    def _linkPipeline(self):
        # Link from Color/Mono components to Sync inputs
        self.color.isp.link(self.sync.inputs["colorSync"])
        self.monoLeft.out.link(self.sync.inputs["leftSync"])
        self.monoRight.out.link(self.sync.inputs["rightSync"])

        # Link Sync output to Demux input
        self.sync.out.link(self.demux.input)

        # # Link Demux outputs to Accumulation Script inputs
        self.demux.outputs["colorSync"].link(self.accumulationScript.inputs["inColor"])
        self.demux.outputs["leftSync"].link(self.accumulationScript.inputs["inLeft"])
        self.demux.outputs["rightSync"].link(self.accumulationScript.inputs["inRight"])

        # Link Accumulation Script trigger input
        self.xinScript.out.link(self.accumulationScript.inputs['inTrigger'])

        # Link Accumulation Script outputs based on configuration
        self.accumulationScript.outputs["outColor"].link(self.manip.inputImage)
        self.accumulationScript.outputs["outLeft"].link(self.stereoDepth.left)
        self.accumulationScript.outputs["outRight"].link(self.stereoDepth.right)

        # Link StereoDepth output to Pass Script input
        self.stereoDepth.depth.link(self.passScript.inputs["inDepth"])

        # Link Undistortion output to Pass Script input if in use
        self.manip.out.link(self.passScript.inputs["inColor"])

        # Link Pass Script outputs based on configuration
        self.passScript.outputs["outColor"].link(self.xoutColor.input)
        self.passScript.outputs["outDepth"].link(self.xoutDepth.input)

        self.passScript.outputs["outStatus"].link(self.xoutStatus.input)


    def _initQueues(self):
        outputFrames = 1

        self.colorQueue = self.device.getOutputQueue(name=self.streamNameColor, maxSize=outputFrames, blocking=True)
        self.depthQueue = self.device.getOutputQueue(name=self.streamNameDepth, maxSize=outputFrames, blocking=False)

        self.scriptQueue = self.device.getInputQueue(name=self.streamNameScript, maxSize=outputFrames, blocking=True)
        self.statusQueue = self.device.getOutputQueue(name=self.streamStatus,    maxSize=outputFrames, blocking=False)


    def _getMesh(
        self,
        calibrationData : dai.CalibrationHandler,
        cameraSocket : dai.CameraBoardSocket,
        ispSize : Tuple[int, int]
    ):
        M1 = np.array(calibrationData.getCameraIntrinsics(cameraSocket, ispSize[0], ispSize[1]))
        d1 = np.array(calibrationData.getDistortionCoefficients(cameraSocket))
        R1 = np.identity(3)

        mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

        meshCellSize = 16
        mesh0 = []
        # Creates subsampled mesh which will be loaded on to device to undistort the image
        for y in range(mapX.shape[0] + 1): # iterating over height of the image
            if y % meshCellSize == 0:
                rowLeft = []
                for x in range(mapX.shape[1]): # iterating over width of the image
                    if x % meshCellSize == 0:
                        if y == mapX.shape[0] and x == mapX.shape[1]:
                            rowLeft.append(mapX[y - 1, x - 1])
                            rowLeft.append(mapY[y - 1, x - 1])
                        elif y == mapX.shape[0]:
                            rowLeft.append(mapX[y - 1, x])
                            rowLeft.append(mapY[y - 1, x])
                        elif x == mapX.shape[1]:
                            rowLeft.append(mapX[y, x - 1])
                            rowLeft.append(mapY[y, x - 1])
                        else:
                            rowLeft.append(mapX[y, x])
                            rowLeft.append(mapY[y, x])
                if (mapX.shape[1] % meshCellSize) % 2 != 0:
                    rowLeft.append(0)
                    rowLeft.append(0)

                mesh0.append(rowLeft)

        mesh0 = np.array(mesh0)
        meshWidth = mesh0.shape[1] // 2
        meshHeight = mesh0.shape[0]
        mesh0.resize(meshWidth * meshHeight, 2)

        mesh = list(map(tuple, mesh0))

        return mesh, meshWidth, meshHeight

if __name__ == "__main__":
    camera = CameraInterface()

    windowSize = (1280, 980)

    cv2.namedWindow("color", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("color", windowSize)

    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("depth", windowSize)

    exitNow = False

    for idxOut in range(37):
        camera.startCapture()

        delay = 0.000

        if idxOut % 5 == 0:
            delay = random.uniform(0.0, 3.0)

        for idxIn in range(1337):
            print(f"============== Run: {idxOut}:{idxIn} | Delay: {delay:.3} ==============")
            time.sleep(delay)

            camera.triggerFramePass()
            EOS, colorBuffer, depthBuffer = camera.getBuffers()

            if EOS:
                print("Hard Timeout EOS!")
                break

            colorFrame = colorBuffer.getCvFrame()
            depthFrame = depthBuffer.getCvFrame()

            depthFrame = (depthFrame / 10).astype(np.uint8)

            cv2.waitKey(1)
            cv2.imshow("color", colorFrame)
            cv2.imshow("depth", depthFrame)

        camera.stopCapture()

jakaskerl

The script you have sent is only able to get about 0.5 FPS. Was not able to get more than that even by changing the FPS or accumulation value.
It runs fine though albeit slow.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl
The question I was asking is whether setting the FPS to 40 would cause the pipeline to freeze for you, and if so, why. Because when I run it at 40 FPS, the pipeline processes frames successfully the first time, but on the second iteration, it freezes.

NOTE: If you want to speed it up, you can comment/remove the sleep function in the main.

Thanks,
vkm

jakaskerl

Hi @vkmNuller
This is taking me too long to debug because the code has too many parts (syncing/triggering,..) which could all be the culprit. I see the ColorCamera stops producing frames, likely because the queues are full - first run always makes it through. Can't say for sure because of the many components in the script doing things at the same time.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl
It sounds frustrating. Let me know if I can help you with something to make it easier for you.

jakaskerl

Hi @vkmNuller
Make the code shorter and remove the parts that are unnecessary on this issue. https://stackoverflow.com/help/minimal-reproducible-example.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl

I managed to reproduce the issue after removing sync, demux, ImageManip and the second script nodes, it crashes when the FPS exceeds 20.

import json
import time
import random

import cv2
import numpy as np

import depthai as dai

from typing import Tuple
from datetime import timedelta


class CameraInterface:
    def __init__(
        self
    ):
        self.framesToAccumulation = 5
        self.FPS                  = 60

        self.streamNameColor     = "color"
        self.streamNameDepth     = "depth"
        self.streamNameScript    = "script"

        self.nameAccumulationScript = "Stream Accumulation Valve"
        self.namePassScript         = "Stream Pass Valve"

        self.COLOR_CAMERA_SOCKET = dai.CameraBoardSocket.RGB
        self.LEFT_CAMERA_SOCKET  = dai.CameraBoardSocket.LEFT
        self.RIGHT_CAMERA_SOCKET = dai.CameraBoardSocket.RIGHT

        self.device = dai.Device()

        self.device.setLogLevel(dai.LogLevel.DEBUG)
        self.device.setLogOutputLevel(dai.LogLevel.DEBUG)

        self.calibration = self.device.readCalibration()

        self.COLOR_RESOLUTION = dai.ColorCameraProperties.SensorResolution.THE_4_K
        self.MONO_RESOLUTION  = dai.MonoCameraProperties.SensorResolution.THE_480_P

        # 4K
        self.imageWidth  = 3840
        self.imageHeight = 2160


    def startCapture(self):
        self._initPipeline()

        self._setProperties()

        self._linkPipeline()

        self.device.startPipeline(self.pipeline)

        self._initQueues()


    def stopCapture(self):
        if self.device:
            self.device.close()

        self.device   = None
        self.pipeline = None


    def triggerFramePass(self):
        if self.scriptQueue:
            print(f"Trigger at: {dai.Clock.now().total_seconds()}")

            buf = dai.Buffer()
            buf.setData(True)

            self.scriptQueue.send(buf)
        else:
            print("Pipeline not inited")


    def getBuffers(self):
        colorBuffer = self.colorQueue.get()
        depthBuffer = self.depthQueue.get()

        return (
            colorBuffer,
            depthBuffer
        )


    def _initPipeline(self):
        self.pipeline = dai.Pipeline()

        # Improves latency: https://discuss.luxonis.com/d/731-how-to-increase-oakd-lite-s-stereo-pair-frame-rate
        self.pipeline.setXLinkChunkSize(0)

        self.pipeline.setCalibrationData(self.calibration)

        self.color     = self.pipeline.create(dai.node.ColorCamera)
        self.xoutColor = self.pipeline.createXLinkOut()
        self.xoutColor.setStreamName(self.streamNameColor)


        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)


        self.stereoDepth = self.pipeline.create(dai.node.StereoDepth)
        self.xoutDepth   = self.pipeline.create(dai.node.XLinkOut)
        self.xoutDepth.setStreamName(self.streamNameDepth)


        self.accumulationScript = self.pipeline.create(dai.node.Script)

        self.xinScript = self.pipeline.create(dai.node.XLinkIn)
        self.xinScript.setStreamName(self.streamNameScript)


    def _setProperties(self):
        self.color.setBoardSocket(self.COLOR_CAMERA_SOCKET)
        self.color.setResolution(self.COLOR_RESOLUTION)
        self.color.setFps(self.FPS)

        self.monoLeft.setBoardSocket(self.LEFT_CAMERA_SOCKET)
        self.monoLeft.setFps(self.FPS)
        self.monoLeft.setResolution(self.MONO_RESOLUTION)

        self.monoRight.setBoardSocket(self.RIGHT_CAMERA_SOCKET)
        self.monoRight.setFps(self.FPS)
        self.monoRight.setResolution(self.MONO_RESOLUTION)

        self.stereoDepth.setDepthAlign(self.COLOR_CAMERA_SOCKET)

        accumulationScript = f"""
        passed = 0

        while True:
            # Drop all color/left/right frames except the last one
            colorFrame = node.io['inColor'].get()
            leftFrame  = node.io['inLeft'].get()
            rightFrame = node.io['inRight'].get()

            trigger = node.io['inTrigger'].tryGet()

            if trigger:
                passed = {self.framesToAccumulation}

            if passed > 0:
                node.warn("Accumulation: " + str(passed))
                passed -= 1

                # Send only the last color frame
                if passed == 0:
                    node.io['outColor'].send(colorFrame)

                    monoSN = leftFrame.getSequenceNum()
                    colorFrame.setSequenceNum(monoSN)

                node.io['outLeft'].send(leftFrame)
                node.io['outRight'].send(rightFrame)

                node.warn("Mono SN: " + str(leftFrame.getSequenceNum()))
        """

        self.accumulationScript.setScript(
            script = accumulationScript,
            name   = self.nameAccumulationScript
        )


    def _linkPipeline(self):
        # Link from Color/Mono components
        self.color.isp.link(self.accumulationScript.inputs["inColor"])
        self.monoLeft.out.link(self.accumulationScript.inputs["inLeft"])
        self.monoRight.out.link(self.accumulationScript.inputs["inRight"])

        # Link Accumulation Script trigger input
        self.xinScript.out.link(self.accumulationScript.inputs['inTrigger'])

        # Link Accumulation Script outputs based on configuration
        self.accumulationScript.outputs["outColor"].link(self.xoutColor.input)
        self.accumulationScript.outputs["outLeft"].link(self.stereoDepth.left)
        self.accumulationScript.outputs["outRight"].link(self.stereoDepth.right)

        # Link StereoDepth output to Pass Script input
        self.stereoDepth.depth.link(self.xoutDepth.input)


    def _initQueues(self):
        outputFrames = 1

        self.colorQueue = self.device.getOutputQueue(name=self.streamNameColor, maxSize=outputFrames, blocking=True)
        self.depthQueue = self.device.getOutputQueue(name=self.streamNameDepth, maxSize=outputFrames, blocking=False)

        self.scriptQueue = self.device.getInputQueue(name=self.streamNameScript, maxSize=outputFrames, blocking=True)


    def _getMesh(
        self,
        calibrationData : dai.CalibrationHandler,
        cameraSocket : dai.CameraBoardSocket,
        ispSize : Tuple[int, int]
    ):
        M1 = np.array(calibrationData.getCameraIntrinsics(cameraSocket, ispSize[0], ispSize[1]))
        d1 = np.array(calibrationData.getDistortionCoefficients(cameraSocket))
        R1 = np.identity(3)

        mapX, mapY = cv2.initUndistortRectifyMap(M1, d1, R1, M1, ispSize, cv2.CV_32FC1)

        meshCellSize = 16
        mesh0 = []
        # Creates subsampled mesh which will be loaded on to device to undistort the image
        for y in range(mapX.shape[0] + 1): # iterating over height of the image
            if y % meshCellSize == 0:
                rowLeft = []
                for x in range(mapX.shape[1]): # iterating over width of the image
                    if x % meshCellSize == 0:
                        if y == mapX.shape[0] and x == mapX.shape[1]:
                            rowLeft.append(mapX[y - 1, x - 1])
                            rowLeft.append(mapY[y - 1, x - 1])
                        elif y == mapX.shape[0]:
                            rowLeft.append(mapX[y - 1, x])
                            rowLeft.append(mapY[y - 1, x])
                        elif x == mapX.shape[1]:
                            rowLeft.append(mapX[y, x - 1])
                            rowLeft.append(mapY[y, x - 1])
                        else:
                            rowLeft.append(mapX[y, x])
                            rowLeft.append(mapY[y, x])
                if (mapX.shape[1] % meshCellSize) % 2 != 0:
                    rowLeft.append(0)
                    rowLeft.append(0)

                mesh0.append(rowLeft)

        mesh0 = np.array(mesh0)
        meshWidth = mesh0.shape[1] // 2
        meshHeight = mesh0.shape[0]
        mesh0.resize(meshWidth * meshHeight, 2)

        mesh = list(map(tuple, mesh0))

        return mesh, meshWidth, meshHeight

if __name__ == "__main__":
    camera = CameraInterface()

    windowSize = (1280, 980)

    cv2.namedWindow("color", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("color", windowSize)

    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    cv2.resizeWindow("depth", windowSize)

    # Test for stability
    for idxOut in range(37):
        camera.startCapture()

        delay = 0.000

        if idxOut % 5 == 0:
            delay = random.uniform(0.0, 3.0)

        for idxIn in range(1337):
            print(f"============== Run: {idxOut}:{idxIn} | Delay: {delay:.3} ==============")
            # time.sleep(delay)

            camera.triggerFramePass()
            colorBuffer, depthBuffer = camera.getBuffers()

            colorFrame = colorBuffer.getCvFrame()
            depthFrame = depthBuffer.getCvFrame()

            depthFrame = (depthFrame / 10).astype(np.uint8)

            cv2.waitKey(1)
            cv2.imshow("color", colorFrame)
            cv2.imshow("depth", depthFrame)

        camera.stopCapture()

jakaskerl

Hi @vkmNuller
Better. I think that by accumulating the frames, the amount of pointers surpasses the allocated pool number.
Try setting (experiment with different values)

self.color     = self.pipeline.create(dai.node.ColorCamera)
        self.color.setNumFramesPool(10, 10, 0, 0, 0)
        self.xoutColor = self.pipeline.createXLinkOut()
        self.xoutColor.setStreamName(self.streamNameColor)


        self.monoLeft  = self.pipeline.create(dai.node.MonoCamera)
        self.monoLeft.setNumFramesPool(20)
        self.monoRight = self.pipeline.create(dai.node.MonoCamera)
        self.monoRight.setNumFramesPool(20)

Was able to run constantly at 60FPS/5accumulation without crashing.

Thanks,
Jaka

vkmNuller

Hi @jakaskerl,

Thank you for the quick response. This looks to be the solution.

I have some questions regarding the setNumFramesPool:

I tried to find in the C++ source how exactly the FramesPool works but only found the default values:
luxonis/depthai-sharedblob/9762f334f39fc841a5bbea9b1114bbac933710e0/include/depthai-shared/properties/ColorCameraProperties.hpp

    int numFramesPoolRaw = 3;
    int numFramesPoolIsp = 3;
    int numFramesPoolVideo = 4;
    int numFramesPoolPreview = 4;
    int numFramesPoolStill = 4;

Can you direct me to some documentation describing this functionality?

I saw this conversation:
https://discuss.luxonis.com/d/1477-high-latency-for-two-stereo-pairs-on-oak-ffc-4p/3
If I understand correctly, some of the problems mentioned in the conversation are related to latency because of the pools.
Can we configure it to start dropping frames if the pool gets full, and in what order will the frames be dropped (FIFO/LIFO)?

Thank you!

jakaskerl

vkmNuller Can you direct me to some documentation describing this functionality?

I'll document it probably this week if I have some spare time.

vkmNuller I saw this conversation:
https://discuss.luxonis.com/d/1477-high-latency-for-two-stereo-pairs-on-oak-ffc-4p/3
If I understand correctly, some of the problems mentioned in the conversation are related to latency because of the pools.

Pools are set too low, so the frames have to wait until the whole processing completes and the pools become available.

vkmNuller Can we configure it to start dropping frames if the pool gets full, and in what order will the frames be dropped (FIFO/LIFO)?

If queues are non-blocking, then frames will be dropped (depending on the queue sized) and pool will become available.

Thanks,
Jaka

vkmNuller

jakaskerl If queues are non-blocking, then frames will be dropped

How can I set the pools to be non-blocking?

In the conversation, I saw this:

depth_cam.left.setBlocking(False)
depth_cam.right.setBlocking(False)

If I can make the camera's pools/queues non-blocking, it might fix some problems I'm facing right now:

Currently, the pools are too big and take up a lot of space (even more if using 12MP).
The program crashes at random points on the host side (I have no idea why, or if it's related to this change).

Thanks

jakaskerl

Hi @vkmNuller
Every queue input has option to make it non blocking.
eg: xout_rgb.input.setBlocking(False).
Make sure the pool sizes are larger than queues, otherwise the messages in queue will not be dropped, but new messages can not be created because the pool is empty.

Thanks,
Jaka