Script behaviour on 2 cameras

Mmartin181818 · Mar 25, 2024

Dear community ,

Usually when I test a script on OAK 1 PoE and OAK 1 Lite I get the same results and the same performance.
Now, I am using a complicated script using an object detection model , an object tracker node and also a segmentation neural network activated via dai.Buffer() and script node.
The script works fine without any issues when I run it on a OAK 1 PoE. but when I try it on a OAK 1 Lite with USB , it doesn't work and the following error is displayed eventhough the 2 cameras have the same processor :

[1844301011186D0E00] [1.3] [2.193] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

F: [global] [ 0] [ThreadN] dispatcherResponseServe:928 no request for this response: XLINK_WRITE_RESP 1

What is the reason of this error since the 2 cams have the same processor ?
Thank you in advance

jakaskerl · Mar 26, 2024

Hi @martin181818
Did you account for the sensor change from LITE to standard POE? Did you set the correct script node CPU (script.setProcessor(dai.ProcessorType.LEON_CSS))?

Run the USB device script with DEPTHAI_DEBUG=1 python script.py and post the terminal output.

Thanks,
Jaka

Mmartin181818 · Mar 26, 2024

jakaskerl

Thank you for your reply.
Yes am setting it to LEON_CSS.

This is the output of the debug:

[2024-03-26 09:32:42.989] [depthai] [debug] Python bindings - version: 2.25.0.0 from 2024-03-08 20:02:05 +0100 build: 2024-03-09 11:08:16 +0000

[2024-03-26 09:32:42.990] [depthai] [debug] Library information - version: 2.25.0, commit: eba7400699e69c81f04dc0dc1e2c2b6aafc3b962 from 2024-03-08 19:52:47 +0100, build: 2024-03-09 11:08:06 +0000, libusb enabled: true

[2024-03-26 09:32:42.996] [depthai] [debug] Initialize - finished

[2024-03-26 09:32:43.062] [depthai] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.26.tar.xz' open: 2ms, archive read: 68ms

C:\Users\z4.py:208: DeprecationWarning: Use constructor taking 'UsbSpeed' instead

with dai.Device(pipeline, device_info) as device:

[2024-03-26 09:32:43.215] [host] [debug] Found an actual device by given DeviceInfo: DeviceInfo(name=1.3, mxid=1844301011186D0E00, X_LINK_UNBOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS)

[2024-03-26 09:32:43.216] [host] [debug] Device - OpenVINO version: 2022.1

[1844301011186D0E00] [1.3] [1711441963.216] [host] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":null,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":4,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}

libnop:

0000: b9 12 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb

0020: 00 be be be be be be be 00 bb 00 be be

[2024-03-26 09:32:43.367] [depthai] [debug] Resources - Archive 'depthai-device-fwp-7665aed8712bb941958660a0103868908726eb56.tar.xz' open: 1ms, archive read: 374ms

[2024-03-26 09:32:44.796] [depthai] [debug] Searching for booted device: DeviceInfo(name=1.3, mxid=1844301011186D0E00, X_LINK_BOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only

[1844301011186D0E00] [1.3] [0.651] [system] [info] Memory Usage - DDR: 0.12 / 333.39 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 7.50 / 82.31 MiB, LeonRT Heap: 2.89 / 40.50 MiB / NOC ddr: 25 MB/s

[1844301011186D0E00] [1.3] [0.651] [system] [info] Temperatures - Average: 31.83C, CSS: 32.49C, MSS 32.01C, UPA: 31.77C, DSS: 31.04C

[1844301011186D0E00] [1.3] [0.651] [system] [info] Cpu Usage - LeonOS 38.65%, LeonRT: 1.94%

[1844301011186D0E00] [1.3] [1711441965.472] [host] [debug] Schema dump: {"connections":[{"node1Id":8,"node1Output":"out","node1OutputGroup":"","node2Id":9,"node2Input":"in","node2InputGroup":""},{"node1Id":0,"node1Output":"video","node1OutputGroup":"","node2Id":1,"node2Input":"in","node2InputGroup":""},{"node1Id":3,"node1Output":"out","node1OutputGroup":"","node2Id":2,"node2Input":"toggle","node2InputGroup":"io"},{"node1Id":0,"node1Output":"preview","node1OutputGroup":"","node2Id":2,"node2Input":"rgb","node2InputGroup":"io"},{"node1Id":2,"node1Output":"nn","node1OutputGroup":"io","node2Id":4,"node2Input":"in","node2InputGroup":""},{"node1Id":4,"node1Output":"out","node1OutputGroup":"","node2Id":5,"node2Input":"in","node2InputGroup":""},{"node1Id":0,"node1Output":"preview","node1OutputGroup":"","node2Id":6,"node2Input":"in","node2InputGroup":""},{"node1Id":6,"node1Output":"out","node1OutputGroup":"","node2Id":7,"node2Input":"in","node2InputGroup":""},{"node1Id":6,"node1Output":"passthrough","node1OutputGroup":"","node2Id":8,"node2Input":"inputDetectionFrame","node2InputGroup":""},{"node1Id":6,"node1Output":"passthrough","node1OutputGroup":"","node2Id":8,"node2Input":"inputTrackerFrame","node2InputGroup":""},{"node1Id":6,"node1Output":"out","node1OutputGroup":"","node2Id":8,"node2Input":"inputDetections","node2InputGroup":""}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"sippBufferSize":18432,"sippDmaBufferSize":16384,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":[[["","inputConfig"],{"blocking":false,"group":"","id":1,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":6,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","still"],{"blocking":false,"group":"","id":7,"name":"still","queueSize":8,"type":0,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":2,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}],[["","video"],{"blocking":false,"group":"","id":3,"name":"video","queueSize":8,"type":0,"waitForMessage":false}],[["","isp"],{"blocking":false,"group":"","id":4,"name":"isp","queueSize":8,"type":0,"waitForMessage":false}],[["","preview"],{"blocking":false,"group":"","id":5,"name":"preview","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":8,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}]],"name":"ColorCamera","properties":[185,26,185,31,0,3,0,136,0,0,0,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,189,0,255,0,0,0,129,128,2,129,128,2,133,128,2,133,128,2,255,255,0,136,0,0,240,65,0,136,0,0,128,191,136,0,0,128,191,1,185,4,2,3,2,3,3,3,4,4,4,190]}],[1,{"id":1,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":9,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,3,114,103,98,0]}],[2,{"id":2,"ioInfo":[[["io","toggle"],{"blocking":true,"group":"io","id":10,"name":"toggle","queueSize":8,"type":3,"waitForMessage":false}],[["io","nn"],{"blocking":false,"group":"io","id":12,"name":"nn","queueSize":8,"type":0,"waitForMessage":false}],[["io","rgb"],{"blocking":true,"group":"io","id":11,"name":"rgb","queueSize":8,"type":3,"waitForMessage":false}]],"name":"Script","properties":[185,3,189,14,97,115,115,101,116,58,95,95,115,99,114,105,112,116,189,8,60,115,99,114,105,112,116,62,0]}],[3,{"id":3,"ioInfo":[[["","out"],{"blocking":false,"group":"","id":13,"name":"out","queueSize":8,"type":0,"waitForMessage":false}]],"name":"XLinkIn","properties":[185,3,189,2,105,110,130,0,0,80,0,8]}],[4,{"id":4,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":14,"name":"in","queueSize":5,"type":3,"waitForMessage":true}],[["","out"],{"blocking":false,"group":"","id":15,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","passthrough"],{"blocking":false,"group":"","id":16,"name":"passthrough","queueSize":8,"type":0,"waitForMessage":false}]],"name":"NeuralNetwork","properties":[185,5,130,154,146,102,0,189,12,97,115,115,101,116,58,95,95,98,108,111,98,8,0,0]}],[5,{"id":5,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":17,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,2,110,110,0]}],[6,{"id":6,"ioInfo":[[["","in"],{"blocking":false,"group":"","id":18,"name":"in","queueSize":5,"type":3,"waitForMessage":true}],[["","out"],{"blocking":false,"group":"","id":19,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","passthrough"],{"blocking":false,"group":"","id":20,"name":"passthrough","queueSize":8,"type":0,"waitForMessage":false}]],"name":"DetectionNetwork","properties":[185,6,130,24,242,92,0,189,12,97,115,115,101,116,58,95,95,98,108,111,98,8,2,0,185,7,0,136,154,153,153,62,4,4,186,0,187,0,136,0,0,0,63]}],[7,{"id":7,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":21,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,5,100,101,116,110,110,0]}],[8,{"id":8,"ioInfo":[[["","inputTrackerFrame"],{"blocking":false,"group":"","id":22,"name":"inputTrackerFrame","queueSize":4,"type":3,"waitForMessage":true}],[["","inputDetections"],{"blocking":false,"group":"","id":23,"name":"inputDetections","queueSize":4,"type":3,"waitForMessage":true}],[["","inputDetectionFrame"],{"blocking":false,"group":"","id":24,"name":"inputDetectionFrame","queueSize":4,"type":3,"waitForMessage":true}],[["","out"],{"blocking":false,"group":"","id":25,"name":"out","queueSize":8,"type":0,"waitForMessage":false}],[["","passthroughTrackerFrame"],{"blocking":false,"group":"","id":26,"name":"passthroughTrackerFrame","queueSize":8,"type":0,"waitForMessage":false}],[["","passthroughDetectionFrame"],{"blocking":false,"group":"","id":27,"name":"passthroughDetectionFrame","queueSize":8,"type":0,"waitForMessage":false}],[["","passthroughDetections"],{"blocking":false,"group":"","id":28,"name":"passthroughDetections","queueSize":8,"type":0,"waitForMessage":false}]],"name":"ObjectTracker","properties":[185,6,136,0,0,0,0,60,188,4,0,0,0,0,6,1,1]}],[9,{"id":9,"ioInfo":[[["","in"],{"blocking":true,"group":"","id":29,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,9,116,114,97,99,107,108,101,116,115,0]}]]}

[1844301011186D0E00] [1.3] [1711441965.474] [host] [debug] Asset map dump: {"map":{"/node/2/__script":{"alignment":64,"offset":0,"size":311},"/node/4/__blob":{"alignment":64,"offset":320,"size":6722202},"/node/6/__blob":{"alignment":64,"offset":6722560,"size":6091288}}}

[1844301011186D0E00] [1.3] [0.822] [ObjectTracker(8)] [info] Init node object tracker

[1844301011186D0E00] [1.3] [0.824] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '18432'B, DMA buffer size: '16384'B

[1844301011186D0E00] [1.3] [0.860] [system] [info] ImageManip internal buffer size '258560'B, shave buffer size '34816'B

[1844301011186D0E00] [1.3] [0.860] [system] [info] NeuralNetwork allocated resources: shaves: [0-11] cmx slices: [0-11]

ColorCamera allocated resources: no shaves; cmx slices: [12-14]

ImageManip allocated resources: shaves: [14-14] no cmx slices.

ObjectTracker allocated resources: shaves: [15-15] cmx slices: [15-15]

[1844301011186D0E00] [1.3] [1.083] [NeuralNetwork(4)] [info] Needed resources: shaves: 4, ddr: 12486144

[1844301011186D0E00] [1.3] [1.083] [NeuralNetwork(4)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance

[1844301011186D0E00] [1.3] [1.085] [DetectionNetwork(6)] [info] Needed resources: shaves: 6, ddr: 9011200

[1844301011186D0E00] [1.3] [1.099] [NeuralNetwork(4)] [warning] The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary

[1844301011186D0E00] [1.3] [2.159] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

[1844301011186D0E00] [1.3] [1711441968.984] [host] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)

[2024-03-26 09:32:48.984] [depthai] [debug] DataOutputQueue (rgb) closed

[2024-03-26 09:32:48.984] [depthai] [debug] DataOutputQueue (detnn) closed

[2024-03-26 09:32:48.984] [depthai] [debug] DataOutputQueue (nn) closed

[1844301011186D0E00] [1.3] [1711441968.984] [host] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)

[2024-03-26 09:32:48.984] [depthai] [debug] DataOutputQueue (tracklets) closed

[1844301011186D0E00] [1.3] [1711441968.984] [host] [debug] Device about to be closed...

[1844301011186D0E00] [1.3] [1711441968.990] [host] [debug] RPC error: Couldn't write data to stream: '__rpc_main' (X_LINK_ERROR)

[1844301011186D0E00] [1.3] [1711441968.991] [host] [debug] shutdown call error: Device already closed or disconnected: io error

[2024-03-26 09:32:48.991] [depthai] [debug] XLinkResetRemoteTimeout returned: X_LINK_ERROR

[1844301011186D0E00] [1.3] [1711441969.255] [host] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)

[2024-03-26 09:32:49.549] [depthai] [debug] XLinkResetRemote of linkId: (0)

[1844301011186D0E00] [1.3] [1711441969.977] [host] [debug] Getting crash dump...

[2024-03-26 09:32:49.996] [host] [debug] Device - OpenVINO version: 2022.1

[1844301011186D0E00] [1.3] [1711441969.997] [host] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":null,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":4,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}

libnop:

0000: b9 12 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb

0020: 00 be be be be be be be 00 bb 00 be be

[2024-03-26 09:32:51.191] [depthai] [debug] Searching for booted device: DeviceInfo(name=1.3, mxid=1844301011186D0E00, X_LINK_BOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only

[1844301011186D0E00] [1.3] [1711441972.019] [host] [warning] Device crashed. Crash dump saved to C:\Users\MAHDI~1.MSE\AppData\Local\Temp\1844301011186D0E00-depthai_crash_dump.json

[1844301011186D0E00] [1.3] [1711441972.586] [host] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)

[2024-03-26 09:32:52.675] [depthai] [debug] XLinkResetRemote of linkId: (1)

[1844301011186D0E00] [1.3] [1711441973.331] [host] [debug] Device closed, 4346

[2024-03-26 09:32:53.333] [depthai] [debug] DataInputQueue (in) closed

Traceback (most recent call last):

File "C:\Users\z4.py", line 227, in <module>

frame = qRgb.get().getCvFrame()

RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'rgb' (X_LINK_ERROR)'

Thank you very much

jakaskerl · Mar 26, 2024

Hi @martin181818
Hmm, how much is the code changed between the devices? I see usually the 000 trap error happens due to bad pipeline configurations.

Could you add the crash dump as well?

martin181818 [1844301011186D0E00] [1.3] [1711441972.019] [host] [warning] Device crashed. Crash dump saved to C:\Users\MAHDI₁.MSE\AppData\Local\Temp\1844301011186D0E00-depthai_crash_dump.json

Thanks,
Jaka

Mmartin181818 · Mar 26, 2024

Hi jakaskerl

The code is the same between devices and it runs 100% correct on the PoE device.

Thank you

This is content of crash_dump.json file:

jakaskerl · Mar 28, 2024

Hi @martin181818
Can you recheck if the different resolutions of camera sensors are the culprit? Try setting the resolution to something they both support then check if the issue remains.

Thanks,
Jaka