erik Here is the dump. Let me know if you need additional info. Thanks a lot for your help.

My pipeline setup this way: poe, 3 lens, 3 jpeg encoder, all 10fps, two monos 720p, rgb 1080p, jpeg to script node, pumping out urls: /mjpeg, /left, /right, all in mjpeg multipart frames.

This dump is produced today.

{"crashReports":[{"crashedThreadId":167837697,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2204218472,"trapName":"Bad trap","trapNumber":9}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2200385356,"calledTarget":2200381832,"context":"","framePointer":2211279768},{"callSite":2205098404,"calledTarget":0,"context":"Thread handler","framePointer":2211279864},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211279960},{"callSite":2205098556,"calledTarget":1881154720,"context":"","framePointer":2211280080},{"callSite":2204985004,"calledTarget":2205098536,"context":"","framePointer":2211280184},{"callSite":2205931196,"calledTarget":0,"context":"Thread handler","framePointer":2211280288},{"callSite":2204298320,"calledTarget":0,"context":"Thread handler","framePointer":2213073296},{"callSite":2204221036,"calledTarget":2204218436,"context":"","framePointer":2213073592},{"callSite":2204208956,"calledTarget":0,"context":"Thread handler","framePointer":2213073864},{"callSite":2204201828,"calledTarget":0,"context":"Thread handler","framePointer":2213074032},{"callSite":2204165556,"calledTarget":0,"context":"Thread handler","framePointer":2213074272},{"callSite":2204064028,"calledTarget":0,"context":"Thread handler","framePointer":2213074392},{"callSite":2204052024,"calledTarget":2204064100,"context":"","framePointer":2213074488},{"callSite":2204054424,"calledTarget":2204051528,"context":"","framePointer":2213074584},{"callSite":2204064028,"calledTarget":0,"context":"Thread handler","framePointer":2213074680},{"callSite":2204050908,"calledTarget":2204064100,"context":"","framePointer":2213074776},{"callSite":2200546904,"calledTarget":0,"context":"Thread handler","framePointer":2213074872},{"callSite":2200707600,"calledTarget":2200546064,"context":"","framePointer":2213074984},{"callSite":2200964480,"calledTarget":0,"context":"Thread handler","framePointer":2213075080},{"callSite":2200973128,"calledTarget":0,"context":"Thread handler","framePointer":2213075176},{"callSite":167837698,"calledTarget":1195661140,"context":"Thread exit","framePointer":0}],"instructionPointer":2200385356,"stackBottom":2212944504,"stackPointer":2211279768,"stackTop":2213075575,"threadId":167837697,"threadName":"CBTH","threadStatus":"READY"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2213731360},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2213731464},{"callSite":2201052240,"calledTarget":2205066912,"context":"","framePointer":2213731584},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2213731688},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213731784},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213731880}],"instructionPointer":2205067204,"stackBottom":2213600904,"stackPointer":2213731360,"stackTop":2213731975,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214125496},{"callSite":2205072104,"calledTarget":2205067040,"context":"","framePointer":2214125600},{"callSite":2205075072,"calledTarget":2205071996,"context":"","framePointer":2214125720},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214125824},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214125920},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214126016}],"instructionPointer":2205067204,"stackBottom":2213995040,"stackPointer":2214125496,"stackTop":2214126111,"threadId":167837699,"threadName":"TIME","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214256720},{"callSite":2205072104,"calledTarget":2205067040,"context":"","framePointer":2214256824},{"callSite":2204990304,"calledTarget":2205071996,"context":"","framePointer":2214256944},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214257048},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214257144},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214257240}],"instructionPointer":2205067204,"stackBottom":2214126264,"stackPointer":2214256720,"stackTop":2214257335,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214387944},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2214388048},{"callSite":2204647304,"calledTarget":2205066912,"context":"","framePointer":2214388168},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214388272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214388368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214388464}],"instructionPointer":2205067204,"stackBottom":2214257488,"stackPointer":2214387944,"stackTop":2214388559,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214519176},{"callSite":2203893028,"calledTarget":2203950744,"context":"","framePointer":2214519280},{"callSite":2204655088,"calledTarget":2203892796,"context":"","framePointer":2214519376},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214519496},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214519592},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214519688}],"instructionPointer":2203950260,"stackBottom":2214388712,"stackPointer":2214519176,"stackTop":2214519783,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214650400},{"callSite":2203893028,"calledTarget":2203950744,"context":"","framePointer":2214650504},{"callSite":2203951260,"calledTarget":2203892796,"context":"","framePointer":2214650600},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214650720},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214650816},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214650912}],"instructionPointer":2203950260,"stackBottom":2214519936,"stackPointer":2214650400,"stackTop":2214651007,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214781616},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2214781720},{"callSite":2204647304,"calledTarget":2205066912,"context":"","framePointer":2214781840},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214781944},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214782040},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214782136}],"instructionPointer":2205067204,"stackBottom":2214651160,"stackPointer":2214781616,"stackTop":2214782231,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214912848},{"callSite":2203893028,"calledTarget":2203950
"crash_dump.json" [Incomplete last line] 1 line, 94228 bytes

  • erik replied to this.

      jkchen46033 could you please upload the json to gdrive and send the link? as your reply is incomplete

          Hi erik

          In the original post I already included our camera pipeline. Can you use that as an MRE?

          Our robots run on Ubuntu 22.04 with Ros2 Humble or Ubuntu 24.04 with Ros2 Jazzy. We use mostly Oak-D Pro USB cameras (https://shop.luxonis.com/products/oak-d-pro?variant=42455252369631), but still have some older Oak-D PoE cameras. Our depthai version is 2.27.0.0. The cameras are connected with provided USB3 cables. We also tested with the Oak-D Y adapter to make sure it's not a power problem.

          We have been looking at the telemetrics of our robots. Our robots drive back and forth across a field. At the end of the field a robot stops, makes a u-turn, then starts driving back again. It seems that the cameras fail most of the times when a robot makes a u-turn or when it starts driving back again. So it might have to do with acceleration. For instance that the feature detector can't handle too much acceleration? But it only happens after some hours.

          I hope that helps. Please let me know if you need more info. We're building 16 robots with each 3 or 4 cameras, so we would be very grateful if these issues are fixed.

          Kind regards

            Hi @WouterOddBot ,
            My guess would be that happens due to USB3 cable, perhaps you could try using USB-C with screws? I tried running your script (from the OP), but it's not really reproducible, tons of things aren't specified (create_downsample_script, parameters, model, json...), could you make it fully reproducible (.zip that contains everything)? And please check whether it also crashes if the camera is completely still (no movement in USB cable).
            Thanks, Erik

              @jkchen46033 I requested the access to your gdrive. Could you also share full minimal repro example, so we can test it locally?

                erik

                Ah danged, sorry forgot about the things unspecified in the example. Yes I'll make it fully reproducible.

                We actually use the USB-C with screws. The camera doesn't seem to crash when it's completely still.

                  @WouterOddBot that seems.. strange. Why would My bet would be that it's either bad connection (perhaps between USB-A and computer?), or perhaps voltage drops - ripple on source, maybe because a bunch of current goes through the motor, and it's being powered from the same battery?

                    erik

                    Ok I send you the MRE. We have the exact same MRE running on our robot on the field for quite some hours now, but the camera is still producing depth images and features.

                      erik

                      Please note that the camera stops producing only depth images and features. It keeps producing color images and detections. And we suspect that the camera needs to be in motion.

                        Hi @erik,

                        We haven't been able to reproduce the issues with the MRE so far. But we found that we normally set the yolo detection confidence very low (0.1) so that we can filter detections dynamically. We're now running an experiment with super low detection confidence (0.01) to see if that makes a difference. I also sent you the updated script by mail.

                        Interesting, please let us know if that makes a difference.