@WouterOddBot could you pehraps provide minimal repro example, so we can try to repro it locally? feel free to send it over to erik@luxonis.com and ill set it up locally & connect debugger to the camera. Please also mention depthai version, camera used, host computer/os used (perhaps relevant), cable/powering specifics, etc. just so we can have as close to 1:1 repro as possible.
Thanks, Erik

    Hi jakaskerl
    We can lower the quality of the hi res images but the camera sends only very few of them. Most of them are dismissed. So they should not take too much bandwidth?

      erik Here is the dump. Let me know if you need additional info. Thanks a lot for your help.

      My pipeline setup this way: poe, 3 lens, 3 jpeg encoder, all 10fps, two monos 720p, rgb 1080p, jpeg to script node, pumping out urls: /mjpeg, /left, /right, all in mjpeg multipart frames.

      This dump is produced today.

      {"crashReports":[{"crashedThreadId":167837697,"errorSource":"RTEMS_FATAL_SOURCE_EXCEPTION","errorSourceInfo":{"assertContext":{"fileName":"","functionName":"","line":0},"errorId":0,"trapContext":{"trapAddress":2204218472,"trapName":"Bad trap","trapNumber":9}},"processor":0,"threadCallstack":[{"callStack":[{"callSite":2200385356,"calledTarget":2200381832,"context":"","framePointer":2211279768},{"callSite":2205098404,"calledTarget":0,"context":"Thread handler","framePointer":2211279864},{"callSite":1881154788,"calledTarget":0,"context":"Thread handler","framePointer":2211279960},{"callSite":2205098556,"calledTarget":1881154720,"context":"","framePointer":2211280080},{"callSite":2204985004,"calledTarget":2205098536,"context":"","framePointer":2211280184},{"callSite":2205931196,"calledTarget":0,"context":"Thread handler","framePointer":2211280288},{"callSite":2204298320,"calledTarget":0,"context":"Thread handler","framePointer":2213073296},{"callSite":2204221036,"calledTarget":2204218436,"context":"","framePointer":2213073592},{"callSite":2204208956,"calledTarget":0,"context":"Thread handler","framePointer":2213073864},{"callSite":2204201828,"calledTarget":0,"context":"Thread handler","framePointer":2213074032},{"callSite":2204165556,"calledTarget":0,"context":"Thread handler","framePointer":2213074272},{"callSite":2204064028,"calledTarget":0,"context":"Thread handler","framePointer":2213074392},{"callSite":2204052024,"calledTarget":2204064100,"context":"","framePointer":2213074488},{"callSite":2204054424,"calledTarget":2204051528,"context":"","framePointer":2213074584},{"callSite":2204064028,"calledTarget":0,"context":"Thread handler","framePointer":2213074680},{"callSite":2204050908,"calledTarget":2204064100,"context":"","framePointer":2213074776},{"callSite":2200546904,"calledTarget":0,"context":"Thread handler","framePointer":2213074872},{"callSite":2200707600,"calledTarget":2200546064,"context":"","framePointer":2213074984},{"callSite":2200964480,"calledTarget":0,"context":"Thread handler","framePointer":2213075080},{"callSite":2200973128,"calledTarget":0,"context":"Thread handler","framePointer":2213075176},{"callSite":167837698,"calledTarget":1195661140,"context":"Thread exit","framePointer":0}],"instructionPointer":2200385356,"stackBottom":2212944504,"stackPointer":2211279768,"stackTop":2213075575,"threadId":167837697,"threadName":"CBTH","threadStatus":"READY"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2213731360},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2213731464},{"callSite":2201052240,"calledTarget":2205066912,"context":"","framePointer":2213731584},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2213731688},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2213731784},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2213731880}],"instructionPointer":2205067204,"stackBottom":2213600904,"stackPointer":2213731360,"stackTop":2213731975,"threadId":167837698,"threadName":"TWDG","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214125496},{"callSite":2205072104,"calledTarget":2205067040,"context":"","framePointer":2214125600},{"callSite":2205075072,"calledTarget":2205071996,"context":"","framePointer":2214125720},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214125824},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214125920},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214126016}],"instructionPointer":2205067204,"stackBottom":2213995040,"stackPointer":2214125496,"stackTop":2214126111,"threadId":167837699,"threadName":"TIME","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214256720},{"callSite":2205072104,"calledTarget":2205067040,"context":"","framePointer":2214256824},{"callSite":2204990304,"calledTarget":2205071996,"context":"","framePointer":2214256944},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214257048},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214257144},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214257240}],"instructionPointer":2205067204,"stackBottom":2214126264,"stackPointer":2214256720,"stackTop":2214257335,"threadId":167837700,"threadName":"IRQS","threadStatus":"WAITING_FOR_SYSTEM_EVENT"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214387944},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2214388048},{"callSite":2204647304,"calledTarget":2205066912,"context":"","framePointer":2214388168},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214388272},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214388368},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214388464}],"instructionPointer":2205067204,"stackBottom":2214257488,"stackPointer":2214387944,"stackTop":2214388559,"threadId":167837701,"threadName":"swi6","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214519176},{"callSite":2203893028,"calledTarget":2203950744,"context":"","framePointer":2214519280},{"callSite":2204655088,"calledTarget":2203892796,"context":"","framePointer":2214519376},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214519496},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214519592},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214519688}],"instructionPointer":2203950260,"stackBottom":2214388712,"stackPointer":2214519176,"stackTop":2214519783,"threadId":167837702,"threadName":"config","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214650400},{"callSite":2203893028,"calledTarget":2203950744,"context":"","framePointer":2214650504},{"callSite":2203951260,"calledTarget":2203892796,"context":"","framePointer":2214650600},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214650720},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214650816},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214650912}],"instructionPointer":2203950260,"stackBottom":2214519936,"stackPointer":2214650400,"stackTop":2214651007,"threadId":167837703,"threadName":"kqueue","threadStatus":"WAITING_FOR_BSD_WAKEUP"},{"callStack":[{"callSite":2205067204,"calledTarget":1881154236,"context":"","framePointer":2214781616},{"callSite":2205067016,"calledTarget":2205067040,"context":"","framePointer":2214781720},{"callSite":2204647304,"calledTarget":2205066912,"context":"","framePointer":2214781840},{"callSite":2205091048,"calledTarget":0,"context":"Thread handler","framePointer":2214781944},{"callSite":1881154668,"calledTarget":0,"context":"Thread handler","framePointer":2214782040},{"callSite":1881154572,"calledTarget":0,"context":"Thread exit","framePointer":2214782136}],"instructionPointer":2205067204,"stackBottom":2214651160,"stackPointer":2214781616,"stackTop":2214782231,"threadId":167837704,"threadName":"swi5","threadStatus":"WAITING_FOR_EVENT"},{"callStack":[{"callSite":2203950260,"calledTarget":1881154236,"context":"","framePointer":2214912848},{"callSite":2203893028,"calledTarget":2203950
      "crash_dump.json" [Incomplete last line] 1 line, 94228 bytes

      • erik replied to this.

          jkchen46033 could you please upload the json to gdrive and send the link? as your reply is incomplete

              Hi erik

              In the original post I already included our camera pipeline. Can you use that as an MRE?

              Our robots run on Ubuntu 22.04 with Ros2 Humble or Ubuntu 24.04 with Ros2 Jazzy. We use mostly Oak-D Pro USB cameras (https://shop.luxonis.com/products/oak-d-pro?variant=42455252369631), but still have some older Oak-D PoE cameras. Our depthai version is 2.27.0.0. The cameras are connected with provided USB3 cables. We also tested with the Oak-D Y adapter to make sure it's not a power problem.

              We have been looking at the telemetrics of our robots. Our robots drive back and forth across a field. At the end of the field a robot stops, makes a u-turn, then starts driving back again. It seems that the cameras fail most of the times when a robot makes a u-turn or when it starts driving back again. So it might have to do with acceleration. For instance that the feature detector can't handle too much acceleration? But it only happens after some hours.

              I hope that helps. Please let me know if you need more info. We're building 16 robots with each 3 or 4 cameras, so we would be very grateful if these issues are fixed.

              Kind regards

                Hi @WouterOddBot ,
                My guess would be that happens due to USB3 cable, perhaps you could try using USB-C with screws? I tried running your script (from the OP), but it's not really reproducible, tons of things aren't specified (create_downsample_script, parameters, model, json...), could you make it fully reproducible (.zip that contains everything)? And please check whether it also crashes if the camera is completely still (no movement in USB cable).
                Thanks, Erik

                  @jkchen46033 I requested the access to your gdrive. Could you also share full minimal repro example, so we can test it locally?

                    erik

                    Ah danged, sorry forgot about the things unspecified in the example. Yes I'll make it fully reproducible.

                    We actually use the USB-C with screws. The camera doesn't seem to crash when it's completely still.

                      @WouterOddBot that seems.. strange. Why would My bet would be that it's either bad connection (perhaps between USB-A and computer?), or perhaps voltage drops - ripple on source, maybe because a bunch of current goes through the motor, and it's being powered from the same battery?

                        erik

                        Ok I send you the MRE. We have the exact same MRE running on our robot on the field for quite some hours now, but the camera is still producing depth images and features.

                          erik

                          Please note that the camera stops producing only depth images and features. It keeps producing color images and detections. And we suspect that the camera needs to be in motion.