Hi, We're using a camera with the pipeline underneath. The camera runs fine for around 4 hours. Then its stops producing depth images and features. \`\`\` pipeline = Pipeline() color = pipeline.createColorCamera() color.setBoardSocket(CameraBoardSocket.RGB) color.setFps(10.0) color.setResolution(ColorCameraProperties.SensorResolution.THE_4_K) color.setVideoSize(parameters.color_video_width, parameters.color_video_height) color.setInterleaved(False) color.initialControl.setSharpness(parameters.sharpness) color.initialControl.setLumaDenoise(parameters.luma_denoise) color.initialControl.setChromaDenoise(parameters.chroma_denoise) color.initialControl.setAutoFocusLensRange(parameters.focus_range_infinity_position, parameters.focus_range_macro_position) color.initialControl.setAutoExposureLimit(parameters.exposure_limit_us) # The detection model can't handle 10fps so we drop every 2nd frame and downsample to 5fps color_downsample_script = pipeline.createScript() color_downsample_script.setScript(create_downsample_script(DOWNSAMPLE_SCRIPT_STREAM_IN, DOWNSAMPLE_SCRIPT_STREAM_OUT, 2)) color_resize = pipeline.createImageManip() color_resize.initialConfig.setResize(parameters.color_preview_width, parameters.color_preview_height) color_resize.initialConfig.setFrameType(ImgFrame.Type.BGR888p) # Sending high resolution images at 5fps takes too much bandwidth, so we downsample the high resolution frame rate to 1 frame every 4s. hi_res_downsample_script = pipeline.createScript() hi_res_downsample_script.setScript(create_downsample_script(DOWNSAMPLE_SCRIPT_STREAM_IN, DOWNSAMPLE_SCRIPT_STREAM_OUT, parameters.high_resolution_image_modulo)) hi_res_encoder = pipeline.createVideoEncoder() hi_res_encoder.setProfile(VideoEncoderProperties.Profile.MJPEG) hi_res_encoder.setQuality(parameters.high_resolution_image_quality) left = pipeline.createMonoCamera() left.setBoardSocket(CameraBoardSocket.LEFT) left.setFps(10.0) left.setResolution(MonoCameraProperties.SensorResolution.THE_400_P) # left.initialControl.setManualExposure(100, 400) # left.initialControl.setAutoExposureLock(True) right = pipeline.createMonoCamera() right.setBoardSocket(CameraBoardSocket.RIGHT) right.setFps(10.0) right.setResolution(MonoCameraProperties.SensorResolution.THE_400_P) # right.initialControl.setManualExposure(100, 400) # right.initialControl.setAutoExposureLock(True) depth = pipeline.createStereoDepth() # depth.setDepthAlign(CameraBoardSocket.RIGHT) # depth.setExtendedDisparity(True) depth.setLeftRightCheck(True) # LR-check required for depth alignment depth.initialConfig.setConfidenceThreshold(parameters.depth_confidence_threshold) depth.initialConfig.setMedianFilter(StereoDepthProperties.MedianFilter.MEDIAN_OFF) depth.initialConfig.setSubpixel(True) depth.initialConfig.setBilateralFilterSigma(0) depth.initialConfig.setSubpixelFractionalBits(5) depth.initialConfig.setDisparityShift(30) depth.initialConfig.setDepthUnit(RawStereoDepthConfig.AlgorithmControl.DepthUnit.CUSTOM) depth_config = depth.initialConfig.get() depth_config.algorithmControl.customDepthUnitMultiplier = parameters.depth_unit_multiplier depth.initialConfig.set(depth_config) depth.enableDistortionCorrection(True) features = pipeline.createFeatureTracker() features.setHardwareResources(numShaves=2, numMemorySlices=2) model_name = parameters.detection_model_name model_config_path = MODEL_FOLDER / (model_name + '.json') with open(model_config_path) as fp: model_config = json.load(fp) model_blob_path = MODEL_FOLDER / (model_name + '.blob') labels = model_config['labels'] coordinate_size = model_config['coordinate_size'] anchors = model_config['anchors'] anchor_masks = model_config['anchor_masks'] iou_threshold = model_config['iou_threshold'] detector_confidence_threshold = model_config['confidence_threshold'] detection = pipeline.createYoloDetectionNetwork() detection.setBlobPath(model_blob_path) detection.setAnchors(anchors) detection.setAnchorMasks(anchor_masks) detection.setConfidenceThreshold(detector_confidence_threshold) detection.setNumClasses(len(labels)) detection.setCoordinateSize(coordinate_size) detection.setIouThreshold(iou_threshold) detection.setNumInferenceThreads(2) detection.input.setBlocking(False) detection.input.setQueueSize(1) control_in = pipeline.createXLinkIn() color_out = pipeline.createXLinkOut() hi_res_out = pipeline.createXLinkOut() depth_out = pipeline.createXLinkOut() features_out = pipeline.createXLinkOut() detection_out = pipeline.createXLinkOut() control_in.setStreamName(CONTROL_STREAM_NAME) color_out.setStreamName(COLOR_STREAM_NAME) hi_res_out.setStreamName(HI_RES_STREAM_NAME) depth_out.setStreamName(DEPTH_RIGHT_STREAM_NAME) features_out.setStreamName(FEATURES_STREAM_NAME) detection_out.setStreamName(DETECTIONS_STREAM_NAME) # Also tried without these 2 lines but it didnt make a difference. depth_out.input.setBlocking(False) depth_out.input.setQueueSize(1) control_in.out.link(color.inputControl) color.video.link(color_downsample_script.inputs[DOWNSAMPLE_SCRIPT_STREAM_IN]) color_downsample_script.outputs[DOWNSAMPLE_SCRIPT_STREAM_OUT].link(color_resize.inputImage) color_downsample_script.outputs[DOWNSAMPLE_SCRIPT_STREAM_OUT].link(hi_res_downsample_script.inputs[DOWNSAMPLE_SCRIPT_STREAM_IN]) color_resize.out.link(color_out.input) color_resize.out.link(detection.input) hi_res_downsample_script.outputs[DOWNSAMPLE_SCRIPT_STREAM_OUT].link(hi_res_encoder.input) hi_res_encoder.bitstream.link(hi_res_out.input) left.out.link(depth.left) right.out.link(depth.right) depth.depth.link(depth_out.input) depth.rectifiedRight.link(features.inputImage) features.outputFeatures.link(features_out.input) detection.out.link(detection_out.input) \`\`\` and \`\`\` device = Device(self.__pipeline, self.__device_info) color_queue = device.getOutputQueue(name=COLOR_STREAM_NAME, maxSize=1, blocking=False) hi_res_queue = device.getOutputQueue(name=HI_RES_STREAM_NAME, maxSize=1, blocking=False) depth_queue = device.getOutputQueue(name=DEPTH_RIGHT_STREAM_NAME, maxSize=1, blocking=False) features_queue = device.getOutputQueue(name=FEATURES_STREAM_NAME, maxSize=1, blocking=False) detections_queue = device.getOutputQueue(name=DETECTIONS_STREAM_NAME, maxSize=1, blocking=False) color_queue.addCallback(self.__get_and_publish_color_image) hi_res_queue.addCallback(self.__get_and_publish_hi_res_image) depth_queue.addCallback(self.__get_and_publish_depth_image) detections_queue.addCallback(self.__get_and_publish_detections) features_queue.addCallback(self.__get_and_publish_features) \`\`\` We also collected soms logs (see attached file). No errors whatsoever. Around line 84215 the camera stops producing depth images and features. The logs then only show a drop in CPU and NOC ddr. Do you have any idea what could be wrong? Thanks in advance!

Having a hard time attaching the complete log file. So here some screen shots: [upl-image-preview uuid=5b49f68c-81b9-4767-a78a-4f0ab70e143c url=https://discuss.luxonis.com/assets/files/2025-02-19/1739965357-321465-vision-logs-cpu.png alt={TEXT?}] [upl-image-preview uuid=09089f03-4237-40e4-bdd6-da0a0a19e1f6 url=https://discuss.luxonis.com/assets/files/2025-02-19/1739965369-99485-vision-logs-memory.png alt={TEXT?}] [upl-image-preview uuid=d72d6a37-6307-45bd-9a61-59c0893cf47e url=https://discuss.luxonis.com/assets/files/2025-02-19/1739965373-416072-vision-logs-temps.png alt={TEXT?}]

@"WouterOddBot"#p28562 Very difficult to know what the exact issue is. My first guess would be that some node in the pipeline runs out of frames in a pool.. This causes the node to stop producing (processing) images. Check at which node this stops, then up the `setNumFramesPool(num)` for that node and nodes before that. Thanks, Jaka

Hi @"jakaskerl"#1112 , Ok thanks for the tip. We'll have to find a time where we can run the camera for a long time again to see if it makes a difference. In the mean time, it seems that the camera keeps running for 20h+ in a static environment, but only 4h when it's moving. Does that make sense to you?

@"WouterOddBot"#p28598 I do - Encoder, detection and features will change the amount of messages or their size when movement is introduced. Encoder will lower information transfer when images are static because the information in an image stays the same. When moving, the entropy increases so more info is required. Detector will produce more detections when objects are in the scene. This potentially increases the processing required for following nodes since they have to process each detections separately. Same goes for features. Thanks, Jaka

Ok so I increased numframespool for the left and right camera nodes and the depth node from 3 (default value) to 5, but it didn't make a difference. What information do we need to see what breaks the pipeline? And how can I retrieve that information from the camera? Apart from that, I don't remember seeing this until perhaps few months. And afaik we didn't make significant code changes to the affected part of the pipeline\*. Could it be due to a depthai update? Since it only happens after 4h it feels a bit like a memory leak or something? At least I can check if we forgot to update any firmware. \*We did make a significant change in the detection part of the pipeline. We used to have the color camera node run at 5fps and the left/right camera nodes at 10fps. But then the color and depth frames aren't closely synced. So now we have the color and left/right nodes run at 10fps but we downsample the color frames to 5fps (we drop every 2nd frame). Could that still make a difference?

@"WouterOddBot"#p28624 > @"WouterOddBot"#p28624 What information do we need to see what breaks the pipeline? And how can I retrieve that information from the camera? It's not currently reported unless you are looking at a crash dump which might give some additional info, but it's difficult to find the cause. We are working on making this logging better. > @"WouterOddBot"#p28624 Could it be due to a depthai update? Since it only happens after 4h it feels a bit like a memory leak or something? At least I can check if we forgot to update any firmware. It does sound like a memory leak as well, perhaps you can try some older FW where all needed features are still present. Though I would say the change you have made could affect this as well. We have seen such crashes previously as well, but many turned out to be intel's fault (MyriadX memory issues).. Hope that is not the case here. Thanks, Jaka

Ok so I made something that we can retrieve crash dumps. I suggested it may be a memory leak, but the logs don't show any changes or out of memory (see images attached). Perhaps it still something else?

Tried retrieving a crash dump, but the camera doesn't make a crash dump when it stops working.

@"jakaskerl"#1112 We suspect that this could relate to how fast the camera moves. Which makes sense according to your entropy story. But we'd still like to get a clue for a solution? Thanks in advance.

@"WouterOddBot"#p28768 If that is the case and videoEncoder is the problem then you should check the bandwidth that is consumed by the pipeline.. How about you lower the quality to a minimum. That way, even when moving the difference in bitstream size should be negligible - or perhaps use `setBitrate()` to lock the size. Thanks, Jaka

Camera stops producing depth images and features after 4h.

WouterOddBot

Hi erik

In the original post I already included our camera pipeline. Can you use that as an MRE?

Our robots run on Ubuntu 22.04 with Ros2 Humble or Ubuntu 24.04 with Ros2 Jazzy. We use mostly Oak-D Pro USB cameras (https://shop.luxonis.com/products/oak-d-pro?variant=42455252369631), but still have some older Oak-D PoE cameras. Our depthai version is 2.27.0.0. The cameras are connected with provided USB3 cables. We also tested with the Oak-D Y adapter to make sure it's not a power problem.

We have been looking at the telemetrics of our robots. Our robots drive back and forth across a field. At the end of the field a robot stops, makes a u-turn, then starts driving back again. It seems that the cameras fail most of the times when a robot makes a u-turn or when it starts driving back again. So it might have to do with acceleration. For instance that the feature detector can't handle too much acceleration? But it only happens after some hours.

I hope that helps. Please let me know if you need more info. We're building 16 robots with each 3 or 4 cameras, so we would be very grateful if these issues are fixed.

Kind regards

jkchen46033

erik Here is the link of the dump and code,
https://drive.google.com/drive/u/1/folders/1e7gqpPiIWf40YNNkfOvkihGs5RrKrLkt?dmr=1&ec=wgc-drive-hero-goto

I produced it today. 3 streams, 10fps, mjpeg. script node.

erik

Hi @WouterOddBot ,
My guess would be that happens due to USB3 cable, perhaps you could try using USB-C with screws? I tried running your script (from the OP), but it's not really reproducible, tons of things aren't specified (create_downsample_script, parameters, model, json...), could you make it fully reproducible (.zip that contains everything)? And please check whether it also crashes if the camera is completely still (no movement in USB cable).
Thanks, Erik

WouterOddBot

erik

Ah danged, sorry forgot about the things unspecified in the example. Yes I'll make it fully reproducible.

We actually use the USB-C with screws. The camera doesn't seem to crash when it's completely still.

erik

@jkchen46033 I requested the access to your gdrive. Could you also share full minimal repro example, so we can test it locally?

jkchen46033

erik Shared.

erik

@WouterOddBot that seems.. strange. Why would My bet would be that it's either bad connection (perhaps between USB-A and computer?), or perhaps voltage drops - ripple on source, maybe because a bunch of current goes through the motor, and it's being powered from the same battery?

WouterOddBot

erik

Ok I send you the MRE. We have the exact same MRE running on our robot on the field for quite some hours now, but the camera is still producing depth images and features.

WouterOddBot

erik

Please note that the camera stops producing only depth images and features. It keeps producing color images and detections. And we suspect that the camera needs to be in motion.

WouterOddBot

Hi @erik,

We haven't been able to reproduce the issues with the MRE so far. But we found that we normally set the yolo detection confidence very low (0.1) so that we can filter detections dynamically. We're now running an experiment with super low detection confidence (0.01) to see if that makes a difference. I also sent you the updated script by mail.

erik

Interesting, please let us know if that makes a difference.

WouterOddBot

Ok we ran an experiment with confidence level 0.01. The camera gave between 80 and 160 detections per frame. The depth and features stopped after 12h. Can you confirm that with an experiment too?

We also ran an experiment where we omitted the downsampling node for the hi res images. With the ethernet camera we had to limit the amount of hi res images. But with usb3 we can just transfer all of them. We're not entirely sure yet, but it looks like omitting this downsampling node helps too.

(We still keep the downsampling node from 10Hz to 5Hz because 10Hz is too fast for the network.)

erik

@jkchen46033 we weren't able to repro your MRE, it has been working as expected. After how long did it crash?

« Previous Page