I'm trying to debug a rare camera freeze issue. I'm thinking/hoping it's a software implementation issue on my end and not a camera hardware related issue. If you Google (vs. forum search) "luxonis forums frozen camera" there are more results that have helped my infer what might be happening. I'm currently under the impression the queue gets full but I don't know this for sure yet. When I have `rgbQueue = device->getOutputQueue(STREAM_PREVIEW, 4, false);` as part of my pipeline initialization and then the below code block executing at an adaptive frame rate targeting 140fps (via OV9782): 1. What happens if the queue gets "full"? Would my current implementation keep reading the same frame which gives the illusion of being frozen/stuck? 2. I track `frameRepeatCount` against a threshold to determine this "frozen" state and it validates rarely for most users, where one user gets the issue consistently after at most 3-5 minutes of runtime. Any ideas what might be happening? 3. `hasUpdatedFrame` seems to never get reset to `true` due to the early returns (by design), but I'd like to know what can cause getting into this state to begin with? 4. Is there a way (and is it a good idea) to clear the queue if I get into this "froze" state? ``` void UpdateFrame() { try { hasUpdatedFrame = false; auto imgFrames = rgbQueue->tryGetAll(); int count = imgFrames.size(); if (count getSequenceNum(); bool isNewFrame = frameSequenceNum != num; if (!isNewFrame) { frameRepeatCount++; return; } frameSequenceNum = num; hasUpdatedFrame = true; cameraSettingsSummary.exposureTimeUs = imgFrame->getExposureTime().count(); cameraSettingsSummary.wbColorTemp = imgFrame->getColorTemperature(); cameraSettingsSummary.sensitivityIso = imgFrame->getSensitivity(); std::pair result = toMatsFromPlanar(imgFrame->getData(), imgFrame->getWidth(), imgFrame->getHeight()); frameRGB = result.first; frameBGR = result.second; cv::cvtColor(frameRGB, frameRGBA, cv::COLOR_RGB2RGBA); UpdateFpsActual(); } catch (const std::exception &e) { std::cerr

Camera freeze and/or stale frame read

luxd

I'm trying to debug a rare camera freeze issue. I'm thinking/hoping it's a software implementation issue on my end and not a camera hardware related issue.

If you Google (vs. forum search) "luxonis forums frozen camera" there are more results that have helped my infer what might be happening. I'm currently under the impression the queue gets full but I don't know this for sure yet.

When I have rgbQueue = device->getOutputQueue(STREAM_PREVIEW, 4, false); as part of my pipeline initialization and then the below code block executing at an adaptive frame rate targeting 140fps (via OV9782):

What happens if the queue gets "full"? Would my current implementation keep reading the same frame which gives the illusion of being frozen/stuck?
I track frameRepeatCount against a threshold to determine this "frozen" state and it validates rarely for most users, where one user gets the issue consistently after at most 3-5 minutes of runtime. Any ideas what might be happening?
hasUpdatedFrame seems to never get reset to true due to the early returns (by design), but I'd like to know what can cause getting into this state to begin with?
Is there a way (and is it a good idea) to clear the queue if I get into this "froze" state?

    void UpdateFrame()
    {
        try
        {
            hasUpdatedFrame = false;

            auto imgFrames = rgbQueue->tryGetAll<dai::ImgFrame>();
            int count = imgFrames.size();
            if (count <= 0)
                return;

            std::shared_ptr<dai::ImgFrame> imgFrame = imgFrames[count - 1];
            if (!imgFrame)
                return;

            auto num = imgFrame->getSequenceNum();
            bool isNewFrame = frameSequenceNum != num;
            if (!isNewFrame)
            {
                frameRepeatCount++;
                return;
            }

            frameSequenceNum = num;
            hasUpdatedFrame = true;

            cameraSettingsSummary.exposureTimeUs = imgFrame->getExposureTime().count();
            cameraSettingsSummary.wbColorTemp = imgFrame->getColorTemperature();
            cameraSettingsSummary.sensitivityIso = imgFrame->getSensitivity();

            std::pair<cv::Mat, cv::Mat> result = toMatsFromPlanar(imgFrame->getData(), imgFrame->getWidth(), imgFrame->getHeight());
            frameRGB = result.first;
            frameBGR = result.second;
            cv::cvtColor(frameRGB, frameRGBA, cv::COLOR_RGB2RGBA);

            UpdateFpsActual();
        }
        catch (const std::exception &e)
        {
            std::cerr << "UpdateFrame error occurred: " << e.what() << std::endl;
        }
    }

erik

Hi luxd , could you also post the actual pipeline? Camera can freeze due to filled node pools causing a "deadlock", more info here:
https://docs.luxonis.com/software/depthai-components/nodes/#Nodes-Inputs%20and%20outputs

luxd

@erik Yes, below is the pipeline setup. I'll check that link in the meantime.

    bool InitPipeline(int fpsCamera)
    {
        // 1. Create pipeline
        dai::Pipeline corePipeline;

        // 2. Define sources and outputs
        colorCam = corePipeline.create<dai::node::ColorCamera>();
        auto manip = corePipeline.create<dai::node::ImageManip>();
        auto controlIn = corePipeline.create<dai::node::XLinkIn>();
        auto configIn = corePipeline.create<dai::node::XLinkIn>();
        auto xLinkPreviewOut = corePipeline.create<dai::node::XLinkOut>();

        // 3. Properties
        colorCam->setPreviewSize(512, 512);
        colorCam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_720_P);
        colorCam->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);
        colorCam->setInterleaved(false);
        colorCam->setImageOrientation(dai::CameraImageOrientation::NORMAL);
        colorCam->setFps(fpsCamera);

        controlIn->setStreamName(STREAM_CONTROL_IN);

        manip->initialConfig.setCenterCrop(cropRatio);
        manip->initialConfig.setResize(previewSize.width, previewSize.height);
        manip->setMaxOutputFrameSize(colorCam->getResolutionHeight() * colorCam->getResolutionWidth() * 3);

        configIn->setStreamName(STREAM_CONFIG_IN);

        xLinkPreviewOut->setStreamName(STREAM_PREVIEW);

        // 4. Linking
        controlIn->out.link(colorCam->inputControl);
        configIn->out.link(manip->inputConfig);
        colorCam->preview.link(manip->inputImage);
        manip->out.link(xLinkPreviewOut->input);

        try
        {
            // 5. Connecting
            device = std::shared_ptr<dai::Device>(new dai::Device(corePipeline, deviceInfo));

            // 6. Queries
            rgbQueue = device->getOutputQueue(STREAM_PREVIEW, 4, false);
            controlQueue = device->getInputQueue(STREAM_CONTROL_IN, 4, false);
            configQueue = device->getInputQueue(STREAM_CONFIG_IN, 4, false);
        }
        catch (const std::exception &e)
        {
            std::cerr << "Exception during pipeline initialization: " << e.what() << std::endl;
        }

        return true;
    }

erik

@luxd not sure if it's relevant, but why don't you save the device instance? i converted it to python (via gpt) and used standard with Device(pipeline) as device: to init pipeline, and it works as expected...

luxd

why don't you save the device instance?

@erik Can you be more specific? I don't quite follow. I'm caching the device via device = std::shared_ptr<dai::Device>(new dai::Device(corePipeline, deviceInfo)); which is how I do cleanup.

As mentioned in the original post it works as expected most of the time but one user seems to get it more frequently.

Do you have any other ideas based on the code of what might be causing this? Is there anything I can do different to prevent the "deadlock" you mentioned?

If I have blocking as false vs. the default true would you expect my setup to be immune to this "deadlock"?
Would you expect tryGet vs. tryGetAll to be favorable?
Would increasing the maxSize be favorable?

erik

@luxd I see, perhaps that was lost when i converted it to python via chatgpt. Yes, setting (all) queues blocking to false should eliminate this. As long as you're reading fast enough it doesn't matter. Since you have only 1 queue you could also use blocking .get() to preserve CPU cycles (you're hotlooping otherwise).
You can try to also increase maxSize, it would help if some queue gets filled up, yes.

luxd

erik Yes, setting (all) queues blocking to false should eliminate this.

Thanks for confirming this expectation.

erik Since you have only 1 queue you could also use blocking .get() to preserve CPU cycles (you're hotlooping otherwise)

Since I have an adaptive fps solution that calls into UpdateFrame I'm under the impression I'm not hotlooping. Is that your expectation too based on that info? Does this fact change any of your suggestions?

erik You can try to also increase maxSize, it would help if some queue gets filled up, yes.

OK, if it doesn't hurt I'll try this as well.

erik Could you provide full MRE (minimal repro example) in python, so we can debug it locally?

I would love to, but I can't repro this myself. As mentioned it's very rare, but then one user recently is able to trigger the issue somewhat consistently. I'll try to determine why that's the case, but thank you in the meantime for confirming expectations related to my various questions.

I'm on v2.24.0 and in looking at the release notes v2.26.0 has an "Improved ImageManip stability" bullet point (which is the only line item I interpreted as potentially improving my situation). Are there any other updates since v2.24.0 you'd anticipate positively impacting my situation?

erik

Could you provide full MRE (minimal repro example) in python, so we can debug it locally?

erik

@luxd
I am not sure how fast you call UpdateFrame() - if there's while True: UpdateFrame() then that'd be hotlooping, but that wouldn't result in issue anyways.

There were quite some changes yes, some also improved stability, so I'd suggest using latest depthai version either way.

luxd

erik I am not sure how fast you call UpdateFrame() - if there's while True: UpdateFrame() then that'd be hotlooping, but that wouldn't result in issue anyways.

Can you elaborate on the "but that wouldn't result in issue anyways"? Any details that come to mind would be great. I think I'm starting to hone in on where the problem might be coming from.

Thanks again, you've been super helpful and your responses have helped me better understand what is and isn't expected.

luxd

@erik Regarding my use of hasUpdatedFrame, I initially added it as I was under the impression I could grab the same frame (via tryGetAll) on back to back UpdateFrame calls. I'm now thinking this isn't even possible if grabbing the frame then also removes it from the queue. Am I thinking about this right? Is my notion of hasUpdatedFrame irrelevant?

erik

@luxd I am not completely sure what you mean. If you want to just check if there's a frame (and not remove it), you should use if outputQueue.has(): instead of get/tryGet (which removes msg from queue).

erik

luxd I meant that even if you just use 100% of CPU (hotlooping) on the host computer it doesn't matter depthai/oak-wise. It should still work, your CPU on computer would just be a bit warmer, but it wouldn't cause issues you were mentioning