What does cam.setPreviewNumFramesPool(10) do, what is its purpose?

AdamPolak

I have seen this used in the experiments and demos but I can't figure out what it is meant to accomplish.

The documentation does not provide much info:

I see it used here for pedestrian reidentification:https://github.com/luxonis/depthai-experiments/blob/6a872165bd383e8a0768308a2ca7f63186d07e45/gen2-pedestrian-reidentification/api/main_api.py#L31

    # This manip uses 15*4.7 MB => 70 MB of RAM.

    copy_manip = pipeline.create(dai.node.ImageManip)

    copy_manip.setNumFramesPool(15)

    copy_manip.setMaxOutputFrameSize(1632 * 960 * 3)

    cam.preview.link(copy_manip.inputImage)

jakaskerl

Hi AdamPolak
The use of setNumFramesPool is described in this feature request. Apart from RAM usage, the preview pool size (default 4; now 10) is set so the camera node doesn't block if the next node (say NN node) doesn't process the frames fast enough.

Hope this helps,
Jaka

AdamPolak

jakaskerl

Sorry I am not clever enough to understand the implications in pipelines.

So the pool is a "waiting area" for further processing of output streams (for nn or xlinkout, or any other types of nodes)?
So that color frames can still be produced while a downstream NN works on them?
Can it be set to only 1 so it blocks the color camera or does it cause the color camera to crash?
How does this interace with the fps? If I have a color camera that in the pipeline I set to 30 fps and a pool size of 10, but a nn that can only run at 9 fps, what happens for color image generation, does it not add another image to the pool until the nn processes it?

erik

@AdamPolak

The pool is the memory block allocated for frames. So you can upfront specify number of messages (eg frames) a node can produce, and since you know the frame size, pools get allocated statically. Input queues (to the node) are "waiting area", but there we only save pointers to the actual message in the "pool memory".
copy_manip was only added so more full-res frames can be saved in the pipeline - back then, ColorCamera didn't have API to select number of preview/video/isp/raw frames for pools. If pool is full (so frames are waiting to be processed somewhere in the pipeline), colorcamera can't produce new frames, as it has nowhere to save them.
I think minimum is 2 due to how pipelines work, but I may be wrong.
Depends if the input queue to NN is blocking/nonblocking, and what the queue size is.
Example:
- if NN queue size is eg. 15, then it doesn't matter - all 10 frames will be waiting in the NN input queue, and camera won't be able to produce new frames
- If NN queue size is 5 and non-blocking, whenever there will be new frame, it will replace the old one in the queue (if queue fills up)
- If NN queue size is 5 and blocking, then it won't replace the old frame, but camera will still produce new frames. So if you link other ndoes, eg. XLinkOut, colorcamera will still send frames to it. Whereas in the first example it wouldn't - as new frames wouldn't be able to generate.

Additional docs here: https://docs.luxonis.com/projects/api/en/latest/components/nodes/
Thanks, Erik

AdamPolak

erik

Really appreciate the thorough response. I am having trouble with fps and will explore blocking/non blockign to try to get the fps up.

I have set the rgb camera at 30 fps that goes into manip 1 to downscale small so it doesn't take much bandwidth:
camRgb = pipeline->create<dai::node::ColorCamera>();
camRgb->setPreviewSize(st->rgb_dai_preview_x, st->rgb_dai_preview_y);
camRgb->setResolution(st->rgb_dai_res);
camRgb->setInterleaved(false);
camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::RGB);
camRgb->setBoardSocket(dai::CameraBoardSocket::RGB);
camRgbManip = pipeline->create<dai::node::ImageManip>(); camRgbManip-> initialConfig.setResize(408, 240); camRgbManip->initialConfig.setFrameType(dai::ImgFrame::Type::BGR888p); camRgb->preview.link(camRgbManip->inputImage);
rgbOut = pipeline->create<dai::node::XLinkOut>();
// Here I will take the camRgb frames and downscale them before sending them out
rgbOut->setStreamName("rgb");
I link manip 1 to xlinkout
camRgbManip->out.link(rgbOut->input);
// camera Properties
camRgb->setFps(st->rgb_dai_fps);
camRgb->setPreviewNumFramesPool(10);

The pool also goes to manip 2 to resize for a person detection nn:
person_det_manip = pipeline->create<dai::node::ImageManip>();
person_det_manip->initialConfig.setResize(544, 320); // This seems to downscale it by 1/3 (544x320)
person_det_manip->initialConfig.setFrameType(dai::ImgFrame::Type::RGB888p);
camRgb->preview.link(person_det_manip->inputImage);

So I believe that means the preview frames go to 2 different manips. manip 1 goes to XlinkOut, manip 2 goes to MobileDetectionNetwork.

I have my output queues like so:
// Set up the output queues
for (const auto &name : {"rgb", "detection", "recognition"}) {
queues[name] = device->getOutputQueue(name, 10, false);
}

And I calculate fps for each of the queues:
if (queues["rgb"]->has()) {
auto rgb_message = queues["rgb"]->get();
sync.add_msg(rgb_message, "color");
current_rgb_frame_counter++;
uint64_t capture_timestamp = CurrentTimeMs();
auto framesASecond = (float)current_rgb_frame_counter/((float)(capture_timestamp - start_time)*.001);
auto fps_string = std::to_string(framesASecond);
std::cerr << "color fps: " << fps_string << std::endl << std::flush;
}

I cannot get the color fps sent after manip 1 through xlink to go higher than 10fps. Is it known that ImageManips can't operate at faster than 10fps?
color fps: 10.808226
recognition fps: 7.240854
detection fps: 9.986684

(reducing to 1 manip and xlinking out the persondetection_manip doesn't increase the fps, still ~10 fps)

erik

Hi AdamPolak ,
Please follow pipeline issue MRE: https://docs.luxonis.com/en/latest/pages/support/#depthai-issue
Also please provide python MRE, as I am not familiar with C++.
Thanks, Erik

AdamPolak

@erik

I can't wrap my head around this frame pool concept.

I have a ImageManip node that takes in from the color camera and then outputs to a nn

person_det_manip = pipeline->create<dai::node::ImageManip>();
person_det_manip->initialConfig.setResize(rgb_person_det_nn_in_x_res, rgb_person_det_nn_in_y_res); // requied for nn (544, 320)
person_det_manip->initialConfig.setFrameType(dai::ImgFrame::Type::RGB888p);
person_det_manip->setNumFramesPool(10);
camRgb->preview.link(person_det_manip->inputImage)
// Person detection spatial nn setup cout << "Creating Person Detection Spatial Neural Network..." << endl;
person_nn = pipeline->create<dai::node::MobileNetSpatialDetectionNetwork>();
person_nn->setBoundingBoxScaleFactor(0.25);
person_nn->setDepthLowerThreshold(50);
person_nn->setDepthUpperThreshold(12000);
person_nn->setBlobPath(model_person_detection_path);
person_nn->setNumPoolFrames(10);

Down stream pulling from the XLinkOut queues the number of frames I can pull is equal to the amount of the frames in the frame pool. If I change it to 20 I can pull 20 frames, if I change it to 5, I can only pull 5 frames.

What does this imply?

jakaskerl

Hi @AdamPolak
By changing the pool size, you are essentially allocating more space on ram, which can be used by the node (for storing frames). These frames stay in the pool until they are discarded or read (when no node is pointing to it anymore). Increasing the pool size will increase the demand for memory (which is very limited).