setSrcCrops()

pkyleUPI

[edited to add code]
Note, I accidently typed 'setSrcCrops()' for the title, where as this discussion/question should be about verifying cropping outcome via 'getSrcCrops()

Hi,
I'm writing hardware test to confirm the functionality of a Luxonis Implementation of our base camera class. I'm having trouble "proving" that the crop is set correctly.

Here is our software setup:

depthai_core v3.0.0-rc4
OpenCV 4.10

And I'm running tests using:

Oak-D S2 Fixed Focus

Contextual Goals:

Capture an image at full resolution
Use Cropping to achieve 'digital zoom'
Downscale to 1080p
Send across the usb to host machine.

Here is some pseudo-code describing:

How I'm Setting up my pipeline:
How I'm applying a crop dynamically during runtime

// Setup Device/pipeline
device_ = std::make_shared_<dai:: Device>("our_device's_MxID", UsbSpeed::SUPER_PLUS);
pipeline_ = std::make_shared<dai:: Pipeline>(device_);

// Setup Camera Node
auto cam_node = pipeline_->create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_A);
auto cam_node_output = cam_node->requestFullResolutionOutput(std::nullopt, 30);

// Create Image Manip Node
auto manip_node = pipeline->create<dai::node::ImageManip>()->build();
cam_node_output->link(manip_node->inputImage);
// always downscale the image to 1080p
manip_node->initialConfig->setOutputSize( 1920, 1080,ImageManipConfig::ResizeMode::CENTER_CROP );
manip_node->initialConfig->setFrameType(dai::ImgFrame::Type::NV12);
manip_node->setMaxOutputFrameSize(1920*1080*3);
image_manip_queue_ = manip_node->inputConfig.createInputQueue();

// Link Output of Image Manip to Image Message Queue
image_queue_ = manip_node->out.createOutputQueue();
  
// Start Pipeline
pipeline_->start();

And for setting crop dynamically i call :

auto cfg = std::make_shared<dai::ImageManipConfig>();
cfg->addCrop(x,y,w,h);
cfg->setOutputSize(1920,1080, ImageManipConfig::ResizeMode::CENTER_CROP);
cfg->setFrameType(dai::ImgFrame::Type::NV12);
image_manip_queue_->send(cfg);

Ok, given all of this. I'm trying to prove that when I 'setCrop' the image is being cropped correctly. How do you reccommend doing this?
I've tried:

auto img_frame = image_queue_->get<dai::ImgFrame>();
auto crops = img_frame->transformation.getSrcCrops();
// then, compare the data in crops to expected values

For example, if I call SetCrop(10, 15, 2880, 1620);
I expect the image dimensions to be:
width == 1920
height == 1080
crop x = 10
crop y = 15
crop width = 2880
crip height = 1620

I get the following:

I20250811 15:25:39.661936 130749977570880 luxonis_oak_d_s2_tests.cpp:232] SOCKET: Center
I20250811 15:25:39.661956 130749977570880 luxonis_oak_d_s2_tests.cpp:233] img->getWidth(): 1920
I20250811 15:25:39.661959 130749977570880 luxonis_oak_d_s2_tests.cpp:234] img->getHeight(): 1080
I20250811 15:25:39.661961 130749977570880 luxonis_oak_d_s2_tests.cpp:235] img->transformation.getSize().first: 1920
I20250811 15:25:39.661965 130749977570880 luxonis_oak_d_s2_tests.cpp:236] img->transformation.getSize().second: 1080
I20250811 15:25:39.661987 130749977570880 luxonis_oak_d_s2_tests.cpp:238] crops.size(): 2
I20250811 15:25:39.661989 130749977570880 luxonis_oak_d_s2_tests.cpp:240] 	crop.size.width: 2880
I20250811 15:25:39.661998 130749977570880 luxonis_oak_d_s2_tests.cpp:241] 	crop.size.height: 1620
I20250811 15:25:39.662000 130749977570880 luxonis_oak_d_s2_tests.cpp:242] 	crop.center.x: 1450
I20250811 15:25:39.662003 130749977570880 luxonis_oak_d_s2_tests.cpp:243] 	crop.center.y: 825
I20250811 15:25:39.662005 130749977570880 luxonis_oak_d_s2_tests.cpp:244] 	crop.getPoints().begin()->x: 10
I20250811 15:25:39.662008 130749977570880 luxonis_oak_d_s2_tests.cpp:245] 	crop.getPoints().begin()->y: 15
I20250811 15:25:39.662011 130749977570880 luxonis_oak_d_s2_tests.cpp:240] 	crop.size.width: 2880
I20250811 15:25:39.662013 130749977570880 luxonis_oak_d_s2_tests.cpp:241] 	crop.size.height: 1620
I20250811 15:25:39.662015 130749977570880 luxonis_oak_d_s2_tests.cpp:242] 	crop.center.x: 1450
I20250811 15:25:39.662017 130749977570880 luxonis_oak_d_s2_tests.cpp:243] 	crop.center.y: 825
I20250811 15:25:39.662019 130749977570880 luxonis_oak_d_s2_tests.cpp:244] 	crop.getPoints().begin()->x: 10
I20250811 15:25:39.662021 130749977570880 luxonis_oak_d_s2_tests.cpp:245] 	crop.getPoints().begin()->y: 15

Why is the crop vector retrieved from getSrcCrops() size 2?

IF i change the input width/height to be non even numbers...
For example, if I call SetCrop(10, 15, 2881, 1621);
I expect the image dimensions to be:
width == 1920
height == 1080
crop x = 10
crop y = 15
crop width = 2881
crip height = 1621

I get the following:

I20250811 15:29:16.823418 133967056932416 luxonis_oak_d_s2_tests.cpp:232] SOCKET: Center
I20250811 15:29:16.823457 133967056932416 luxonis_oak_d_s2_tests.cpp:233] img->getWidth(): 1920
I20250811 15:29:16.823460 133967056932416 luxonis_oak_d_s2_tests.cpp:234] img->getHeight(): 1080
I20250811 15:29:16.823462 133967056932416 luxonis_oak_d_s2_tests.cpp:235] img->transformation.getSize().first: 1920
I20250811 15:29:16.823466 133967056932416 luxonis_oak_d_s2_tests.cpp:236] img->transformation.getSize().second: 1080
I20250811 15:29:16.823499 133967056932416 luxonis_oak_d_s2_tests.cpp:238] crops.size(): 2
I20250811 15:29:16.823502 133967056932416 luxonis_oak_d_s2_tests.cpp:240] 	crop.size.width: 2881
I20250811 15:29:16.823516 133967056932416 luxonis_oak_d_s2_tests.cpp:241] 	crop.size.height: 1621
I20250811 15:29:16.823518 133967056932416 luxonis_oak_d_s2_tests.cpp:242] 	crop.center.x: 1450.5
I20250811 15:29:16.823520 133967056932416 luxonis_oak_d_s2_tests.cpp:243] 	crop.center.y: 825.5
I20250811 15:29:16.823522 133967056932416 luxonis_oak_d_s2_tests.cpp:244] 	crop.getPoints().begin()->x: 10
I20250811 15:29:16.823526 133967056932416 luxonis_oak_d_s2_tests.cpp:245] 	crop.getPoints().begin()->y: 15
I20250811 15:29:16.823528 133967056932416 luxonis_oak_d_s2_tests.cpp:240] 	crop.size.width: 2881
I20250811 15:29:16.823530 133967056932416 luxonis_oak_d_s2_tests.cpp:241] 	crop.size.height: 1620.56
I20250811 15:29:16.823532 133967056932416 luxonis_oak_d_s2_tests.cpp:242] 	crop.center.x: 1450.5
I20250811 15:29:16.823535 133967056932416 luxonis_oak_d_s2_tests.cpp:243] 	crop.center.y: 825.5
I20250811 15:29:16.823537 133967056932416 luxonis_oak_d_s2_tests.cpp:244] 	crop.getPoints().begin()->x: 10
I20250811 15:29:16.823539 133967056932416 luxonis_oak_d_s2_tests.cpp:245] 	crop.getPoints().begin()->y: 15.2188

Why are the 'repeated' crops slightly different?

If you're curious, here is the code for my test so far:

      uint32_t x=10;
      uint32_t y=15;
      auto w = 2881;
      auto h = 1621;
      // I'm not sure why 'crops' contains more than one element, So I'm being certain its not 
      // old data by clearing the buffer.. then setting the Crop
      oak_d_s2->TryGetAllFramesFromSensor(socket);
        
      // Crop
      EXPECT_TRUE(oak_d_s2->UpdateSensorCrop(x, y, w, h, socket));
      
      //Then wait for a few frames to stack up and get all frames from the buffer
      std::this_thread::sleep_for(std::chrono::milliseconds(1000));
      oak_d_s2->TryGetAllFramesFromSensor(socket);
      
      //Now the next image should be 'guaranteed' to contain the new cropping
      auto lane = oak_d_s2->GetLanes()[socket];
      bool timed_out = true;
      auto img = lane.received_image_queue->get<dai::ImgFrame>(
          std::chrono::milliseconds(1000), timed_out);
      
      // Confirm we have an image
      ASSERT_FALSE(timed_out);
      
      // Inspect the crop
      LOG(INFO) << "SOCKET: " << nv::camera::DeviceSocketToString(socket);
      LOG(INFO) <<"img->getWidth(): " << img->getWidth();
      LOG(INFO) <<"img->getHeight(): " << img->getHeight();
      LOG(INFO) <<"img->transformation.getSize().first: " << img->transformation.getSize().first;
      LOG(INFO) <<"img->transformation.getSize().second: " << img->transformation.getSize().second;
      auto crops = img->transformation.getSrcCrops();
      LOG(INFO) <<"crops.size(): " << crops.size();
      for(const auto& crop : crops) {
        LOG(INFO) <<"\tcrop.size.width: " << crop.size.width;
        LOG(INFO) <<"\tcrop.size.height: " << crop.size.height;
        LOG(INFO) <<"\tcrop.center.x: " << crop.center.x;
        LOG(INFO) <<"\tcrop.center.y: " << crop.center.y;
        LOG(INFO) <<"\tcrop.getPoints().begin()->x: " << crop.getPoints().begin()->x;
        LOG(INFO) <<"\tcrop.getPoints().begin()->y: " << crop.getPoints().begin()->y;
      }

jakaskerl

pkyleUPI
Can you give me a full repro on this? As minimal as you can possibly make it?

Thanks,
Jaka

pkyleUPI

Jaka,
Here is a simple demonstration of the 'problem'/ confusing data: luxonis_crop_data. I really just need help deciphering why there is a 'vector' of Crops, and what each crop object in the vector means conceptually. I don't understand why there would be more than one crop in this vector considering the 'workflow' or pipeline I described.

Cheers,
Kyle

pkyleUPI

Ok, Playing around with this and crunching the numbers on the Aspect Ratio(AR) I've learned the following:

When you start the app (before pressing up/down arrows), there is 1 crop in the vector.
- I believe this is the 'crop' which occurs when the image is downscaled/cropped from full resolution to (in this example) 1280x720.
When you Adjust the 'zoom' by pressing 'Up-Arrow' or 'Down-Arrow', a second crop is added. Now...
- Crop 0 : crops the full resolution image, maintaining the AR
- Crop 1: downscales/crops the image from full resolution to 1280x720.

Given this, my questions are as follows:

Can you confirm the crop ordering above?
Is this a reasonable workflow/pipeline to achieve digital zoom? Are there improvements that can be made to reduce latency?

jakaskerl

pkyleUPI
The order matters, but the setOutputSize will always be the last operation. It will upscale/downscale the current image to the desired size. As far as the optimizations go, make sure the outputSize is not larger than your current frame. Upscaling is costly.

Thanks,
Jaka