high performance sensor crop / region of interest

Asa

I would like to crop the raw stream of a color camera to a square aspect ratio with a precise integer pixel offset from the left. The sensorCrop function only takes a single float (percentage of sensor) for the left, and does not offer a right side. How can I assure that I am employing the most low level region of interest crop so that the data coming off of the sensor is lessened for each frame. Hopefully this would give me a little more FPS as there is 1.33x less data.

Is this possible?

erik

Hi Asa ,
WRT camRgb.setSensorCrop(x,y), it's used for ISP->video cropping (img below). By default, it will be center cropped (so setSensorCrop(0.25, 0.25)), but you could also configure it to get top-left quarter of the img (.setSensorCrop(0, 0)), or bottom-right quarter of the img (.setSensorCrop(0.5, 0.5)).

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
# camRgb,setSensorCrop(0.25, 0.25) # By default already, center crop
camRgb.setVideoSize(1920, 1080)

It doesn't actually do on-sensor crop (and MIPI stream only that region), but it does crop in postprocessing on RVC. I will update the documentation for this function as well.
Thoughts?
Thanks, Erik

erik

(Updated docs PR)

Timmm

@erik Can you tell me what is the range of (X,y) coordinates we can set in camRgb,setSensorCrop(0.25, 0.25). Eg. what is the initial point and end point?

Asa

Timmm these are "fractions of whole width and height" I believe.

Asa

I am looking for something that would be before isp. I am trying for the lowest latency, closest to the metal experience. Reducing the width of the readout ROI on the sensor should let you reduce the rolling shutter artifacting by a decent amount as well(less pixels to read out each line at the same clock rate = less overall time between rows). I would like to ideally set this up as my own SensorResolution (THE_12_MP_CENTER_CROP_1_TO_1?). Is this possible? I know the Sony sensors support this ROI type of setup. Being able to specify the correct integer column offset would be important so that we don't change the debayering order of course. I guess its up to you to setup support for something like this as you currently are pulling whole frames off all the time, even if you crop to 4k downstream?

erik

Asa I don't think you can currently achieve that with OAK cameras, and we would need to implement this first. And since implementing such things takes very large efforts, I'm quite sure we wouldn't want to go down this rabbit hole unless there's a commitment for 1000s of devices.I hope you understand.
And you are right, it's 0..1 values, so 0.25 is a quarter, and if we have isp 4k and video 1080P, then it will do center cropping (from 0.25,0.25 to 0.75,0.75).
Thanks, Erik

Asa

Understood, albeit dissappointing. You would improve the performance of all of the cameras if you can support this as you are currently exposing and reading out a third more pixels than you need to. If you are center cropping 4k you are throwing away even more at around 4mp of data every frame. This has to limit the aggregate isp performance of all of your systems, increases the jello effect etc. As for this affecting a large commitment of devices, you may not be too far off. At any rate, I would love to track this issue. Perhaps this is an area that will open up in the SDK over time?

erik

Hi Asa , I agree - in some cases, this would improve the system's performance. But the effort/benefit ratio would need to be good enough to work on this for RVC2. We will likely focus most of our energy on RVC3 / RVC4 (_Q1 2024) in the future, as they provide a significant advantage (for the user) over RVC2. We will still maintain RVC2 in the future, but likely wouldn't add such large features to it.

Asa

Is the sensor crop pipeline different in RVC3? Are we able to have more of this kind of control there?

erik

Asa very likely no by default. I am not very familiar with sensors & their configs, but my guess would be that you'd need custom camera config for each such "sensor cropping", which would likely be manual work.

Asa

I am happy do do whatever is necessary as a manual task as long there is some kind of api exposed.Any hooks for that I can look at yet?