ISP Size vs Video Size

jjc999

Hi, I am having some difficulty understanding how the ISP size, camera resolution, and video size all work together. I am using the OAK-1 MAX and have been experimenting with streaming video at 1080p and 4K. I have called setVideoSize() to make the video window size 1920x1080. When I set the camera resolution to 4K, I downscale the ISP by 1/2 to make the ISP have the same resolution as the video window. The resulting feed looks great (high quality) and I get the expected FOV. However, when I set the camera resolution to 1080p, I do not downscale the ISP (already at 1920x1080 since the camera resolution has been set to this) but see a cropped in video feed that has a much smaller FOV. This confuses me because I am expecting to need to do anything to the ISP size to see the full image. If I downscale the ISP by 1/2 so the ISP size is now 960x540, the video has the expected FOV and is higher quality. This doesn't make sense though because I would not expect I would need to downscale the ISP in this case. Let me know if you need more information on my end to understand my question or if you want code snippets.

Thanks,

Jordan

erik

The resulting feed looks great (high quality) and I get the expected FOV.

Hi jjc999 , Yes, that's expected. I asked gpt4:

Using 4K camera resolution and 1/2 isp downscaling provides better image quality compared to 1080P (quad binning). Why is that the case?

These 2 points are quite relevant:

Higher resolution: 4K resolution (3840 x 2160 pixels) is four times higher than 1080p (1920 x 1080 pixels). When downscaling from 4K to 1080p using a 1/2 ISP, the image retains a higher amount of detail and sharpness, as more pixel data is being used to create the final image.

Downscaling algorithms: When using an ISP to downscale an image, the processor typically employs advanced algorithms to average pixel values and retain more detail. This results in a final image with lower noise and better color accuracy compared to quad binning, which is a simpler process that combines pixel data by averaging the values of four neighboring pixels into one.

jjc999

Ok great, thanks Eric. As a follow-up question, shouldn't I get a fairly similar image in terms of FOV when downscaling the 4K image to 1080p and showing the 1080p video on a 1920x1080 window? When I set the resolution to 1080p with no ISP scaling and show the video on a 1920x1080 window, the frames are pretty heavily cropped. I would expect to see nearly the same image (in terms of FOV) as showing the downscaled 4K image. However, if I downscale the 1080p image by 1/2 and show that on a 1920x1080 window, the video looks very similar to downscaling the 4K image. Do you know why this would be the case?

erik

jjc999 yes, it should be the same FOV, could you share the minimal repro code?

jjc999

I apologize, as I went back to gather my code I realized I made a simple mistake. I am using the OAK-1 MAX, but the camera does not support 1080p resolution. When I try to set the resolution to 1080p, it says that it picks 4K by default instead. Now it makes sense why I need to still scale the ISP in this case.

I have another question about this though. When I try to set the resolution to 1080p on the OAK-1 MAX, it says that it is not a supported resolution and instead sets it to 4K. However, when I print out the ISP size it still lists it as 1920x1080 even though this isn't the resolution it is actually set to (if I set the resolution to 4K, the ISP would be 4K size). So even though the resolution is updated to support the camera, does the ISP size not also get updated accordingly (at least when I ask it the size)? It must be updating this in the background (this is what my experience suggests), but not when I call cam_rgb.getIspSize(). Is there a way to see what the actual ISP size and resolution for the camera are independent of what I tried to set them to?

Update: with some more thought, I realized that it is because I am referencing the resolution and ISP size from my ColorCamera node (cam_rgb). Once I send this in the pipeline to the device, this must be when the resolution and ISP size are updated and they are not updated in the ColorCamera node. Is there a way for me to access these values from the device?

erik

Hi jjc999 ,
My guess is that it will set to 4K (as 1080P isn't supported), but it will also set the ISP downscaling to achieve 1080P, as that's what it was set to. To get the supported sensor resolutions, you can use:

with dai.Device(pipeline) as device:
    for sensor in device.getConnectedCameraFeatures():
        for config in sensor.configs:
            print(f"{sensor.name} Resolution: {config.width}x{config.height}")

Thanks, Erik