Inconsistent image quality between 4k and 1080p

patrick52682

Hello, we are using a stationary Fixed-Focus POE OAK-D Pro to capture images. We are looking to get 1080p images from the camera. We have noticed that capturing at 4K and resizing (on the ISP) to 1080p gives higher quality images than taking images directly at 1080p. However, the resize on the ISP takes too long (on the order of ~300ms) for our application, so we would like to receive the 1080p frames.

Using this code, I created two captures of the same scene at 1080p & 4k:

import depthai as dai
import time
import cv2

def create_pipeline(resolution):
    # Create a pipeline
    pipeline = dai.Pipeline()

    # Create a ColorCamera node and set its properties
    cam = pipeline.createColorCamera()
    cam.setResolution(resolution)
    cam.setInterleaved(False)
    cam.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

    # Disable ISP processing features
    cam.initialControl.setManualExposure(800, 1200)
    cam.initialControl.setLumaDenoise(0)
    cam.initialControl.setChromaDenoise(0)
    cam.initialControl.setSaturation(0)
    cam.initialControl.setSharpness(0)

    xout = pipeline.createXLinkOut()
    xout.setStreamName("video")
    cam.video.link(xout.input)

    return pipeline

def capture_and_save(pipeline, filename):
    # Connect to the device and start the pipeline
    with dai.Device(pipeline) as device:
        # Get the video output queue
        q = device.getOutputQueue(name="video", maxSize=4, blocking=False)

        # Warm-up let capture a few frames to ignore any intial adjustments 
        for _ in range(5):
            q.get()

        # Capture and save the image
        img = q.get().getCvFrame()
        cv2.imwrite(filename, img)

# Create 4k image
pipeline_4k = create_pipeline(dai.ColorCameraProperties.SensorResolution.THE_4_K)
capture_and_save(pipeline_4k, "4k_res_isp.png")

# Create 1080p image
pipeline_1080p = create_pipeline(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
capture_and_save(pipeline_1080p, "1080p_isp.png")

I hope you can see below, there is a clear difference in visual quality (beyond 4k having more pixels). It seems as if there is some ISP processing happening that is more apparent at 1080p. I tried to control for every manual set function in CameraControl but this was the best set of settings I could find. I also read through this doc on improving image quality, but didn't see any improvement from the suggestions.

1080p:

4K:

Is there anything I might be missing, or any suggestions? I'm also wondering if we can bypass the ISP but keep the resolution crop to 4k/1080p. Thank you

jakaskerl

patrick52682 However, the resize on the ISP takes too long (on the order of _300ms) for our application, so we would like to receive the 1080p frames.

ISP shouldn't take that long, it's pretty performant. My measurements using the same configuration (4K res, 1/2 downscale to 1080p):

[1844301091CF5C1200] [192.168.118.110] [7.960] [ColorCamera(0)] [trace] Color ISP took '13.026026' ms.
[1844301091CF5C1200] [192.168.118.110] [7.965] [ColorCamera(0)] [trace] ColorCamera post processing took '3.459535' ms.

Thanks,
Jaka

patrick52682

Can you provide any information/code on how you achieved that? We are using the ImageManip node. By simply removing this block we see the latency measurement decrease from ~300ms to ~20ms. Is there a better way to do it?

        inference_resolution = (1920, 1080)

        resize_node = self.pipeline.create(dai.node.ImageManip)

        rgb_camera.video.link(resize_node.inputImage)

        resize_node.setMaxOutputFrameSize(

            inference_resolution[0] \* inference_resolution[1] \* 3,

        )

        resize_node.initialConfig.setResize(

            inference_resolution[0],

            inference_resolution[1],

        )

        resize_node.initialConfig.setKeepAspectRatio(False)

        resize_node.out.link(encoder.input)

… and then we receive the encoded frame

jakaskerl

Hi patrick52682 and graceziwei

patrick52682 Can you provide any information/code on how you achieved that?

I used the .setIspScale() to downscale the image:

camRgb.setInterleaved(False)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
camRgb.setIspScale(1, 2) # this should scale the 4K to 1080p. Use video output for VideoEncoder

I think also setting the setInterleaved() to false should help since image manipulations are more efficient on planar images.

graceziwei Is it expected that the image quality when capturing at 1080p looks worse than when capturing at 4K and resizing? Why is that happening?

I'm just going to paste the response froma similar thread:

These 2 points are quite relevant:

Higher resolution: 4K resolution (3840 x 2160 pixels) is four times higher than 1080p (1920 x 1080 pixels). When downscaling from 4K to 1080p using a 1/2 ISP, the image retains a higher amount of detail and sharpness, as more pixel data is being used to create the final image.

Downscaling algorithms: When using an ISP to downscale an image, the processor typically employs advanced algorithms to average pixel values and retain more detail. This results in a final image with lower noise and better color accuracy compared to quad binning, which is a simpler process that combines pixel data by averaging the values of four neighboring pixels into one.

Hope this explains it.

Thanks,
Jaka

graceziwei

Hi @jakaskerl, thanks for your help with understanding ISP resize latency.

I work with @patrick52682, and in addition to his questions about how to reduce resize latency, I was wondering: Is it expected that the image quality when capturing at 1080p looks worse than when capturing at 4K and resizing? Why is that happening?

I ask because we're ultimately just interested in getting high-quality 1080p images at low latency, whether that's getting the 4K resize to be faster or getting the images captured at 1080p to look better.

patrick52682

jakaskerl,

Thank you for your response. I just tested using setIspScale and the results look good. The e2e latency measurement (using this method) is 50-60ms, which is within a tolerable range for our application. The images do appear to look of higher quality using this over 1080p & ImageManip.

We appreciate your helpfulness here!

graceziwei

@jakaskerl - Thank you for the help with how to efficiently resize the images.

It's also great to know that 4K resized to 1080p looks better because more pixel data is used to create the final image. That makes sense. I had done a quick search of the forum to see if this question had been answered before, but I will search more carefully next time. Thank you!