image pipelines, compression, and filtering

pbarsic

I'm streaming RGB video from an OAK-D PoE to run an object detector on a GPU, and I'm having some image quality issues. I hope somebody here can shed some light on this for me. We are currently using a different camera for detection. I want to move to the OAK-D, but am having problems with fine scale image texture. Our networks respond strongly to texture, so we have to be careful with image pre-processing.

This image is an example from the other camera. It was saved as H265 video and extracted as a PNG. You can see the texture in the rock face and in the vegetation.

This is an example OAK-D image of the same scene taken at the same time. This was streamed from the camera as H265 video and extracted as PNG. If you zoom in 3x to 5x, you can see that image looks filtered or compressed. We've seen similar things with JPEG compression, and it has caused problems for our network. However, there are so many things to recommend the OAK-D that I want to see if this problem can be solved.

The RGB image pipeline indicates that there are different processing pipelines.

Can somebody tell me the differences among these image types?

Are the artifacts that I'm seeing applied in the "Image Post-Processing" module?

Can I manipulate the way that operates to get the data in the form that I need?

Thanks in advance,
Paul

erik

Hello pbarsic ,
I believe this is caused by the VideoEncoder when compressing the image. Regarding the ColorCamera and ISP - it's mostly just different frame types (isp=YUV420 / video,still = NV12 / preview=RGB) and sizes, so cropping is applied, so these outputs don't have any additional artifacts. One option would also be lossless JPEG encoding, which preserves about 40% of bandwidth and is, well, lossless, so there won't be any encoding artifacts. Thoughts?
Thanks, Erik

pbarsic

Hi Erik,

I'll try the lossless encoding and report back. It will be a couple weeks because other issues have taken priority.

Best,
Paul

pbarsic

I've tried the different image modes (uncompressed, lossless JPEG) but still see the same artifacts. I also experimented with acquiring a raw image, and the artifacts are not present there. Sample images are attached below. I am showing luminance only because I am interested in texture, not color.

This is the uncompressed image:

This is the raw image (debayered only):

From the color camera documentation, the uncompressed image is processed through the ISP and then the Image Post Processing module. The raw image, on the other hand, is not. When you look at the images as a whole, the processed image is more pleasing. However, the application to which we wish to apply this sensor is a rock detection algorithm, which relies upon the fine scale texture.

I want to know if there is a way to manipulate the filters in the ISP or Image Post Processing module to preserve the fine scale details.

Luxonis-Alex

@pbarsic What resolution was configured for the OAK-D camera color sensor, THE_1080_P or one of THE_4_K/THE_12_MP?

pbarsic

It was configured for THE_4_K. The code I used is based on the rgb_rull_resolution_saver demo. If it helps, I've pasted it below.

# modified from
# https://docs.luxonis.com/projects/api/en/latest/samples/VideoEncoder/rgb_full_resolution_saver/#demo

import time
from pathlib import Path
import cv2 
import depthai as dai 

# Create pipeline
pipeline = dai.Pipeline()

# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
xoutRgb = pipeline.create(dai.node.XLinkOut)
controlIn = pipeline.create(dai.node.XLinkIn)

xoutRgb.setStreamName("rgb")
controlIn.setStreamName("control")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)

# Linking
camRgb.still.link(xoutRgb.input)
controlIn.out.link(camRgb.inputControl)

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    # Output queue will be used to get the rgb frames from the output defined above
    qRgb = device.getOutputQueue(name="rgb", maxSize=30, blocking=True)
    controlQueue = device.getInputQueue("control")

    # Make sure the destination path is present before starting to store the examples
    dirName = "data_still"
    Path(dirName).mkdir(parents=True, exist_ok=True)

    start_time = time.time()
    while True:
        inRgb = ( 
            qRgb.tryGet()
        )  # Non-blocking call, will return a new data that has arrived or None otherwise

        if inRgb is not None:
            rgbFrame = inRgb.getCvFrame()
            cv2.imshow("rgb", rgbFrame)
            cv2.imwrite(f"{dirName}/{int(time.time() * 1000)}.png", inRgb.getCvFrame())
            if cv2.waitKey(1) == ord("q"):
                break
        else:
            ctrl = dai.CameraControl()
            ctrl.setCaptureStill(True)
            controlQueue.send(ctrl)
            print(f"The capture mode is {ctrl.getCaptureStill()}")
            time.sleep(0.1)

Luxonis-Alex

pbarsic Thanks!
For those fine artifacts, setting some ISP controls to 0 might help: sharpness, luma denoise, chroma denoise.
Can experiment with this app, starting it as: python3 main.py -raw -res 4k
https://github.com/luxonis/depthai-experiments/blob/master/gen2-color-isp-raw/main.py

Luxonis-Alex

Running with default controls (sharpness = 1, luma_denoise = 1, chroma_denoise = 1, range for all being 0..4), I get:

And with: sharpness = 0, luma_denoise = 0, chroma_denoise = 4:

pbarsic

Thanks, @Luxonis-Alex ! This is exactly what I need.
This is the boulder with default filters:

This is the boulder without:

AndresViana

pbarsic Congratulations Paul!

I am a beginner with Oak D camera and really interesting in to apply VideoEncoder as H265 video for other kind of texture but high quality.
Can you please share your code or some example in an easy way please?

my e-mail is andres.viana.viana@gmail.com

Thank you!

erik

For anyone else interested: We have just created docs about Image Quality, PR here.

Ushavilash

Hi Paul,

The differences among the image types in the RGB image pipeline typically involve various processing stages, including compression and filtering. The artifacts you're observing could indeed be applied during the "Image Post-Processing" module.

To mitigate these issues and achieve the desired image quality, you can experiment with manipulating the filters in the ISP or Image Post Processing module. Adjusting settings such as sharpness, luma denoise, and chroma denoise might help preserve fine-scale details for that check Image quality comparison — DepthAI Hardware Documentation 1.0.0 documentation (luxonis.com) and every third-party application such as https://jpegcompressor.com/. They maintain the quality as well as sharpness, luma denoise, and chroma denoise.