• Community
  • image pipelines, compression, and filtering

I'm streaming RGB video from an OAK-D PoE to run an object detector on a GPU, and I'm having some image quality issues. I hope somebody here can shed some light on this for me. We are currently using a different camera for detection. I want to move to the OAK-D, but am having problems with fine scale image texture. Our networks respond strongly to texture, so we have to be careful with image pre-processing.

This image is an example from the other camera. It was saved as H265 video and extracted as a PNG. You can see the texture in the rock face and in the vegetation.

This is an example OAK-D image of the same scene taken at the same time. This was streamed from the camera as H265 video and extracted as PNG. If you zoom in 3x to 5x, you can see that image looks filtered or compressed. We've seen similar things with JPEG compression, and it has caused problems for our network. However, there are so many things to recommend the OAK-D that I want to see if this problem can be solved.

The RGB image pipeline indicates that there are different processing pipelines.

Can somebody tell me the differences among these image types?

Are the artifacts that I'm seeing applied in the "Image Post-Processing" module?

Can I manipulate the way that operates to get the data in the form that I need?

Thanks in advance,
Paul

  • erik replied to this.

    Hello pbarsic ,
    I believe this is caused by the VideoEncoder when compressing the image. Regarding the ColorCamera and ISP - it's mostly just different frame types (isp=YUV420 / video,still = NV12 / preview=RGB) and sizes, so cropping is applied, so these outputs don't have any additional artifacts. One option would also be lossless JPEG encoding, which preserves about 40% of bandwidth and is, well, lossless, so there won't be any encoding artifacts. Thoughts?
    Thanks, Erik

    Hi Erik,

    I'll try the lossless encoding and report back. It will be a couple weeks because other issues have taken priority.

    Best,
    Paul

    a month later

    I've tried the different image modes (uncompressed, lossless JPEG) but still see the same artifacts. I also experimented with acquiring a raw image, and the artifacts are not present there. Sample images are attached below. I am showing luminance only because I am interested in texture, not color.

    This is the uncompressed image:

    This is the raw image (debayered only):

    From the color camera documentation, the uncompressed image is processed through the ISP and then the Image Post Processing module. The raw image, on the other hand, is not. When you look at the images as a whole, the processed image is more pleasing. However, the application to which we wish to apply this sensor is a rock detection algorithm, which relies upon the fine scale texture.

    I want to know if there is a way to manipulate the filters in the ISP or Image Post Processing module to preserve the fine scale details.

    @pbarsic What resolution was configured for the OAK-D camera color sensor, THE_1080_P or one of THE_4_K/THE_12_MP?

    It was configured for THE_4_K. The code I used is based on the rgb_rull_resolution_saver demo. If it helps, I've pasted it below.

    # modified from
    # https://docs.luxonis.com/projects/api/en/latest/samples/VideoEncoder/rgb_full_resolution_saver/#demo
    
    import time
    from pathlib import Path
    import cv2 
    import depthai as dai 
    
    # Create pipeline
    pipeline = dai.Pipeline()
    
    # Define sources and outputs
    camRgb = pipeline.create(dai.node.ColorCamera)
    xoutRgb = pipeline.create(dai.node.XLinkOut)
    controlIn = pipeline.create(dai.node.XLinkIn)
    
    xoutRgb.setStreamName("rgb")
    controlIn.setStreamName("control")
    
    # Properties
    camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
    
    # Linking
    camRgb.still.link(xoutRgb.input)
    controlIn.out.link(camRgb.inputControl)
    
    # Connect to device and start pipeline
    with dai.Device(pipeline) as device:
    
        # Output queue will be used to get the rgb frames from the output defined above
        qRgb = device.getOutputQueue(name="rgb", maxSize=30, blocking=True)
        controlQueue = device.getInputQueue("control")
    
        # Make sure the destination path is present before starting to store the examples
        dirName = "data_still"
        Path(dirName).mkdir(parents=True, exist_ok=True)
    
        start_time = time.time()
        while True:
            inRgb = ( 
                qRgb.tryGet()
            )  # Non-blocking call, will return a new data that has arrived or None otherwise
    
            if inRgb is not None:
                rgbFrame = inRgb.getCvFrame()
                cv2.imshow("rgb", rgbFrame)
                cv2.imwrite(f"{dirName}/{int(time.time() * 1000)}.png", inRgb.getCvFrame())
                if cv2.waitKey(1) == ord("q"):
                    break
            else:
                ctrl = dai.CameraControl()
                ctrl.setCaptureStill(True)
                controlQueue.send(ctrl)
                print(f"The capture mode is {ctrl.getCaptureStill()}")
                time.sleep(0.1)

      Running with default controls (sharpness = 1, luma_denoise = 1, chroma_denoise = 1, range for all being 0..4), I get:

      And with: sharpness = 0, luma_denoise = 0, chroma_denoise = 4:

      3 months later
      4 months later

      pbarsic Congratulations Paul!

      I am a beginner with Oak D camera and really interesting in to apply VideoEncoder as H265 video for other kind of texture but high quality.
      Can you please share your code or some example in an easy way please?

      my e-mail is andres.viana.viana@gmail.com

      Thank you!

      a year later

      Hi Paul,

      The differences among the image types in the RGB image pipeline typically involve various processing stages, including compression and filtering. The artifacts you're observing could indeed be applied during the "Image Post-Processing" module.

      To mitigate these issues and achieve the desired image quality, you can experiment with manipulating the filters in the ISP or Image Post Processing module. Adjusting settings such as sharpness, luma denoise, and chroma denoise might help preserve fine-scale details for that check Image quality comparison — DepthAI Hardware Documentation 1.0.0 documentation (luxonis.com) and every third-party application such as https://jpegcompressor.com/. They maintain the quality as well as sharpness, luma denoise, and chroma denoise.