Encoding RAW16 Depth values

AnthonyRomm

Hello!

We are using subpixel depth and I was wondering if there was any clever workaround to encode the RAW16 depth values. I've tried to pass the depth through an ImageManip node to change the frame type to GRAY8 but the device errors that this is invalid. I've also tried a custom Script node to use ImgFrame.setType() to GRAY8, which seems to work for a few frames but then silently stops working.

Another idea was to split the RAW16 into two GRAY8 ImgFrames and then combine them on host, but I haven't figured out a way to split them yet.

I figure there must be some way to change the RAW16 ImgFrame on the device to 1+ GRAY8/NV12 ImgFrame(s) > encode it > send to host > decode > convert back to uint16, but I'm stumped.

Any theories or ideas? Thank you!

jakaskerl

Hi AnthonyRomm
There really isn't a way to do this efficiently when using subpixel. Without it, streaming disparity is an option.
With RAW16, the only way you could do it is to use the script node, but that would be terribly slow since the HW is not meant to handle general purpose computations.
You can still do host side encoding if your device has onboard host (like the CM4 devices).

Thanks,
Jaka

AnthonyRomm

jakaskerl Thank you for the reply!

I would be curious to try a script node approach just to see what it's like. High-level, what were you thinking the script node would do? Take in the RAW16 ImgFrame and output a new GRAY8 ImgFrame? Or would it try to do the encoding too?

Host side encoding is a great idea and we can consider using a CM4 device.

jakaskerl

Hi AnthonyRomm
Only the splitting part. Even this will be a very demanding task for the node.

## asume image is in bytes
image = [int.from_bytes(data[i:i+2], byteorder='little') for i in range(0, len(data), 2)]

def split_to_8bit(image, width, height):
    high_bytes = [((value >> 8) & 0xFF) for value in image]
    low_bytes = [(value & 0xFF) for value in image]
    return high_bytes, low_bytes

Thanks,
Jaka

AnthonyRomm

Thank you Jaka for the help!

As you predicted, running that loop put a significant load on the script node.

I was able to manage something that kindof worked. Passing the depth into the following script node allowed the output depth to be video encoded.

while True:
    depth = node.io['depth_in'].get()  # loads a 1440x1080 RAW16 depth
    depth.setType(ImgFrame.Type.YUV400p)
    depth.setWidth(depth.getWidth() * 2)
    node.io['depth_out'].send(depth)

It could then be unencoded and reconstructed on host using

encoded_depth_data = depth_packet.getData()
depth_frame = cv2.imdecode(encoded_depth_data, cv2.IMREAD_GRAYSCALE)  # returns a 2880x1080 uint8
depth_frame = depth_frame.view(np.uint16).reshape(1080, 1440)  # reconstructs original 1440x1080 uint16

Seems hacky, but it is working consistently and giving a 50% reduction in bandwidth (w/ default mjpeg quality of 97) which is nice 🙂 However, it introduces more encoding/decoding processing and adds JPEG artifacts to the depth image which is something to consider.

Thank you all!