• DepthAI-v2

  • Sending depth node depth to neural network

Hello,

How can I create a custom NN model which can operate on the depth image as an input? More specifically, what types do I assign to the NN model (uint16 or float32 or float16) and how?

Here's where I got to:
I've created a basic neural network using onnx and converted it to blob. I used id on depth_node.disparity and everything seemed to work as I expected. However when I pass the depth_node.depth input to it I get incorrect data, which seems like it's trying to read an array of uint16's as an array of uint8's.

Here's how I try to pass it in:

depth = pipeline.create(dai.node.StereoDepth)
nn = pipeline.create(dai.node.NeuralNetwork)
nn.setBlobPath(...)
depth.depth.link(nn.input)

As a test I made a neural network that just copies the input and returns it. I printed the outputs of the NN and the depth image and go the following:
NN result: [122.0, 19.0, 122.0, 19.0, 122.0, 19.0, 122.0, 19.0, 122.0, 19.0, ...]
Depth image: [4986 4986 4986 4986 4986 4986 4986 4986 4986 4986, ....]
It seems that the network is being fed the the depth image as uint8's instead of unit16s: 4986 = 19 * 256 + 122.

How do I get the input correctly? Onnx supports uint16 as an input type, however when I tried it I wasn't able to convert the network to a blob using blobconverter.from_onnx. The data_type of the method can only be FP32 or FP16. If I try U16 or UINT16 or UI16 it gives me this error: Unknown precisions specified: U16.. If I try FP32 or FP16 and the onnx model has uint16 type then it throws a different error. If I try FP32 or FP16 on the onnx model the input doesn't get properly formatted.

I found a hacky work-around. It would be nice if there was a better way to do it, but if not here's what I did:

I set the input to be double the size: h.make_tensor_value_info('x', TensorProto.FLOAT, [HEIGHT, WIDTH, 2]), then added a few onnx layers to combine the low and the high bits together. Note the shape ends up having a redundant dimensions at the end.

nodes = [
  onnx.helper.make_node('Constant', inputs=[], outputs=['c_256'], value=numpy_helper.from_array(np.array([256], dtype=np.float32))),
  onnx.helper.make_node('Split', inputs=['x'], outputs=['x_low', 'x_high'], axis=2),
  onnx.helper.make_node('Mul', inputs=['x_high', 'c_256'], outputs=['x_high_multed']),
  onnx.helper.make_node('Add', inputs=['x_low', 'x_high_multed'], outputs=['xx']),
  # Rest of your processing
]
  • erik replied to this.