How to send array to neural network node from Script node?

Aabhanupr · 10 Mar

Hi,

I am trying to send an array to neural network node in depthai using Script node. Tried using the same principle as in this example.

To simplify things, I just have an Identity network (output = input). However, I just get [0.] as output from the NN though I pass [1.0] from the script node.

Please see simplified code below

This is my ONNX file in Netron app

Please let me know where I am going wrong. I'm not sure if I am setting the size of the NNData buffer properly. Let me know how we need to calculate this size.

Thanks in advance

jakaskerl · 11 Mar

abhanupr
NNData should be in bytes.
What happens if you try a different number instead of 1.0? Does the output change? Perhaps only the scaling in wrong during conversion.

Thanks,
Jaka

Aabhanupr · 11 Mar

jakaskerl

Thanks for the thoughts. I tried integers up to 21, and didn't have patience to try more ;-) Please see results below. See any pattern that can help?

Also an additional detail: My ONNX model is configured to be float16. I am assuming this is OK? Thanks

jakaskerl · 12 Mar

abhanupr
cc @JanCuhel on quick insight

Aabhanupr · 12 Mar

jakaskerl and @JanCuhel

If it helps to reproduce the issue I have created a github repo with the python + ONNX + blob files

Cheers

JanCuhel · 13 Mar

abhanupr Thanks, I'll check it out and get back to you!

JanCuhel · 14 Mar

Hey @abhanupr,

dai.node.NeuralNetwork expects input to be uint8. That’s why passing an integer (or a uint8 value) works as expected:

nn_data.setLayer("uint8", [21]) => 21.0
nn_data.setLayer("fp16", [21]) => 21.0

However, when you use a floating-point value with fp16 (like nn_data.setLayer("fp16", [21.0])), it doesn't convert the value to uint8. Instead, it interprets the first byte of the fp16 representation as a uint8, resulting in 64.0. This behavior is due to the absence of type checks in DepthAI v2.29.0.0. DepthAI v3 will address this issue, but for now, you'll need to ensure you're providing the correct data type.

Kind regards,
Jan

Aabhanupr · 14 Mar

Hi JanCuhel ,

Many thanks for the clarification.
That's disappointing though - I think the uint8 input limitation is quite a restrictive one. I was hoping this wouldn't be the case!

In fact, the way I landed here is as follows: I was trying to implement a Kalman filter on the on-board processor of the OAK device, to avoid having to stream data to the host and back. This is crucial to me as I am using the OAK as part of a real-time audio application and latency needs to be minimised. Now, as you would know, the Kalman filter being a recursive filter requires the state to be memorised every time step. I did not find a straightforward way to memorise the state of the neural network node. Therefore, as a workaround, I thought about passing the state as output, saving it in a Script node, and then feeding it back with one time step delay to the input of the NN. This is where I observed this datatype conversion issue, which is why I created this minimal reproducible example.

Is there a way to save the state of the NN node other than feeding back the delayed output via a Script node? If so, please let me know how to do it and I can happily avoid the Script node-based workaround.

If there is no other way, then it implies that one cannot implement any recursive/IIR temporal filter on non-uint8 data on the device! Am I right in thinking so, or am I missing something?

Best,
Abhi