General DepthAI v3 thoughts so far

TheHiddenWaffle

Just thought I would put some of my feedback for v3 down in writing so I don't forget. Have been rewriting and old application with a lot of complexity for RVC4 and the depthai-nodes repo is amazing! Being able to make custom host nodes and use numpy for postprocessing is so wonderful.

If you add a script node to the pipeline, the visualizer(dai.RemoteConnection) will just not display any frames and throws no errors? EDIT: not adding a script node, but the moment you connect it to the rest of the graph this occurs.

This is a much older complaint but: All messages that store arrays of smaller structs(IE: ImgDetections) have docs that are relatively unhelpful implementation-wise. When writing script nodes and parsing such messages you end up needing to dig through to the nested struct, having a link to the nested struct on the page for the parent would be helpful.

Adding Caveats to WIP nodes in the docs would be nice, got my pipeline all rebuilt for RVC4 just to have it let me know that ObjectTracker node "doesn't exist"(direct quote from the error).

Running 3.0.0-rc2 but I've seen this issues since alpha 14.

jakaskerl

Hi TheHiddenWaffle
Thanks for the feedback! We'll keep in mind and update the documentation.

TheHiddenWaffle

jakaskerl One thing I really need clarification on as well is the new conversion process.

I have the RTMPose3D ONNX that accepts RGB img float16[1,3,384,288]. Based off my best understanding of the docs, I built the config.json as follows:

"inputs": [ { "name": "input", "dtype": "float16", "input_type": "image", "shape": [ 1, 3, 384, 288 ], "layout": "NCHW", "preprocessing": { "mean": [ 123.675, 116.28, 103.53 ], "scale": [ 58.395, 57.12, 57.375 ], "dai_type": "RGBF16F16F16p" } } ],

My first question: dai_type, is it(in this context of the onnx model) correct?

So then I move on to RVC4 conversion on hub, all my stuff is prefilled by this JSON(nice!) except for the encoding types

Do I just select none->none here? Or BGR->RGB? It seems like the default for RVC4 in all the examples shows an input of BGR888i, is this taken care of by the converter? I guess it feels like a lot more guidance is needed because when the documentation and UI say "the model" at various points I have no idea whether they mean "the base ONNX model" or "the post RVC2/4 conversion model"

TheHiddenWaffle

jakaskerl

Some more stuff:

SpatialDetectionNetwork::build() has some ambiguities and assumptions that limit it's usefulness:

It only accepts Camera inputs, which would be great if the ResizeMode was configurable and the default documented somewhere(based on some quick digging it seems to be CenterCrop which is not optimal)
It only takes StereoDepth, which is not a huge issue as the OAK4 TOF is not built yet but even so the RVC4 device could potentially be controlling an RVC2 device with a TOF sensor.

TheHiddenWaffle

I should add that post conversion it seems that the modelconverter adjusts dai_type as well as adds the flags reverse_channels and interleaved_to_planar back in(which I had remove because Hub CLI give deprecation warnings), are these generated based on the ONNX dai_type vs desired RVC4 dai_type?

Currently the modelconverter still feels like a very large black box.

jakaskerl

TheHiddenWaffle
cc @KlemenSkrlj

KlemenSkrlj

Hi @TheHiddenWaffle ,
Thanks for the feedback and we'll work on better documentation and UI improvements so that the conversion doesn't look like a black box anymore. To answer your question:

dai_type, is it(in this context of the onnx model) correct?

Yes, this is correct in the context of your ONNX model. The NNArchive (the config.json file inside the tar.gz) describes the model that it refers to in its model.metadata section. So in your case, the NNArchive you shared refers to the ONNX model, and since the input to the ONNX model is float16 RGB then the dai_type should be RGBF16F16F16p (p because input is planar).

Do I just select none->none here? Or BGR->RGB?

This is something we'll improve the UI on to automatically detect the Encoding From part from the given NNArchive of the base model. Since this is not done yet (will be very soon) you have to for now manually set it properly. In your case, since the base model is trained on RGB images you have to set the Encoding From to RGB. As for the Encoding To depends on your use case but you are correct, we usually use BGR color space in our examples - the reason is mostly historical because OpenCV uses BGR by default so it makes the interaction between it and DepthAI more straightforward.

modelconverter adjusts dai_type as well as adds the flags reverse_channels and interleaved_to_planar back in(which I had remove because Hub CLI gives deprecation warnings), are these generated based on the ONNX dai_type vs desired RVC4 dai_type?

The reverse_channels and interleaved_to_planar flags are deprecated but still put into NNArchives by default (set to null). They were introduced first but we then replaced both with single dai_type flag (the first part of dai_type is the color space and the last part, "p" or "i", is the interleaved/planar information). The logic in the modelconverter is that first dai_type is checked and if possible we infer everything based on it but to keep back-compatibility we still have a fallback to these deprecated flags.
And yes, it is expected that the dai_type of ONNX NNArchive and of RVC4 NNArchive is different because NNArchive is tied to the model and we automatically handle that the RVC4 dai_type is then the one of your actual RVC4 exported model. The dai_type of the target (in case of ONNX-> RVC4 conversion the target is RVC4) is inferred based on the selected Encoding to value (for color space ie. RGB or BGR or Gray etc.) and target platform (RVC4 needs interleaved input whereas RVC2 needs planar input so that is the "i" or "p" at the end of the dai_type value)

Feel free to let m know if there is anything still unclear.
Klemen

TheHiddenWaffle

KlemenSkrlj Thank you, this is exactly what I needed! Model outputs are looking beautiful now.

TheHiddenWaffle

KlemenSkrlj

Sorry one more thing, it appears that the hub modelconverter CLI doesn't allow for quantization: none, and it currently defaults to random if none is specified. Am I correct about that or have I just overlooked a switch?

jakaskerl

TheHiddenWaffle
There are plans to make it more generalized to all types of input. Thanks for additional push to make it happen.
cc @MatevzMorato

Thanks,
Jaka

KlemenSkrlj

TheHiddenWaffle This depends on the platform and precision you want to export your model for. For RVC2 and RVC3 models we don't use calibration datasets as the models are not quantized. Same goes for RVC4 FP32 and FP16 exported models, these have disable_calibration flag set to True. Whereas if you are exporting RVC4 model with INT8 precision (which is the default export for RVC4) you do need the calibration data to quantize the model with and this can't be None (or if it is random data will be used).
I assume you were probably confused about the logs where it said that "Random calibration is being used for input..." even if you tried exporting an FP16 precision model. This was due to bad logging on our part and we have since fixed the issue and will be added in the next release.

MatevzMorato

SpatialDetectionNetwork::build() has some ambiguities and assumptions that limit it's usefulness:

It only accepts Camera inputs, which would be great if the ResizeMode was configurable and the default documented somewhere(based on some quick digging it seems to be CenterCrop which is not optimal)
It only takes StereoDepth, which is not a huge issue as the OAK4 TOF is not built yet but even so the RVC4 device could potentially be controlling an RVC2 device with a TOF sensor.

This is some great feedback @TheHiddenWaffle!
We will improve the build() function with the ability to set the type of crop mode and add one for the ToF too.

Note that you can still use the old school type of linking, it's just a bit more involved due to having to know the NN model input size upfront, but it's still doable. See this for an example luxonis/depthai-coreblob/v3_windows_improvements/examples/python/ModelZoo/model_zoo.py#L12-L13.