Resource Allocation in OAK-D

Wilbur

Version 1: OAK-D computes depth, calculates (x, y) coordinates, and converts them to world coordinates (x, y, z). Output: (x, y, z) world coordinates

Version 2: OAK-D computes depth and calculates (x, y) coordinates. Output: (x, y) coordinates along with a disparity map and a grayscale image from the right camera, and world coordinates are computed on the host side.

Q1. Does computing world coordinates within OAK-D (Version 1) require the use of most of the resources, as shown in Figure 1?

Q2. In Version 2, there seem to be available computational resources (Figure 2). Are there any other areas for improvement in terms of configuration?

jakaskerl

Hi @Wilbur
I believe spatial calculator should be relatively efficient and shouldn't impact performance much.
Are you using multi ROI calculations perhaps?

Thanks,
Jaka

Wilbur

jakaskerl ,

Version 1 and Version 2 use different resources for depth calculation, while the rest of the AI model architecture remains the same.

I don't understand what "multi ROI calculations perhaps" means. Could you please explain it to me?

Or is it because Version 1 utilizes the Script module?

jakaskerl

Wilbur

Version 1 shouldn't use the script node, if you are using the SpatialLocationCalculator node.

Wilbur I don't understand what "multi ROI calculations perhaps" means. Could you please explain it to me?

I'm talking about this feature: https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_calculator_multi_roi/.

Wilbur

jakaskerl ,

How can I dynamically update the (x, y) coordinates on an image to (x, y, z) world coordinates without using a script node?
Additionally, how much resources are available for neural networks to use in the absence of a depth node?
Does the usage of CMX affect the inference times of NN models?

jakaskerl

Wilbur How can I dynamically update the (x, y) coordinates on an image to (x, y, z) world coordinates without using a script node?

Not sure what you mean. SpatialLocationCalculator node will automatically convert x, y to x, y, z.

Wilbur Additionally, how much resources are available for neural networks to use in the absence of a depth node?

Well, however much more shaves and slices the stereo node takes when running. You can check that with DEBUG mode.

Wilbur Does the usage of CMX affect the inference times of NN models?

It does, because the NN's need use both CMX and shaves (same quantity), and if you lower the CMX slices, the shaves will lower, resulting in worse performance.

Thanks,
Jaka

Wilbur

jakaskerl ,
Not sure what you mean. SpatialLocationCalculator node will automatically convert x, y to x, y, z.

Things like this.

   image_manip_script->setScript(R"(
while True:
    config=SpatialLocationCalculatorConfigData()
    config.depthThresholds.lowerThreshold = 300
    config.depthThresholds.upperThreshold = 800
    cfg = SpatialLocationCalculatorConfig()
    face=node.io['face_det'].tryGet()
    if face is not None:
        x=face.getLayerFp16('pred')[0]
        y=face.getLayerFp16('pred')[1]
        if x-0.0390625 < 0 or y-0.0390625 < 0:
            continue
        if x+0.0390625 > 1 or y+0.0390625 > 1:
            continue
        topLeft = Point2f(x-0.0390625, y-0.0390625)
        bottomRight = Point2f(x+0.0390625, y+0.0390625)
        config.roi = Rect(topLeft, bottomRight)
        config.calculationAlgorithm = SpatialLocationCalculatorAlgorithm.AVERAGE
        cfg.addROI(config)
        node.io['to_manip'].send(cfg)
    )");

jakaskerl ,
It does, because the NN's need use both CMX and shaves (same quantity), and if you lower the CMX slices, the shaves will lower, resulting in worse performance.

This means that even though there are resources available, the usage of CMX has reached its limit.
Therefore, it's not possible to maximize the inference times of the neural network?

jakaskerl

Hi @Wilbur
Oh, you can't really. You'd need a custom decoding function for Calculator node which is currently only doable inside the script node and host side.

Wilbur Therefore, it's not possible to maximize the inference times of the neural network?

Depthai will tell you, you could benefit from increasing the number of shaves if they are available. If you get no warning this means you have set the shave count optimally to account for CMX slices used by other nodes and processes.

Thanks,
Jaka