Undistortion compensation in SpatialDetectionNetwork (and some other questions)

frmssmd

Hi,

I am using a SpatialDetectionNetwork. I run the mono cams at 640x400 (1.6 AR) and the color camera at 768x576 (1.33 AR) on an OAK-D Wide (OV9782 color sensor).

For reference, both of those cameras have a DFOV / HFOV / VFOV of 150° / 127° / 79.5°.

My problem:

In the image, a trailer (maybe 20 ft away?) has been spotted beyond the hill. The trailer gets a bounding box (correctly) - so the nueral network gets an A+. The SpatialDetectionNetwork gets an F, because instead of giving that detection a location of 0, it places it just right on the hill.

Or here, where a car is spotted on the side of the FOV. The truck is clearly further than 3.6m. I don't even understand how the SpatialDetectionNetwork found any points in this box which are 3.6m away.

1) I do not undistort either of the cameras. But I assume StereoDepth probably does undistort? When the nueral network bounding box outputs (which are done on an original image) are overlayed on the undistorted depth image, would that lead to issues? (This could be impacting my truck question, where the color camera places the truck lower than on the depth image.)
2) How does it handle the difference in aspect ratio? Just a stretch? The detections come off the Nueral network as xmin ymin etc, aka fractions of a frame. It that goes right into the depth as fractions of a frame, then it should line up, regardless of my resolutions.
3) I know there is a small difference between FOV of stereo and FOV of cameras. At 3-5m, it should be ₀.5 degrees of FOV lost on each side. Does SpatialDetectionNetwork account for this? Should I?
4) What is the default value of BoundingBoxScaleFactor? For these tests I was not setting it myself.

frmssmd

also notice how in the truck image, even though the item is all the way to the left of the FOV, when plotting from above the location of the item is not even close to the FOV bounds!

(the x coordinate (vertical axis) comes directly from the detection.spatialCoordinates.x, but the z coordinate (horizontal axis) comes from math.hypot(d.spatialCoordinates.z, d.spatialCoordinates.y) - the camera is pointed down 30 degrees, so to compensate the depth for the tilt I add in the y coordinate)

Oskar-Luxonis

Hey @frmssmd - SpatialDetectionNetwork should already account for the RGB/depth transform for you. In v3, it enables undistortion on the NN camera path and aligns stereo depth to that path internally. The default BoundingBoxScaleFactor is 1.0, and the default spatial algorithm is MEDIAN.

For your two examples, the behavior looks more like ROI contamination than an NN issue. In the trailer image, the box likely contains a lot of nearer hill pixels, so the median depth lands on the hill. In the truck image, the object is very close to the edge of the RGB FOV, where valid stereo coverage is usually reduced, so the spatial ROI can end up sampling nearer ground/foreground pixels instead of the truck.

On your specific questions:

StereoDepth does rectification/alignment internally. If you are using SpatialDetectionNetwork normally, RGB-depth alignment is supposed to be handled for you.
Aspect-ratio differences are not handled as a simple stretch of normalised bbox coordinates, the pipeline uses frame transformation metadata/remapping.
The stereo/depth-valid FOV is still smaller than the RGB full FOV at the edges, and that can absolutely cause bad spatial results near the borders. SpatialDetectionNetwork can remap the ROI, but it cannot create valid depth where stereo has none.
Default BoundingBoxScaleFactor is 1.0.

I would suggest visualizing detection.boundingBoxMapping on the depth frame first. If that mapped ROI sits on the hill/ground instead of the detected object, that explains the result immediately. In practice, the usual fixes are reducing BoundingBoxScaleFactor, trying MIN instead of MEDIAN for some targets, and rejecting detections near the RGB edges when they fall outside reliable stereo coverage.

Best,
Oskar

frmssmd

Oskar, thanks for your reply, and especially thanks for those suggestions. Will do!

Oskar-Luxonis

frmssmd - happy to help!

aljaz

Hi @frmssmd,

Would you mind sharing which depthai version you are using? In depthai==3.4.0 we changed the logic on how spatial locations are calculated. Regarding your questions:

Yes StereoDepth applies undistortion and rectification to left and right. In older depthai versions the code assumes RGB is undistorted. After depthai 3.4.0 it should throw an error.
In depthai < 3.4.0 the code assumed the same aspect ratio and resize operation. After 3.4.0 it checks that RGB and depth frames are perfectly aligned (same size, distortion coeffs, intrinsics etc. )
It does account for the FOV yes.

From your screenshots my guess as to what is happening:

NeuralNetwork detects on wide FOV in fractions of a frame.
StereoDepth automatically undistorts and rectifies left and right cameras, lowering the FOV.
SpatialCoordinates then take the fractions of bboxes and use those on the smaller FOV stereo depth frame. That would result as to why the truck is not on FOV bounds in the second example. It would also explain the distance of 3.6m as the actual depth is taken in some smaller FOV region and calculates some nonsense depth region of grass.

One simple test you can do to validate the above is to draw bboxes on the depth frame aswell and you will be able to see which region the bbox takes. For further debugging I would need to see the part of the pipeline where you link cameras and SpatialDetectionNetwork + which depthai you are using.

If you are using newer depthai, you can better control the bahavior by using NN + DetectionParser + SpatialLocationCalculator nodes instead of SpatialDetectionNetwork (its just a wrapper around those three). Here is an example that would fit for your use-case (just switch the model out for your model in the NN).

Hope this helps,

Aljaz

frmssmd

Thank you Aljaz - am I to understand that the example you linked is a segmentation model? I was under the impression that a segmentation model needed postprocessing on the host that the camera could not do. (On RVC2). But segmentation would be much better than just using bounding boxes - is this the recommended approach? (It looks like segmentation support was added in 3.4.0 - so makes sense why I thought it wasn't available at the time I developed this.)

I am using depthai==3.3.0. but, I think I will upgrade to 3.7.1, for this alignment issue, and the autocalibration that comes with 3.4.0.

Thanks!

frmssmd

also just to be clear - your guess

SpatialCoordinates then take the fractions of bboxes and use those on the smaller FOV stereo depth frame. That would result as to why the truck is not on FOV bounds in the second example. It would also explain the distance of 3.6m as the actual depth is taken in some smaller FOV region and calculates some nonsense depth region of grass.

would only be valid while I am on 3.3.0 - once I move to 3.7.1, this is not possible anymore, right?

Thank you!