• DepthAI

  • Incorrect spatial location for yolov8 detections when using depthai-core

When using the DepthAI Core C++ API, the depth accuracy is terrible, and the measurements are often output as a single fixed value.

For more details, please refer to this GitHub link: luxonis/depthai-coreissues/1272.

    jakaskerl I built and ran the Spatial Location Calculator C++ example, but the depth values were still output as single fixed values, such as 5979mm and 7469mm. Additionally, the depth measurements were not accurate.

    However, when using Python, the depth values were correct, which suggests that this is not a calibration issue.
    (I have already performed calibration using a checkerboard.)

    I suspect that when using the C++ API, the camera's calibration extrinsic and intrinsic parameters might be default values instead of the actual calibrated ones.

    Could you check if the C++ API is correctly retrieving the calibration parameters?

        shAhn Perhaps you are using an older version of core? I checked and they both work the same for me. X and Y are 0 unless you move the roi around.

        Thanks,
        Jaka

            jakaskerl
            No, both depthai-core and the bootloader are up to date.

            X and Y are not my focus—I am only checking the Z values, but the depth measurements are inaccurate.
            For example, all objects detected within the range of 5.6m to 6.8m are measured as 7.469m, regardless of their actual position.

            I built the C++ example using the following steps:

            1. git clone https://github.com/luxonis/depthai-core.git

            2. git submodule update --init --recursive

            3. cmake -S. -Bbuild -D'DEPTHAI_BUILD_EXAMPLES=ON'
              cmake --build build

            4. cd build/examples
              ./SpatialDetection/spatial_location_calculator.cpp

            Could you verify if there is any issue with how the C++ API handles depth measurements compared to Python?

            Also, could you please take a look at issue
            [BUG] incorrect spatial location for yolov8 detections when using depthai-core (#1272)
            in this link: https://github.com/luxonis/depthai-core/issues?

                shAhn
                Could you please try running the following C++ code and check if the depth values are correctly displayed during object detection?

                I used yolov8n_coco_640x352.blob in the CMakeLists.txt.

                $$
                #include <chrono>
                #include "utility.hpp"
                #include "depthai/depthai.hpp"

                // Label map for YOLO
                static const std::vector<std::string> labelMap = {
                "person", "bicycle", "car", "motorbike", "aeroplane", "bus", "train", "truck", "boat",
                "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse",
                "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag",
                "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove",
                "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon",
                "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza",
                "donut", "cake", "chair", "sofa", "pottedplant", "bed", "diningtable", "toilet", "tvmonitor",
                "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink",
                "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"
                };

                int main(int argc, char** argv) {
                std::string nnPath(BLOB_PATH);

                if(argc > 1) {
    nnPath = std::string(argv[1]);
}
printf("Using blob at path: %s\n", nnPath.c_str());

dai::Pipeline pipeline;
auto camRgb = pipeline.create<dai::node::ColorCamera>();
auto spatialDetectionNetwork = pipeline.create<dai::node::YoloSpatialDetectionNetwork>();
auto monoLeft = pipeline.create<dai::node::MonoCamera>();
auto monoRight = pipeline.create<dai::node::MonoCamera>();
auto stereo = pipeline.create<dai::node::StereoDepth>();

// Camera and network settings
camRgb->setPreviewSize(640, 352);
camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
camRgb->setInterleaved(false);
camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);

monoLeft->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
monoLeft->setCamera("left");
monoRight->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
monoRight->setCamera("right");

stereo->setDefaultProfilePreset(dai::node::StereoDepth::PresetMode::HIGH_DENSITY);
stereo->setDepthAlign(dai::CameraBoardSocket::CAM_A);
stereo->setOutputSize(monoLeft->getResolutionWidth(), monoLeft->getResolutionHeight());

spatialDetectionNetwork->setBlobPath(nnPath);
spatialDetectionNetwork->setConfidenceThreshold(0.5f);
spatialDetectionNetwork->setBoundingBoxScaleFactor(0.5);
spatialDetectionNetwork->setDepthLowerThreshold(100);
spatialDetectionNetwork->setDepthUpperThreshold(5000);
spatialDetectionNetwork->setNumClasses(80);
spatialDetectionNetwork->setIouThreshold(0.5f);

// Linking nodes
monoLeft->out.link(stereo->left);
monoRight->out.link(stereo->right);
camRgb->preview.link(spatialDetectionNetwork->input);
stereo->depth.link(spatialDetectionNetwork->inputDepth);

dai::Device device(pipeline);
auto previewQueue = device.getOutputQueue("rgb", 4, false);
auto depthQueue = device.getOutputQueue("depth", 4, false);

while(true) {
    auto imgFrame = previewQueue->get<dai::ImgFrame>();
    auto depth = depthQueue->get<dai::ImgFrame>();

    cv::Mat frame = imgFrame->getCvFrame();
    cv::Mat depthFrame = depth->getFrame();
    
    cv::imshow("depth", depthFrame);
    cv::imshow("rgb", frame);

    if(cv::waitKey(1) == 'q') {
        break;
    }
}
return 0;

                }
                $$

                Could you check if the depth values are accurate when performing object detection using this code?

                Let me know if you experience the same issue with incorrect or fixed depth values.

                    shAhn
                    The code you have sent seems incomplete. Can you send the full MRE?

                    Thanks,
                    Jaka