Hi,

I currently use Oak-D S2 FF cameras and I would like to put two pointclouds together with ICP. First I would like to make a realistic correct pointcloud but I cannot. I use the code from the examples on github, but I get deformed results. I also tried to make a pointcloud with the Oak-D and Oak-D Pro and with another 3 Oak-D S2 camera but the same problem with every one of them. I tried to change the parameters and use the filters too but still deformed result. It looks like okay from the front but when I take a look at it from the side it's useless. I attached some photoes. to demonstrate the problem. I mean what is wrong with my face from the side. I assure you it does not look like this in real life. I also tried it with a stable doll, same problem. Is it a parameter problem I just cannot figure the right values?

Thanks,

Szabina

    Hi @jakaskerl

    Okay so, sorry about my late reply. I wanted to try the multi device example. You can see my results in the images, where there is a calibration board on a chair. So I calibreted two cameras and made two pointclouds and I viewed them in meshlab. The board was totally flat but in the images about the mered pointclouds it seems like there is a curve in the calibration board. I don't know… I don' think this should be the result.

    On the other hand. I made a pointcloud with only one camera today from the doll (I also attached today's result images and you can see the wavy behavior) with these parameters and setup:

    lrcheck = True
    extended = False
    subpixel = True
    median = dai.StereoDepthProperties.MedianFilter.KERNEL_5x5

    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
    stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)

    config.postProcessing.speckleFilter.enable = False
    config.postProcessing.speckleFilter.speckleRange = 50
    config.postProcessing.temporalFilter.enable = True
    config.postProcessing.spatialFilter.enable = True
    config.postProcessing.spatialFilter.holeFillingRadius = 2
    config.postProcessing.spatialFilter.numIterations = 1
    config.postProcessing.thresholdFilter.minRange = 400
    config.postProcessing.thresholdFilter.maxRange = 200000
    config.postProcessing.decimationFilter.decimationFactor = 1

    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camRgb.initialControl.setManualFocus(130)

    depth_trunc=3.0
    depth_scale=1000.0

    Thank you,
    Szabina




    • erik replied to this.

      Hi szzekany ,
      2 Things I'd suggest:

      • Decimation mode should be median instead of (default) pixel skipping, which should provide good filtering. 
         config.postProcessing.decimationFilter.decimationMode = dai.RawStereoDepthConfig.PostProcessing.DecimationFilter.DecimationMode.NON_ZERO_MEDIAN
      • Configure filters for your application - max range (thresholdFilter) of 20m is far too much, I'd go with 6m max, after that the accuracy drops quite fast.

      Let us know if this helps to improve your pointclouds.

        Hi @erik ,

        Thank you for your help. I tried these settings. I think the result is better right now. I attached an image. The body is so much better not so wavy but the neck, face and forhead is still has a weird form. It would be important to reconstruate the face in the best way possible but they look really weird. Do you have a tip why is that?

        Thanks,
        Szabina

        • erik replied to this.

          Hi szzekany ,
          It seems like besides some noise (that you got rid of), depth accuracy also isn't the best. We are now updating our calibration process which should improve the accuracy. We can replace your camera with a newly calibrated device. Please send an email to support@luxonis.com with the link to this post and we can arrange the replacement.
          Thanks, Erik

            Dear @erik

            Thank you for your support and helpfulness. We ordered 12+2 cameras. Our main goal is to build a 3D scanner for the full human body with 4 colums and we plan to put 3 cameras on each colume (and we wanted to have 2 more cameras in case one of the 12 used cameras went wrong).
            First we wanted to make a useable and mostly accurate pointcloud with one camera this is where we are now and have problems.

            I only wanted to inform you beacuse like this maybe you have a bigger picture about our goal and can be more helpful. I have three questions.

            1. Can't we calibrate the cameras with the charuco board and the calibration python script? Do you need to replace the cameras anyway?

            2. I also tried the multi camera calibration and rgb pointcloud code from here: https://github.com/luxonis/depthai-experiments/tree/master/gen2-multiple-devices/rgbd-pointcloud-fusion) I put the calibration chessboard onto the floor and I tried to scan a simple box, with two cameras, about 40° far from each other in the same height from the ground. The result was pretty great regarding the alignment, but the same problem occuerd with the depth accuracy, so the boxes were wavy. My questiun would be: If the depth accuracy problem works out and we get a good result and want to scan a whole body, is there a way to calibrate all of the 12 cameras to each other? It can be tricky and not easy because all of the cameras won't see the calibration board for sure, it is impossible. Do we need a calibration box? Or to calibrate the camerast to each other in chain somhow?

            3. In my head there are two kind of calibration. One is where we calibrate each/one camera, we get the matrix with the intrinsic parameters about the focal length, etc. and this calibration can helpsű correct the distorsions too. The other calibration gives a transformation matrix which wirtes down the different positions of two or more cameras from each other? Am I right or I don't have the right picture about this?

            Thank you in advance,
            Szabina

            9 days later

            Hi @szzekany ,
            I apologize for the late response, seems like I didn't get a notification on it.

            1. Oh it is, we have a documentation on it here. The video is a bit out of date, but the information in it is relevant.
            2. Hm so that would be quite a fun challenge. One option would be to simply do feature extraction from all cameras, and try to match features between them (from which you can do extrinsics caluclation) until all cameras have extrinsics calculated to at least one other camera. It likely won't provide such
              Other option would be quite similar, but with the calibration board, so to check which cameras see the calib board at any given moment, and calculate extrinsics between them.
            3. So there are 2 calibrations:
              • single camera calibration: get intrinsic, distortion model, and extrinsics for each camera sensor on the OAK camera. This is done in the factory, but you can re-calibrate your OAK camera. You can also calculate rectification matrix from this data, for stereo.
              • Multi-camera setup: extrinsics between OAK cameras, which provide translation/rotation from 1 OAK camera to another.

            Thoughts?
            Thanks, Erik

            6 days later

            Hi @erik

            I tried to calibrate the cameras, I printed and made the charuco board and everything but I run into errors after errors running the calibration code. I try to figure out but this is really an old code, some of the called opencv functions has other names and I have to correct a lot of things but still it does not work for me.

            Could we replace these cameras as you suggested earlier? We have a serious deadline and running out of time.

            If we would get a pretty good pointcloud without waves, that would be a big step and we could work on the multicamera setup but with the above shoved results in the images this is not possible because we have pointclouds we cannot use. They have a big error and we can't make caculations in the scanned bodies because of the bad results.

            Thanks,
            Szabina

            Hi @szzekany ,
            Did you install the required requirements? As library versions specified there should be correct. If they are not, please also provide a full terminal output.
            Thanks, Erik