Running Pointcloud NN app for OAK-D Pro Wide

JennaDzitsiuk

I'm trying to run Pointcloud NN sample from here for OAK-D Wide camera and getting the following errors:

[1844301051D9880E00] [1.9] [0.698] [ColorCamera(7)] [warning] Unsupported resolution set for detected camera OV9782, needs 800_P or 720_P. Defaulting to 800_P

Traceback (most recent call last):

File "…/depthai-experiments/gen2-pointcloud/rgbd-pointcloud/main.py", line 117, in <module>

with dai.Device(pipeline) as device:

RuntimeError: StereoDepth(2) - Disparity/depth width must be multiple of 16, but RGB camera width is 427. Set output size explicitly using 'setOutputSize(width, height)'.

With stereo.setOutputSize(480, 320)the script started but I got some opencv exception later

Is there a way to adapt the sample code to work with OAK Wide camera? I assume the sample was written for a regular cam?

My main goal is to export the colored 3D pointcloud from the camera and its ok if depth to points conversion happens on host. Looking for the easiest ways to get this working. Thanks

Update: the device-pointcloud sample not working for this camera either

jakaskerl

Hi JennaDzitsiuk
The script should (tested) work with wide FOV cameras as well. Perhaps you are having troubles with depthai version / example specific requirements? Did you run the pip install -r install_requirements.txt?

Regards,
Jaka

JennaDzitsiuk

If I install from requirements.txt it uninstalls depthai 2.20 and then does not run at all (no cams detected). Because my SDK is newer.

So I just re-installed 2.20 with python3 install_requirements.py. Installed open3d separately with pip3 install open3d. And the result is the main post above.

Should I downgrade my SDK to make this run? Which depthai-sdk is compatible with 2.16.0?

jakaskerl

Hi JennaDzitsiuk
No that's fine, just use the latest version, it should work. I failed to see your device uses a wide rgb camera (with max 800p resolution). Could you try running the example at full resolution (without isp scaling). I won't be as fast but might work better. Also, what following openCV exception are you getting?

Thanks,
Jaka

JennaDzitsiuk

jakaskerl Thanks, once I commented out camRbg.setIspScale, it started working!!

Another question about this demo. I see that not all 3D points are computed. Are there any thresholds defined to get the full pointcloud?

Update:
o3d.geometry.TriangleMesh.create_coordinate_frame() size input has some effect, but 3D starts looking wrong. I also played with view_control.set_constant_z_faranddepth_trunc input to o3d.geometry.RGBDImage.create_from_color_and_depth,no luck..

JennaDzitsiuk

I just exported the pointcloud from the default example to ply. In projector_3d.py: o3d.io.write_point_cloud(filename, self.pcl)
No modifications except commented-out setIspScale. The pointcloud looks wrong.. I point the camera on my two monitors and table and the monitors are not flat in 3D.. Here is the video. I wonder if that NN pointcloud generation is not compatible with the OAK-wide cam?

Pointcloud video - link

erik

JennaDzitsiuk What about with isp scale not being commented out?

JennaDzitsiuk

erik Hi Erik, do you have any updates on this demo for Oak-D Wide camera?

I tried to check other threads and see that only NN model is advised. I'm ok with doing it with a classical no ML way if this demo not adapted for Oak-D Wide. Because currently I'm just evaluating the quality of pointclouds and deciding which cam to use.

Do you have any sample C++ code useful for depth back projection for getting the 3D pointcloud? Also I will need a calibration between depth and color cam to apply RGB to 3D. I'm new to this API so looking for any useful samples to get some speed up

JennaDzitsiuk

then the demo not working, see my first post here.
Getting runtime error
RuntimeError: StereoDepth(2) - Disparity/depth width must be multiple of 16, but RGB camera width is 427. Set output size explicitly using 'setOutputSize(width, height)'

erik

Hi JennaDzitsiuk ,
No updates yet, I can't really try out as we don't have wide FOV OV9782 OAK-D in our office.

We are working on Pointcloud node, so you could use that in C++. More info here: luxonis/depthai-core791 (feel free to subscribe to get notification about it)

erik

Hi JennaDzitsiuk ,
I believe what you are seeing is open3d cutoff of points - perhaps you could look into rerun/depthai-viewer visualizer. We have a demo code and on-device pointcloud node (on custom branch as of now) + video here: luxonis/depthai-core791
I hope this helps!

JennaDzitsiuk

Thanks, I dont worry about cutoff for now. My main concern is the depth quality - link from the thread above.

Thank you for adding the pointcloud demo! I tried it and suspect the issue remains. I asked another question on the dedicated thread you linked above