• DepthAI

  • Summary of Stereo Mapping Cameras

So DepthAI's purpose is to allow real-time perception and interaction with the physical world. So it provides per-frame results of what objects are and where they are in physical space - exactly what all robots need.

So unlike almost all stereo cameras, DepthAI is not intended for only algorithmically generating 3D maps - doing algorithmic simultaneous location and mapping (SLAM) - or generating 3D models of objects. Such is a more traditional use of stereo cameras, and is largely all that these could be used for before the advent of being able to run neural inference directly in the camera. So here is a quick summary of cameras if you are looking to do only stereo vision and/or SLAM and/or 3D recreation of objects:

  • Occipital, here
  • Stereolabs, here
  • Xvisio, here
  • Intel T265, here, Intel D435, Intel SR305 here, Intel D455
  • Kudan, here
  • MyntAI, here
  • Azure Kinect DK, here
  • Duo3D, here
  • QooCam (more consumer-facing), here
  • ArduCam stereo pair for Jetson Nano, here
  • PiEyeCam ToF, here
  • WithRobot, here
  • ncam, here
  • Zivid, here/here/here
  • Onion Tao here
  • Orbbec here
  • In terms of 3D scanning, this KickStarter Revopoint POP seems to make AMAZING maps of objects, here
  • Good comparison review here
  • ToF camera here

So these stereo cameras could be combined with a host computer and an AI processor to accomplish the same thing that DepthAI does all in one camera.

The below figure summarizes it:

So for applications that need depth sensing only (and don't need AI and don't need high-res RGB), the above solutions are probably of more interest.

Where DepthAI is applicable is where Spatial AI is needed in a small, integrated unit. So to enable things to real-time interact with the world like a human would - in a small, modular, open-source eco-system.

And... there is application to use the onboard AI processing in DepthAI to make AWESOME depth (using AI to fill in disparity gaps, improve dense depth) and AWESOME SLAM results (using AI to better track features, etc.)... and this is currently under development, enabled by ArduCam.

See below for a bunch of pertinent threads and Github issues:

  • Original request for this here
  • M12-Mount IMX378 request here
  • M12-Mount OV9281 request here
  • Fish-Eye OV9281 (for better SLAM) request here
  • Global-Shutter Color Camera (OV9782) with same intrinsics as OV9282 grayscale here

Thoughts?

UPDATE 3 August 2021:
It looks like there is now another cameras that has onboard depth and AI on the market:

  1. Modular-lens 4K Depth camera, here. The specs look quite good actually. And the form-factor is nice.

UPDATE 21 March 2022:
We think we have the tools now to make just simply the world's best depth camera. The how is by using all the onboard AI, CV, and general compute capability onboard to OAK-D to be 100% dedicated to just making the best depth possible.

A bit more on the background effort here:
https://github.com/luxonis/depthai/issues/173#issue-675248313

This also is in response to a bunch of RealSense customers moving to OAK-D, who have entire robotics systems running already where the depth camera is "just a depth camera" and so they are OK with 100% of OAK-D's resources going to producing great depth.

So expect progress on this in 2022.

Cheers,
Brandon and the Luxonis team!

10 months later

Hi Brandon, how are you ?

Thanks for this excelent report.
Let me ask you something. Today I'm developing an algorith with T265 and D465, and is a rover, this rover detect an object and follow this object moving inside a room. With this setup, I'm 50% satisfied because how this moving object is near to ground, the D465 has problem measuring the distance from the rover to the object.

That is the major problem, I bought the OAK-D intending to replace the both intel cameras by just one. But I didn't find in the examples a SLAM example and a distance measurement example.
Do you think this my scenario (with the T and D cameras) be replicated with the OAK-D camera? There is a way to improve the distance measurement with OAK-D ?

really thanks.

    Hi juvinski ,

    Quick question - which Intel D-series are you using? I'm not familiar with the D465. Are you using the D455 perhaps?

    We don't have a SLAM example for OAK-D currently. But one of our community members did do a proof of concept with OAK-D with ORBSLAM:
    https://github.com/duncanrhamill/oakd_orbslam3

    And there are some other ongoing OAK-D for SLAM application in our #slam channel in our Discord, here.

    Do you think this my scenario (with the T and D cameras) be replicated with the OAK-D camera? There is a way to improve the distance measurement with OAK-D ?

    Probably? Are you able to share what the object is though? It's hard to say without knowledge of what the object is.

    Here's an example of doing hand-tracking in 3D space for example:

    Also it could actually be a filtering thing on the settings of D455 - as I think they do a good amount of host-side filtering that tries to make planes look more like planes - which can make objects "disappear" when they're close to a plain and not large.

    Thanks,
    Brandon

      Hi Brandon,

      thanks for all insights, my mistake the camera isD435i.
      In true, what I'm intending to do is follow a person, and if the person is near the rover, we can just see their ankle or foot, and when this happen I having this bad performance - because there is this "ground effect" interference.
      I'm joing discord righ now, thanks a lot.