• DepthAI

  • Luxonis DepthAI and megaAI | Overview and Status

And here's a video view of the MVP:

a month later

Our intern went ahead and got DepthAI working natively on Mac OS X:

We’ll be writing up instructions soon. Almost all of the work is actually just setting a Mac up for Python development using Homebrew... so if your Mac is already set up for that it pretty much ‘just works’.

Meant to share this a while ago. So we have our initial online custom training for DepthAI now live on Colab.

https://colab.research.google.com/drive/1Eg-Pv7Amgc3THB6ZbnSaDJm0JAr0QPPU

So there are two notable limitations currently:

  1. DepthAI currently supports OpenVINO 2019 R3, which itself requires older versions of TensorFlow and so on. So this flow has all those old versions, which causes a lot of additional steps in Colab... a lot of uninstalling current versions of stuff and installing old versions. We are currently in the process of upgrading our DepthAI codebase to support OpenVINO 2020.1, see here. We'll release an updated training flow when that's done.
  2. The final conversion for DepthAI (to .blob) for some reason will not run on Google Colab. So it requires a local machine to do it. We're planning on just making our own server for this purpose that Google Colab can talk to to do the conversion.

To test the custom training we took some images of apples and oranges and did a terrible job labeling them and then trained and converted the network and ran it on DepthAI. It's easy to get WAY better accuracies and detection rates by using something like basic.ai to generate a larger dataset.

Cheers,
Brandon

7 days later
8 days later

We now have a more complete training flow:
https://docs.luxonis.com/tutorials/object_det_mnssv2_training/

And we used it to train a mask/no-mask model for DepthAI with a quick effort at it over the weekend:

More images of validation/testing on Google Colab here:
https://photos.app.goo.gl/FhhUCLTsm6tqBgqL8

And here's the Google Colab used to train DepthAI on mask/no-mask face detection:
https://colab.research.google.com/drive/1uY5vekGK7S6uD88d28G861SIRh9yYbjJ

Hi DepthAI Fans,

As promised, we have open sourced the DepthAI hardware!

All the carrier boards for the DepthAI System on Module (SoM), including the Altium design files and all supporting information are below:

https://github.com/luxonis/depthai-hardware


So now you can integrate the power of DepthAI into your custom prototypes and products at the board level using the DepthAI System on Module (SoM).

We can't wait to see what you build with it (and we've already seen some really cool things!).

Cheers, Brandon & the Luxonis Team

The Power over Ethernet (PoE) variant of DepthAI is starting to trickle in (after COVID19 delays)...

We now have the baseboard (which actually implements the PoE portion):

So now you can deploy DepthAI all over the place and with 328.1 feet of cable between you and the device! The power of DepthAI, with the convenience of power over ethernet deployment.

6 days later

DepthAI on Jetson Tx2. Followed the same build instructions used on Mac OS X (here) and it built w/out even a single complaint and worked first try:

14 days later

The PoE boards work great! Tested 1,000FDX over PoE (from UniFi Switch) and they work exactly as intended.

Here's DepthAI running on our Power over Ethernet prototypes:

8 days later

We launched megaAI on Crowd Supply today!


4K Video at 30FPS on a Pi, while running object detection in parallel!

Get yours now before the early bird and roadrunner specials sell out! Only 14 left!

https://www.crowdsupply.com/luxonis/megaai

17 days later

Hi DepthAI (and megaAI) fans!

So we have a couple customers who are interested in IR-only variants of the global-shutter cameras used for Depth, so we made a quick variant of DepthAI with these.

We actually just made adapter boards which plug directly into the BW1097 (here) by unplugging the existing onboard cameras. We tested with this IR flashlight here.


It's a bit hard to see, but you can tell the room is relatively dark to visible light and the IR cameras pick up the IR light quite well.

Cheers,

The Luxonis Team

More great news coming at you! We've accomplished so much so fast recently that it's hard to keep up with the updates.

Over the weekend we wrote a driver for the IMX477 used in the Raspberry Pi HQ Camera.

So now you can use the awesome new Raspberry Pi HQ camera with DepthAI FFC (here​). Below are some videos of it working right after we wrote the driver this weekend.

​​

Notice that it even worked w/ an extra long FFC cable! ​​

​More details on how to use it are here​. And remember DepthAI is open source, so you can even make your own adapter (or other DepthAI boards) from our Github here​.

And you can buy the adapter here: https://shop.luxonis.com/products/rpi-hq-camera-imx477-adapter-kit

Cheers,

Brandon & the Luxonis team

8 days later

We have a super-interesting feature-set coming to DepthAI:

  • 3D feature localization (e.g. finding facial features) in physical space
  • Parallel-inference-based 3D object localization
  • Two-stage neural inference support

And all of these are initially working (in this PR, here).

So to the details and how this works:

We are actually implementing a feature that allows you to run neural inference on either or both of the grayscale cameras.

This sort of flow is ideal for finding the 3D location of small objects, shiny objects, or objects for which disparity depth might struggle to resolve the distance (z-dimension), which is used to get the 3D position (XYZ). So this now means DepthAI can be used two modalities:

  1. As it's used now: The disparity depth results within a region of the object detector are used to re-project xyz location of the center of object.
  2. Run the neural network in parallel on both left/right grayscale cameras, and the results are used to triangulate the location of features.

An example where 2 is extremely useful is finding the xyz positions of facial landmarks, such as eyes, nose, and corners of the mouth.

Why is this useful for facial features like this? For small features like this, the risk of disparity depth having a hole in the location goes up, and even worse, for faces with glasses, the reflection of the glasses may throw the disparity depth calculation off (and in fact it might 'properly' give the depth result for the reflected object).

When running the neural network in parallel, none of these issues exist, as the network finds the eyes, nose, and mouth corners per image, and then the disparity in location of these in pixels from the right and left stream results gives the z-dimension (depth = 1/disparity), and then this is reprojected through the optics of the camera to get the full XYZ position of all of these features.

And as you can see below, it works fine even w/ my quite-reflective anti-glare glasses:

Thoughts?

Cheers,
Brandon and the Luxonis Team

15 days later
Brandon changed the title to Luxonis DepthAI and megaAI | Overview and Status .

​Hi DepthAI Backers and Fans,

So we've proof-of-concepted an SPI-only interface for DepthAI and it's working well (proof of concept done with MSP430 and Raspberry Pi over SPI).

So to make it easier for engineers to leverage this power (and also for us internally to develop it), we're making a complete hardware and software/AI reference design for the ESP32, with the primary interface between DepthAI and the ESP32 being SPI.

The design will still have USB3C for DepthAI, which will allow you to see live high-bandwidth results/etc. on a computer while integrating/debugging communication to your ESP32 code (both running in parallel, which will be nice for debugging). Similarly, the ESP32 will have an onboard UART-USB converter and micro-USB connector for programming/interfacing w/ the ESP32 for easy development/debug.

For details on the effort and to see progress see here

For details and progress on the hardware effort see [here​] and to check out the SPI support enhancement on DepthAI API see [here​]

In short here's the concept:

And here's a first cut at the placement:

And please let us know if you have any thoughts/comments/questions on this design!

Best,
Brandon & The Luxonis Team

7 days later

Our production run of the megaAI CrowdSupply campaign is complete and now shipping to us:

We had 97% yield on the first round of testing and 99% yield after rework of the 3% that had issues in the first testing.

5 days later

Today, our team is excited to release to you the OpenCV AI Kit, OAK, a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training - that allows you to embed Spatial AI and CV super-powers into your product.

And best of all, you can buy this complete solution today and integrate it into your product tomorrow.

Back our campaign today!

a month later

Our KickStarter rose $1,357,995! Quite exciting! Now we're heads-down building around 10,000 units and preparing the logistics to ship all of these out!

25 days later

So we're now heads-down designing and prototyping the aluminum enclosures we promised to our KickStarter backers as a result of hitting our $1-million-raise stretch goal. Some in-progress photos.

Most of the work involves making sure that the designs are thermally sound - given that they fully encase the heat-generator.

We were able to make a mode (for both OAK-1 and OAK-D) that uses more power in the chip than we've ever been able to make it use previously. And we used this mode for all the testing, so the thermal results below are for the absolute-worst-case power use of the part (or as close as we could get to it).

OAK-1:

We did a quick/dirty version to test thermal limits. It looks pretty good.



Our calculations estimated that at this max power use that the max surface temperature would be about 60C (the max recommended for metallic surfaces). And it came out to about that, at 61.3C:

To make sure the maximum surface temperature is below this 60C max recommendation, we increased the heat-sinking and size a bit, from a total thickness of 2.4cm top 2.7cm, and added some vertical aspects as well:

Prototypes of that will likely be in next week. We expect to see sub-60C external temperatures.

Even with the initial/smaller enclosure, we are seeing a max die temperature of less than 75C, and the max safe die temperature is 105C, so this gives nice margin.

a month later
2 months later

Hi DepthAI Backers!

Thanks again for all the continued support and interest in the platform.

So we've been hard at work adding a TON of DepthAI functionalities. You can track a lot of the progress in the following Github projects:

As you can see, there are a TON of features we have released since the last update. Let's highlight a few below:

RGB-Depth Alignment

We have the calibration stage working now. And future DepthAI builds (after this writing) are actually having RGB-right calibration performed. An example with semantic segmentation is shown below:
image

The right grayscale camera is shown on the right and the RGB is shown on the left. You can see the cameras are slightly different aspect ratios and fields of view, but the semantic segmentation is still properly applied. More details on this, and to track progress, see our Github issue on this feature here: https://github.com/luxonis/depthai/issues/284

Subpixel Capability

DepthAI now supports subpixel. To try it out yourself, use the example here. And see below for my quickly using this at my desk:

image
image
image
image

Host Side Depth Capability

We also now allow performing depth estimation from images sent from the host. This is very convenient for test/validation - as stored images can be used. And along with this, we now support outputting the rectified-left and rectified-right, so they can be stored and later used with DepthAI's depth engine in various CV pipelines.

See here on how to do this with your DepthAI model. And see some examples below from the MiddleBury stereo dataset:

image

image

For the bad looking areas, these are caused by the objects being too close to the camera for the given baseline, exceeding the 96 pixels max distance for disparity matching (StereoDepth engine constraint):
image

These areas will be improved with extended = True, however Extended Disparity and Subpixel cannot operate both at the same time.

RGB Focus, Exposure, and Sensitivity Control

We also added the capability (and examples on how to use) manual focus, exposure, and sensitivity controls. See here for how to use these controls.
Here is an example of increasing the exposure time:
![image](

And here is setting it quite low:
image

It's actually fairly remarkable how well the neural network still detects me as a person even when the image is this dark.

Pure Embedded DepthAI

We mentioned in our last update (here), we mentioned that we were making a pure-embedded DepthAI.

We made it. Here's the initial concept:
image

And here it is working!
image

And here it is on a wrist to give a reference of its size:
image

And eProsima even got microROS running on this with DepthAI, exporting results over WiFi back to RViz:
image

RPi Compute Module 4

We're quite excited about this one. We're fairly close to ordering it. Some initial views in Altium below:
image
image

There's a bunch more, but we'll leave you with our recent interview with Chris Gammel at the Amp Hour!

https://theamphour.com/517-depth-and-ai-with-brandon-gilles-and-brian-weinstein/

Cheers,
Brandon & The Luxonis Team

2 months later

It's been a while... and there are more status updates than this, but check out the new POE IP67 version of DepthAI:

The enclosure is actually filled with water in this photo, and the single-directional vent is doing it's job by letting the water slowly seep out.

3 months later
2 months later

It's cool to see the other safety products that are being built off of this. For example recently BlueBox Labs released their collision-deterring (among many other features) spatial AI dashcamera based on DepthAI on KickStarter:

So this could work inside the vehicle to even detect and alert if a (distracted) driver is going to hit a person riding a bike or a pedestrian.

More safety products are in the works. Very satisfying to see.

Cheers,
Brandon

6 months later
a year later
erik unstickied the discussion .