Luxonis DepthAI and megaAI | Overview and Status

The Myriad X is a vision processor capable of doing real time object detection and stereo depth at over 30FPS.

Let's unleash this power!

How?
We're making a Myriad X System on Module (SoM) which allows embedding the power of the Myriad X into your own products, with firmware support for 3D object detection/location.

And we're making a carrier board that includes all the cameras, the Myriad X, and the Raspberry Pi Compute Module all together to allow you to get up and running in seconds.

This allows:

1. The video data path to skip the Pi, eliminating that additional latency and bottlenecking
2. Stereo depth capability of the Myriad X for 3D object localization
3. Significant reduction in the CPU load on the Raspberry Pi

So you, the Python programmer, now have real-time 3D position of all the objects around - on an embedded platform - and backed by the power of the Raspberry Pi Community!

For the full back story, let's start with the why:

• There’s an epidemic in the US of injuries and deaths of people who ride bikes
• Majority of cases are distracted driving caused by smart phones (social media, texting, e-mailing, etc.)
• We set out to try to make people safer on bicycles in the US
  • We’re technologists
  • Focused on AI/ML/Embedded
  • So we’re seeing if we can make a technology solution

Making People Who Ride Bikes Safer

(If you'd like to read more about CommuteGuardian, see here)

DepthAI Platform

• In prototyping the Commute Guardian, we realized how powerful the combination of Depth and AI is.
• And we realized that no such embedded platform existed
• So our milestone on the path to CommuteGuardian is to build this platform – and sell it as a standard product.
• We’re building it for the Raspberry Pi (Compute Module)
  • Human-level perception on the world’s most popular platform
  • Adrian’s PyImageSearch Raspberry Pi Computer Vision Kickstarter sold out in 10 seconds – validating demand for Computer Vision on the Pi (that, and validating that Adrian is AWESOME!)

So below is a rendering of our first prototype of DepthAI for Raspberry Pi.

The key difference between this, and say using the Raspberry Pi with an NCS2 is the data path. With the NCS2 approach all the video/image data has to flow through (and be resized/etc.) by the host, whereas in this system the video data goes directly to the Myriad X, as below, unburdening the host from these tasks - which increases frame-rate and drastically reduces latency:

Development Steps

The first thing we made was a dev board for ourselves. The Myriad X is a complicated chip, with a ton of useful functionality... so we wanted a board where we could explore this easily, try out different image sensors, etc. Here's what that looks like:

BW0235

We made the board with modular camera boards so we could easily test out new image sensors w/out the complexity of spinning a new board. So we'll continue to use this as we try out new image sensors and camera modules.

While waiting on our development boards to be fabricated, populated, etc. we brainstormed how to keep costs down (working w/ fine-pitch BGAs that necessitate laser vias means prototypes are EXPENSIVE), while still allowing easy experimentation w/ various form-factors, on/off-board cameras, etc. We landed on making ourselves a Myriad X System on Module, which is the board w/ all the crazy laser vias, stacked vias, and over all High-Density-Integration (HDI) board stuff that makes them expensive. This way, we figure, we can use this as the core of any Myriad X designs we do, without having to constantly prototype w/ expensive boards.

BW1099

We exposed all that we needed for our end-goal of 3D object detection (i.e. MobileNet-SSD object detection + 3D reprojection off of stereo depth data). So that meant exposing a single 4-lane MIPI for handling high-res (e.g. 12MP) color camera sensors and 2x 2-lane MIPI for cameras such as _1MP global-shutter image sensors for depth.

And we threw a couple other interfaces, boot methods, etc. on there for good measure, which are default de-pop to save cost when not needed, and can be populated if needed.

So of course in making a module, you also need to make a board on which to test the module. So in parallel to making the SoM, we started attacking a basic breakout carrier board:

It's basic, but pulls out all the important interfaces, and works with the same modular camera-board system as our development board. So it's to some degree our 'development board lite'.

And once we got both of these ordered, we turned our attention to what we set out to build, for you, the DepthAI for Raspberry Pi system. And here it is, in all it's Altium-rendered glory:

So what does this thing do? The key bit is it's completely self-contained. If you need to give something autonomy, you don't need anything more than this. It has the vision accelerator (Myriad X), all the cameras, all the connectivity, and interfaces you need - and the Raspberry Pi Compute Module on-board.

So it's a self-contained system, allowing you to write some simple Python to solve problems that 3 years ago were not yet solvable by humanity! And now you can do it all on this one system.

To visualize what this gives you, see below, noting that DepthAI will give over 30FPS instead of the 3FPS in this example:

And while we are still working to integrate object detection and depth data together directly on the Myriad X, as well as tweaking our depth filtering, here's an example of depth alone running on our platform at 30FPS on DepthAI for Raspberry Pi (see here for more details):

Cheers,
The Luxonis Team

Hi DepthAI fans,

So we've done SO MUCH since we last updated here. The only thing we haven't done is keep this post active.

So what have we done:

• We delivered our Crowd Supply on time! Backers are happily using DepthAI now, and are discussing ideas on our luxonis-community.slack.com public slack group.
• We got our first set of documentation out. https://docs.luxonis.com/
• We made a couple new models which are available now (at https://shop.luxonis.com/) And we will have these on Crowd Supply soon.
• We are in the process of making a power-over-ethernet version of DepthAI.
• Our MVP Python API is running (and super fund to play with)

New Models
On the new hardware models since the Crowd Supply started. These include a USB3 Edition with onboard cameras and a USB3 Edition that's tiny and single camera (which we're calling μAI):

USB3C with Onboard Cameras (BW1098OBC):

μAI (BW1093):

Upcoming Model

This is our first engineering-development version of our PoE version of DepthAI. Some interesting new features include:

1. A new module (the BW1099) with:

2. Built in 128GB eMMC

3. SD-Card interface for base-board SD-Card support

4. PCIE support for Ethernet

5. A reference-design carrier board with:

6. PoE (100/1000)

7. SD-Card

8. 4-lane MIPI for 12MP camera (BG0249)

MVP Functionality

So the core functionality gives 3D object localization as the output from the DepthAI - with all processing done on the Myriad X - and no other hardware required. The Raspberry Pi here is used purely to show the results.

So what you can see is the person - a little person at that - at 1.903 meters away from the camera, 0.427 meters below the camera, and 0.248 meters to the right of that camera.

And you can also see the chair, which is 0.607 meters to the left of the camera, 0.45 meters below the camera, and 2.135 meters away from the camera.

And for good measure, here is our test subject walking over to the chair:

The results are returned real-time. And the video is optional. We even have a special version that outputs results over SPI for using DepthAI with microcontrollers like the MSP430. (Contact us at support@luxonis.com if this is of interest)

Cheers,
Brandon & the Luxonis Team

Update on the training. To use the latest of everything (as of this writing), including OpenVINO (R2020.1), etc. use the following:

Training: https://colab.research.google.com/drive/1_bjLv6QH_SPQ4QQ4TX1l_45acBSbjQBu
Running on DepthAI: https://github.com/luxonis/depthai-python-extras/tree/host_watchdog_r10.15

We now have a more complete training flow:
https://docs.luxonis.com/tutorials/object_det_mnssv2_training/

And we used it to train a mask/no-mask model for DepthAI with a quick effort at it over the weekend:

More images of validation/testing on Google Colab here:
https://photos.app.goo.gl/FhhUCLTsm6tqBgqL8

And here's the Google Colab used to train DepthAI on mask/no-mask face detection:
https://colab.research.google.com/drive/1uY5vekGK7S6uD88d28G861SIRh9yYbjJ

Hi DepthAI Fans,

As promised, we have open sourced the DepthAI hardware!

All the carrier boards for the DepthAI System on Module (SoM), including the Altium design files and all supporting information are below:

https://github.com/luxonis/depthai-hardware

So now you can integrate the power of DepthAI into your custom prototypes and products at the board level using the DepthAI System on Module (SoM).

We can't wait to see what you build with it (and we've already seen some really cool things!).

Cheers, Brandon & the Luxonis Team

And we just finished layout of an adapter board which will allow the new Raspberry Pi HQ camera to work with DepthAI
(including the above PoE version and the FFC version here).

Design files are here:
https://github.com/luxonis/depthai-hardware/tree/master/BW0253_R0M0E0_RPIHQ_ADAPTER

We launched megaAI on Crowd Supply today!

4K Video at 30FPS on a Pi, while running object detection in parallel!

Get yours now before the early bird and roadrunner specials sell out! Only 14 left!

https://www.crowdsupply.com/luxonis/megaai

More great news coming at you! We've accomplished so much so fast recently that it's hard to keep up with the updates.

Over the weekend we wrote a driver for the IMX477 used in the Raspberry Pi HQ Camera.

So now you can use the awesome new Raspberry Pi HQ camera with DepthAI FFC (here​). Below are some videos of it working right after we wrote the driver this weekend.

​​

Notice that it even worked w/ an extra long FFC cable! ^​​

​More details on how to use it are here​. And remember DepthAI is open source, so you can even make your own adapter (or other DepthAI boards) from our Github here​.

And you can buy the adapter here: https://shop.luxonis.com/products/rpi-hq-camera-imx477-adapter-kit

Cheers,

Brandon & the Luxonis team

We have a super-interesting feature-set coming to DepthAI:

• 3D feature localization (e.g. finding facial features) in physical space
• Parallel-inference-based 3D object localization
• Two-stage neural inference support

And all of these are initially working (in this PR, here).

So to the details and how this works:

We are actually implementing a feature that allows you to run neural inference on either or both of the grayscale cameras.

This sort of flow is ideal for finding the 3D location of small objects, shiny objects, or objects for which disparity depth might struggle to resolve the distance (z-dimension), which is used to get the 3D position (XYZ). So this now means DepthAI can be used two modalities:

1. As it's used now: The disparity depth results within a region of the object detector are used to re-project xyz location of the center of object.
2. Run the neural network in parallel on both left/right grayscale cameras, and the results are used to triangulate the location of features.

An example where 2 is extremely useful is finding the xyz positions of facial landmarks, such as eyes, nose, and corners of the mouth.

Why is this useful for facial features like this? For small features like this, the risk of disparity depth having a hole in the location goes up, and even worse, for faces with glasses, the reflection of the glasses may throw the disparity depth calculation off (and in fact it might 'properly' give the depth result for the reflected object).

When running the neural network in parallel, none of these issues exist, as the network finds the eyes, nose, and mouth corners per image, and then the disparity in location of these in pixels from the right and left stream results gives the z-dimension (depth = 1/disparity), and then this is reprojected through the optics of the camera to get the full XYZ position of all of these features.

And as you can see below, it works fine even w/ my quite-reflective anti-glare glasses:

Thoughts?

Cheers,
Brandon and the Luxonis Team

​Hi DepthAI Backers and Fans,

So we've proof-of-concepted an SPI-only interface for DepthAI and it's working well (proof of concept done with MSP430 and Raspberry Pi over SPI).

So to make it easier for engineers to leverage this power (and also for us internally to develop it), we're making a complete hardware and software/AI reference design for the ESP32, with the primary interface between DepthAI and the ESP32 being SPI.

The design will still have USB3C for DepthAI, which will allow you to see live high-bandwidth results/etc. on a computer while integrating/debugging communication to your ESP32 code (both running in parallel, which will be nice for debugging). Similarly, the ESP32 will have an onboard UART-USB converter and micro-USB connector for programming/interfacing w/ the ESP32 for easy development/debug.

For details on the effort and to see progress see here

For details and progress on the hardware effort see [here​] and to check out the SPI support enhancement on DepthAI API see [here​]

In short here's the concept:

And here's a first cut at the placement:

And please let us know if you have any thoughts/comments/questions on this design!

Best,
Brandon & The Luxonis Team

Today, our team is excited to release to you the OpenCV AI Kit, OAK, a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training - that allows you to embed Spatial AI and CV super-powers into your product.

And best of all, you can buy this complete solution today and integrate it into your product tomorrow.

Back our campaign today!