Luxonis DepthAI and megaAI | Overview and Status

Brandon

And much more to come on firmware, software, and hardware updates. Oh and also our Crowd Supply pre-launch page is up. Need to update that too!

https://www.crowdsupply.com/luxonis/depthai

Brandon

JanT

Good job! Please, which accelerator on Myriad X do you use for the Stereo Depth implementation? Do you use the new "Stereo depth block" mentioned on this webpage: https://www.movidius.com/myriadx or do you use the SHAVE cores only?
Thank you!
Jan

Brandon

Hi JanT

Thanks!

Yes we're using the stereo depth block and not the SHAVEs for the stereo implementation. This leaves the SHAVEs free for neural processing and also filtering on the depth, etc.:

A bit more information is here:
https://github.com/Luxonis-Brandon/DepthAI

Thanks again,
Brandon

GOB

Hi Brandon! This project looks amazing. What is the frame rate to be expected for depth camera + AI usage? In the sample video it looked like the framerate was around 2 or 3 fps. My project would need a higher rate.

Brandon

GOB Thanks and good question GOB . So we're expecting 25 frames per second of depth and AI operating at the same time. We were expecting to have a demo of this for the launch, but don't have both stacks working correctly yet - so we used our slower, prototype video for now, which is yes slower at 2-3 frames per second.

GOB

Brandon Thanks for the update! If it's okay, can I email you with some more questions I have?

Brandon

Yes for sure. Brandon at luxonis dot com

JanT

Thank you for your answer, Brandon!
I have an additional question. I'm sorry if it's already explained somewhere else.
Which libraries do you use for the Stereo block and SHAVE cores programming? You mention you are working on some stack to put together the Depth processing and AI detection. I would expect there is some stack or framework delivered from Intel.. I know there is the OpenVINO, but it's for the AI block only..
So, if I summarize my question, do you use enablement (drivers, libs, stacks) from Intel, or do you need to write everything from scratch by yourself?

Thank you!
Jan

Brandon

Hi JanT,

We’re writing our own system, based on an architecture we put together and have been implementing for a while now. This is composed of a binary (custom, that we make) that runs in the Myriad X, a library that runs on the host OS (again, that we make) that works with OpenVINO cleanly and has additional open source Python code which exposes the additional functionality that does exist yet in OpenVINO.

We are also planning on working with OpenCV and OpenVINO to integrate this functionality directly into both.

Expect some announcements soon (and I think some are already out, from the President of OpenCV for example).

Thoughts?

Thanks,
Brandon

JanT

Hi Brandon,
I just saw the announcement from the President of OpenCV yesterday. That's great news! You are doing a great work! I'm happy to see the Myriad X is much more capable than what the NCS2 can deliver. Thank you for putting effort into it.

Best regards,
Jan

Brandon

Hey JanT,

Thanks a ton! Really looking forward to working with OpenCV on this to make it easy to use and wicked useful.

Thanks again,
Brandon

Brandon

Hi DepthAI fans,

So we've done SO MUCH since we last updated here. The only thing we haven't done is keep this post active.

So what have we done:

We delivered our Crowd Supply on time! Backers are happily using DepthAI now, and are discussing ideas on our luxonis-community.slack.com public slack group.
We got our first set of documentation out. https://docs.luxonis.com/
We made a couple new models which are available now (at https://shop.luxonis.com/) And we will have these on Crowd Supply soon.
We are in the process of making a power-over-ethernet version of DepthAI.
Our MVP Python API is running (and super fund to play with)

New Models
On the new hardware models since the Crowd Supply started. These include a USB3 Edition with onboard cameras and a USB3 Edition that's tiny and single camera (which we're calling μAI):

USB3C with Onboard Cameras (BW1098OBC):

μAI (BW1093):

Upcoming Model

This is our first engineering-development version of our PoE version of DepthAI. Some interesting new features include:

A new module (the BW1099) with:
Built in 128GB eMMC
SD-Card interface for base-board SD-Card support
PCIE support for Ethernet
A reference-design carrier board with:
PoE (100/1000)
SD-Card
4-lane MIPI for 12MP camera (BG0249)

MVP Functionality

So the core functionality gives 3D object localization as the output from the DepthAI - with all processing done on the Myriad X - and no other hardware required. The Raspberry Pi here is used purely to show the results.

So what you can see is the person - a little person at that - at 1.903 meters away from the camera, 0.427 meters below the camera, and 0.248 meters to the right of that camera.

And you can also see the chair, which is 0.607 meters to the left of the camera, 0.45 meters below the camera, and 2.135 meters away from the camera.

And for good measure, here is our test subject walking over to the chair:

The results are returned real-time. And the video is optional. We even have a special version that outputs results over SPI for using DepthAI with microcontrollers like the MSP430. (Contact us at support@luxonis.com if this is of interest)

Cheers,
Brandon & the Luxonis Team

Brandon

And here's a video view of the MVP:

Brandon

Our intern went ahead and got DepthAI working natively on Mac OS X:

We’ll be writing up instructions soon. Almost all of the work is actually just setting a Mac up for Python development using Homebrew... so if your Mac is already set up for that it pretty much ‘just works’.

Brandon

Meant to share this a while ago. So we have our initial online custom training for DepthAI now live on Colab.

https://colab.research.google.com/drive/1Eg-Pv7Amgc3THB6ZbnSaDJm0JAr0QPPU

So there are two notable limitations currently:

DepthAI currently supports OpenVINO 2019 R3, which itself requires older versions of TensorFlow and so on. So this flow has all those old versions, which causes a lot of additional steps in Colab... a lot of uninstalling current versions of stuff and installing old versions. We are currently in the process of upgrading our DepthAI codebase to support OpenVINO 2020.1, see here. We'll release an updated training flow when that's done.
The final conversion for DepthAI (to .blob) for some reason will not run on Google Colab. So it requires a local machine to do it. We're planning on just making our own server for this purpose that Google Colab can talk to to do the conversion.

To test the custom training we took some images of apples and oranges and did a terrible job labeling them and then trained and converted the network and ran it on DepthAI. It's easy to get WAY better accuracies and detection rates by using something like basic.ai to generate a larger dataset.

Cheers,
Brandon

Brandon

Update on the training. To use the latest of everything (as of this writing), including OpenVINO (R2020.1), etc. use the following:

Training: https://colab.research.google.com/drive/1_bjLv6QH_SPQ4QQ4TX1l_45acBSbjQBu
Running on DepthAI: https://github.com/luxonis/depthai-python-extras/tree/host_watchdog_r10.15

Brandon

And here's training for MobileNetSSDv2:
https://colab.research.google.com/drive/1n7uScOl8MoqZ1guQM6iaU1-BFbDGGWLG

You can label using this tool:
https://github.com/tzutalin/labelImg

Brandon

We now have a more complete training flow:
https://docs.luxonis.com/tutorials/object_det_mnssv2_training/

And we used it to train a mask/no-mask model for DepthAI with a quick effort at it over the weekend:

More images of validation/testing on Google Colab here:
https://photos.app.goo.gl/FhhUCLTsm6tqBgqL8

And here's the Google Colab used to train DepthAI on mask/no-mask face detection:
https://colab.research.google.com/drive/1uY5vekGK7S6uD88d28G861SIRh9yYbjJ

Brandon

Hi DepthAI Fans,

As promised, we have open sourced the DepthAI hardware!

All the carrier boards for the DepthAI System on Module (SoM), including the Altium design files and all supporting information are below:

https://github.com/luxonis/depthai-hardware

So now you can integrate the power of DepthAI into your custom prototypes and products at the board level using the DepthAI System on Module (SoM).

We can't wait to see what you build with it (and we've already seen some really cool things!).

Cheers, Brandon & the Luxonis Team