The Myriad X is a vision processor capable of doing real time object detection and stereo depth at over 30FPS. Let's unleash this power! How? We're making a Myriad X System on Module (SoM) which allows embedding the power of the Myriad X into your own products, with firmware support for 3D object detection/location. And we're making a carrier board that includes all the cameras, the Myriad X, and the Raspberry Pi Compute Module all together to allow you to get up and running in seconds. This allows: The video data path to skip the Pi, eliminating that additional latency and bottlenecking Stereo depth capability of the Myriad X for 3D object localization Significant reduction in the CPU load on the Raspberry Pi So you, the Python programmer, now have real-time 3D position of all the objects around - on an embedded platform - and backed by the power of the Raspberry Pi Community! For the full back story, let's start with the why : There’s an epidemic in the US of injuries and deaths of people who ride bikes Majority of cases are distracted driving caused by smart phones (social media, texting, e-mailing, etc.) We set out to try to make people safer on bicycles in the US We’re technologists Focused on AI/ML/Embedded So we’re seeing if we can make a technology solution Making People Who Ride Bikes Safer (If you'd like to read more about CommuteGuardian, see here ) DepthAI Platform In prototyping the Commute Guardian, we realized how powerful the combination of Depth and AI is. And we realized that no such embedded platform existed So our milestone on the path to CommuteGuardian is to build this platform – and sell it as a standard product. We’re building it for the Raspberry Pi (Compute Module) Human-level perception on the world’s most popular platform Adrian’s PyImageSearch Raspberry Pi Computer Vision Kickstarter sold out in 10 seconds – validating demand for Computer Vision on the Pi (that, and validating that Adrian is AWESOME!) So below is a rendering of our first prototype of DepthAI for Raspberry Pi. The key difference between this, and say using the Raspberry Pi with an NCS2 is the data path. With the NCS2 approach all the video/image data has to flow through (and be resized/etc.) by the host, whereas in this system the video data goes directly to the Myriad X, as below, unburdening the host from these tasks - which increases frame-rate and drastically reduces latency: Development Steps The first thing we made was a dev board for ourselves. The Myriad X is a complicated chip, with a ton of useful functionality... so we wanted a board where we could explore this easily, try out different image sensors, etc. Here's what that looks like: BW0235 We made the board with modular camera boards so we could easily test out new image sensors w/out the complexity of spinning a new board. So we'll continue to use this as we try out new image sensors and camera modules. While waiting on our development boards to be fabricated, populated, etc. we brainstormed how to keep costs down (working w/ fine-pitch BGAs that necessitate laser vias means prototypes are EXPENSIVE), while still allowing easy experimentation w/ various form-factors, on/off-board cameras, etc. We landed on making ourselves a Myriad X System on Module, which is the board w/ all the crazy laser vias, stacked vias, and over all High-Density-Integration (HDI) board stuff that makes them expensive. This way, we figure, we can use this as the core of any Myriad X designs we do, without having to constantly prototype w/ expensive boards. BW1099 We exposed all that we needed for our end-goal of 3D object detection (i.e. MobileNet-SSD object detection + 3D reprojection off of stereo depth data). So that meant exposing a single 4-lane MIPI for handling high-res (e.g. 12MP) color camera sensors and 2x 2-lane MIPI for cameras such as 1MP global-shutter image sensors for depth. And we threw a couple other interfaces, boot methods, etc. on there for good measure, which are default de-pop to save cost when not needed, and can be populated if needed. So of course in making a module, you also need to make a board on which to test the module. So in parallel to making the SoM, we started attacking a basic breakout carrier board: It's basic, but pulls out all the important interfaces, and works with the same modular camera-board system as our development board. So it's to some degree our 'development board lite'. And once we got both of these ordered, we turned our attention to what we set out to build, for you, the DepthAI for Raspberry Pi system. And here it is, in all it's Altium-rendered glory: So what does this thing do? The key bit is it's completely self-contained. If you need to give something autonomy, you don't need anything more than this. It has the vision accelerator (Myriad X), all the cameras, all the connectivity, and interfaces you need - and the Raspberry Pi Compute Module on-board. So it's a self-contained system, allowing you to write some simple Python to solve problems that 3 years ago were not yet solvable by humanity! And now you can do it all on this one system. To visualize what this gives you, see below, noting that DepthAI will give over 30FPS instead of the 3FPS in this example: And while we are still working to integrate object detection and depth data together directly on the Myriad X, as well as tweaking our depth filtering, here's an example of depth alone running on our platform at 30FPS on DepthAI for Raspberry Pi (see here for more details): Cheers, The Luxonis Team

Hi everyone, So the first prototypes of the DepthAI for Raspberry Pi just finished population late last night, and shipped to us this morning, likely to arrive this week! Some images below: It's going to be a busy week of testing, as we're getting these, and also the Myriad X modules one day apart. On this board our test plan is to exercise all the standard Raspberry Pi features and IO (as it's supposed to act as a standard Raspberry Pi, minus WiFi for this revision), take notes on any errors/etc. for fixing in Altium, and then if all is OK-enough, connect the Myriad X module to it, and see if the whole thing runs together (with the code we have working so far). Any comments or questions? Feel free to drop them here or shoot an email to brandon at luxonis dot com. Best, The Luxonis Team

Hi again! So our first Myriad X modules finally shipped! So we're expecting to have these in-hand tomorrow, Tuesday August 20th. So back-story on these is that we ordered them on June 26th w/ 3-week turn from MacroFab (who we really like, and have used a lot), and the order was unlucky enough to fall in w/ 2 other orders that were subject to a bug in MacroFab's automation (which is super-impressive, by the way). So what was the bug? (You may ask!) Well, their front-end and part of the backend were successfully initializing all the correct actions (e.g. components order, bare PCB order, scheduling of assembly) at the correct times. However, the second half of the backend was apparently piping these commands straight to dev/null, meaning that despite the system showing and thinking that all the right things were being done, nothing was actually happening. So on July 15th, when the order was supposed to ship, and despite our every-2-day prodding up until then, it was finally discovered that the automation had done nothing, at all. So then this was debugged, the actual status was discovered, and the boards were actually started around July 22nd. Fast-forward to now, and this 3-week order is now a 8-week order - which should arrive tomorrow! Unfortunately, the only photo we got of the units, was one from a confirmation that the JTAG connector was populated in the right orientation (and it was), so here's a reminder of what the module looks like, rendering in Altium: And for good measure, the only photo we have of the boards so far, which is of the JTAG connector: So hopefully tomorrow we'll have working modules! And either way we'll have photos to share. Cheers! The Luxonis Team

Woefully behind on updates. We got the BW1099 in, and they work, and we got the BW1097 in, and they work! Pictures only because out of time these days with all the exciting hardware to play with and write code for!

And much more to come on firmware, software, and hardware updates. Oh and also our Crowd Supply pre-launch page is up. Need to update that too! https://www.crowdsupply.com/luxonis/depthai

Good job! Please, which accelerator on Myriad X do you use for the Stereo Depth implementation? Do you use the new "Stereo depth block" mentioned on this webpage: https://www.movidius.com/myriadx or do you use the SHAVE cores only? Thank you! Jan

Hi JanT Thanks! Yes we're using the stereo depth block and not the SHAVEs for the stereo implementation. This leaves the SHAVEs free for neural processing and also filtering on the depth, etc.: A bit more information is here: https://github.com/Luxonis-Brandon/DepthAI Thanks again, Brandon

Hi Brandon! This project looks amazing. What is the frame rate to be expected for depth camera + AI usage? In the sample video it looked like the framerate was around 2 or 3 fps. My project would need a higher rate.

GOB Thanks and good question GOB . So we're expecting 25 frames per second of depth and AI operating at the same time. We were expecting to have a demo of this for the launch, but don't have both stacks working correctly yet - so we used our slower, prototype video for now, which is yes slower at 2-3 frames per second.

Brandon Thanks for the update! If it's okay, can I email you with some more questions I have?

Luxonis DepthAI and megaAI | Overview and Status

Brandon

Our KickStarter rose $1,357,995! Quite exciting! Now we're heads-down building around 10,000 units and preparing the logistics to ship all of these out!

Brandon

So we're now heads-down designing and prototyping the aluminum enclosures we promised to our KickStarter backers as a result of hitting our $1-million-raise stretch goal. Some in-progress photos.

Most of the work involves making sure that the designs are thermally sound - given that they fully encase the heat-generator.

We were able to make a mode (for both OAK-1 and OAK-D) that uses more power in the chip than we've ever been able to make it use previously. And we used this mode for all the testing, so the thermal results below are for the absolute-worst-case power use of the part (or as close as we could get to it).

OAK-1:

We did a quick/dirty version to test thermal limits. It looks pretty good.

Our calculations estimated that at this max power use that the max surface temperature would be about 60C (the max recommended for metallic surfaces). And it came out to about that, at 61.3C:

To make sure the maximum surface temperature is below this 60C max recommendation, we increased the heat-sinking and size a bit, from a total thickness of _2.4cm top 2.7cm, and added some vertical aspects as well:

Prototypes of that will likely be in next week. We expect to see sub-60C external temperatures.

Even with the initial/smaller enclosure, we are seeing a max die temperature of less than 75C, and the max safe die temperature is 105C, so this gives nice margin.

Brandon

Our first example/reference for using the Gen2 Pipeline Builder system is now live!

https://github.com/luxonis/depthai-experiments/pull/8

And we have our first aluminum enclosure samples for BW1093 and BW1098OBC:

Brandon

Hi DepthAI Backers!

Thanks again for all the continued support and interest in the platform.

So we've been hard at work adding a TON of DepthAI functionalities. You can track a lot of the progress in the following Github projects:

Gen1 Feature-Complete: https://github.com/orgs/luxonis/projects/3
Gen2 December-Delivery: https://github.com/orgs/luxonis/projects/2
Gen2 2021 Efforts: (Some are even already in progress) https://github.com/orgs/luxonis/projects/4

As you can see, there are a TON of features we have released since the last update. Let's highlight a few below:

RGB-Depth Alignment

We have the calibration stage working now. And future DepthAI builds (after this writing) are actually having RGB-right calibration performed. An example with semantic segmentation is shown below:

The right grayscale camera is shown on the right and the RGB is shown on the left. You can see the cameras are slightly different aspect ratios and fields of view, but the semantic segmentation is still properly applied. More details on this, and to track progress, see our Github issue on this feature here: https://github.com/luxonis/depthai/issues/284

Subpixel Capability

DepthAI now supports subpixel. To try it out yourself, use the example here. And see below for my quickly using this at my desk:

Host Side Depth Capability

We also now allow performing depth estimation from images sent from the host. This is very convenient for test/validation - as stored images can be used. And along with this, we now support outputting the rectified-left and rectified-right, so they can be stored and later used with DepthAI's depth engine in various CV pipelines.

See here on how to do this with your DepthAI model. And see some examples below from the MiddleBury stereo dataset:

For the bad looking areas, these are caused by the objects being too close to the camera for the given baseline, exceeding the 96 pixels max distance for disparity matching (StereoDepth engine constraint):

These areas will be improved with extended = True, however Extended Disparity and Subpixel cannot operate both at the same time.

RGB Focus, Exposure, and Sensitivity Control

We also added the capability (and examples on how to use) manual focus, exposure, and sensitivity controls. See here for how to use these controls.
Here is an example of increasing the exposure time:
![image](

And here is setting it quite low:

It's actually fairly remarkable how well the neural network still detects me as a person even when the image is this dark.

Pure Embedded DepthAI

We mentioned in our last update (here), we mentioned that we were making a pure-embedded DepthAI.

We made it. Here's the initial concept:

And here it is working!

And here it is on a wrist to give a reference of its size:

And eProsima even got microROS running on this with DepthAI, exporting results over WiFi back to RViz:

RPi Compute Module 4

We're quite excited about this one. We're fairly close to ordering it. Some initial views in Altium below:

There's a bunch more, but we'll leave you with our recent interview with Chris Gammel at the Amp Hour!

https://theamphour.com/517-depth-and-ai-with-brandon-gilles-and-brian-weinstein/

Cheers,
Brandon & The Luxonis Team

Brandon

It's been a while... and there are more status updates than this, but check out the new POE IP67 version of DepthAI:

The enclosure is actually filled with water in this photo, and the single-directional vent is doing it's job by letting the water slowly seep out.

Brandon

We won product of the year!

https://www.edge-ai-vision.com/2021/05/2021-edge-ai-and-vision-product-of-the-year-award-winner-showcase-luxonis-and-opencv-cameras-and-sensors/

Brandon

It's cool to see the other safety products that are being built off of this. For example recently BlueBox Labs released their collision-deterring (among many other features) spatial AI dashcamera based on DepthAI on KickStarter:

So this could work inside the vehicle to even detect and alert if a (distracted) driver is going to hit a person riding a bike or a pedestrian.

More safety products are in the works. Very satisfying to see.

Cheers,
Brandon

Brandon

POE is now fully supported by DepthAI, including 2x IP67 DepthAI Models:
https://docs.luxonis.com/en/latest/pages/tutorials/getting-started-with-poe/

Brandon

Our lowest-cost model launched late last year and has been shipping for a while now.

Our next gen is also releasing now. OAK-D Series 2.

And these are available for early-adopters on our Beta store:
https://shop.luxonis.com/collections/beta-store

« Previous Page