• DepthAI
  • Initial BW1092 ESP32 Proof of Concept Code

We have an initial implementation for the DepthAI SPI communication (streaming of NN metadata for now). The SPI protocol that Martin described above is not integrated yet, for now the packet consists of a NULL-terminated string.
The DepthAI boot and initialization is still done by USB, and afterwards an SPI controller can pull the data off SPI. Tested with ESP32:
https://github.com/luxonis/depthai/compare/spi_esp32
https://github.com/luxonis/depthai-python/compare/master...spi_esp32
6:19
The ESP32 app is based on this example: https://github.com/espressif/esp-idf/tree/v4.0.1/examples/peripherals/spi_slave/sender
with these modifications: https://github.com/luxonis/depthai-python/commit/619d485

As DepthAI streams the NN metadata on its own (when the controller initiates communication), to test that the COPI (MOSI) line works well, the ESP32 app sends a NULL-terminated string like esp32-cnt:253 that is echoed back by DepthAI, together with the NN metadata. After DepthAI boots up (by running ./depthai.py), the prints in the ESP32 app should look like:

I (0) cpu_start: Starting scheduler on APP CPU.
I (305) gpio: GPIO[2]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:1 
[RECV-0] 
[RECV-1] 
[RECV-2] 
[RECV-3] 
[RECV-4] 
... 
[RECV-106] 
[RECV-107] 
[RECV-108] 
[RECV-109] 
[RECV-110] 
[RECV-111] 
[RECV-112]  | DepthAI pkt 0:
[RECV-113] esp32-cnt:108 | DepthAI pkt 1:
[RECV-114] esp32-cnt:109 | DepthAI pkt 2:
[RECV-115] esp32-cnt:110 | DepthAI pkt 3:
[RECV-116] esp32-cnt:111 | DepthAI pkt 4:
[RECV-117] esp32-cnt:112 | DepthAI pkt 5:
[RECV-118] esp32-cnt:113 | DepthAI pkt 6:
[RECV-119] esp32-cnt:114 | DepthAI pkt 7:
[RECV-120] esp32-cnt:115 | DepthAI pkt 8:
[RECV-121] esp32-cnt:116 | DepthAI pkt 9:
[RECV-122] esp32-cnt:117 | DepthAI pkt 10:
[RECV-123] esp32-cnt:118 | DepthAI pkt 11:
[RECV-124] esp32-cnt:119 | DepthAI pkt 12:
[RECV-125] esp32-cnt:120 | DepthAI pkt 13:
[RECV-126] esp32-cnt:121 | DepthAI pkt 14:
[RECV-127] esp32-cnt:122 | DepthAI pkt 15:
[RECV-128] esp32-cnt:123 | DepthAI pkt 16:
[RECV-129] esp32-cnt:124 | DepthAI pkt 17:
[RECV-130] esp32-cnt:125 | DepthAI pkt 18:
[RECV-131] esp32-cnt:126 | DepthAI pkt 19:
[RECV-132] esp32-cnt:127 | DepthAI pkt 20:
[RECV-133] esp32-cnt:128 | DepthAI pkt 21:
[RECV-134] esp32-cnt:129 | DepthAI pkt 22:
[RECV-135] esp32-cnt:130 | DepthAI pkt 23:
[RECV-136] esp32-cnt:131 | DepthAI pkt 24:
[RECV-137] esp32-cnt:132 | DepthAI pkt 25:
[RECV-138] esp32-cnt:133 | DepthAI pkt 26:
[RECV-139] esp32-cnt:134 | DepthAI pkt 27:
[RECV-140] esp32-cnt:135 | DepthAI pkt 28:
[RECV-141] esp32-cnt:136 | DepthAI pkt 29:
[RECV-142] esp32-cnt:137 | DepthAI pkt 30:
[RECV-143] esp32-cnt:138 | DepthAI pkt 31:
[RECV-144] esp32-cnt:139 | DepthAI pkt 32:
[RECV-145] esp32-cnt:140 | DepthAI pkt 33:
					Det0: tvmonitor    57.67% (-0.01,0.02)->(1.00,1.00)
[RECV-146] esp32-cnt:141 | DepthAI pkt 34:
[RECV-147] esp32-cnt:142 | DepthAI pkt 35:
[RECV-148] esp32-cnt:143 | DepthAI pkt 36:
[RECV-149] esp32-cnt:144 | DepthAI pkt 37:
					Det0: person       78.27% (0.23,0.23)->(0.41,0.47)
[RECV-150] esp32-cnt:145 | DepthAI pkt 38:
					Det0: person       83.30% (0.22,0.22)->(0.41,0.47)
[RECV-151] esp32-cnt:146 | DepthAI pkt 39:
					Det0: person       92.43% (0.21,0.23)->(0.41,0.47)
[RECV-152] esp32-cnt:147 | DepthAI pkt 40:
					Det0: person       95.61% (0.20,0.22)->(0.41,0.48)
					Det1: person       62.06% (0.04,0.26)->(0.23,0.48)
[RECV-153] esp32-cnt:148 | DepthAI pkt 41:

An initial version of the USB flasher/bootloader is implemented: branch spi_esp32, these commits:
https://github.com/luxonis/depthai/compare/127ff77...c22bac6
Initial flashing procedure:
(during these steps it's handy to have a dmesg -wH terminal running, to check device state)

  1. Boot the 'loader.cmd' firmware over USB: ./depthai.py -boot loader.cmd. A new DFU device should appear: 03e7:f003.
  2. Download the 'loader.cmd' firmware, in order to be flashed: dfu-util -D loader.cmd . At the end of the process, the DFU device should be re-enumerated by the host.
  3. At this point it is possible to switch the board to Flash boot mode, recommended boot mode 0x8 (Quad-SPI 50MHz) on DIP switches:
    001000
    123456
  4. Download the 'depthai.cmd' firmware, in order to be flashed: dfu-util -D depthai.cmd. Note that the flashing process takes longer and is not finished by the time dfu-util exits (this will be fixed). At the end of the process, depthai should boot and create the usual 03e7:f63b device. The device will reboot every few seconds (because of the watchdog), until the USB host connects: ./depthai.py -boot '' -- this specifies that no firmware has to be loaded.
    Special pin functions:
    • header SPI2.5 (GPIO35_3V3) asserted low, followed by a hard-reset: the device enters in DFU mode, allowing to update the application (or the bootloader).
    • header SPI2.6 (TAO_WAKE/GPIO34_3V3) asserted low, followed by a hard-reset: the device switches to USB-boot mode, allowing to run for example ./depthai.py. The device stays in USB-boot mode during soft-resets (depthai being closed - reopened), until it is hard reset.

TODO (WIP): add options to bootloader to write application sections to Flash (config, network blob), and modify depthai.cmd to use them, allowing no-USB operation. This will require switching the Flash part from 24-bit to 32-bit addressing, to access data past 16 MiB boundary. A non-volatile Flash config of 32-bit addressing would be preferred, to allow boot after a hard-reset without power-cycle.

Running NN from Flash (without USB support) is implemented now, please pull the latest changes from https://github.com/luxonis/depthai-python/tree/spi_esp32.
Updated usage guide:

  1. If the board already contains an older loader.cmd bootloader flashed, and the switches are set to boot from flash: Instruct the bootloader to stay in waiting-for-update mode and not boot the existing app: short SPI2.5 to GND and reset the board. Go to step 2.
  2. If the board is set to USB boot (default): Boot the loader.cmd firmware over USB: ./depthai.py -boot loader.cmd. A new DFU device should appear: 03e7:f003.
  3. Download the 'loader.cmd' firmware, in order to be flashed: dfu-util -D loader.cmd . The process will take several seconds, please wait until the DFU tool reports 'dfuIDLE' state and exits. At the end of the process, the device should restart back in DFU mode.
  4. If the board is set to USB boot, switch it to Flash boot, recommended boot mode 0x8 (Quad-SPI 50MHz) on DIP switches:
    001000
    123456
    And then reset it.
  5. Build a DepthAI firmware update package, that will contain the application firmware, config and NN blob(s). Run the 'depthai.py' script with the desired parameters and append -build. For example to use defaults: ./depthai.py -build.
  6. Download the 'depthai_flash.fw' package to the device: dfu-util -D depthai_flash.fw. Note that the flash programming is a slow operation and may take several minutes, depending on the size of the package. Please wait until the DFU tool reports the final state (for success dfuIDLE is expected). Alternately it's possible to check the status on the SoM LED, near the boot switches. While the bootloader waits, or during USB transfers, it will double-blink every second. When flashing is is progress, it will blink fast, 6 times per second.
  7. Reset the board and it should work as an SPI peripheral, from where an SPI controller can pull the NN+depth metadata. The format is still text for now.
12 days later

Quick additional update: YOLO is now supported w/ on-DepthAI parsing before export over SPI:
We merged the SPI capability with the yolo tensor decoding, and this is now in the spi_esp32 branch.

The new text format printed by existing ESP32 app looks like below. It is currently limited to 3 simultaneous detections due to 256 bytes packet size.

[RECV-156478] esp32-cnt:156473 DepthAI 1777
[RECV-156479] esp32-cnt:156474 DepthAI 1778
[RECV-156480] esp32-cnt:156475 DepthAI 1779
0 lbl 27 54.0% xy0xy1 0.537 0.377 0.592 0.527 xyz 0.00 0.00 0.00
[RECV-156481] esp32-cnt:156476 DepthAI 1780
0 lbl 0 67.9% xy0xy1 0.471 0.112 0.652 0.540 xyz 0.00 0.00 0.00
[RECV-156482] esp32-cnt:156477 DepthAI 1781
[RECV-156483] esp32-cnt:156478 DepthAI 1782
0 lbl 0 74.5% xy0xy1 0.514 0.129 0.760 0.583 xyz 0.00 0.00 0.00
1 lbl 27 69.6% xy0xy1 0.634 0.414 0.690 0.556 xyz 0.00 0.00 0.00
2 lbl 27 51.9% xy0xy1 0.618 0.414 0.670 0.552 xyz 0.00 0.00 0.00
[RECV-156484] esp32-cnt:156479 DepthAI 1783
0 lbl 0 52.5% xy0xy1 0.532 0.127 0.739 0.581 xyz 0.00 0.00 0.00
[RECV-156485] esp32-cnt:156480 DepthAI 1784
[RECV-156486] esp32-cnt:156481 DepthAI 1785
[RECV-156487] esp32-cnt:156482 DepthAI 1786
[RECV-156488] esp32-cnt:156483 DepthAI 1787
0 lbl 27 67.6% xy0xy1 0.767 0.421 0.828 0.562 xyz 0.00 0.00 0.00
1 lbl 0 57.6% xy0xy1 0.636 0.135 0.846 0.586 xyz 0.00 0.00 0.00
[RECV-156489] esp32-cnt:156484 DepthAI 1788
0 lbl 27 74.3% xy0xy1 0.765 0.406 0.824 0.551 xyz 0.00 0.00 0.00
[RECV-156490] esp32-cnt:156485 DepthAI 1789

Below is a quick update guide (assuming the new bootloader is already updated):

  1. Pull the latest changes and update git submodules, then build the python lib (for Linux x86_64 the library is also prebuilt).
  2. Put the device in DFU update mode by shorting SPI2.5 pin to ground and reset or power-cycle. The 1099 SoM LED should start double-blinking once per second.
  3. Build the Flash FW update package, like: ./depthai.py -cnn tiny-yolo-v3 -build. In this case its size is about 23.6MB.
  4. Send it over DFU: dfu-util -D depthai_flash.fw. In this case the entire process will take about 2m57s. The first part (USB transfer) takes 34 seconds -- that's too much and related to DFU, we'll eventually look how to shorten it.
  5. The updated app+blob will boot automatically. If SPI2.5 is still asserted, release it, to not have the device re-enter bootloader after another reset.

We also tried with the full YOLO model, -cnn yolo-v3 , it works. The package gets to 130.3MB (124.3MiB), so using almost the entire capacity of the SPI Flash -- 128MiB (besides the FW+blob, we also have 0.5MiB reserved for the bootloader).

The entire update process takes 14m49s, from where the USB transfer is 3m07s.

15 days later

For updating the firmware package (depthai_flash.fw) on BW1092, currently GPIO4 of ESP32 should be asserted low, to put the DepthAI bootloader in DFU update mode. The ESP32 demo app was updated: https://github.com/luxonis/depthai-python/commit/99ffed0
The macro ENABLE_FW_UPDATE should be set to 1, and the ESP32 app reflashed. Then the DepthAI firmware can be upgraded using the dfu-util tool. When finished, revert the change to the ESP32 app.

2 months later