Hello there.
I have recently embarked on a slightly crazy journey. Before I get too far into it, I would appreciate feedback from the Luxonis team on where I might see difficulties.
Goals and challenges
We would like to use a microcontroller (the ESP32-S3 in particular) as a host for the OAK-D Pro camera, connected over USB.
Our minimal proof-of-concept goals:
- Boot up the Myriad X with DepthAI firmware
- Keep the device alive with XLink pings
- Open an XLink stream for RPC and use it to load in a pre-serialized pipeline schema and start the pipeline
- Open an XLink stream to receive metadata
Here are the features I get on this platform (ESP-IDF):
- USB On-The-Go (OTG) host capable of control, bulk, interrupt and isochronous transfers
- Full C and C++ standard libraries and C++20 compiler support
- Most (?) of POSIX, notably including pthread (wraps FreeRTOS task routines)
- Basic filesystem (FAT32 on an SD card)
The main hardware limitations are:
- Less than 1 MiB RAM
- Only full-speed USB (i.e. 12 Mbps in theory, but in practice even slower)
The RAM limitation is my biggest concern. Do you think it's feasible to work with small buffers here? For example, my bulk transfers are currently being chunked at 16 KiB.
Progress so far
Having read a previous thread on this subject, I examined the XLink library and how depthai-core utilises it. I ported some XLink platform routines, and soon I was able to transfer and boot the DepthAI firmware (takes a breezy 30 seconds to do so). I had to make a few small modifications to XLink, and write a bunch of custom routines, to get this working.
At this point, I looked at XLinkConnection and XLinkStream in the core, and decided that hand-rolling these would be a nightmare, so I pulled in these classes into my project. So far, I'm just instantiating oneXLinkConnection and seeing what happens.
My sticking point is that XLink is unable to keep the connection alive to the device. Here's a sample log (starting upon device boot just after firmware is sent; a few seconds after this log, the device shuts down):
D: [global] [ 33855] [ThreadName_N/A] XLinkConnect:240 XLinkConnect() device name x glHandler 0x3fccee78 protocol 1
D: [global] [ 33858] [ThreadName_N/A] dispatcherLocalEventGetResponse:119 XLINK_PING_REQ
D: [global] [ 33865] [ThreadName_N/A] dispatcherLocalEventGetResponse:254 XLINK_PING_REQ - do nothing
D: [global] [ 33875] [ThreadName_N/A] dispatcherEventSend:51 Send event: XLINK_PING_REQ, size 0, streamId 0.
W: [global] [ 34117] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
D: [global] [ 34123] [ThreadName_N/A] handleIncomingEvent:723 XLINK_PING_RESP, size 687903012, streamId 67.
D: [global] [ 34133] [ThreadName_N/A] dispatcherRemoteEventGetResponse:288 XLINK_PING_RESP
D: [global] [ 34134] [ThreadName_N/A] handleIncomingEvent:723 XLINK_CREATE_STREAM_REQ, size 128, streamId 3735936685.
F: [global] [ 34154] [ThreadName_N/A] dispatcherResponseServe:930 no request for this response: XLINK_PING_RESP 1
D: [global] [ 34154] [ThreadName_N/A] handleIncomingEvent:723 XLINK_CREATE_STREAM_REQ, size 5242880, streamId 3735936685.
D: [global] [ 34165] [ThreadName_N/A] dispatcherRemoteEventGetResponse:288 XLINK_CREATE_STREAM_REQ
D: [global] [ 34178] [ThreadName_N/A] handleIncomingEvent:723 XLINK_CREATE_STREAM_REQ, size 131072, streamId 3735936685.
I (34395) main: Connection created
D: [global] [ 34199] [ThreadName_N/A] XLinkAddOrUpdateStream:40 name: __watchdog, writeSize: 0, readSize: 128, forcedId: -559030611
D: [global] [ 34216] [ThreadName_N/A] XLinkStreamInitialize:20 name: __watchdog, id: 0
D: [global] [ 34225] [ThreadName_N/A] XLinkAddOrUpdateStream:107 The stream "__watchdog" created, id = 0, writeSize = 0, readSize = 128
D: [global] [ 34238] [ThreadName_N/A] dispatcherRemoteEventGetResponse:393 creating stream 0
D: [global] [ 34248] [ThreadName_N/A] dispatcherEventSend:51 Send event: XLINK_CREATE_STREAM_RESP, size 128, streamId 0.
W: [global] [ 34459] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
D: [global] [ 34459] [ThreadName_N/A] dispatcherRemoteEventGetResponse:288 XLINK_CREATE_STREAM_REQ
D: [global] [ 34474] [ThreadName_N/A] XLinkAddOrUpdateStream:40 name: __rpc_main, writeSize: 0, readSize: 5242880, forcedId: -559030611
D: [global] [ 34487] [ThreadName_N/A] XLinkStreamInitialize:20 name: __rpc_main, id: 1
D: [global] [ 34496] [ThreadName_N/A] XLinkAddOrUpdateStream:107 The stream "__rpc_main" created, id = 1, writeSize = 0, readSize = 5242880
D: [global] [ 34510] [ThreadName_N/A] dispatcherRemoteEventGetResponse:393 creating stream 1
D: [global] [ 34519] [ThreadName_N/A] dispatcherEventSend:51 Send event: XLINK_CREATE_STREAM_RESP, size 5242880, streamId 1.
W: [global] [ 34734] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
D: [global] [ 34734] [ThreadName_N/A] dispatcherRemoteEventGetResponse:288 XLINK_CREATE_STREAM_REQ
D: [global] [ 34749] [ThreadName_N/A] XLinkAddOrUpdateStream:40 name: __log, writeSize: 0, readSize: 131072, forcedId: -559030611
D: [global] [ 34762] [ThreadName_N/A] XLinkStreamInitialize:20 name: __log, id: 2
D: [global] [ 34770] [ThreadName_N/A] XLinkAddOrUpdateStream:107 The stream "__log" created, id = 2, writeSize = 0, readSize = 131072
D: [global] [ 34783] [ThreadName_N/A] dispatcherRemoteEventGetResponse:393 creating stream 2
D: [global] [ 34793] [ThreadName_N/A] dispatcherEventSend:51 Send event: XLINK_CREATE_STREAM_RESP, size 131072, streamId 2.
E (35259) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 34999] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (35524) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 35263] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (35789) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 35528] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (36054) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 35793] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (36319) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 36058] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (36584) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 36323] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (36849) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 36588] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (37114) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 36853] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
E (37379) Device: rx bulk_transfer timed out, resetting endpoint
W: [global] [ 37118] [ThreadName_N/A] dispatcherEventReceive:96 dispatcherEventReceive() Read failed -1
These logs are difficult to parse. I'm not sure yet which reads are failing or why. Some responses seem to contain garbage, but I can't yet tell you whether that's uninitialized memory or junk from the USB.
Any advice at all would be appreciated. I understand that I'm venturing way beyond what you guys officially support. That being said, it would be very exciting to be able to run a mainline OAK-D camera from a small cheap device like an ESP32. 🙂
