python3 test.py - Failing XLinkReadData

jim

Hello All,

My first post. Great work, Brandon. I'm very excited to come across DepthAI, and to promptly receive my USB3 with modular cameras, over here in the UK.

I believe I've followed the relevant documentation for setting up the USB3 and for setting up the host side on different platforms.

Unfortunately, I appear to be hitting an initial data issue. I'm posting here to gain guidance, and potentially document for other new starters who may, in the future, face a similar issue.

python3 test.py output below:

`python3 depthai.py
No calibration file. Using Calibration Defaults.
Using Arguments= {'config_overwrite': None, 'board': None, 'field_of_view': 71.86, 'rgb_field_of_view': 68.7938, 'baseline': 9.0, 'rgb_baseline': 2.0, 'swap_lr': True, 'store_eeprom': False, 'clear_eeprom': False, 'override_eeprom': False, 'device_id': '', 'dev_debug': None, 'force_usb2': None, 'cnn_model': 'mobilenet-ssd', 'disable_depth': False, 'draw_bb_depth': False, 'full_fov_nn': False, 'streams': ['metaout', 'previewout'], 'video': None}
depthai.version == 0.0.10a
depthai.dev_version == 38cbd9ac41ff7e817dc4faa3e050f7033f71f797
XLink initialized.
Sending device firmware "cmd_file": /home/jim/depthai/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
watchdog started 3000
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
WARNING: Version (dev) does not match (unknown & 38cbd9ac41ff7e817dc4faa3e050f7033f71f797)
EEPROM data: invalid / unprogrammed
Available streams: ['meta_d2h', 'left', 'right', 'disparity', 'depth_sipp', 'metaout', 'previewout', 'jpegout', 'video', 'object_tracker']
CNN configurations read: /home/jim/depthai/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd_depth.json
depthai: Calibration file is not specified, will use default setting;
config_h2d json:
{"board":{"homography_right_to_left":[0.988068163394928,0.0029474012553691864,5.067617416381836,-0.008765067905187607,0.9921473264694214,-8.795275688171387,-8.449587767245248e-06,-3.603489403758431e-06,1.0]},"load_inBlob":true,"pipeline":{"_streams":[{"name":"metaout"},{"name":"previewout"}]},"ai":{"calc_dist_to_bb":true,"keep_aspect_ratio":true},"board":{"clear-eeprom":false,"left_fov_deg":71.86000061035156,"left_to_rgb_distance_m":0.019999999552965164,"left_to_right_distance_m":0.09000000357627869,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":68.7938003540039,"stereo_center_crop":false,"store-to-eeprom":false,"swap-left-and-right-cameras":true},"depth":{"confidence_threshold":0.5,"depth_limit_mm":10000,"padding_factor":0.30000001192092896},"ot":{"confidence_threshold":0.5,"max_tracklets":20}}
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1000 bytes to config_h2d
!!! XLink write successful: config_h2d (1000)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Creating observer stream host_capture: ...
Attempting to open stream host_capture
Successfully opened stream host_capture with ID #0!
Creating observer stream host_capture: DONE.
Read: 14514560
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14514560 bytes to inBlob
!!! XLink write successful: inBlob (14514560)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file /home/jim/depthai/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #2!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
CNN input width: 300
CNN input height: 300
CNN input num channels: 3
CNN to depth bounding-box mapping: start(318, 78), max_size(643, 643)
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Successfully opened stream metaout with ID #1!
Starting thread for stream: metaout
Host stream start:previewout
Opening stream for read: previewout
Attempting to open stream previewout
Started thread for stream: metaout
Successfully opened stream previewout with ID #2!
Starting thread for stream: previewout
depthai: INIT OK!
Started thread for stream: previewout
E: [global] [ 473466] [python3] XLinkReadData:165 addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
Closing stream metaout: ...
E: [global] [ 473466] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [ 473466] [Scheduler00Thr] sendEvents:1036 Event sending failed
Closing stream metaout: DONE.
Thread for metaout finished.
E: [global] [ 473468] [python3] XLinkReadData:165 addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
Closing stream previewout: ...
E: [global] [ 473468] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [ 473468] [Scheduler00Thr] sendEvents:1036 Event sending failed
Closing stream previewout: DONE.
Thread for previewout finished.
watchdog triggered
Stopping threads: ...
Stopping threads: DONE 0.000s.
Closing all observer streams: ...
Closing stream host_capture: ...
E: [global] [ 475399] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [ 475399] [Scheduler00Thr] sendEvents:1036 Event sending failed
Closing stream host_capture: DONE.
Closing all observer streams: DONE.
Reseting device: 0.
E: [global] [ 475399] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -4) | event XLINK_RESET_REQ

E: [xLink] [ 475399] [Scheduler00Thr] sendEvents:1036 Event sending failed
Reseting: DONE.
XLink initialized.
Sending device firmware "cmd_file": /home/jim/depthai/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
<snip>
`

The lines ..
WARNING: Version (dev) does not match (unknown & 38cbd9ac41ff7e817dc4faa3e050f7033f71f797)
EEPROM data: invalid / unprogrammed
.. sound a bit scary

Notes:

upon running test.py, the USB product ID changes from 2485 to a f63b, as detailed in the docs
pressing the reset button on the board appears to make no discernible difference to the output, nor leaving the board to cycle through the same process many times
I get the same output, irrespective of using a laptop or a Raspberry Pi

Laptop> Ubuntu 18.0.4.4 LTS, Python 3.6.9
RaspberryPi> RaspberryPi OS, Python 3.7.3

Any and all guidance, most gratefully received.

I'm hoping it's something obvious.

Looking forward to seeing the depth & object detection in action,

Best regards,
Jim

Brandon

Hi Jim,

Thanks for the post!

So the warning about (dev) does not match (unknown & 38cbd9ac41ff7e817dc4faa3e050f7033f71f797) is likely the issue.

So we should probably make this an error actually rather than a warning (but we haven't yet as sometimes internally and externally we want to allow running unmatched versions).

So when the versions don't match, the host-side API and the device-side API may be sending data using different formats, hence the system will fail to talk properly and reset.

And on the EEPROM not being programmed: For the BW1098FFC this is expected. Details below:

So on the FFC variant, the EEPROM is left unprogrammed so that the user can program in their own spacing between the cameras, stereo calibration (which depends on how the cameras are mounted) etc. It even allows for using different field-of-view cameras (which is possible, but we only have very limited number of units).

So when you build a specific setup (say a given mechanical configuration that you may re-use) you can save the parameters of which to the EEPROM. See below on how to do so:
https://docs.luxonis.com/tutorials/stereo_calibration/#modular_cameras

The name ACME01 is just an example, you could have for example 3Dprotodesign1 or whatever you'd like.

So back to the issue at hand:

How are you currently using the DepthAI repo? So if you modify a file in the depthai folder, and then do a git pull, it can update everything else except that one file, causing a conflict in terms of versions/communication.
- I'd be curious if you do a fresh git clone https://github.com/luxonis/depthai into some new folder if the system will now work.

I'm also going to ask the engineers to see if I'm missing anything here.

Thoughts?

Thanks,
Brandon

Brandon

Oh also another question:

Could you post a picture of the setup?

We've seen before where if the FFC are not properly seated (i.e. partially seated) they can cause a failure to initialize.

And the following will help diagnose if this is the case. I'd recommend first doing the fresh clone of the repo before doing these in case something is off w/ the current cloned version:

Test the left camera alone:
python3 test.py -s left

Test the right camera alone:
python3 test.py -s right

Test the color camera alone (with no neural inference):
python3 test.py -s previewout

So when running python3 test.py the system pulls data from the left, right, and color and combines them together to show the 3D object localization (xyz of the object). So if any of the cameras don't have a proper connection the system can fail.

And if this is the case (that one of the cameras isn't working), if the color cameras is working, it is possible to run neural inference w/out depth information with:

python3 test.py -s metaout previewout -dd

Where -dd stands for "disable depth".

Thanks,
Brandon

Brandon

So @jim - talking to the team, we think that likely the color camera is not properly connected. Can you post a picture of the setup? It could be the FFC is just inserted upside down. I need to post a video on how to insert the FFC... I took one a bit ago but don't immediately remember where it is.

I'll plan on doing that soon.

Thoughts?

Thanks,
Brandon

jim

Excellent, thank you. The FFC connections do appear to be the culprit. The colour camera is now operational, after some careful squeezing at the camera end.

The left & right cameras continue to give the same XLinkReadData issue. I'll wait for your input as whether I should flip the cables over. Pictures below.

Unfortunately in my checking I've managed to break the end of the FFC clasp for the left camera's connection to the main board. So I'll have to find a similar FFC or find some way of keeping the pressure sustained.

Many thanks,
Jim

Brandon

Hi jim ,

Sorry about the delay here. So I meant to share a video like this in our docs forever ago, and just kept not getting to it. So I've finally uploaded it to youtube and here it is below:

So the FFC you have there are actually all upside-down. So if the color camera did work, it's likely because of capacitive coupling through the little blue plastic bit on the flexible flat cable.

So here's a view of what the flexible flat cable should look like when inserted:

And be super careful when opening the FFC latch. Slowly open it by iterating from left to right side slowly to make sure to not rip apart the assembly.

Thoughts?

Thanks,
Brandon

Brandon

And WRT the broken CAM_L connector, if the black latch is still there and can still be pushed in above the flexible flat cable (FFC) then it will likely still work.

jim

Hey Brandon,

Well, to be honest, I don't think I could have got those connections any more wrong ...

The very good news, it all seems to be working. No damage done.

I'm very impressed the colour camera worked at all!

Thank you. The photo was just what I needed,

Many thanks,
Jim

Brandon

Thanks @jim for the quick feedback! I'm glad it's all running and apologies for not having these really documented well at all on our site. I have on my to-do list to rectify that today!

Will make it 10000% clearer in the future how to connect everything. And also you're not alone on the install as you did it, one of our local customers did the exact same thing... so when that happened I should have updated the documentation but just didn't get to it.

So on calibrating the stereo pair now that you have it working, the instructions for that are here:
https://docs.luxonis.com/tutorials/stereo_calibration/

One thing to note though:
With your install of the cameras (where the RGB is above the grayscale), if you run inference off of the color camera, it will be offset in y dimension by the offset of the color camera. This may not be a big deal, but just wanted to mention it. If you're looking for very-little things it could matter for example. For looking at people or not-small things (i.e. not something like a grape for example) it won't matter.

If it does matter, we are actually implementing a feature that allows you to run neural inference on either or both of the grayscale cameras. This itself then makes the color camera not an issue, even for tiny objects. The in-progress pull-request for this feature is here.

And in fact, this sort of flow is ideal for finding the 3D location of small objects. So this now means DepthAI can be used two modalities:

Pull the disparity depth results that are 'behind' the object detector, use those to reproject xyz location of the center of object.
Run the neural network in parallel on both left/right grayscale cameras, use the results to triangulate the location of features.

So 1 has been the only way to use DepthAI before this 2 option. An example where 2 is extremely useful is finding the xyz location of eyes, nose, and corners of the mouth, like below:

So if using 1 to do this, it could be SUPER difficult, as the locations of the eyes, nose, edges of mouth are all very little... so the probability that there might be a 'hole' in the disparity depth is high (the smaller the object, the higher the probability).

So using this 2nd modality totally circumvents that problem.

Thoughts?

Thanks again,
Brandon

jim

Excellent. To be fair, the paired left/right grayscale cameras is my prime motivator, from a 3D/depth understanding of the environment, so developments such as NN on both left/right is greatly of interest.

My intention, I don't know how practical, is to bring in the colour camera to use only when focal tasks require it, such as checking the colour of a surface e.g traffic light detection.

Once I've experimented a bit, I hope to send period commands to the module to restrict the colour json stream to only a specific part of the image at any one time, to mirror cortical attentional type processing. Any thoughts about the likely pitfalls for this approach, greatly appreciated

p.s The new YouTube video is very good, a very clear explanation. I don't think anyone else will be wiring the FFC incorrectly

Brandon

Hi @jim ,

Great! Thanks.

My intention, I don't know how practical, is to bring in the colour camera to use only when focal tasks require it, such as checking the colour of a surface e.g traffic light detection.

Once I've experimented a bit, I hope to send period commands to the module to restrict the colour json stream to only a specific part of the image at any one time, to mirror cortical attentional type processing. Any thoughts about the likely pitfalls for this approach, greatly appreciated

So as our pipeline builder (v1, JSON config here) stands now the streams requested are 'all or nothing' sort of thing, but you can always selectively throw away results on the host.

In the v2 pipeline builder we will support returning only feeds from regions of interest (say just the image data inside a bounding box). So you'd be be able to do what you're talking about in this v2 pipeline builder. So if a neural network detects a region of interest, you can just get results from that region when detection occurs, sort of thing. Which I think fits with what you are describing here.

And thanks for the comments on the Youtube video. So I actually took another one, here: https://docs.luxonis.com/products/bw1098ffc/#connect-your-modular-cameras

Thanks again,
Brandon

jim

Excellent. The v2 pipeline builder sounds just right.

Regarding the video, that was the one I actually meant to reference. I gave it a like : )

Looking forward to getting in deep with the code, starting tomorrow,

Best,
Jim

Brandon

Excellent; thanks!