Hi, I've recently purchased an OAK-FFP-3P along with three Arducam AR0234 camera modules and I'm having difficulty with pretty much all the examples, I'm entirely expecting this to be user error and figure there's something fundamental I've missed. If I write my own simple scripts using bits from various examples I can get a list of the three camera modules and they are recognised fine, I can even run one of the RGB examples that shows the source and cropped image, but I can't get IMU data or seem to get images from the left or right cameras. Is there an idiots guide to getting started with the 3P board please? I'm a software engineer with a lot of experience but still relatively new to computer vision, if anyone could give me pointers to get started I'd appreciate it. Photo for attention, this is the bot I'll be putting the 3P in to once I've got it working 🙂

Hi kneave , I think most demos don't work because you have 3x ColorCamera, and many demos use MonoCamera depthai nodes. After OAK FFC calibration you can also do stereodepth, demo here , which also shows how to use multuple ColorCamera nodes (instead of MonoCamera nodes). Thoughts? And that robot looks really cool!! Would love to see picture with the OAK FFC on there🙂 Thanks, Erik

Ah! I thought that mono would just return a mono image, I didn't realise that it would also require or expect a mono camera. This all looks ideal, I'll give it a go today. Thanks for the tips! The plan is for wide angle colour cameras for the stereo pair and a narrower lens on the centre camera. I figure that'll give a good combination for SLAM and a more zoomed in camera for distance viewing. I'm hoping to find a small camera for the centre so that I can hide it behind the panel between his eyes and not spoil the aesthetic 🙂

I've switched to and built the develop branch of depthai_python, I saw that the multi_cam_support branch was merge in a few weeks ago, and I've gone through calibration.py from the depthai library and modified it for three colour cameras. I'm really struggling to more than a few calibration images accepted by the calibration script, none at all unless I disable the RGB sensor, could this be because I'm using fisheye lenses? The left and right camera have 160 degree lenses and the RGB has a 90 degree lens, do I have to do anything different to enable a fisheye mode? I know that's something I've needed to do with ROS before. Here is the calibration file I created in case it helps. { "board_config": { "name": "NE3P", "revision": "V1.0", "swap_left_and_right_cameras": false, "left_fov_deg": 160, "rgb_fov_deg": 90, "left_to_right_distance_cm": 8.0, "left_to_rgb_distance_cm": 4.0 } } Any tips would be welcome, I'm happy I'm at least getting images now but it's still a little frustrating.

Hi @kneave , Try to calibrate using this branch ( https://github.com/luxonis/depthai/tree/ar0234_x3 ), you have to add a custom board config to work, try adding something like this in depthai/resoruces/boards/FFC-3P.json, and then be sure to select the board while doing the calibration for example python3 calibrate.py -brd FFC-3P. Be sure that you use the HFOV, not DFOV or something else, and check if the left/right/rgb cameras are in the right place (change the CAM_A/B/C ports as needed) { "board_config": { "name": "NE3P", "revision": "R1M0E1", "cameras":{ "CAM_B": { "name": "right", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_C", "specTranslation": { "x": 8, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_C": { "name": "left", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_A", "specTranslation": { "x": 4, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_A": { "name": "rgb", "hfov": 90, "type": "color" } }, "stereo_config":{ "left_cam": "CAM_B", "right_cam": "CAM_C" } } } if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); }

Thanks, I'll have a look at this today hopefully

I've gone through this a few times today and gotten much further so thanks for the tips! Pretty sure one of the issues I was facing was due to a poor quality calibration image, the printer put out something more grey than black so I reprinted on photo paper and it's much happier. First few times I got XLink errors, pretty sure I knocked a cable, and then when I was almost done the battery in my laptop died as it was taking the last image! I just got through them all for the first time after a recharge, it collected them all then hit an error in OpenCV. I've not had a chance to debug it yet, I'll have another look tomorrow. OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-5rb_9df3\opencv_contrib\modules\aruco\src\charuco.cpp:448: error: (-215:Assertion failed) _markerCorners.total() == _markerIds.getMat().total() && _markerIds.getMat().total() > 0 in function 'cv::aruco::_interpolateCornersCharucoLocalHom' Though it now looks fairly obvious, "appveyor" isn't my username so I suspect that there is a hard coded path somewhere in the calibration script. I'll have a look tomorrow, hopefully it's an easy win. 🙂

Hmm, no, AppVeyor isn't mentioned in the repo at all that I can find. How very odd! Edit:just realised AppVevor is a CI tool so that's likely the path for the file on the build server and I've the resulting assembly on my machine. Will dig in to the code and figure out what it's doing, or not, as the case may be.

Just for those who hit a similar issue, with the branch above I needed to downgrade OpenCV to the following version manually: opencv-contrib-python==4.5.5.64 I'm now hitting an issue where it seems to get through everything but complain there's too big a difference between stereo cameras and RGB, I'm assuming as stereo are 160deg and the RGB is 90deg. I'm going to try with the -drgb flag to see if it can calibrate the stereo cameras at least.

Hi kneave , We apologize for the inconvenience - could you copy the full error that you get? Thanks, Erik

Hi Erik, will do. I'll go through again and grab it shortly. Are the images saved to the disk too by the way? It'd be handy if I could run calibration again without having to recapture the images each time. It'd speed up debugging no end!

OAK-FFP-3P User Guide For Dummies?

kneave

Hey erik, I'll have a look and see if I can reproduce it consistently and put together all the config files and whatnot that I have so far.

Thanks DanielCalaver, each time with the 160/90/160 lens combo I got the epipolar errors, ignoring the error_text to see what should happen next seemed like a good idea and I think you said that the errors meant that the performance wouldn't be great but that it should still work with errors like that? With it ignored I got the message saying that upload had completed followed by "done" so I think it may be worth changing that to "done with errors" in cases where it doesn't work? It may help customers with troubleshooting for sure.

Could I ask what the maximum FOV of lenses you've tested with for stereo cameras please? I had a look at the Pro W and it seems that they have a HFOV of 120 so my lenses shouldn't be that far out from this setup. The calculated HFOV I got was 122.91 and with the centre camera having a HFOV of 90 I think that's really close to what you have on the W too?

Another thing I'd like to confirm is that I haven't messed up my board config. I'm more confident now of the HFOV values but the specTranslation bit I'm not certain of. I based the positive and negative values off the other board definitions in the depthai repo, I assume that the translation values are relative to the to_cam reference? So in my config the left camera is 8cm to the left of the right, and the right camera is 4cm to the right of the RGB cam.

{ "board_config": { "name": "FFC-3P", "revision": "R1M0E1", "cameras":{ "CAM_A": { "name": "rgb", "hfov": 90, "type": "color" }, "CAM_B": { "name": "left", "hfov": 122.91, "type": "color", "extrinsics": { "to_cam": "CAM_C", "specTranslation": { "x": -8, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_C": { "name": "right", "hfov": 122.91, "type": "color", "extrinsics": { "to_cam": "CAM_A", "specTranslation": { "x": 4, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } } }, "stereo_config":{ "left_cam": "CAM_B", "right_cam": "CAM_C" } } }

I was playing around with a few things last night so I'll get it back in to the same state and see what results I get from the depth checking scripts. I do have three colour cameras though so I'll go through and update the scripts for that first. I'll get the firmware updated too for the IMU, thanks for links for those.

DanielCalaver

kneave

Could I ask what the maximum FOV of lenses you've tested with for stereo cameras please? I had a look at the Pro W and it seems that they have a HFOV of 120 so my lenses shouldn't be that far out from this setup. The calculated HFOV I got was 122.91 and with the centre camera having a HFOV of 90 I think that's really close to what you have on the W too?

The maximum HFOV we've calibrated is 127, here you can find the captured used to calibrate an OAK-D-PRO-W-POE(127 HFOV for Left and Right, 108HFOV for RGB, 7.5 cm between Left and Right), maybe it will give you some hints on how to make the captures . The Charuco Board used is 16x22 squares with 4cm square size (it's about 930mm x 700mm)

So in my config the left camera is 8cm to the left of the right, and the right camera is 4cm to the right of the RGB cam.

Config looks good, left camera is -8cm from the right, and right camera is +4cm from the center one.

ignoring the error_text to see what should happen next seemed like a good idea and I think you said that the errors meant that the performance wouldn't be great but that it should still work with errors like that? With it ignored I got the message saying that upload had completed followed by "done" so I think it may be worth changing that to "done with errors" in cases where it doesn't work? It may help customers with troubleshooting for sure.

Thank you for your suggestion, note that if epipolar error is too high Depth perception might not work at all in some cases.

kneave

Thanks for the clarification Daniel, I'll take a look at the examples you've link to.
Regarding my comments on "done", I just meant that a return value for the end of the script could be useful. So if there was an error it could say "exited with error code X" and only say "done" if successful.

Ignoring the error_text was 100% a hack on my part!

kneave

I just tried to get the files you linked to have a look at and it's throwing an error, it then prompted me to download something else which was a little worrying and could catch folks out if they aren't careful.

Coul you try uploading to We Transfer please? I use that myself and know it's a trustworthy site 🙂

DanielCalaver

kneave

sorry @kneave not sure how it got deleted (should stay for 2 weeks). Anyways, here is the wetransfer link

kneave

Brilliant, I've got them. Thanks!

kneave

Finally got my new modern art delivered, I can finally test this again!

erik

That's quite large kneave ! When you are calibrating it, just make sure the board is actually flat (eg on table, or on the ground), as currently, it looks like it's bending a bit, which would cause worse calibration.
Thanks, Erik

kneave

Thanks Erik, I had it made to the same specifications @DanielCalaver mentioned that you use for your wide-angle products. I used it on the ground and managed to avoid the cats "helping" but sadly I'm still getting the epipolar errors.

I'll try again tomorrow with natural light rather than the indoor lights we have and more images too.

DanielCalaver

kneave thanks, yeah, if the board reflects too much light from ahead it can be disruptive, hope it'll work this time!

kneave

Sadly it hasn't, though I'm down to only one epipolar error so that's progress I guess:

Reprojection error threshold -> 100
rgb Reprojection Error: 0.688267
1200
Reprojection error threshold -> 166.66666666666666
left Reprojection Error: 0.614719
1200
Reprojection error threshold -> 166.66666666666666
right Reprojection Error: 0.577443
['high epipolar error between right and rgb']
py: DONE.

I've uploaded the calibration images and board definition here: https://we.tl/t-5uGyFtsNkM
The command I ran was python calibrate.py -s 4 -ms 3.1 -brd FFC-3Pw -nx 22 -ny 16 -c 3 and this is from the ar0234x3 branch of DepthAI.

One thing I'm still unsure of which I suspect could have an impact is the HFOV value I'm using for the stereo cameras. They are the same as the centre with a 160deg lens instead of the 90 that the 0234 come with. I couldn't find a definition for the HFOV for these lenses but calculated it at 122deg, I'm not sure of my maths however as optics isn't my forte to say the least.

I'm out of ideas and kinda annoyed I can't get this working to be honest. If you've any advice, even suggestions for other lenses to try, I'd appreciate it. Heck, if I can just calibrate with the stereo cameras and ignore the centre could that be an option? I'm planning on using the colour stereo for SLAM and point cloud generation and the centre with narrower field for object detection. If I don't actually need the centre can we get rid of that as it's the RGB to right cam that's throwing the error?

kneave

I just ran the following to disable RGB and got the same error 🙁

python calibrate.py -s 4 -ms 3.1 -brd FFC-3Pw -nx 22 -ny 16 -c 3 -drgb -m process

Displaying Stereo Pair for visual inspection. Press the [ESC] key to exit.
1200
Reprojection error threshold -> 100
rgb Reprojection Error: 0.688267
1200
Reprojection error threshold -> 166.66666666666666
left Reprojection Error: 0.614719
1200
Reprojection error threshold -> 166.66666666666666
right Reprojection Error: 0.577443
['high epipolar error between right and rgb']
py: DONE.

kneave

I saw someone downloaded the dataset file, hopefully that means someone is looking at this please?
@erik, @DanielCalaver?

erik

(@DanielCalaver would be the best person to check this - I have no exp with calibration.)

DanielCalaver

Yes, I downloaded the file but didn't managed to take a close look, will get back on this soon, sorry for delay @kneave

kneave

DanielCalaver cool, thanks Daniel. I appreciate the help

DanielCalaver

Hi @kneave sorry for delay, can you check if the lenses for both LEFT and RIGHT are the same? They look a bit different for me and this might affect calibration, could you try to get some better lighting? It was a bit dark in the pictures, maybe try to put the calib board on a wall and the room to be well light? As this way, the lights from above won't be reflected.

kneave

Hi Daniel,

They both came off the same camera model so should be the same. I finally found the specs for hfov and they're 120 rather than 122. Not sure if that 2deg difference is enough to cause a problem.

Will check lighting when home too, thanks for the tips!

Keegan

kneave

Hi Folks, I had to take care of some other projects and I'm just picking this back up. Could I ask if the 3xar0234 branch is still the best one to follow for multi-cam calibration please? I see there has been a new release since I looked at this last and wanted to check there isn't some kind of new Easy Mode you've added! 😅

erik

Hi kneave ,
For calibration, I believe the modular-calibration branch (PR here) would be the best to follow, as it will also be used for calibrating OAK-D-LR (which have 3x ar0234). Thoughts?
Thanks, Erik

« Previous Page Next Page »