Hi, I've recently purchased an OAK-FFP-3P along with three Arducam AR0234 camera modules and I'm having difficulty with pretty much all the examples, I'm entirely expecting this to be user error and figure there's something fundamental I've missed. If I write my own simple scripts using bits from various examples I can get a list of the three camera modules and they are recognised fine, I can even run one of the RGB examples that shows the source and cropped image, but I can't get IMU data or seem to get images from the left or right cameras. Is there an idiots guide to getting started with the 3P board please? I'm a software engineer with a lot of experience but still relatively new to computer vision, if anyone could give me pointers to get started I'd appreciate it. Photo for attention, this is the bot I'll be putting the 3P in to once I've got it working :) [URL=https://i.imgur.com/rMM2Oqg.jpg][IMG]https://i.imgur.com/rMM2Oqgh.jpg[/IMG][/URL]

Hi @"kneave"#p5209 , I think most demos don't work because you have 3x ColorCamera, and many demos use MonoCamera depthai nodes. After [OAK FFC calibration ](https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/calibration.html#modular-cameras-calibration)you can also do stereodepth, [demo here](https://raw.githubusercontent.com/alex-luxonis/depthai-python/e1c02e070d8ad3073c2dfbda7c8be45814d49999/examples/StereoDepth/rgb_depth_aligned.py), which also shows how to use multuple ColorCamera nodes (instead of MonoCamera nodes). Thoughts? And that robot looks really cool!! Would love to see picture with the OAK FFC on there:) Thanks, Erik

Ah! I thought that mono would just return a mono image, I didn't realise that it would also require or expect a mono camera. This all looks ideal, I'll give it a go today. Thanks for the tips! The plan is for wide angle colour cameras for the stereo pair and a narrower lens on the centre camera. I figure that'll give a good combination for SLAM and a more zoomed in camera for distance viewing. I'm hoping to find a small camera for the centre so that I can hide it behind the panel between his eyes and not spoil the aesthetic :)

I've switched to and built the develop branch of depthai_python, I saw that the multi_cam_support branch was merge in a few weeks ago, and I've gone through calibration.py from the depthai library and modified it for three colour cameras. I'm really struggling to more than a few calibration images accepted by the calibration script, none at all unless I disable the RGB sensor, could this be because I'm using fisheye lenses? The left and right camera have 160 degree lenses and the RGB has a 90 degree lens, do I have to do anything different to enable a fisheye mode? I know that's something I've needed to do with ROS before. Here is the calibration file I created in case it helps. `{ "board_config": { "name": "NE3P", "revision": "V1.0", "swap_left_and_right_cameras": false, "left_fov_deg": 160, "rgb_fov_deg": 90, "left_to_right_distance_cm": 8.0, "left_to_rgb_distance_cm": 4.0 } }` Any tips would be welcome, I'm happy I'm at least getting images now but it's still a little frustrating.

Hi @"kneave"#898, Try to calibrate using this branch (https://github.com/luxonis/depthai/tree/ar0234_x3), you have to add a custom board config to work, try adding something like this in depthai/resoruces/boards/FFC-3P.json, and then be sure to select the board while doing the calibration for example python3 calibrate.py -brd FFC-3P. Be sure that you use the HFOV, not DFOV or something else, and check if the left/right/rgb cameras are in the right place (change the CAM_A/B/C ports as needed) ``` { "board_config": { "name": "NE3P", "revision": "R1M0E1", "cameras":{ "CAM_B": { "name": "right", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_C", "specTranslation": { "x": 8, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_C": { "name": "left", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_A", "specTranslation": { "x": 4, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_A": { "name": "rgb", "hfov": 90, "type": "color" } }, "stereo_config":{ "left_cam": "CAM_B", "right_cam": "CAM_C" } } } ```

Thanks, I'll have a look at this today hopefully

I've gone through this a few times today and gotten much further so thanks for the tips! Pretty sure one of the issues I was facing was due to a poor quality calibration image, the printer put out something more grey than black so I reprinted on photo paper and it's much happier. First few times I got XLink errors, pretty sure I knocked a cable, and then when I was almost done the battery in my laptop died as it was taking the last image! I just got through them all for the first time after a recharge, it collected them all then hit an error in OpenCV. I've not had a chance to debug it yet, I'll have another look tomorrow. `OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-5rb_9df3\opencv_contrib\modules\aruco\src\charuco.cpp:448: error: (-215:Assertion failed) _markerCorners.total() == _markerIds.getMat().total() && _markerIds.getMat().total() > 0 in function 'cv::aruco::_interpolateCornersCharucoLocalHom'` Though it now looks fairly obvious, "appveyor" isn't my username so I suspect that there is a hard coded path somewhere in the calibration script. I'll have a look tomorrow, hopefully it's an easy win. :)

Hmm, no, AppVeyor isn't mentioned in the repo at all that I can find. How very odd! Edit:just realised AppVevor is a CI tool so that's likely the path for the file on the build server and I've the resulting assembly on my machine. Will dig in to the code and figure out what it's doing, or not, as the case may be.

Just for those who hit a similar issue, with the branch above I needed to downgrade OpenCV to the following version manually: opencv-contrib-python==4.5.5.64 I'm now hitting an issue where it seems to get through everything but complain there's too big a difference between stereo cameras and RGB, I'm assuming as stereo are 160deg and the RGB is 90deg. I'm going to try with the -drgb flag to see if it can calibrate the stereo cameras at least.

Hi @"kneave"#p5297 , We apologize for the inconvenience - could you copy the full error that you get? Thanks, Erik

Hi Erik, will do. I'll go through again and grab it shortly. Are the images saved to the disk too by the way? It'd be handy if I could run calibration again without having to recapture the images each time. It'd speed up debugging no end!

OAK-FFP-3P User Guide For Dummies?

kneave

Getting closer, now stuck on RuntimeError: No PROTECTED permissions to override protected EEPROM fields so something else on the EEPROM isn't quite right, or is locked in some way?

erik

Hi kneave ,
If you set environmental variable to DEPTHAI_ALLOW_FACTORY_FLASHING=235539980 (so on linux you can do DEPTHAI_ALLOW_FACTORY_FLASHING=235539980 python3 flash_eeprom.py) it will allow you to do so. Note that usually you wouldn't want to override factory calibration.
Thanks, Erik

kneave

Thanks Erik, I was able to use that to update the calibration but getting other errors permissions errors now 😅
Also, it doesn't have a factory calibration apparently. Is this supposed to be the case with the 3P? It doesn't have cameras included so I can see why it wouldn't, but also it seems like not having a calibration at all may be the cause of a bunch of other issues.

Thanks for all your help so far Erik, and colleagues, I think I'm going to have to spend some time properly getting my head around calibration and how to get it to work with my setup. I was previously using a StereoPi with two fisheye lenses and using the ROS calibration method without issue so I think I got complacent. With the extra power of the 3P it seems it needs more attention to get working. That said, it's tempting just to publish the images from each cameras to ROS and see if I can use the same method to generate the calibration. A bit hacky but I'm willing to try anything at this point! 🤣

For Windows users seeing the previous post regarding the environmental variable, the equivalent is:
in command prompt:
set DEPTHAI_ALLOW_FACTORY_FLASHING="235539980"

in powershell:
$env:DEPTHAI_ALLOW_FACTORY_FLASHING="235539980"

Then run the python script. This will persist until the cmd/powershell session is closed.

erik

kneave Good point, not sure why depthai complains about eeprom, even if there's no calibration present.. Will bring it up internally with the team. What kind of other permission erros are you getting?

kneave

One I got the calibration done and uploaded to the 3P using the calibration_flash script (it failed to upload via calbration) I then tried to run the depthai_demo.py script and got these when I tried to switch to depth preview:

I'm on Windows so no udev rules to set which confused me

kneave

Hey folks, other projects have been wrapped up and I'll be able to look at this again next week. It looks like a lot of work has happened in the interim, could I ask which branch I should be using please?

There are also a few PRs pending that involve calibration, are they worth waiting for them being merged?

erik

Hi kneave ,
You should used latest main branch when calibrating, we have updated the script 2 times since then🙂

kneave

It's been a very long fight getting this far but it looks like the work done to improve the calibration process has paid off so thank you Luxonis and friends for that! My IMX378 isn't being detected any more, hopefully just a dodgy ribbon cable, but I'm at least able to generate a depth map from the two AR0234 cameras.

Now the hard work of tuning everything to reduce noise and get some usable data out of it 🙂

kneave

erik Hi Erik, sadly it seems the fix didn't work and I can no longer detect the camera. I managed to get some images once but it hasn't worked again since. I've tried replacing the ribbon cable too, and tried another camera on that port to confirm it works (it does) so the sensor is likely broken. I'll email support as per the errata to get a replacement. 🙁

erik

kneave What kind of errata? If it worked once, I don't understand how it could be errata.
What doesn't work - could you share terminal output of eg. cam_test.py?

kneave

Hi Erik, this one. I may not have cut the trace correctly I guess but trying the fix was worth a try, sadly it didn't work out for some reason. Regarding terminal output, it's simply not detected.

erik

Hi kneave , I see, I think it would be best to send an email to support@luxonis.com (with the link to this discussion) and we will replace the faulty HW asap.

kneave

Replacement camera arrived, I just need to figure out how to calibrate the centre along with the sides. I'm going to try to calibrating the sides first then, the centre, then all three together and hoping for the best!

kneave

Success! I have all three camera's calibrated! I had to mess around with the calibration script quite a bit to override the error bounds for the projection errors between narrow FOV centre and very wide left and right, seems to work though!

Also, I love how our black cat, Toby, is literally a void in the depth map! 😅

Thank you to everyone for your help, I'll post my code up once I've got it all checked in.

kneave

Another update, I've got the depth alignment example working! 😃

« Previous Page