Hi, I've recently purchased an OAK-FFP-3P along with three Arducam AR0234 camera modules and I'm having difficulty with pretty much all the examples, I'm entirely expecting this to be user error and figure there's something fundamental I've missed. If I write my own simple scripts using bits from various examples I can get a list of the three camera modules and they are recognised fine, I can even run one of the RGB examples that shows the source and cropped image, but I can't get IMU data or seem to get images from the left or right cameras. Is there an idiots guide to getting started with the 3P board please? I'm a software engineer with a lot of experience but still relatively new to computer vision, if anyone could give me pointers to get started I'd appreciate it. Photo for attention, this is the bot I'll be putting the 3P in to once I've got it working 🙂

Hi kneave , I think most demos don't work because you have 3x ColorCamera, and many demos use MonoCamera depthai nodes. After OAK FFC calibration you can also do stereodepth, demo here , which also shows how to use multuple ColorCamera nodes (instead of MonoCamera nodes). Thoughts? And that robot looks really cool!! Would love to see picture with the OAK FFC on there🙂 Thanks, Erik

Ah! I thought that mono would just return a mono image, I didn't realise that it would also require or expect a mono camera. This all looks ideal, I'll give it a go today. Thanks for the tips! The plan is for wide angle colour cameras for the stereo pair and a narrower lens on the centre camera. I figure that'll give a good combination for SLAM and a more zoomed in camera for distance viewing. I'm hoping to find a small camera for the centre so that I can hide it behind the panel between his eyes and not spoil the aesthetic 🙂

I've switched to and built the develop branch of depthai_python, I saw that the multi_cam_support branch was merge in a few weeks ago, and I've gone through calibration.py from the depthai library and modified it for three colour cameras. I'm really struggling to more than a few calibration images accepted by the calibration script, none at all unless I disable the RGB sensor, could this be because I'm using fisheye lenses? The left and right camera have 160 degree lenses and the RGB has a 90 degree lens, do I have to do anything different to enable a fisheye mode? I know that's something I've needed to do with ROS before. Here is the calibration file I created in case it helps. { "board_config": { "name": "NE3P", "revision": "V1.0", "swap_left_and_right_cameras": false, "left_fov_deg": 160, "rgb_fov_deg": 90, "left_to_right_distance_cm": 8.0, "left_to_rgb_distance_cm": 4.0 } } Any tips would be welcome, I'm happy I'm at least getting images now but it's still a little frustrating.

Hi @kneave , Try to calibrate using this branch ( https://github.com/luxonis/depthai/tree/ar0234_x3 ), you have to add a custom board config to work, try adding something like this in depthai/resoruces/boards/FFC-3P.json, and then be sure to select the board while doing the calibration for example python3 calibrate.py -brd FFC-3P. Be sure that you use the HFOV, not DFOV or something else, and check if the left/right/rgb cameras are in the right place (change the CAM_A/B/C ports as needed) { "board_config": { "name": "NE3P", "revision": "R1M0E1", "cameras":{ "CAM_B": { "name": "right", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_C", "specTranslation": { "x": 8, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_C": { "name": "left", "hfov": 160, "type": "color", "extrinsics": { "to_cam": "CAM_A", "specTranslation": { "x": 4, "y": 0, "z": 0 }, "rotation":{ "r": 0, "p": 0, "y": 0 } } }, "CAM_A": { "name": "rgb", "hfov": 90, "type": "color" } }, "stereo_config":{ "left_cam": "CAM_B", "right_cam": "CAM_C" } } } if(window.hljsLoader && !document.currentScript.parentNode.hasAttribute('data-s9e-livepreview-onupdate')) { window.hljsLoader.highlightBlocks(document.currentScript.parentNode); }

Thanks, I'll have a look at this today hopefully

I've gone through this a few times today and gotten much further so thanks for the tips! Pretty sure one of the issues I was facing was due to a poor quality calibration image, the printer put out something more grey than black so I reprinted on photo paper and it's much happier. First few times I got XLink errors, pretty sure I knocked a cable, and then when I was almost done the battery in my laptop died as it was taking the last image! I just got through them all for the first time after a recharge, it collected them all then hit an error in OpenCV. I've not had a chance to debug it yet, I'll have another look tomorrow. OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-5rb_9df3\opencv_contrib\modules\aruco\src\charuco.cpp:448: error: (-215:Assertion failed) _markerCorners.total() == _markerIds.getMat().total() && _markerIds.getMat().total() > 0 in function 'cv::aruco::_interpolateCornersCharucoLocalHom' Though it now looks fairly obvious, "appveyor" isn't my username so I suspect that there is a hard coded path somewhere in the calibration script. I'll have a look tomorrow, hopefully it's an easy win. 🙂

Hmm, no, AppVeyor isn't mentioned in the repo at all that I can find. How very odd! Edit:just realised AppVevor is a CI tool so that's likely the path for the file on the build server and I've the resulting assembly on my machine. Will dig in to the code and figure out what it's doing, or not, as the case may be.

Just for those who hit a similar issue, with the branch above I needed to downgrade OpenCV to the following version manually: opencv-contrib-python==4.5.5.64 I'm now hitting an issue where it seems to get through everything but complain there's too big a difference between stereo cameras and RGB, I'm assuming as stereo are 160deg and the RGB is 90deg. I'm going to try with the -drgb flag to see if it can calibrate the stereo cameras at least.

Hi kneave , We apologize for the inconvenience - could you copy the full error that you get? Thanks, Erik

Hi Erik, will do. I'll go through again and grab it shortly. Are the images saved to the disk too by the way? It'd be handy if I could run calibration again without having to recapture the images each time. It'd speed up debugging no end!

OAK-FFP-3P User Guide For Dummies?

kneave

I saw someone downloaded the dataset file, hopefully that means someone is looking at this please?
@erik, @DanielCalaver?

erik

(@DanielCalaver would be the best person to check this - I have no exp with calibration.)

DanielCalaver

Yes, I downloaded the file but didn't managed to take a close look, will get back on this soon, sorry for delay @kneave

kneave

DanielCalaver cool, thanks Daniel. I appreciate the help

DanielCalaver

Hi @kneave sorry for delay, can you check if the lenses for both LEFT and RIGHT are the same? They look a bit different for me and this might affect calibration, could you try to get some better lighting? It was a bit dark in the pictures, maybe try to put the calib board on a wall and the room to be well light? As this way, the lights from above won't be reflected.

kneave

Hi Daniel,

They both came off the same camera model so should be the same. I finally found the specs for hfov and they're 120 rather than 122. Not sure if that 2deg difference is enough to cause a problem.

Will check lighting when home too, thanks for the tips!

Keegan

kneave

Hi Folks, I had to take care of some other projects and I'm just picking this back up. Could I ask if the 3xar0234 branch is still the best one to follow for multi-cam calibration please? I see there has been a new release since I looked at this last and wanted to check there isn't some kind of new Easy Mode you've added! 😅

erik

Hi kneave ,
For calibration, I believe the modular-calibration branch (PR here) would be the best to follow, as it will also be used for calibrating OAK-D-LR (which have 3x ar0234). Thoughts?
Thanks, Erik

kneave

I'll have a look, thanks. Just to complicate matters I've an oak-ffc-imx378 I'm hoping to use for the centre camera now. I've not been able to get it to be detected by the 3P yet which is annoying but it's been that long since I've looked at this I've likely forgotten a trick needed. That or the one USB cable I have that works reliably has gone AWOL 😅

erik

kneave is it the case that only FFC-IMX378 isn't being recognized, while other cameras are? If so, then it's not an USB issue. I would also suggest using latest depthai version.
Thanks, Erik

kneave

erik It's a bit random at the minute, it's sometimes saying no cameras detected but as it's getting that response it at least means it's connected to the 3P and able to get a response back. I'm currently printing a new enclosure, once I've got everything in place I'll do some more tests.

Quick render of the new enclosure attached, it'll be 3d printed so not quite as shiny! You can see the centre camera hidden in the black separator between the eyes, I'm keen to make it look like he only has two as much as possible.

erik

Hi kneave ,
Could you copy the full terminal log of cam_test script, please? If it doesn't work as expected we will send a replacement unit. The render looks dope!! Please let us know how it will turn out🙂
Thanks, Erik

kneave

Sure thing, does it matter which branch I run cam-test from or can I install the production libraries and run against them?

kneave

Ok, I've done a lot of digging and should really be in bed but I may have stumbled upon something in this issues;
https://github.com/luxonis/depthai-hardware/issues/292

Current configuration is:
Left: AR0234
Centre: IMX378
Right: AR0234

I have a simple camera preview script which I've successfully used to focus the cameras in the past, it has this line that lists all the cameras:

with dai.Device(pipeline) as device:
    print('Connected cameras: ', device.getConnectedCameras())

First off, I ran this against the develop branch of depthai, making sure to run ./install_requirements.py:

I then switch to the ar0234_x3 branch which is the only one I've been able to get the cameras working on in the past:

This shows that the two AR0234 are detected, if I replace the center with the other AR0234 I have it is detected too and I'm able to preview from all three.

This made me wonder about the IMX378 which led me to the issue I linked above. Taking a macro photo of the board it looks like it does indeed have the K28 power regulator on board and it also has all the pins connected, I'm starting to think I have one of the broken ones?

Could I cut the problem pin and see if it works or having plugged it in would it have damaged it? I don't want to do any mods that may prevent a warranty claim but if all I need to so is cut that pin then it could be an easy win?

erik

Hi kneave ,
Yes, you are right, you have the bad board - we apologize for the inconvenience. Full Errata here. We can either send a replacement, or you could try to fix it locally.
Thanks, Erik

kneave

erik Hi Erik, sadly it seems the fix didn't work and I can no longer detect the camera. I managed to get some images once but it hasn't worked again since. I've tried replacing the ribbon cable too, and tried another camera on that port to confirm it works (it does) so the sensor is likely broken. I'll email support as per the errata to get a replacement. 🙁

kneave

Success! Kinda...

If I run on the Main branch I can get cam_test to run and show the IMX378 camera but the two AR0234 cameras aren't detected. If I run on the ar0234x3 branch I can see all three cameras in my preview.py script however cam_test.py won't run as it doesn't recognise THE_1200_P as a valid resolution.

I also tried the modular-calibration branch however I couldn't run install_requirements.py as it failed with the following error:
ERROR: Could not find a version that satisfies the requirement depthai==2.19.1.0.dev+1eb01248fb7e994b51ceec91984bff46e0a5d5f2

I suspect my combination of cameras is throwing things off slighting and that the IMX378 is supported on one branch and the AR0234 on another but not both together maybe? Either that or depthai_python just doesn't have an updated version of cam_test?

I'm once again a little lost to be honest, if you know of a branch that has success with this camera combo could you point me towards it please?

erik

Hi kneave ,
Looks like we didn't update requirements.txt. I just updated it on that branch (to 2.20.2), so it should work as expected - could you try again, please?
Thanks, Erik

kneave

Hi Eric,

To confirm, I'm now on the following branches for:
depthai: modular-calibration (Your commit ₃₀ minutes ago)
depthai-python: main (for cam-test.py in utilities, latest commit)

I ran `install-requirements.py' from the root of depthai and the utilities folder of depthai-python.

When I run cam_test.py I only see the IMX378, it comes up in the preview window quite happily and the console reports this:

Enabled cameras:
    rgb : color
   left : color
  right : color
DepthAI version: 2.20.2.0
DepthAI path: C:\Users\keegan\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\depthai.cp39-win_amd64.pyd
[184430105182410E00] [1.5] [0.865] [ColorCamera(6)] [error] Camera not detected on socket: 2
[184430105182410E00] [1.5] [0.866] [ColorCamera(4)] [error] Camera not detected on socket: 1
Connected cameras:
 -socket RGB   : IMX378 4056 x 3040 focus:auto  - COLOR
USB speed: SUPER
IR drivers: []
Cam:      rgb           left          right    [host | capture timestamp]
FPS:  30.00| 30.00   0.00|  0.00   0.00|  0.00

If I switch to the ar0234x3 branch the cam_test.py script doesn't run as it looks as though there was a bit refactor that deprecated or changed a lot of the methods in depthai between then and 2.20.

It looks as though if I try and run with any version of depthai other than depthai==2.17.2.0.dev+211dc1c0a135d5615dba75ebf2098ba8cdc05cd0 it isn't able to see the AR0234 cameras. If I look at the depthai-python branch it looks as though ar0234x3 was never merged in to any other branch either.

I may be misreading the commit graph, I've been digging between branches for a while now and I've lost track, but is the AR0234 supported on either main or develop of depthai-python? To me it looks like it isn't and that it's only that old 2.17 branch that it's available in.

kneave

I'm once again trying to calibrate against the AR0234x3 branch as it seems to be the only one that supports all my cameras, this time all new errors. Before I start digging even further and spending more time on this can I get a yes/no answer to the following please?

Has the combination of 2x AR0234 and single IMX378 been tested by yourselves or anyone else that you know of?

Right now I've no idea if this newest issue will be the last one and I can get passed it and do the actual work I want to with these cameras, or if this is the tip of an iceberg and there will be yet more work to do afterwards.

I understand that what I'm doing isn't simple and that the majority of your customers just use the standard integrated and tested cameras. I also understand that these things can take time, but right now I have an awful lot of unknowns and other projects I need to work on. If I know that this has worked for others I can continue, if I'm on my own with a new and untested combination I may have to shelve this project until the 3P has better support for multi-cameras.

If you could let me know if the light at the end of the tunnel is sunlight or an oncoming train, I would greatly appreciate it. 🤣

kneave

It's looking like the issue was with the image preview. The AR0234 output at 1200p and the IMX378 wants to output at 1080p/4k.

I'm thinking if ignoring the centre camera for now and just trying to get the depth camera working. I think that the difference between HFOV of the centre and side cameras is too great for calibrating together. At least with the side cameras being rgb I should still be able to get a coloured point cloud. Hopefully I'll be able to use the centre camera independently, I'll have to have a play around with it at some point.

erik

Hi kneave

Has the combination of 2x AR0234 and single IMX378 been tested by yourselves or anyone else that you know of?

No, we have only tried our "standard" configuration, which is 2x OV9x82 for stereo and IMX214/IMX378 for color camera. Any other configuration (like yours) has likely not been tested, but should likely work, perhaps with some modifications, like to calibration script. If streaming 3 camera sources (cam_test) doesn't work with latest develop branch, please open an issue on depthai-core.
Thanks, Erik

« Previous Page Next Page »