Jello effect when using OAK-1 on drone

dhiraj

We are testing the OAK-1 camera on a drone. However, we see a very strong Jello effect with the camera.

We have tried to reduce the Exposure Time to 1000 ms (effective shutter speed of 1/1000). This improved things a bit, however, there still is a significant effect.

However, manually controlling Exposure time is forcing us to control the ISO as well, which means we have to figure out optimum lighting settings as per the scene.

Anti-vibration mounting is also difficult because of the very stiff camera cable.

Any thoughts on what can be done from the camera side?

dhiraj

Attaching an image showing ArUco codes, which have straight edges.

erik

Partly resolved on Github issue.

Brandon

Also discussed offline is using the AR0234 for the color sensor so as to benefit from global shutter and 2.3MP. https://github.com/luxonis/depthai-hardware/issues/87

JohnSymonds

Brandon I am wondering if this is enough reason to consider giving more thought towards support for the option of using a color camera module with a fixed focus lens (on non-FFC modules). Something with a similar depth of field to the b&w depth cameras. If the upcoming OpenCV AI kit Lite is being made smaller with the suggestion it may be able to be mounted on drones and other moving vehicles etc, this issue with vibration on the non-fixed lens cameras modules may become more apparent. Is there any fixed focus, pin / driver compatible modules out there for the color camera port?

Brandon

JohnSymonds Yes, I think this is an excellent point.

This won't be on OAK-D-Lite because we architected that and then had to order parts/cameras/etc. for it 9 months ago (in volume - so that we'd have them in time).

But we are working with ArduCam to make fixed-focus versions of the IMX378 now. We should have samples in about 2 weeks.

And we're doing this with the FOV we have now and also wide-ish FOV for some applicability to OAK-D-Pro-W (formerly OAK-WD-Pro).

Also we are working on OV9782 global shutter as the color camera as well.

Thoughts?

Thanks,
Brandon

JohnSymonds

Brandon
Happy to hear you are considering a fixed focus IMX 378 - i know I would be keen on that myself as I would like to use the OAK-D-POE versions in industrial applications and the current variable focus for me is not ideal due to possible vibration / and pixel size calibration complications. Variable focus may be an advantage for some uses, but it is a disadvantage for others.

A FOV the same as current color FOV is fine. However, you could also consider that if the IMX378 had exactly the same FOV as the depth cameras then it may be easier in software for overlaying depth data and positions onto the RGB image more easily. I am not sure if that is an option available to you. When arducam end up stocking these fixed focus cameras modules it will be great. I would be prepared to uncemented the current camera module and replace it to gain fixed focus.

erik

Hello JohnSymonds,
depth alignment to color camera is already available, see code snippet here. With this alignment we have created this demo (code here) - inference running on color stream, then inference results were applied to depth frames which were cropped accordingly.
Thanks, Erik

Brandon

Thanks JohnSymonds .

Yes, that's a good point. Likely a lot of applications could benefit from fixed focus. We'll think on this and discuss what would be best here.

Question - do you need the 12MP of IMX378 for color or would say global shutter OV9782 1MP be sufficient?

JohnSymonds

erik Thanks for your response. I have looked at that example previously. I am wondering however how well the alignment actually is as you travel away from the centre of the image. The set IspScaling 2/3, seems to use that ratio to adjust the 1080 of the color camera down to 720p. However the FOV of the color and mono lens are slightly different to each other so perhaps therefore pixel alignment away from the centre of the lens is less accurate the further you go out? Just wondering really - as I don't know what coding is under the hood to compensate for the alignment besides the IspScaling ratio.

Also noticed in the code snippet above this comment:

For now, RGB needs fixed focus to properly align with depth.
This value was used during calibration
camRgb.initialControl.setManualFocus(130)

This also highlights one of the earlier issues raised with non-fixed lens cameras, that being the difficulty experienced when trying to maintain a given pixel / mm size with an adjustable focus lens. (In this case the size needed to remain the same for alignment purposes against the mono cameras pixels.)

JohnSymonds

Brandon I actually need the higher pixel resolution of the IMX378 since I need to perform a digital zoom (via ImageManip crop) to effectively reduce the FOV. (with the camera running in 4K mode to expose the native resolution of the sensor).

I have considered the OV9782 as something I will try sometime - but I haven't got around to ordering any of those modules yet since the drivers are not fully integrated yet.

Off topic, do you have any plans in the future to make enclosures for the OAK-D-CM4 modules - similar in design to the OAK-D-POE enclosure?. I feel like these would improve the usability somewhat for this model which has a lot of potential (as the CM4 can run OpenCV code on the CM4 device, and hence can be used to create standalone networkable smart cameras with in device OpenCV support). You will remember the type of feedback received during the original kickstarter campaign (for the wifi version) which somewhat reinforces the value given to a design with an enclosure.

Brandon

Thanks JohnSymonds this is clutch feedback.

And yes, we learned the hard way that enclosures are super important in this market.

For OAK-D-IoT (which I dumbly named OAK-D-WiFi initially) we recently finished prototyping it and opened tooling for the enclosure design we did based on that feedback. It will be ready likely December-ish and then folks will be able to buy the enclosure for their units:

We did WiFi testing and it seems to be the same with and without enclosure, which was the main reason we didn't do this the first time. (It did take several iterations to make that performance OK; so the concern was valid.)

In regards to a CM4 version with enclosure, yes, agreed. We're working to make such a version. See bellow for the prototype:

More details here:
https://github.com/luxonis/depthai-hardware/issues/53#issuecomment-909527674

Thoughts on this?

We're thinking of doing follow-on of this with M12 connector instead and active illumination (laser and LED) like OAK-D-Pro, here.

Thanks,
Brandon

erik

Hello JohnSymonds ,

I agree that enclosures are extremely valuable and we are working on having enclosures for all our modules. We are currently testing it (WRT wireless connectivity) for our IOT line. I believe that the upcoming OAK-D-POE-CM4 device will also be sold with an enclosure, so as you said it will allow for a standalone networkable smart camera with OpenCV support.

About the rgb-depth alignment - from my testing, it seems to be fairly accurate even at some distance (not perfect, depends on the calibration though). If you would like to know more about this feature I can ask Sachin to elaborate on any concerns you may have. We currently only have the correct modeling for rgb-depth alignment at lens position of 130, since the lens position will affect zoom. We plan on adding support for non-fixed focus as well in the future.

Thanks, Erik

JohnSymonds

The enclosure for the OAK-D-Iot looks great!

I have an OAK-D_Iot 75 sitting unused in the box here so I will be keen to buy one of those cases when they become available.

The OAK-D-POE-CM4 sounds awesome. Any rough idea when those may be available to purchase?

I hope that you do make the follow-on with the M12 connector and active illumination. Since USB is not very practical for industrial applications (due to cable length restrictions). I found the current depth perception on non-textured surfaces was not good enough to use for my use case unfortunately. Could I ask the part numbers for the laser / LED illumination components. I would like to look up the specs on the datasheets to anticipate what it may be capable of (if possible).

erik

Hello JohnSymonds, I'm not sure about ETA of the POE-CM4 (I assume around the end of this year), but I can share that we plan on using this dot projector for the PRO series.
Thanks, Erik