Color stereo images from left and right streams

ttsesm

Hi,

I have the Luxonis OAK-D Pro Wide (OV9782), OAK-D Pro Wide and OAK-4D Pro Wide cameras

My question is, whether it would be possible somehow to obtain RGB stereo images from the left and right streams. As far as I understand the left and right streams are providing only monocolor output which is synced/aligned with the rgb but I would be interested to see if I can obtain native RGB output for the left and right streams.

Thanks.

OskarSonc

Hey @ttsesm - So on the OAK-D Pro Wide / OAK-4D Pro Wide, the center camera is RGB, but the left/right stereo pair are OV9282 monochrome sensors. So the stereo streams themselves can only be mono – there’s no way to get native RGB left/right images from these devices.
If you want to view all cameras just run this example and you will see one RGB and two mono streams...

Thanks,
Oskar

ttsesm

I see, thanks for the clarification.

ttsesm

Another question, related to this. Can you let me know which luxonis models can give me native RGB stereo streams. I am also interested on the custom units as well.

From a quick search I found that OAK-D LR, OAK-D SR and then I guess all the OAK-FFC series meaning 1P/2P/3P/4P from custom modules. Regarding the OAK-FFC units as I understood I can plug combination of cameras meaning for example in the 3P I can have 1 RGB and 2 monochrome (can I have a combo of 1 mono and 2 RGB?). Also I this custom units can I add (or replace the monochrome) some IR sensors for low light visibility?

Thanks.

OskarSonc

Out of the standard devices, OAK-D LR and OAK-D SR give you RGB stereo, since their left/right sensors are color. The other OAK-D / OAK-D Pro variants use mono stereo(since it gives better depth info) + one RGB in the center.
With the OAK-FFC you can mix modules however you like – e.g. 1 mono + 2 RGB is fine.
For low-light you can use this IR module.

Curious: what’s the exact reason you need native RGB stereo (vs mono stereo + a single RGB camera)

ttsesm

Perfect, thanks for the feedback.
Really helpful.

Well, the reason that I am looking on an RGB stereo solution, is the fact that I would like to take advantage of some good AI models for encoding the depth/point-cloud information directly from a dual rgb stream. For example FoundationStereo, BridgeDepth, etc.. that's why I would be interested to check also the process unit of the board (though this is not a deal break). For example, as I understood all the OAK-FFC boards are coming with an RVC2 architecture on-board features, right? Is there anything already, or possibly coming soon with something more powerful like an RVC4 architecture board? Otherwise, the RVC2 architecture I guess can handle at least compression and synchronization.

OskarSonc

I see, that makes sense now 🙂 For running things like FoundationStereo / BridgeDepth you’d typically run the heavy model on a host GPU and use the OAK mainly to deliver synced dual-RGB (and optionally depth/IMU) to the host. Yes, all current OAK-FFC carrier boards are based on RVC2, which is more than enough for sync, compression and basic processing. We’re currently focused on the official launch of the OAK4 (RVC4) devices, so there isn’t an RVC4-based FFC carrier available yet.

ttsesm

@OskarSonc another question the IR sensor that you linked above connects to a MIPI connector on the OAK-FCC boards or it has its own connection? And I guess the 2-lane MIPI and 4-lane MIPI connections are for the mono and RGB camera sensor modules respectively, right? Meaning that I cannot connect an RGB camera sensor to a 2-lane MIPI or I can?

ttsesm

ok, as I see this is not always the case... there are some RGB sensors that can also work on a 2-lanes MIPI and the 4-lanes MIPI is for the high resolution RGB sensors with 12MP and 32MP. So, then the only questions that remains is whether the IR sensor has its own connector or takes one of the MIPI connections.

OskarSonc

It has its own connector:

This is the link to this page:
https://docs.luxonis.com/hardware/platform/deploy/ffc/#Connecting%20an%20IR%20Module

ttsesm

@OskarSonc is it possible to integrate the OAK 4 SoM with the OAK-FFC units? In principle on the OAK 4 SoM, I should be able to run some heavier models.
I've seen that there is a RCV4 devkit that it is supposed to be released mid-next month. However, the board dimensions might be too big for my needs, meaning for integrating it to my installation since I have some dimension restrictions regarding what I can fit in specific space. Thus, I was looking whether it would be possible to take advantage of the OAK 4 SoM without the "lenthy" board... 🤔

Also is actually the devkit board needed for the operation of the OAK 4 SoM, or it is used for loading/pre-programming the model/operations to the OAK 4 SoM unit? and then the OAK 4 SoM unit can operate on its own.

OskarSonc

@ttsesm - no, you can’t mix the current OAK-FFC boards with the OAK4 SoM. The platform (RVC4 vs RVC2) and pinout are completely different, so they’re not electrically compatible.

The OAK4 devkit is essentially the carrier board for the SoM and is required for operation, unless you design your own custom carrier around the SoM. We do plan an OAK4 SoM carrier in an FFC-4P style form factor, but that’s still in early stages and not ready yet.

Thanks,
Oskar

ttsesm

I see, thanks!
Do you have an approx. ETA for the new carrier?

OskarSonc

Hey @ttsesm - timeline is still TBD, but roughly mid-end next year. We’re actively working on it – it’s basically a smaller, more focused and cheaper spin of the existing DevKit. If you need a dev platform now, they can just use the current DevKit and later port to the FFC carrier.

Thanks,
Oskar