• DepthAI

  • Stereo camera and visual SLAM

Dear Luxonis Experts,
Currently we are looking for an alternative solution of the following : https://www.dynimlabs.com/industrialstereocamera
For that purpose, we are really interested in both of your Robotics Vision Cores 2 (within OAK-SoM Pro / OAK-SoM ) and 3 (within OAK-SoM Pro S3/OAK-SoM MAX), especially with the later that is equipped with the Intel Keem Bay VPU disposing quad-core ARM Cortex A53 (similar to the NXP's iMX8M Plus in the above link) as a SLAM-able platform. Therefore, we would like to know :
A. For RVC3 :

  1. Could we have access to its Yocto Linux BSP to bringup custom MIPI-CSI2 RAW Bayer camera sensors?
  2. Could we do the ISP calibration ourselves (with provided documentation and tooling)? Or do you or does any partner / 3rd party provide calibration service for custom camera sensors ?
  3. Do you or does any partner / 3rd party provide custom hardware design for the Intel Keem Bay VPU ?

B. For RVC2:

  1. Could we use an external SoC (such as Raspberry Pi 4 or the NXP iMX8M Plus om the above link) to interface with OAK-SoM Pro / OAK-SoM for similar functionalities as RVC3 (with custom camera sensors)?

General questions :

  1. In case we would like to use a stereo pair of your AR0234 or OV9782 based camera modules for evaluation purpose, could we use them as master-slave scheme (i.e is there hardware synchronization inputs/outputs on the board as well is does the current driver support this) ?
  2. How would we stream video (encoded or not, 1080p or 4K) over the network as both RVC or SOM do not have Ethernet interface?

Best Regards,
Khang

  • erik replied to this.

    Hi l4es,
    Thank you for your interest in our devices!

    1. We are not yet sure whether we want to expose full access to Linux. Which sensor support would you need?
    2. As in sensor tuning? For that, you would need Intel propriatery camera tuning tools, which we can't provide. But we can suggest 3rd party companies that can provide this (as they have these tools).
    3. Yes, we usually recommend partners to handle such custom HW requests, most of the time based on one of the OAK SOMs.
    4. Yes, you could, that's essentially what our OAK RPi line does.
    5. Yep, both of these sensors are supported on RVC2 already (docs here), and will be supported on RVC3 as well (hardware syncing as well).
    6. We use PCIe to ethernet bridge on our OAK PoE cameras, eg. schematics here (see page 2 for block diagram).

    Thoughts?
    Thanks, Erik

    • l4es replied to this.

        erik

        Thank you for your prompt replies

        1. We are not yet sure whether we want to expose full access to Linux. Which sensor support would you need?

        We have ideas for different products/solutions :
        For stereo / multiple camera systems used in autonomous navigation (including VIO/vSLAM), we would like to use Sony global shutter sensors IMX296/IMX297.
        For 4K cameras, we would like to use Sony IMX678.
        We also have ideas about high resolution camera such as 'Hawk-Eye' Pi camera (most likely IMX686) for photogrammetry cameras.

        We are happy with Sony sensors until now. I see that you have the AR0234/OV9782 which could be candidates for IMX296/IMX297 but we haven't had practical evaluation with sensors others than Sony.

        By the way, could you share what Linux distribution would be running on Intel Keem Bay VPU based SOM if possible? Could we develop applications (deep learning inferences, navigation ROS, VIO/vSLAM, ...) on top of it?

        1. As in sensor tuning? For that, you would need Intel propriatery camera tuning tools, which we can't provide. But we can suggest 3rd party companies that can provide this (as they have these tools).

        Yes it is ISP tuning for custom sensors. It is good to know that there's 3rd party companies that can provide such services.

        Thanks,
        Khang

        • erik replied to this.

            Hi l4es ,
            I believe the plan is to make kernel/tools to build OS available to companies, so you would be able to add custom sensor drivers. For some common sensors, we might also look into integrating them ourselves.

            KeemBay is running Yocto OS. And by default we will expose running custom apps on the VPU via docker, and (I believe) companies will have full access to Yocto (ssh).

            Yes, definitely possible, we (or 3rd party companies) have done this for quite a few customers already.

            Thanks, Erik

              Hi @erik,
              Thanks alot for sharing useful information!

              We use PCIe to ethernet bridge on our OAK PoE cameras, eg. schematics here (see page 2 for block diagram).

              By the way, is it possible to combine SJ2088POE_PoE_Board with OAK-SoM Pro S3? Or does it come with OAK-SoM/RVC2 inside by default to form OAK-D PoE camera?
              Would you also have plan to design baseboard for OAK-SoM MAX, please ?

              Best Regards,
              K.

              • erik replied to this.

                Hi l4es ,
                Yes it's possible🙂 OAK-SoM-Pro rvc2/3 should be directly compatible. Let me check whether we can already share some designs with OAK-SOM-Max (I believe most are still WIP).
                THanks, Erik

                  l4es OAK-FFC-6P uses the SOM-MAX and is already opensource. I hope that helps🙂
                  Thanks, Erik

                    Very interesting! However, I am still not able to use Ethernet with that baseboard. Thank you!