Depth perception 360 degrees

BradleyDillon

Depth perception 360 degrees using OAK-FFC-IMX378-W (Camera sensor) (Qty: 6) OAK-FFC-6P (Qty: 1):

Customer originally thought that they could connect the cameras that they bought in 3 stereo pairs at a 120 degree angle. This comes with the problem that there is no depth perception between the camera pairs. Does anybody have an idea about mounting the cameras at a 60 degree angle and still being able to do depth perception?
Some hints of the python libraries that would help would be very kind.

erik

Hi BradleyDillon ,
As a background, IMX378-W CCMs (PY060) have the following FOVs:

Diagonal: 120°
Horizontal: 95°
Vertical: 70°

Since these are a bit distorted, after rectification (+undistrotion), the horizontal FOV will likely drop in a ballpark of 85°. So you would actually need 4-5 of these pairs to be able to cover full 360° view.

Mounting the two cameras (of the stereo camera pair) at an angle is currently not supported.

For an OAK-FFC 360 view I would actually suggest using OV9282/OV9782-W CCMs (PY059) which have wider FOV, after rectification of about 97°, so you would only need 4 and still have some overlap.
Thanks, Erik

codejocky

I appreciate your reply and all the hard work it must take to bring these products to market.
I own the OAK-FFC-IMX378-W (Camera sensor) (Qty: 6) OAK-FFC-6P (Qty: 1) Although other cameras system sold by Luxonis may do the job, I would really like to try and get the system I own working. I really think I made the best choice for this in particular task.
It is very interesting that using setting the camera's at a 72 degree angle may give me a 360 view as this would leave a camera to set perpendicular to the 360 view. (straight up or down)
As you were saying, about mounting the camera's at an angle, depth perception is currently not supported. Why not? It just hasn't been done yet or is there a deeper reason I'm missing? For depth perception I thought what I needed was field of view overlap. I'm trying to keep up, but you guys at Luxonis are advancing this technology quickly. It fact I guess my lack of understand is astounding because I don't understand why I cant also mount these cameras at a 45 degree angle additionally and still get depth perception. I thought I understood this stuff, I have been doing it for 8 years (dedicating 4 month a year), please educate me, what am I not getting?
Think of the future and viewing things in VR where you can look around anywhere, the whole 360 view must be capture and recorded, for playback. It's not live stereo vision with the vr headset moving a live stereo camera. We are taking about a whole new way to make video recordings (the whole 360 view, all at once) and I believe with a little thought you will have the market cornered because there are lot's of other applications, like a new form of SLAM, employee monitoring (for safety of course), boating (above (for navigation) and below the water), crowd monitoring, the sky's the limit.
Hopefully you are excited as I am to try at advancing the technology in this direction.

codejocky

codejocky Maybe I'm approaching this from the wrong angle.
Instead of looking toward a Depth perception Luxonis solution via the OAK-SoM-MAX and out put to a raspberry pi 4 I should try a NVIDIA Nerf solution and connect the OAK-FFC-6P to my Jetson Orin SDK and just by pass any intelligence that's equipped on this device. Well, I guess I will leave that as a backup plan. I would hate to treat the OAK-SoM-MAX like a toaster without even trying to figure it out, It seam's like it has some vision processing power.
Let's try with baby steps first and see how far we can get. Let's start with calibrating at the focal distance of interest, using a large Charuco board for the cameras at 60 degree angle so I make sure there is a significate FOV overlay. Sounds like the right place to start. We will see.

erik

Hi codejocky ,

It just hasn't been done yet or is there a deeper reason I'm missing?

It hasn't been done with our cameras/platforms afaik. So likely you would need to hack the calibration process and rectification matrix. But I am not familiar enough with either to be able to provide any additional pointers. Also note that if you do mount 2 camera pairs at 60° angle (that have 85° HFOV), you would get a small portion of an angle (25°/2) that's seen from both stereo cams and can run disparity matching on.

For NERF, that's a bit of a different solution - it's closer to SfM than to stereo depth.
Thoughts?
Thanks, Erik

codejocky

Well baby steps right. With stereo depth perception, on ether side of the FOV there is an area that the other camera in the pair does not see. Is that a 25 degree angle? I guess we are back to our original problem. But we may be further ahead; using the cameras at a 72 degree angle will the entire 360 view be covered by two cameras and at what distance? What about the 60 degree angle and at what distance from the camera system?
This is a new camera system and questions like these are worth investigating.
I must say, I love this! Here we are, doing such exciting things with our lives, pushing the technology forward, making a difference. In my life I have seen countless people live and die without affecting the world in the least bit. Be joyful in the fact that we made a difference. We were not lost in the sands of time but have become the "ghost in the machine" and our ingenuity will be the foundation our descendant build upon. Thank you, for your service to the science of computer vision.

codejocky

I had better go through this documentation more thoroughly and see what answers I can come up with.
https://docs.luxonis.com/projects/hardware/en/latest/pages/DM3390.html
https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/#max-stereo-depth-distance
https://docs.luxonis.com/projects/hardware/en/latest/pages/DM3399.html
I'll get to it right away.

erik

Hi codejocky ,
My understanding of how you thought of mounting the cameras:

Where beta is rectified/depth HFOV. Having cameras tilted even more (72deg) would reduce the overlap. But let me know if I am understanding this idea incorrectly.
And we totally agree on technology, we love being on the "bleeding edge" as we are the ones that are pioneering these fields! And please reach out to erik@luxonis.com, we have a small gift for you for sharing our beliefs and evangelizing this area of technology🙂
Regarding the links - you got it all figured out I see. Let us know if you have any further questions.
Thanks, Erik

codejocky

I had this thought.
using depth perception I could pick out objects and give them bounding boxes, then take the frame of the bounding box and reduce the size of the picture in the bounding box to fix the A.I. models picture input size say 250x250 (as it is a common size to train models) then feed these pics into the A.I. only saving the image search from being the inter picture and now only a few boxes. This should save some time so the A.I. could use that time to scan the 250x250 for an increased library of items it's trying to identify. Has somebody else thought of that?

codejocky

codejocky
Answering my own question, doing more research...
[https://www.stereolabs.com/blog/performance-of-yolo-v5-v7-and-v8/#:~:text=In%20conclusion%2C%20all%20three%20versions,performance%20out%20of%20the%20three.]
This article talks about yolo8 and a depth camera for situational awareness. not exactly what I was hoping for as both are computed separately. I wanted depth perception to create a bounding box and only these feed into yolo8. additionally I wanted the depth perception detection based on the previous frame and only rescan the frame for new objects say 0.2 seconds or have a thread dedicated to this task alone.

codejocky

I meant to say picture size is 224x224 so I could use IMAGENET. for a large library of object to detect.
plus depending on the size of object detected there should be a way to adjust bias toward that type of image identification. like an elephant is bigger than a mouse so they should be a bias toward elephant.