Working in parallel with 2D and Depth Images

palmerini

Hello everyone!
I'm a little newbie and I'm using OAK-D camera to acquire images that I need to elaborate. I'm using external libraries to do some detections inside the acquired 2D image. What I want to perform is visualize also the Depth image, in an other window of the same size and with the same resolution (let's say 640x480 for example), take the 2D coordinates of some points/segments that have been calculated on the 2D image, copy and paste those points in the Depth image and from this window, extract the 3D coordinates (Z values) of the pasted points.
I don't know if I've been clear enough. Is it possible? I have some difficulty also into simply visualize both 2D and Depth images in 2 windows.. Can you help me? Thank you!!

Luxonis-Lukasz

Hi palmerini ,

Only 2 days ago we made a first release (2.4) where RGB-Depth alignment is possible - here is the feature description.

We have also prepared an example with feature is used - https://github.com/luxonis/depthai-python/blob/main/examples/rgb_depth_aligned.py. You can freely copy and modify it to meet your needs

palmerini

Luxonis-Lukasz thank you a lot!! I'm going to test and understand the code in order to adjust it to reach my goals. Thank you very much!

erik

Hello @palmerini ,
to add to @Luxonis-Lukasz comment, you can find the code to convert depth + region of interest (ROI) into spatial coords on the host here. Just wanted to add it in case you also need spatial coords besides the Z values 🙂
Thanks, Erik

palmerini

erik Thank you for the additional informations! I'm going to check it too!!

Luxonis-Lukasz

@palmerini

The depth values can be obtained from depth stream of the StereoDepth node. With it, you receive an array with each point distance in mm.

The example I've shared uses disparity output, as it's better for visualizations

stereo.disparity.link(depthOut.input)

But you can add a depth stream there too to obtain raw depth info.
The example erik shared uses depth stream by default, so no change there is needed

stereo.depth.link(sdn.inputDepth)

So to get the Z coordinate of an X/Y point, you need to take the depth frame (self.depth inside calc_spatials) and check what the depth value at X/Y index

palmerini

I'm having troubles in import find_version module. As soon as I solve this I could come back again.
Thank you!

palmerini

Sorry for bothering you again..
Last question. I've successfully run the example code and I've modified it in order to perform my elaborations on the 2D images. I've also looked the code for getting the spatial coords, but maybe I need some help. I've cheked the "calc_spatials" function, but I can't find a way to obtain the X,Y,Z of a point inside the Depth image. I drew a point in the color image (with a X, Y depending on the width, height of the frame), then I just did a copy-paste on the depth image. There is a possibility to obtain the Z value, knowing the X and Y of a point from the Depth Image? Maybe is a trivial question but I can't find a way to solve this.
Thank you very much!!

palmerini

Hello again Luxonis-Lukasz !!
I'm trying to follow the first approach you have mentioned. I've drawn inside the color image some landmarks, that have some 2d coordinates, depending on the normalizated values and height/width of the image. Then I used the same function to draw the landmarks also on the depth image, and the position of the points are the same since the Color and the Depth image has the same heigth and width after the alignment. What I want is, for each landmark, detect their X, Y, Z (mm) from the StereoDepth image passing as input their X,Y (pixels).
I know i'm asking a lot of help, but i'm still learning and understanding how to do it. Sorry and thank you very much again!

Luxonis-Lukasz

Hi @palmerini,

If you have the X, Y coordinates of the point you want to gather depth info for, and you made sure that both color and depth frames are aligned, you can take the value under X/Y coordinate from the depth frame to get the point distance.

I think we should prepare an example showing how to get the distance of a point (we do have a ROI depth here, but not for single points). I'll circle back when it's available

palmerini

Hi Luxonis-Lukasz!
I have investigated the code to obtain the distance of a point and i used the ROI as described in the shared link, but if it's possible to obtain the depth of a single points it should be perfect! Thank you very much!

erik

Hello palmerini ,
in that case, I would just use the same example code and instead of ROI just provide it with a single pixel. I wouldn't advise it because of the noise in the depth frame. I would just use a very small ROI, so you can filter out the noise.
Thanks, Erik

palmerini

Hi erik! Thank you very much for your reply!
Last week I've exactly used this approach: I've used a really small ROI in order to obtain the depth of the image for let's say 1 pixel, but due to the noise of the depth frame, it's not possible to obtain a valid value of the depth for a pixel with X,Y coordinates. It works fine if I consider a bigger ROI. So I'm searching other way to achieve my goal.
Thank you so much!

erik

Hello palmerini,
in that case, I would look into depth filtering (eg wls filter), since on-device median filtering is disabled. You should also filter out distances above/below threshold (in the script above this means setting appropriate variables DEPTH_THRESH_LOW and DEPTH_THRESH_HIGH) and possibly experiment use a different kind of depth averaging (eg. median, median /w bins, mode etc). Hope this helps🙂
Thanks, Erik

palmerini

@erik thank you for your advices!! Looking very quickly to wls filter, I think it can help me a bit better! I'm going go look the example code and try to figure out which features can help me!
Thank you!!

erik

palmerini happy to help🙂

isabel-roruiz

Hi everyone!!!
I am coming here because I am using this camera for my master's final project and i am new in this field.
What I want is to have a 3D image in which I can represent the human skeleton. I have seen in luxonis documentation examples this code: https://github.com/geaxgx/depthai_blazepose
If I am not wrong this code doesn't use depth. Instead of use it predict the human pose in the 3D plane. We prove this the last semester in the project of a partner and this was so imprecise depending of the movements.

For that reason we have deciced to change for this new project and we would like to try get de image 2D, calculate the depth and obtain as a result of both the 3D image where we can apply the detection of the human pose.

Where should I start looking to work with these two things to get what I need? Any advice is welcome. I read this topic but it is not clear at all for me.

Thank you in advance!!!!

erik

Hello isabel-roruiz,
you are right, depth estimation is already a part of the blazepose network, so that demo doesn't use any DepthAI depth features. You could either use SpatialLocationCalculator to calculate spatial coordinates on the device (based on ROI - in your case a small rectangle around the body keypoints), or calculate spatial coordinates on the host, demo here (again, based on small ROI around keypoints).
On the huma-machine-safety demo: We decode palm detection NN results on the host (quite complex), take the detected bounding box, decrease it a bit and use it to calculate spatial coordinates based on ROI and depth frame. Just note here that depth should be aligned to color camera to achieve best results.
Thanks, Erik