Get depth value in script embeded in camera

Otabiel

Hi,
I'm trying to adapt an existing script to add in the capacity of matching the output coordinate of a NN for pose estimation, with the depth value of the stereo capture.

I've created my depth pipeline as :

            stereo = pipeline.createStereoDepth()
            stereo.setConfidenceThreshold(230)
           
            stereo.setLeftRightCheck(True)
            stereo.setDepthAlign(dai.CameraBoardSocket.RGB)
            stereo.setSubpixel(False) 
            stereo.setMedianFilter(dai.StereoDepthProperties.MedianFilter.MEDIAN_OFF)
            
            left.out.link(stereo.left)
            right.out.link(stereo.right)

            stereo.depth.link(spatial_location_calculator.inputDepth)

            depth_out = pipeline.create(dai.node.XLinkOut)
            depth_out.setStreamName("depth")
            depth_out.input.setQueueSize(1)
            depth_out.input.setBlocking(False)
            stereo.depth.link(manager_script.inputs['depth_data'])

Then in the scipt load in the camera I've added
depth_link = node.io['depth_data'].get()
For getting the object ImgFrame, and when I try to check the type of the object it tells me that it is "lpb.ImgFrame". In this case the method as getFrame() does not exist for this object type.

What I'm trying to make is as this function show :

    def get_depth(self, point):
        depth_video = self.q_depth.get()
        depth_frame = depth_video.getFrame()
        depth = depth_frame[point[1]][point[0]]

Catching the depth value in millimeters of a specific coordinate.

I hope you would be able to help me

Thanks in advance

erik

Hi Otabiel ,
I think you would need to use .getData(), which would be a byte array (1D) of the depth frame. So you would have to calculate a bit to get desired depth (im mm) from your x/y coordinates. Thoughts?
Thanks, Erik

Otabiel

Like, if I have a frame of resolution NxM, if I want the pixel at (x,y) juste getting the getData[y*N+x] ?

erik

Hi Otabiel , Yes, I just think that you would get byte array, and depth is int16, so you would need to read 2 bytes (and also start reading from index 2*(y*N+x) afaik).

Otabiel

That explain why the size of getData return is so big ahah many thanks for your answer, i'll try it !

Otabiel

Hi, I've try something, but value seems to be not really good, did I maybe do it the wrong way ?

        depth_link = node.io['depth_data'].get()
        data = depth_link.getData()

        test = data[2 * (400 * ${_img_w} + 400)]
        test_2 = data[2 * (400 * ${_img_w} + 400) + 1]

        final = (test << 8) + test_2

The value output of final is something like : 17666, but it should be something closer to 1000 mm

erik

Hi Otabiel
Could you also try little endian vs big endian? As I'm not sure which we use. And also I would try averaging a few pixels around the point of interest (similar to how it's done here), as depth images can be noisy.

Otabiel

I managed to solve my problem! Many thanks for your answer ! 😁

erik

Otabiel Could you please share what was the culprit of the error, so I (and other users) know for the future😃?

Otabiel

Oh yes of course aha! I was reading it wrongly, so just swap the order of reading of the two bytes from Big endian to little endian and I receive better value. I will also try to do as the ROI method with the spacial locator

stevex0

Hi! Do you happen to have any code of what that would look like? I'm not sure I understand the big/little endian ordering here.

jakaskerl

Hi @stevex0
For 16bit data (or more), the data is written in two bytes. If you have some number eg 0x1234, big endian way of saving it would be

Address: 0x00 0x01
Value: 0x12 0x34

and little endian would be:

Address: 0x00 0x01
Value: 0x34 0x12

Basically just swap the data (swap the two bytes representing one number) in the array to obtain the correct value.

Thanks,
Jaka