If I subtract 2 StereoDepth frames from each other how to output in OpenCV

AdamPolak · 2023-06-01T20:39:57+00:00

I created a NN that subtracts 2 depth frames: class DiffImgs(nn.Module): def forward(self, img1, img2): if img1.shape != (1, 1, 960, 1632) or img2.shape !...

AdamPolak

jeremie_m

Can you change this line:

diff = np.array(floatVector).reshape(resolution[0], resolution[1])

diff = np.array(floatVector).reshape(resolution[1], resolution[0])

jeremie_m

AdamPolak
Thank you Adam,
It seems better now!

Maybe it's not so clear but right?
You help a lot Adam!
And I'm confused that the timestamp is still not right.

AdamPolak

jeremie_m

Which timestamp is not correct?

ts0, time_diff, or ts 2?

What are these times meant to represent?

Could you post what your console is logging?

AdamPolak

jeremie_m

If you move your hand in front of the oakd do you get what you expect?

jeremie_m

AdamPolak
Here is the log:

ts1 =  9 days, 23:06:52.837212
ts0 =  9 days, 23:06:52.837212
time_diff =  -10 days, 0:53:07.162788
ts 2 =  0:00:00
ts0 =  0:00:00
time_diff =  0:00:00
ts 2 =  0:00:00
ts0 =  0:00:00
time_diff =  0:00:00
ts 2 =  0:00:00
ts0 =  0:00:00
time_diff =  0:00:00

jeremie_m

AdamPolak
yeah, here is the hand, although not very clear.
Thank you Adam!

AdamPolak

jeremie_m

It I am not sure how the information is being represented but it looks like HH:MM:SS

Because the depth frame is usually coming in at ~9/second that would be around a .1 second difference. So python is rounding it down to 0 seconds.

With things like this, you usually want to represent time as time since a point in time in milliseconds. You want to capture or display the time in a different format.

Try running: getTimestamp().time_since_epoch (this should display time in milliseconds since it started)

AdamPolak

jeremie_m

Normalization on a depth frame using diff is a bit weird, it doesn't scale very well. Normalizing with different values might do it to highlight the differences better. Changing the frame so that anything that is not 0 is 1, and then displaying that will make all pixels that are the same black, and all pixels that are different white.

It will be a lot more noisy but it will highlight the difference in a louder way.

jeremie_m

AdamPolak
I tried the transition , but I still cannot get timestamp from depthDiff

Maybe I should not get timestamp there?
But according to the documentation, there should be timestamp.
I don't understand.

jeremie_m

AdamPolak
I don't understand why the scale will be not very well when using diff,
and how to normalize with different values?
Could you give me more clue?
Thank you Adam

jeremie_m

AdamPolak
And all I want, is the speed of the moving objects, I want to get speed = depth_diff / delta_time.

AdamPolak

jeremie_m

depth values go into diff model at: (0..65535) depth data in millimeters
they are calculated from 96 disparity steps
so you can only have 96 different depth readings
the readings at long distances "jump" a lot (it is not linear)
so at from step 1 to 2 it might be 10 millimeters, but from step 90 to 91 it might be 10,000 millimeters
so at far away distances (around 7 meters) the depth reading may have read it at step 90 and then even though nothing has changed it then reads it at step 91
this means even though nothing changed, there might be a diff reading of 1000 millimeters
when you wave your hand the distance of the camera to whatever was behind your hand and the distance to your hand (diff) might be small maybe 1,000 millimeters
So the depth_diff has a range of values of 0 (because either it didn't have a confident reading or it is the exact same reading), or 1,000 (because of your hand movement), or 10,000 (because a far wall changed a step reading even though nothing changed)
So when you run colorize = cv2.normalize(diff, None, 255, 0, cv2.NORM_INF, cv2.CV8UC1)you are asking opencv to change the values that you have in the depth_diff (normalize) so that the smallest value you have in the dept_ diff (0) is 0, and the largest value you have in the depth_diff (10,000) is 255.
From the depth_diff you sent I am guessing that in the top right of the image it is far away walls, and the bottom left of the image is also far away
The largest values are "white" (the far away things)
So your small depth_diff values (like your hand moving) get squashed to be near black (very dark grey), and have a value of 25, only 10% of the largest value (which will be white)
if you set up your depth camera so that there was a large poster only a few meters away from it, then waved your hand infront of THAT, the values of the depth_diff of your hand moving would be larger relative to all readings so it would be more clear
try changing to colorize = cv2.normalize(diff, None, 1, 0, cv2.NORM_INF, cv2.CV_8UC1) and see what happens
it should make it so pixels are only white or black, all 0 values (no difference or unconfident) will be black, and ANY depth diff values that are not 0 will be white

AdamPolak

jeremie_m

it is printing only 1 decimal, it might still be rounding down.

Try this to print to 3 decimal places:

print("ts0 = ", timestamp)
time_diff = depthDiff.getTimestamp() - timestamp
print(f'time_diff = {time_diff:.3f}')  # printing time_diff to 3 decimal places

AdamPolak

jeremie_m

I actually am not sure what you are trying to accomplish with the time diffs. Are you trying to find the difference between since the last depth diff image was generated and the new one was generated?

AdamPolak

jeremie_m

Now I am confused, how does getting the depth_diff of depth frames get you that?

If you want to speed of an object you will want to do a SpatialImgDetection to get the center point and use that overtime.

What objects are you trying to get speed for?

The depth-diff is in a single dimension not in 3 dimensions.

jeremie_m

AdamPolak
You are right Adam.
The depth is just one dimension.
I'm rethinking how to get the change of distance in 3 dimensions now.
Maybe I should train a detection model for SpatialImgDetection firstly?

AdamPolak

jeremie_m

There are a lot of good models out there, just choose one from the model zoo. Use SpatialImgDetections. Find the x,y,z per frame, and then find the difference in distance between distance, and divide by timestamp of frames and you are set.

You will likely want to average this with a window to get rid of noise.

« Previous Page