If I subtract 2 StereoDepth frames from each other how to output in OpenCV

@erik

Hello erik, do you have tutorials or examples in processing depth frames?

Hi jeremie_m
What kind of depth frame processing are you thinking of?

Thanks,
Jaka

jakaskerl

Thank you for reply.

1. As the subtraction of depth frames as we mentioned here.

2. Selection of the farthest or nearest area of pixels in depth frame.

3. Mask of specific shape from or to depth frame.

   (4. Some NN models use both depth and RGB image maybe.)

   .etc

   Some thoughts for now, thank you

jeremie_m does the code Adam provided above not work? Besides the tutorials we have on documentation / depthai-experiments, we don't have any additional ones.

erik

Thank you erik.

I'm not sure if the first code works, cause it doesn't match the inputs with the second code.

And I'm looking into how to generate the shave.blob, I'm not clear in processing the NN model.

Hi jeremie_m
Here are some guides on how to create NN models for image processing.

Hope it helps,
Jaka

jakaskerl thank you

AdamPolak

Hello Adam, I've add the following process for the inputs, but it seems something not right.

Could you let me know how to adapt the depthai code.

Thank you Adam.

script.setScript("""
old = node.io['in'].get()
while True:
frame = node.io['in'].get()
node.io['img1'].send(old)
node.io['img2'].send(frame)
old = frame
""")
script.outputs['img1'].link(nn.inputs['input1'])
script.outputs['img2'].link(nn.inputs['inout2'])

AdamPolak

Thanks for reply.

I got images like this with the diff process above:

AdamPolak This is the "final" version to do a diff between 2 depth map images:

And I added time_diff code:

timestamp = dai.Clock.now();

with dai.Device(p) as device:

…

time_diff = depthDiff.getTimestamp() - timestamp
print('time_diff = ', time_diff)
timestamp = depthDiff.getTimestamp()

Which the output is always 0.0

I'm confused now.

AdamPolak

All I want to do, is just to subtract two frames in sequence, just use the older frame as the 'control' frame.

AdamPolak
And the model code is:

from pathlib import Path
import torch
from torch import nn
import blobconverter
import onnx
from onnxsim import simplify
import sys

class DiffImgs(nn.Module):
    def forward(self, img1, img2):
        img1DepthFP16 = 256.0 \* img1[:,:,:,1::2] + img1[:,:,:,::2]
        img2DepthFP16 = 256.0 \* img2[:,:,:,1::2] + img2[:,:,:,::2]

        # Create binary masks for each image
        # A pixel in the mask is 1 if the corresponding pixel in the image is 0, otherwise it's 0
        img1Mask = (img1DepthFP16 == 0)
        img2Mask = (img2DepthFP16 == 0)

        # If a pixel is 0 in either image, set the corresponding pixel in both images to 0
        img1DepthFP16 = img1DepthFP16 \* (\~img1Mask & \~img2Mask)
        img2DepthFP16 = img2DepthFP16 \* (\~img1Mask & \~img2Mask)

        # Compute the difference between the two images
        diff = torch.sub(img1DepthFP16, img2DepthFP16)
        return diff

\# Instantiate the model
model = DiffImgs()

\# Create dummy input for the ONNX export
input1 = torch.randn(1, 1, 800, 1280 \* 2, dtype=torch.float16)
input2 = torch.randn(1, 1, 800, 1280 \* 2, dtype=torch.float16)

onnx_file = **"diff_images.onnx"**

\# Export the model
torch.onnx.export(model,               # model being run
                  (input1, input2),    # model input (or a tuple for multiple inputs)
                  onnx_file,        # where to save the model (can be a file or file-like object)
                  opset_version=12,    # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = [**'input1'**, **'input2'**],   # the model's input names
                  output_names = [**'output'**])

\# Simplify the model
onnx_model = onnx.load(onnx_file)
onnx_simplified, check = simplify(onnx_file)
onnx.save(onnx_simplified, **"diff_images_simplified.onnx"**)

\# Use blobconverter to convert onnx->IR->blob
blobconverter.from_onnx(
    model=**"diff_images_simplified.onnx"**,
    data_type=**"FP16"**,
    shaves=4,
    use_cache=False,
    output_dir=**"../"**,
    optimizer_params=[],
    compile_params=[**'-ip U8'**],
)

AdamPolak
Thank you Adam,
It seems better now!

Maybe it's not so clear but right?
You help a lot Adam!
And I'm confused that the timestamp is still not right.