Hi there @"Nejc-Luxonis"#695 , I try to run a [multi camera ](https://github.com/luxonis/depthai-experiments/tree/master/gen2-multiple-devices/multi-cam-calibration) calibration by launching main.py. I first validated the calibration on my OAK-D-Pro camera and I confirm a reasonable result. Later on, I tried the same procedure with OAK-D-Pro W camera. The rotation vector looks promising, but the transformation vector shows unreal values. See the values comparison of both cameras calibrated with marker in the same position: **OAK-D-Pro:** Rotation vector: [[-0.06005594] [ 0.09223808] [ 0.0277156 ]] Translation vector: [[-0.03522526] [-0.04736168] [ 0.70119316]] **OAK-D-Pro W** Rotation vector: [[0.00374177] [0.00809129] [0.02615565]] Translation vector: [[-1.55830205] [-0.91438703] [ 2.036714 ]] Any opinions on potential reason ? Thank you

Hi @"marlu"#p9627 , Could you elaborate on why those would be unreal values?

Hi @"erik"#175 , because I physically measured and verified that value shown by OAK-D-Pro calibration is correct. But when OAK-D-Pro W is placed at the same position, the translation vector is apparently wrong.

@"erik"#175 @"Nejc-Luxonis"#695 Is this calibration example tested also with cameras facing other directions? Every camera would find its relative transformation to its own printed pattern, while the transformation between two patterns is known.

Hi @"marlu"#p9741 , It might be that you'd need to undistort the stream, not sure if cv2 provides correct extrinsics if the image is warped.. About second question - current scripts don't support such use-case, but we could add it in the future. So you have 2 calibration boards at a known distance/rotation, and have multiple OAK cameras looking at them, and you'd want to get extrinsics of each cameras based on each other camera?

@"erik"#175 Correct, my calibration setup is visualized on the following scheme. If CAM1 Pattern1 and CAM2 Pattern2 transforms are correct, I can easily compute the transformation between cameras. I interchanged OAK-D-Pro W to standard OAK-D-Pro, although the computed transform is still quite different to distances approximately measured by the distance measuring tool. [URL=https://i.imgur.com/c7OKw9f.png][IMG]https://i.imgur.com/c7OKw9f.png[/IMG][/URL]

@"marlu"#1202 @"erik"#175 do you have any updates on the Multi-Camera Calibration of cameras not facing the same view? And did you find a way to undistort the image? i tried to use OAK-D-W POE, faceing the same issue with wrong vectors with calibration. (i´m using the depthai-experiments/gen2-multiple-devices examples)

Hi @"pfgnoobi"#2244 Could you add the matrices you received from the multi cam calibration and the intrinsic matrix/distortion coefficients from the eeprom? Some visual aid and ground truth will help as well. Thanks, Jaka

Hi @"pfgnoobi"#2244 , We haven't yet pursued it, but likely will in the upcoming months. Regarding the undistortion - could you elaborate on the question? Like wide FOV img undistortion?

Hi, so to give some more context: I´m trying to get the position data (floor x/y) of multiple people walking through a light installation built inside a 30' shipping container. My first Idea was to just use a single OAK-D-W-POE OV9782, mounted in the middle-top facing straight down. Not working as would need to train my own person detection model (available ones don´t like birdseye cam data). So my new test setup would look like this or maybe just 4x OAK-D, one in each corner? [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-02-19/1708377530-95875-image.png] Problem with new Setup: Multicam Calibration not working, as calibration grid placed in the middle of the container is too small do recognice. Need to place in the middle so all cam's can see it. @"erik"#175 Problem combining OAK-D-W-POE and OAK-D-lite with depthai-experiments/gen2-multiple-devices/spatial-detection-fusion: OAK-D-W not lining up with OAK-D-lite, see screenshots bottom half, Bird's Eye View only showing OAK-D-lite position (OAK-D-wide position data likely somewhere off screen). I have the feeling this is due to the 150° FOV. Do in need to add some wide FOV img undistortion in the spatial-detection or calibration process? If yes, where and what would i need to add. [upl-image-preview url=https://discuss.luxonis.com/assets/files/2024-02-19/1708378518-984646-untitled-1.png] \*top two pics are from multicam-calibration process, rest is from running the spatial-fusion-detection @"jakaskerl"#p17491 i uploaded the files here, .json was not allowed in the forum\* https://we.tl/t-h170S5oKo7 thanks for your help, I don´t really know what i´m doing, just getting startet with computer vision. Still a noob. Also please let me know if you know a different approach to the final person position data goal. Maybe also just 2-3 Webcams on top facing straight down and some sort of blob track?

Hi @"pfgnoobi"#2244 , Could you share some screenshots by just using depth cameras instead, as mentioned in the email? https://github.com/luxonis/depthai-experiments/tree/master/gen2-depth-people-counting#depth-people-counting > Do in need to add some wide FOV img undistortion in the spatial-detection or calibration process? Only if the NN requires undistorted images, then you should use Camera node and use undistortion: https://docs.luxonis.com/projects/api/en/latest/samples/Camera/camera_undistort/#undistort-camera-stream > OAK-D-W not lining up with OAK-D-lite, see screenshots bottom half What would be the point of the multi-cam calibration if that's the case? In the example above, it's for spatial detection fusion - as multiple cameras detect the same object (and its 3d detection), to fuse it together for better accuracy. Thanks, Erik

Multi-camera calibration - OAK-D-Pro W faulty translation matrix

jakaskerl

Hi @pfgnoobi
That's strange but I'm glad it's fixed not. I'll keep an eye out if something similar happens. Also make sure to use the latest depthai, many bugs are ironed out compared to older versions.

Thanks,
Jaka

pfgnoobi

erik
Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the canny filter and the morphology.open.
Does someone know how to fix this or where to look further?

Canny filter:

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter

# Name of the model
name = 'threshold'

# Define the model class
class Model(nn.Module):

    def forward(self, image):
        canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1]
        return canny_edges

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)

# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

throws error:
File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 18, in forward canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1] File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\filters\canny.py", line 93, in canny nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2) RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

pfgnoobi

Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the morphology.open.
Does someone know how to fix this or where to look further?

custom nn for cv2.morphologyEx(thresh, cv2.MORPH_OPEN,cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (22, 22))):

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter
# Name of the model
name = 'threshold'
# Define the model class
class Model(nn.Module):

    def forward(self, image):
        # Define structuring element (kernel) for morphological operation
        kernel_size = (22, 22)
        # Create an empty tensor to represent the elliptical structuring element
        structuring_element = torch.zeros(*kernel_size)
        # Calculate the coordinates of the center of the ellipse
        center_x = kernel_size[1] // 2
        center_y = kernel_size[0] // 2
        # Set the values inside the ellipse to 1
        for i in range(kernel_size[0]):
            for j in range(kernel_size[1]):
                if ((j - center_x) / (kernel_size[1] // 2)) ** 2 + ((i - center_y) / (kernel_size[0] // 2)) ** 2 <= 1:
                    structuring_element[i, j] = 1
        # Perform morphology operation (opening) using Kornia
        blob = kornia.morphology.opening(image, structuring_element)
        return blob

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)
# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

error log:

Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 49, in <module>
    torch.onnx.export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 516, in export
    _export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1613, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1139, in _model_to_graph
    graph = _optimize_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 677, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1957, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset12.py", line 331, in unfold
    return opset9.unfold(g, input, dimension, const_size, const_step)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset9.py", line 3149, in unfold
    return symbolic_helper._unimplemented(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 612, in _unimplemented
    _onnx_unsupported(f"{op}, {msg}", value)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 623, in _onnx_unsupported
    raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of operator Unfold, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues  [Caused by the value '18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.] 
    (node defined in C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py(4495): pad
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(174): erosion
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(259): opening
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(34): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1501): _slow_forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(129): wrapper
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(138): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(1296): _get_trace_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(915): _trace_and_get_graph_from_model
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1011): _create_jit_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1135): _model_to_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1613): _export
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(516): export
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(49): <module>
)

    Inputs:
        #0: image defined in (%image : Float(1, 3, 300, 300, strides=[270000, 90000, 300, 1], requires_grad=0, device=cpu) = prim::Param()
    )  (type 'Tensor')
        #1: 18837 defined in (%18837 : Long(8, strides=[1], device=cpu) = onnx::Cast[to=7](%18836), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
        #2: 18838 defined in (%18838 : Float(device=cpu) = onnx::Constant[value={10000}](), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
    Outputs:
        #0: 18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')

Process finished with exit code 1

Does this mean that the onnx export does not support Unfold or something with onnx: Pad. How would i get arround this? Would i need to ask Pytorch ONNX people for help here?

Thanks a lot for any clues 🙂

erik

Hi @pfgnoobi ,
For the 1st post, it seems it's quite evident from the error log:

nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2)
RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

So it seems you didn't supply padding.

For second issue, it seems like pytorch onnx export issue, so best to open a github issue on their github repo. I would also note that the nested loop when creating a model isn't feasible - you should use pytorch (vector) functions.
Thanks, Erik

pfgnoobi

erik
Hi @erik,
thanks for having a look.
i tried fixing the padding like this:

    padding = nms_kernels.shape[-1] // 2    
    nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=(padding, padding))

which got me a bit further, but then i get to following error: something with Atan not supported when converting onnx to blob. Do you have an idea on how to fix this? OpenVino Documentations says it supports atan, sin, cos ...
creating the model with kornia.canny uses angle: Tensor = torch.atan2(gy, gx). i tried to change this to sin, cos, ... but it will throw same error message with Sin,Acos ... instead

Simplifying model to out\threshold_simplified.onnx
Converting ONNX model to blob format...
Downloading ..\models\threshold_simplified_openvino_2022.1_6shave.blob...
{
    "exit_code": 1,
    "message": "Command failed with exit code 1, command: /opt/intel/openvino2022_1/tools/compile_tool/compile_tool -m /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.xml -o /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.blob -c /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/myriad_compile_config.txt -d MYRIAD -ip U8",
    "stderr": "[ GENERAL_ERROR ] \n/home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu20/b/repos/openvino/src/plugins/intel_myriad/graph_transformer/src/frontend/frontend.cpp:592 Failed to compile layer \"/Atan_output_0\": unsupported layer type \"Atan\"\n",
    "stdout": "OpenVINO Runtime version ......... 2022.1.0\nBuild ........... 2022.1.0-7019-cdb9bec7210-releases/2022/1\nNetwork inputs:\n    image : u8 / [...]\nNetwork outputs:\n    magnitude/sink_port_0 : f16 / [...]\n"
}
Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 74, in <module>
    blobconverter.from_onnx(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 421, in from_onnx
    return compile_blob(blob_name=Path(model_name).stem, req_data={"name": Path(model_name).stem}, req_files=files, data_type=data_type, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 307, in compile_blob
    response.raise_for_status()
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://blobconverter.luxonis.com/compile?version=2022.1&no_cache=True

thanks 🙂

erik

Hi @pfgnoobi .
Atan actually isn't supported by the VPU plugin, see their docs here.

pfgnoobi

erik
Hi @erik,
thanks for pointing me to the correct docs. Seems like i´m at a dead end wih canny.

Let´s say i would have a custom nn with all the image proccesing. How would i send back detections as imgDetections from that custom nn to the Objeckt Tracker Node. Or in more detail what would i need to add to the custom nn? If i just add another return value e.g a tensor with [label, confidence, xmin,ymin,xmax,ymax] how would i convert this to imgDetections before passing it to ObjectTracker Node on device?

Rethinking all of this does it even make sense to try doing this all on device to achive realtime tracking values? Or proceed further with just taking the depth image from the camera and do all the rest on Host?
What are your experienced thoughts on this?

big thanks

erik

Hi @pfgnoobi ,
So you'd want to create custom NN that would output imgDetections? What about first using custom NN to do some CV processing, then output that into NN that does detection (Yolo/Mobilenet will output you ImgDetections directly). Thougths?

« Previous Page