Multi-camera calibration - OAK-D-Pro W faulty translation matrix

pfgnoobi

Hi @erik,
thanks a lot, also for pointing me to custom NN - seems to be quite intresting to try after i got these "basics" working.

The last days i did some more testing and tried to get depth-people-counting working with live camera stream, not via recorded video. I have a strange error which i´m not able to fix. When i create the xinFrame = pipeline.createXLinkIn() pipeline Depth looks like Picture one. If i comment out the 2 lines after #Linking (see code below) the depth looks normal. Why is this strange thing happening?
(down below is only reduced projectcode)

import cv2
import numpy as np
import depthai as dai

device = dai.Device()
print("Creating Stereo Depth pipeline")
pipeline = dai.Pipeline()

camLeft = pipeline.create(dai.node.MonoCamera)
camRight = pipeline.create(dai.node.MonoCamera)

camLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
camRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

### Define stereoDepth node and create outputs
#---
stereo = pipeline.create(dai.node.StereoDepth)

#Set StereoDepth config
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)  # KERNEL_7x7 default, 5x5, 3x3, MEDIAN_OFF
stereo.setRectifyEdgeFillColor(0)  # Black, to better see the cutout
stereo.setLeftRightCheck(True)
#stereo.setExtendedDisparity(extended)
stereo.setSubpixel(True)

#Alpha scaling to use 'full' FOV
stereo.setAlphaScaling(0.2)
config = stereo.initialConfig.get()
config.postProcessing.brightnessFilter.minBrightness = 0
stereo.initialConfig.set(config)

xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depthOut")

#Define connection between nodes
#--
camLeft.out.link(stereo.left)
camRight.out.link(stereo.right)
#stereo.syncedLeft.link(xoutLeft.input)
#stereo.syncedRight.link(xoutRight.input)
stereo.disparity.link(xoutDepth.input)

#tracking on Device
objectTracker = pipeline.createObjectTracker()
objectTracker.inputTrackerFrame.setBlocking(True)
objectTracker.inputDetectionFrame.setBlocking(True)
objectTracker.inputDetections.setBlocking(True)
objectTracker.setDetectionLabelsToTrack([1])  # track only person
# possible tracking types: ZERO_TERM_COLOR_HISTOGRAM, ZERO_TERM_IMAGELESS
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
# take the smallest ID when new object is tracked, possible options: SMALLEST_ID, UNIQUE_ID
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)

# Linking
xinFrame = pipeline.createXLinkIn()    #------------------- adding
xinFrame.setStreamName("frameIn")     #-------------------- this creates ugly depth
xinFrame.out.link(objectTracker.inputDetectionFrame)

cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
cvColorMap[0] = [0, 0, 0]
print("Creating DepthAI device")

with device:
    device.startPipeline(pipeline)
    q = device.getOutputQueue(name="depthOut", maxSize=4, blocking=False)

    while True:

        name = q.getName()
        depthFrame = q.get().getCvFrame()

        depthRgb = getDisparityFrame(depthFrame, cvColorMap)

        cv2.imshow(name, depthRgb)
        if cv2.waitKey(1) == ord("q"):
            break

jakaskerl

Hi @pfgnoobi
Not able to reproduce the issue. The code you have sent does not utilize the object tracker and doesn't have getDisparityFrame defined.

You are linking input to ObjectTracker which should not have an effect on the disparity output.

Thanks,
Jaka

pfgnoobi

Hi @jakaskerl,
sry here is the full code. This code results in a strange depth map.
If you also comment out the Linking part, everything looks nice.
All the other parts which are comment out are just to find the trouble maker code part. Thanks for your help

import cv2
import numpy as np
import depthai as dai

DETECTION_ROI = (20,130,650,240) # x,y,w,h 250,130,450,240
THRESH_DIST_DELTA = 0.2 #thrsh minimum distance to get counted

def getDisparityFrame(frame, cvColorMap):
    maxDisp = stereo.initialConfig.getMaxDisparity()
    disp = (frame * (255.0 / maxDisp)).astype(np.uint8)
    disp = cv2.applyColorMap(disp, cvColorMap)

    return disp

class TextHelper:
    def __init__(self) -> None:
        self.bg_color = (0, 0, 0)
        self.color = (255, 255, 255)
        self.text_type = cv2.FONT_HERSHEY_SIMPLEX
        self.line_type = cv2.LINE_AA
    def putText(self, frame, text, coords):
        cv2.putText(frame, text, coords, self.text_type, 1.3, self.bg_color, 5, self.line_type)
        cv2.putText(frame, text, coords, self.text_type, 1.3, self.color, 2, self.line_type)
        return frame
    def rectangle(self, frame, topLeft,bottomRight, size=1.):
        cv2.rectangle(frame, topLeft, bottomRight, self.bg_color, int(size*4))
        cv2.rectangle(frame, topLeft, bottomRight, self.color, int(size))
        return frame

def to_planar(arr: np.ndarray) -> list:
    return arr.transpose(2, 0, 1).flatten()


class PeopleCounter:
    def __init__(self):
        self.tracking = {}
        self.lost_cnt = {}
        self.people_counter = [0,0,0,0] # Up, Down, Left, Right

    def __str__(self) -> str:
        return f"Left: {self.people_counter[2]}, Right: {self.people_counter[3]}"

    def tracklet_removed(self, coords1, coords2):
        deltaX = coords2[0] - coords1[0]
        print('Delta X', deltaX)

        if THRESH_DIST_DELTA < abs(deltaX):
            self.people_counter[2 if 0 > deltaX else 3] += 1
            direction = "left" if 0 > deltaX else "right"
            print(f"Person moved {direction}")

    def get_centroid(self, roi):
        x1 = roi.topLeft().x
        y1 = roi.topLeft().y
        x2 = roi.bottomRight().x
        y2 = roi.bottomRight().y
        return ((x2+x1)/2, (y2+y1)/2)

    def new_tracklets(self, tracklets):
        for t in tracklets:
            # If new tracklet, save its centroid
            if t.status == dai.Tracklet.TrackingStatus.NEW:
                self.tracking[str(t.id)] = self.get_centroid(t.roi)
                self.lost_cnt[str(t.id)] = 0
            elif t.status == dai.Tracklet.TrackingStatus.TRACKED:
                self.lost_cnt[str(t.id)] = 0
            elif t.status == dai.Tracklet.TrackingStatus.LOST:
                self.lost_cnt[str(t.id)] += 1
                # Tracklet has been lost for too long
                if 10 < self.lost_cnt[str(t.id)]:
                    self.lost_cnt[str(t.id)] = -999
                    self.tracklet_removed(self.tracking[str(t.id)], self.get_centroid(t.roi))
            elif t.status == dai.Tracklet.TrackingStatus.REMOVED:
                if 0 <= self.lost_cnt[str(t.id)]:
                    self.lost_cnt[str(t.id)] = -999
                    self.tracklet_removed(self.tracking[str(t.id)], self.get_centroid(t.roi))


device = dai.Device()
print("Creating Stereo Depth pipeline")
pipeline = dai.Pipeline()

camLeft = pipeline.create(dai.node.MonoCamera)
camRight = pipeline.create(dai.node.MonoCamera)

camLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
camRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

### Define stereoDepth node and create outputs
#---
stereo = pipeline.create(dai.node.StereoDepth)

#Set StereoDepth config
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)  # KERNEL_7x7 default, 5x5, 3x3, MEDIAN_OFF
stereo.setRectifyEdgeFillColor(0)  # Black, to better see the cutout
stereo.setLeftRightCheck(True)
#stereo.setExtendedDisparity(extended)
stereo.setSubpixel(True)

#Alpha scaling to use 'full' FOV
stereo.setAlphaScaling(0.2)
config = stereo.initialConfig.get()
config.postProcessing.brightnessFilter.minBrightness = 0
stereo.initialConfig.set(config)

xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depthOut")

#Define connection between nodes
#--
camLeft.out.link(stereo.left)
camRight.out.link(stereo.right)
#stereo.syncedLeft.link(xoutLeft.input)
#stereo.syncedRight.link(xoutRight.input)
stereo.disparity.link(xoutDepth.input)

#tracking on Device
objectTracker = pipeline.createObjectTracker()
objectTracker.inputTrackerFrame.setBlocking(True)
objectTracker.inputDetectionFrame.setBlocking(True)
objectTracker.inputDetections.setBlocking(True)
objectTracker.setDetectionLabelsToTrack([1])  # track only person
# possible tracking types: ZERO_TERM_COLOR_HISTOGRAM, ZERO_TERM_IMAGELESS
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
# take the smallest ID when new object is tracked, possible options: SMALLEST_ID, UNIQUE_ID
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)

# Linking
xinFrame = pipeline.createXLinkIn()     #------this seems to add ...
xinFrame.setStreamName("frameIn")       #------- ... strange depth error, comment out and it looks fine
xinFrame.out.link(objectTracker.inputDetectionFrame)
'''
# Maybe we need to send the old frame here, not sure
xinFrame.out.link(objectTracker.inputTrackerFrame)

xinDet = pipeline.createXLinkIn()
xinDet.setStreamName("detIn")
xinDet.out.link(objectTracker.inputDetections)

trackletsOut = pipeline.createXLinkOut()
trackletsOut.setStreamName("trackletsOut")
objectTracker.out.link(trackletsOut.input)

'''

cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
cvColorMap[0] = [0, 0, 0]
print("Creating DepthAI device")

with device:
    device.startPipeline(pipeline)
    q = device.getOutputQueue(name="depthOut", maxSize=4, blocking=False)
    '''
    trackletsQ = device.getOutputQueue(name="trackletsOut", maxSize=4, blocking=False)
    detInQ = device.getInputQueue("detIn")
    frameInQ = device.getInputQueue("frameIn")
    '''
    disparityMultiplier = 255 / stereo.initialConfig.getMaxDisparity()

    text = TextHelper()
    counter = PeopleCounter()
    #bis hier
    while True:

        name = q.getName()
        depthFrame = q.get().getFrame()
        depthFrame = (depthFrame*disparityMultiplier).astype(np.uint8)    #use for depth
        depthRgb = getDisparityFrame(depthFrame, cvColorMap)
        '''
        trackletsIn = trackletsQ.tryGet()
        if trackletsIn is not None:
            counter.new_tracklets(trackletsIn.tracklets)

        # Crop only the corridor:

        cropped = depthFrame[DETECTION_ROI[1]:DETECTION_ROI[3], DETECTION_ROI[0]:DETECTION_ROI[2]]
        cv2.imshow('Crop', cropped)

        ret, thresh = cv2.threshold(cropped, 20, 145, cv2.THRESH_BINARY)
        cv2.imshow('thr', thresh)

        blob = cv2.morphologyEx(thresh, cv2.MORPH_OPEN,
                                cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (22, 22)))  # editet elipse size 37,37
        cv2.imshow('blob', blob)

        edged = cv2.Canny(blob, 20, 80)
        cv2.imshow('Canny', edged)

        contours, hierarchy = cv2.findContours(edged, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

        dets = dai.ImgDetections()
        # len contorus is count of seperate heads/blobs
        if len(contours) != 0:
            c = max(contours, key=cv2.contourArea)
            x, y, w, h = cv2.boundingRect(c)
            # cv2.imshow('Rect', text.rectangle(blob, (x,y), (x+w, y+h)))
            x += DETECTION_ROI[0]
            y += DETECTION_ROI[1]
            area = w * h
            # print(len(contours), area)

            if 760 < area:
                # Send the detection to the device - ObjectTracker node
                det = dai.ImgDetection()
                det.label = 1
                det.confidence = 1.0
                det.xmin = x
                det.ymin = y
                det.xmax = x + w
                det.ymax = y + h
                dets.detections = [det]

                # Draw rectangle on the biggest countour
                text.rectangle(depthRgb, (x, y), (x + w, y + h), size=2.5)

        detInQ.send(dets)
        imgFrame = dai.ImgFrame()
        imgFrame.setData(to_planar(depthRgb))
        imgFrame.setType(dai.RawImgFrame.Type.BGR888p)
        imgFrame.setWidth(depthRgb.shape[0])
        imgFrame.setHeight(depthRgb.shape[1])
        frameInQ.send(imgFrame)
        '''
        text.rectangle(depthRgb, (DETECTION_ROI[0], DETECTION_ROI[1]), (DETECTION_ROI[2], DETECTION_ROI[3]))
        text.putText(depthRgb, str(counter), (20, 40))

        cv2.imshow(name, depthRgb)
        if cv2.waitKey(1) == ord("q"):
            break

jakaskerl

Hi pfgnoobi

Running script as is:

With commented out lines:

Are you sure you are not also changing socket so something. This looks like switched left and right.

Thanks,
Jaka

pfgnoobi

jakaskerl
I did some more testing but i can´t figure out where this error comes from.
I´m running excactly the same code and it gives me an ugly depth, as soon as i comment out the two lines after #Linking it works fine.

If i invert the sockets, like this and leave the rest of the code the same as in older post it correct depth. So issue fixed thanks

#Define connection between nodes #-- camLeft.out.link(stereo.right) camRight.out.link(stereo.left)

jakaskerl

Hi @pfgnoobi
That's strange but I'm glad it's fixed not. I'll keep an eye out if something similar happens. Also make sure to use the latest depthai, many bugs are ironed out compared to older versions.

Thanks,
Jaka

pfgnoobi

erik
Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the canny filter and the morphology.open.
Does someone know how to fix this or where to look further?

Canny filter:

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter

# Name of the model
name = 'threshold'

# Define the model class
class Model(nn.Module):

    def forward(self, image):
        canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1]
        return canny_edges

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)

# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

throws error:
File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 18, in forward canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1] File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\filters\canny.py", line 93, in canny nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2) RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

pfgnoobi

Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the morphology.open.
Does someone know how to fix this or where to look further?

custom nn for cv2.morphologyEx(thresh, cv2.MORPH_OPEN,cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (22, 22))):

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter
# Name of the model
name = 'threshold'
# Define the model class
class Model(nn.Module):

    def forward(self, image):
        # Define structuring element (kernel) for morphological operation
        kernel_size = (22, 22)
        # Create an empty tensor to represent the elliptical structuring element
        structuring_element = torch.zeros(*kernel_size)
        # Calculate the coordinates of the center of the ellipse
        center_x = kernel_size[1] // 2
        center_y = kernel_size[0] // 2
        # Set the values inside the ellipse to 1
        for i in range(kernel_size[0]):
            for j in range(kernel_size[1]):
                if ((j - center_x) / (kernel_size[1] // 2)) ** 2 + ((i - center_y) / (kernel_size[0] // 2)) ** 2 <= 1:
                    structuring_element[i, j] = 1
        # Perform morphology operation (opening) using Kornia
        blob = kornia.morphology.opening(image, structuring_element)
        return blob

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)
# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

error log:

Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 49, in <module>
    torch.onnx.export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 516, in export
    _export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1613, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1139, in _model_to_graph
    graph = _optimize_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 677, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1957, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset12.py", line 331, in unfold
    return opset9.unfold(g, input, dimension, const_size, const_step)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset9.py", line 3149, in unfold
    return symbolic_helper._unimplemented(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 612, in _unimplemented
    _onnx_unsupported(f"{op}, {msg}", value)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 623, in _onnx_unsupported
    raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of operator Unfold, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues  [Caused by the value '18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.] 
    (node defined in C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py(4495): pad
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(174): erosion
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(259): opening
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(34): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1501): _slow_forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(129): wrapper
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(138): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(1296): _get_trace_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(915): _trace_and_get_graph_from_model
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1011): _create_jit_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1135): _model_to_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1613): _export
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(516): export
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(49): <module>
)

    Inputs:
        #0: image defined in (%image : Float(1, 3, 300, 300, strides=[270000, 90000, 300, 1], requires_grad=0, device=cpu) = prim::Param()
    )  (type 'Tensor')
        #1: 18837 defined in (%18837 : Long(8, strides=[1], device=cpu) = onnx::Cast[to=7](%18836), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
        #2: 18838 defined in (%18838 : Float(device=cpu) = onnx::Constant[value={10000}](), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
    Outputs:
        #0: 18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')

Process finished with exit code 1

Does this mean that the onnx export does not support Unfold or something with onnx: Pad. How would i get arround this? Would i need to ask Pytorch ONNX people for help here?

Thanks a lot for any clues 🙂

erik

Hi @pfgnoobi ,
For the 1st post, it seems it's quite evident from the error log:

nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2)
RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

So it seems you didn't supply padding.

For second issue, it seems like pytorch onnx export issue, so best to open a github issue on their github repo. I would also note that the nested loop when creating a model isn't feasible - you should use pytorch (vector) functions.
Thanks, Erik

pfgnoobi

erik
Hi @erik,
thanks for having a look.
i tried fixing the padding like this:

    padding = nms_kernels.shape[-1] // 2    
    nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=(padding, padding))

which got me a bit further, but then i get to following error: something with Atan not supported when converting onnx to blob. Do you have an idea on how to fix this? OpenVino Documentations says it supports atan, sin, cos ...
creating the model with kornia.canny uses angle: Tensor = torch.atan2(gy, gx). i tried to change this to sin, cos, ... but it will throw same error message with Sin,Acos ... instead

Simplifying model to out\threshold_simplified.onnx
Converting ONNX model to blob format...
Downloading ..\models\threshold_simplified_openvino_2022.1_6shave.blob...
{
    "exit_code": 1,
    "message": "Command failed with exit code 1, command: /opt/intel/openvino2022_1/tools/compile_tool/compile_tool -m /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.xml -o /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.blob -c /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/myriad_compile_config.txt -d MYRIAD -ip U8",
    "stderr": "[ GENERAL_ERROR ] \n/home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu20/b/repos/openvino/src/plugins/intel_myriad/graph_transformer/src/frontend/frontend.cpp:592 Failed to compile layer \"/Atan_output_0\": unsupported layer type \"Atan\"\n",
    "stdout": "OpenVINO Runtime version ......... 2022.1.0\nBuild ........... 2022.1.0-7019-cdb9bec7210-releases/2022/1\nNetwork inputs:\n    image : u8 / [...]\nNetwork outputs:\n    magnitude/sink_port_0 : f16 / [...]\n"
}
Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 74, in <module>
    blobconverter.from_onnx(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 421, in from_onnx
    return compile_blob(blob_name=Path(model_name).stem, req_data={"name": Path(model_name).stem}, req_files=files, data_type=data_type, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 307, in compile_blob
    response.raise_for_status()
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://blobconverter.luxonis.com/compile?version=2022.1&no_cache=True

thanks 🙂

erik

Hi @pfgnoobi .
Atan actually isn't supported by the VPU plugin, see their docs here.

pfgnoobi

erik
Hi @erik,
thanks for pointing me to the correct docs. Seems like i´m at a dead end wih canny.

Let´s say i would have a custom nn with all the image proccesing. How would i send back detections as imgDetections from that custom nn to the Objeckt Tracker Node. Or in more detail what would i need to add to the custom nn? If i just add another return value e.g a tensor with [label, confidence, xmin,ymin,xmax,ymax] how would i convert this to imgDetections before passing it to ObjectTracker Node on device?

Rethinking all of this does it even make sense to try doing this all on device to achive realtime tracking values? Or proceed further with just taking the depth image from the camera and do all the rest on Host?
What are your experienced thoughts on this?

big thanks

erik

Hi @pfgnoobi ,
So you'd want to create custom NN that would output imgDetections? What about first using custom NN to do some CV processing, then output that into NN that does detection (Yolo/Mobilenet will output you ImgDetections directly). Thougths?

« Previous Page