Hi there @"Nejc-Luxonis"#695 , I try to run a [multi camera ](https://github.com/luxonis/depthai-experiments/tree/master/gen2-multiple-devices/multi-cam-calibration) calibration by launching main.py. I first validated the calibration on my OAK-D-Pro camera and I confirm a reasonable result. Later on, I tried the same procedure with OAK-D-Pro W camera. The rotation vector looks promising, but the transformation vector shows unreal values. See the values comparison of both cameras calibrated with marker in the same position: **OAK-D-Pro:** Rotation vector: [[-0.06005594] [ 0.09223808] [ 0.0277156 ]] Translation vector: [[-0.03522526] [-0.04736168] [ 0.70119316]] **OAK-D-Pro W** Rotation vector: [[0.00374177] [0.00809129] [0.02615565]] Translation vector: [[-1.55830205] [-0.91438703] [ 2.036714 ]] Any opinions on potential reason ? Thank you

Hi @"marlu"#p9627 , Could you elaborate on why those would be unreal values?

Hi @"erik"#175 , because I physically measured and verified that value shown by OAK-D-Pro calibration is correct. But when OAK-D-Pro W is placed at the same position, the translation vector is apparently wrong.

@"erik"#175 @"Nejc-Luxonis"#695 Is this calibration example tested also with cameras facing other directions? Every camera would find its relative transformation to its own printed pattern, while the transformation between two patterns is known.

Hi @"marlu"#p9741 , It might be that you'd need to undistort the stream, not sure if cv2 provides correct extrinsics if the image is warped.. About second question - current scripts don't support such use-case, but we could add it in the future. So you have 2 calibration boards at a known distance/rotation, and have multiple OAK cameras looking at them, and you'd want to get extrinsics of each cameras based on each other camera?

@"erik"#175 Correct, my calibration setup is visualized on the following scheme. If CAM1 Pattern1 and CAM2 Pattern2 transforms are correct, I can easily compute the transformation between cameras. I interchanged OAK-D-Pro W to standard OAK-D-Pro, although the computed transform is still quite different to distances approximately measured by the distance measuring tool. [URL=https://i.imgur.com/c7OKw9f.png][IMG]https://i.imgur.com/c7OKw9f.png[/IMG][/URL]

@"marlu"#1202 @"erik"#175 do you have any updates on the Multi-Camera Calibration of cameras not facing the same view? And did you find a way to undistort the image? i tried to use OAK-D-W POE, faceing the same issue with wrong vectors with calibration. (i´m using the depthai-experiments/gen2-multiple-devices examples)

Multi-camera calibration - OAK-D-Pro W faulty translation matrix

jakaskerl

Hi @pfgnoobi
Could you add the matrices you received from the multi cam calibration and the intrinsic matrix/distortion coefficients from the eeprom?
Some visual aid and ground truth will help as well.

Thanks,
Jaka

pfgnoobi

Hi,
so to give some more context:
I´m trying to get the position data (floor x/y) of multiple people walking through a light installation built inside a 30' shipping container. My first Idea was to just use a single OAK-D-W-POE OV9782, mounted in the middle-top facing straight down. Not working as would need to train my own person detection model (available ones don´t like birdseye cam data). So my new test setup would look like this or maybe just 4x OAK-D, one in each corner?

Problem with new Setup:
Multicam Calibration not working, as calibration grid placed in the middle of the container is too small do recognice. Need to place in the middle so all cam's can see it.

@erik Problem combining OAK-D-W-POE and OAK-D-lite with depthai-experiments/gen2-multiple-devices/spatial-detection-fusion:
OAK-D-W not lining up with OAK-D-lite, see screenshots bottom half, Bird's Eye View only showing OAK-D-lite position (OAK-D-wide position data likely somewhere off screen). I have the feeling this is due to the 150° FOV.
Do in need to add some wide FOV img undistortion in the spatial-detection or calibration process? If yes, where and what would i need to add.

*top two pics are from multicam-calibration process, rest is from running the spatial-fusion-detection

jakaskerl
i uploaded the files here, .json was not allowed in the forum*

https://we.tl/t-h170S5oKo7

thanks for your help, I don´t really know what i´m doing, just getting startet with computer vision. Still a noob.
Also please let me know if you know a different approach to the final person position data goal. Maybe also just 2-3 Webcams on top facing straight down and some sort of blob track?

erik

Hi @pfgnoobi ,
We haven't yet pursued it, but likely will in the upcoming months. Regarding the undistortion - could you elaborate on the question? Like wide FOV img undistortion?

erik

Hi @pfgnoobi ,
Could you share some screenshots by just using depth cameras instead, as mentioned in the email?
luxonis/depthai-experimentstree/master/gen2-depth-people-counting#depth-people-counting

Do in need to add some wide FOV img undistortion in the spatial-detection or calibration process?

Only if the NN requires undistorted images, then you should use Camera node and use undistortion:
https://docs.luxonis.com/projects/api/en/latest/samples/Camera/camera_undistort/#undistort-camera-stream

OAK-D-W not lining up with OAK-D-lite, see screenshots bottom half

What would be the point of the multi-cam calibration if that's the case? In the example above, it's for spatial detection fusion - as multiple cameras detect the same object (and its 3d detection), to fuse it together for better accuracy.
Thanks, Erik

pfgnoobi

erik thanks, i did some birdseyeview depth testing with the depthai viewer. It seems like i have visible hight changes by moving within 4.5-4.6m width on groundarea. This massive reduction in HFOV is due to the need to rectify the monocolorimages for depth/disparity calculation, if i understood correctly.
So the 127°HFOV (here 10m) gets rectified to 96° (measured 5.5m) and usable depth HFOV is 84° (my measured) 4.6m. Also tested with OAK-D, mounted the same way, which resulted in usable height changes within an area arround HFOV 70°.

So to cover a groundarea with width of 12m, i would need 3x OAK-D-W or 4x OAK-D mounted arround 2.5-2.6m height.
I will do some further testing and also reading throught the configuring-stereo-depth docs to hopefully pimp the results.

What is your thought after seeing the Screenshots?
Is top-view depth map the way to proceed?
Would this usecase be better suitable for OAK-D-W-POE?

3 more from depth-preview (depthai-python example)

What would be the point of the multi-cam calibration if that's the case?

sry i meant OAK-D-W person tracking position not lining up with OAK-D-lite person tracking position. In Bird#s Eye View there is only the magenta marker (OAK-D) visible, the yellow one is somewhere outside render area. But no need to investigate further.

big thanks for your support 🙂

erik

Hi @pfgnoobi ,

rectify the monocolorimages for depth/disparity calculation, if i understood correctly.

Yes, that's correct, some FOV is lost due to rectification, but you can configure that with stereo alpha param, example here (--alpha): https://docs.luxonis.com/projects/api/en/latest/samples/StereoDepth/stereo_depth_video/#stereo-depth-video

I'd also run detection on depth frames / colorized frames (like blob detection + thresholding) instead of pointcloud.
Thoughts?

pfgnoobi

Hi @erik,
thanks, i tried to implement this alpha param thing into depth-people-counting example (you suggested earlier).
It worked and i got much closer to my final needs and also slowly understand a bit more.

This are my current results, looks quite okay. Will play with some thresholding to run detection more close to groundlevel (e.g Person knee level) so i get more real x/y coordinates. Or do you also have some ideas on improving this accuracy this?

And to double check if i got the workings of depth-people-counting correct:

Replay.Pipeline loads the Videorecordings and sends qued up frames from Host to Device.
Device creates depth frames send to host
Host does some depth value stretching? why do you need to use this disparity multiplier thing? Is it because different devices have different disparityresolution?
Host creates copy of depthFrame and colors it with colormap (rgbDepth)
depthFrame gets cropped to area of intrest and some openCV stuff to create and detect blob
Detections and rgbDepth get sent to Device into ObjectTracker node
ObjecTracker node creates Tracklet and sends data to Host
Host calculates center and moved distance (moved left/right)

did i read this example code correct?

Can i implement this openCV stuff from Step 5. into the device? using a Script Node?
This process seems not really efficient for realtime stuff, sending stuff back and forth multiple times.
Over the weekend, i will hopefully figure out how to implement the live camara into the example. So maybe this back and forth is nowthing ot worry about.

big thanks

erik

Hi @pfgnoobi ,

On accuracy: I'd apply https://docs.luxonis.com/projects/api/en/latest/tutorials/configuring-stereo-depth/#brightness-filter and then lower confidence threshold. You will have less depth points, but they will be more accurate, which can help with the noise. It's mostly trial & error here.

depthFrame = (depthFrame*disparityMultiplier).astype(np.uint8)

Do you mean this? It's because disparity has values between 0..95 (standard disparity search, or 0..190 with extended, or a lot more if using subpixel mode), and you want to normalize it to 0..255 for nicer visualization.

Detections and rgbDepth get sent to Device into ObjectTracker node

Yes, data flow is quite hacky, we could also do object tracking on the host, but we just utilized as much computation on the OAK in this example.

And yes, the whole flow you described is correct.

Can i implement this openCV stuff from Step 5. into the device? using a Script Node?

Please have a look here:
https://docs.luxonis.com/en/latest/pages/tutorials/on-device-programming/
It would likely be possible to make custom NN that does the CV stuff, but would take more time. With script node it would be way too slow.

Thanks, Erik

pfgnoobi

erik
Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the canny filter and the morphology.open.
Does someone know how to fix this or where to look further?

Canny filter:

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter

# Name of the model
name = 'threshold'

# Define the model class
class Model(nn.Module):

    def forward(self, image):
        canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1]
        return canny_edges

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)

# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

throws error:
File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 18, in forward canny_edges = kornia.filters.canny(image, low_threshold=0.1, high_threshold=0.5, kernel_size=(3,3))[1] File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\filters\canny.py", line 93, in canny nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2) RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

pfgnoobi

Hi @erik,
thanks a lot, also for pointing me to custom NN - seems to be quite intresting to try after i got these "basics" working.

The last days i did some more testing and tried to get depth-people-counting working with live camera stream, not via recorded video. I have a strange error which i´m not able to fix. When i create the xinFrame = pipeline.createXLinkIn() pipeline Depth looks like Picture one. If i comment out the 2 lines after #Linking (see code below) the depth looks normal. Why is this strange thing happening?
(down below is only reduced projectcode)

import cv2
import numpy as np
import depthai as dai

device = dai.Device()
print("Creating Stereo Depth pipeline")
pipeline = dai.Pipeline()

camLeft = pipeline.create(dai.node.MonoCamera)
camRight = pipeline.create(dai.node.MonoCamera)

camLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
camRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

### Define stereoDepth node and create outputs
#---
stereo = pipeline.create(dai.node.StereoDepth)

#Set StereoDepth config
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)  # KERNEL_7x7 default, 5x5, 3x3, MEDIAN_OFF
stereo.setRectifyEdgeFillColor(0)  # Black, to better see the cutout
stereo.setLeftRightCheck(True)
#stereo.setExtendedDisparity(extended)
stereo.setSubpixel(True)

#Alpha scaling to use 'full' FOV
stereo.setAlphaScaling(0.2)
config = stereo.initialConfig.get()
config.postProcessing.brightnessFilter.minBrightness = 0
stereo.initialConfig.set(config)

xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depthOut")

#Define connection between nodes
#--
camLeft.out.link(stereo.left)
camRight.out.link(stereo.right)
#stereo.syncedLeft.link(xoutLeft.input)
#stereo.syncedRight.link(xoutRight.input)
stereo.disparity.link(xoutDepth.input)

#tracking on Device
objectTracker = pipeline.createObjectTracker()
objectTracker.inputTrackerFrame.setBlocking(True)
objectTracker.inputDetectionFrame.setBlocking(True)
objectTracker.inputDetections.setBlocking(True)
objectTracker.setDetectionLabelsToTrack([1])  # track only person
# possible tracking types: ZERO_TERM_COLOR_HISTOGRAM, ZERO_TERM_IMAGELESS
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
# take the smallest ID when new object is tracked, possible options: SMALLEST_ID, UNIQUE_ID
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)

# Linking
xinFrame = pipeline.createXLinkIn()    #------------------- adding
xinFrame.setStreamName("frameIn")     #-------------------- this creates ugly depth
xinFrame.out.link(objectTracker.inputDetectionFrame)

cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
cvColorMap[0] = [0, 0, 0]
print("Creating DepthAI device")

with device:
    device.startPipeline(pipeline)
    q = device.getOutputQueue(name="depthOut", maxSize=4, blocking=False)

    while True:

        name = q.getName()
        depthFrame = q.get().getCvFrame()

        depthRgb = getDisparityFrame(depthFrame, cvColorMap)

        cv2.imshow(name, depthRgb)
        if cv2.waitKey(1) == ord("q"):
            break

jakaskerl

Hi @pfgnoobi
Not able to reproduce the issue. The code you have sent does not utilize the object tracker and doesn't have getDisparityFrame defined.

You are linking input to ObjectTracker which should not have an effect on the disparity output.

Thanks,
Jaka

pfgnoobi

Hi @jakaskerl,
sry here is the full code. This code results in a strange depth map.
If you also comment out the Linking part, everything looks nice.
All the other parts which are comment out are just to find the trouble maker code part. Thanks for your help

import cv2
import numpy as np
import depthai as dai

DETECTION_ROI = (20,130,650,240) # x,y,w,h 250,130,450,240
THRESH_DIST_DELTA = 0.2 #thrsh minimum distance to get counted

def getDisparityFrame(frame, cvColorMap):
    maxDisp = stereo.initialConfig.getMaxDisparity()
    disp = (frame * (255.0 / maxDisp)).astype(np.uint8)
    disp = cv2.applyColorMap(disp, cvColorMap)

    return disp

class TextHelper:
    def __init__(self) -> None:
        self.bg_color = (0, 0, 0)
        self.color = (255, 255, 255)
        self.text_type = cv2.FONT_HERSHEY_SIMPLEX
        self.line_type = cv2.LINE_AA
    def putText(self, frame, text, coords):
        cv2.putText(frame, text, coords, self.text_type, 1.3, self.bg_color, 5, self.line_type)
        cv2.putText(frame, text, coords, self.text_type, 1.3, self.color, 2, self.line_type)
        return frame
    def rectangle(self, frame, topLeft,bottomRight, size=1.):
        cv2.rectangle(frame, topLeft, bottomRight, self.bg_color, int(size*4))
        cv2.rectangle(frame, topLeft, bottomRight, self.color, int(size))
        return frame

def to_planar(arr: np.ndarray) -> list:
    return arr.transpose(2, 0, 1).flatten()


class PeopleCounter:
    def __init__(self):
        self.tracking = {}
        self.lost_cnt = {}
        self.people_counter = [0,0,0,0] # Up, Down, Left, Right

    def __str__(self) -> str:
        return f"Left: {self.people_counter[2]}, Right: {self.people_counter[3]}"

    def tracklet_removed(self, coords1, coords2):
        deltaX = coords2[0] - coords1[0]
        print('Delta X', deltaX)

        if THRESH_DIST_DELTA < abs(deltaX):
            self.people_counter[2 if 0 > deltaX else 3] += 1
            direction = "left" if 0 > deltaX else "right"
            print(f"Person moved {direction}")

    def get_centroid(self, roi):
        x1 = roi.topLeft().x
        y1 = roi.topLeft().y
        x2 = roi.bottomRight().x
        y2 = roi.bottomRight().y
        return ((x2+x1)/2, (y2+y1)/2)

    def new_tracklets(self, tracklets):
        for t in tracklets:
            # If new tracklet, save its centroid
            if t.status == dai.Tracklet.TrackingStatus.NEW:
                self.tracking[str(t.id)] = self.get_centroid(t.roi)
                self.lost_cnt[str(t.id)] = 0
            elif t.status == dai.Tracklet.TrackingStatus.TRACKED:
                self.lost_cnt[str(t.id)] = 0
            elif t.status == dai.Tracklet.TrackingStatus.LOST:
                self.lost_cnt[str(t.id)] += 1
                # Tracklet has been lost for too long
                if 10 < self.lost_cnt[str(t.id)]:
                    self.lost_cnt[str(t.id)] = -999
                    self.tracklet_removed(self.tracking[str(t.id)], self.get_centroid(t.roi))
            elif t.status == dai.Tracklet.TrackingStatus.REMOVED:
                if 0 <= self.lost_cnt[str(t.id)]:
                    self.lost_cnt[str(t.id)] = -999
                    self.tracklet_removed(self.tracking[str(t.id)], self.get_centroid(t.roi))


device = dai.Device()
print("Creating Stereo Depth pipeline")
pipeline = dai.Pipeline()

camLeft = pipeline.create(dai.node.MonoCamera)
camRight = pipeline.create(dai.node.MonoCamera)

camLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)
camRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_800_P)

### Define stereoDepth node and create outputs
#---
stereo = pipeline.create(dai.node.StereoDepth)

#Set StereoDepth config
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)  # KERNEL_7x7 default, 5x5, 3x3, MEDIAN_OFF
stereo.setRectifyEdgeFillColor(0)  # Black, to better see the cutout
stereo.setLeftRightCheck(True)
#stereo.setExtendedDisparity(extended)
stereo.setSubpixel(True)

#Alpha scaling to use 'full' FOV
stereo.setAlphaScaling(0.2)
config = stereo.initialConfig.get()
config.postProcessing.brightnessFilter.minBrightness = 0
stereo.initialConfig.set(config)

xoutDepth = pipeline.create(dai.node.XLinkOut)
xoutDepth.setStreamName("depthOut")

#Define connection between nodes
#--
camLeft.out.link(stereo.left)
camRight.out.link(stereo.right)
#stereo.syncedLeft.link(xoutLeft.input)
#stereo.syncedRight.link(xoutRight.input)
stereo.disparity.link(xoutDepth.input)

#tracking on Device
objectTracker = pipeline.createObjectTracker()
objectTracker.inputTrackerFrame.setBlocking(True)
objectTracker.inputDetectionFrame.setBlocking(True)
objectTracker.inputDetections.setBlocking(True)
objectTracker.setDetectionLabelsToTrack([1])  # track only person
# possible tracking types: ZERO_TERM_COLOR_HISTOGRAM, ZERO_TERM_IMAGELESS
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
# take the smallest ID when new object is tracked, possible options: SMALLEST_ID, UNIQUE_ID
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.UNIQUE_ID)

# Linking
xinFrame = pipeline.createXLinkIn()     #------this seems to add ...
xinFrame.setStreamName("frameIn")       #------- ... strange depth error, comment out and it looks fine
xinFrame.out.link(objectTracker.inputDetectionFrame)
'''
# Maybe we need to send the old frame here, not sure
xinFrame.out.link(objectTracker.inputTrackerFrame)

xinDet = pipeline.createXLinkIn()
xinDet.setStreamName("detIn")
xinDet.out.link(objectTracker.inputDetections)

trackletsOut = pipeline.createXLinkOut()
trackletsOut.setStreamName("trackletsOut")
objectTracker.out.link(trackletsOut.input)

'''

cvColorMap = cv2.applyColorMap(np.arange(256, dtype=np.uint8), cv2.COLORMAP_JET)
cvColorMap[0] = [0, 0, 0]
print("Creating DepthAI device")

with device:
    device.startPipeline(pipeline)
    q = device.getOutputQueue(name="depthOut", maxSize=4, blocking=False)
    '''
    trackletsQ = device.getOutputQueue(name="trackletsOut", maxSize=4, blocking=False)
    detInQ = device.getInputQueue("detIn")
    frameInQ = device.getInputQueue("frameIn")
    '''
    disparityMultiplier = 255 / stereo.initialConfig.getMaxDisparity()

    text = TextHelper()
    counter = PeopleCounter()
    #bis hier
    while True:

        name = q.getName()
        depthFrame = q.get().getFrame()
        depthFrame = (depthFrame*disparityMultiplier).astype(np.uint8)    #use for depth
        depthRgb = getDisparityFrame(depthFrame, cvColorMap)
        '''
        trackletsIn = trackletsQ.tryGet()
        if trackletsIn is not None:
            counter.new_tracklets(trackletsIn.tracklets)

        # Crop only the corridor:

        cropped = depthFrame[DETECTION_ROI[1]:DETECTION_ROI[3], DETECTION_ROI[0]:DETECTION_ROI[2]]
        cv2.imshow('Crop', cropped)

        ret, thresh = cv2.threshold(cropped, 20, 145, cv2.THRESH_BINARY)
        cv2.imshow('thr', thresh)

        blob = cv2.morphologyEx(thresh, cv2.MORPH_OPEN,
                                cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (22, 22)))  # editet elipse size 37,37
        cv2.imshow('blob', blob)

        edged = cv2.Canny(blob, 20, 80)
        cv2.imshow('Canny', edged)

        contours, hierarchy = cv2.findContours(edged, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

        dets = dai.ImgDetections()
        # len contorus is count of seperate heads/blobs
        if len(contours) != 0:
            c = max(contours, key=cv2.contourArea)
            x, y, w, h = cv2.boundingRect(c)
            # cv2.imshow('Rect', text.rectangle(blob, (x,y), (x+w, y+h)))
            x += DETECTION_ROI[0]
            y += DETECTION_ROI[1]
            area = w * h
            # print(len(contours), area)

            if 760 < area:
                # Send the detection to the device - ObjectTracker node
                det = dai.ImgDetection()
                det.label = 1
                det.confidence = 1.0
                det.xmin = x
                det.ymin = y
                det.xmax = x + w
                det.ymax = y + h
                dets.detections = [det]

                # Draw rectangle on the biggest countour
                text.rectangle(depthRgb, (x, y), (x + w, y + h), size=2.5)

        detInQ.send(dets)
        imgFrame = dai.ImgFrame()
        imgFrame.setData(to_planar(depthRgb))
        imgFrame.setType(dai.RawImgFrame.Type.BGR888p)
        imgFrame.setWidth(depthRgb.shape[0])
        imgFrame.setHeight(depthRgb.shape[1])
        frameInQ.send(imgFrame)
        '''
        text.rectangle(depthRgb, (DETECTION_ROI[0], DETECTION_ROI[1]), (DETECTION_ROI[2], DETECTION_ROI[3]))
        text.putText(depthRgb, str(counter), (20, 40))

        cv2.imshow(name, depthRgb)
        if cv2.waitKey(1) == ord("q"):
            break

jakaskerl

Hi pfgnoobi

Running script as is:

With commented out lines:

Are you sure you are not also changing socket so something. This looks like switched left and right.

Thanks,
Jaka

pfgnoobi

jakaskerl
I did some more testing but i can´t figure out where this error comes from.
I´m running excactly the same code and it gives me an ugly depth, as soon as i comment out the two lines after #Linking it works fine.

If i invert the sockets, like this and leave the rest of the code the same as in older post it correct depth. So issue fixed thanks

#Define connection between nodes #-- camLeft.out.link(stereo.right) camRight.out.link(stereo.left)

jakaskerl

Hi @pfgnoobi
That's strange but I'm glad it's fixed not. I'll keep an eye out if something similar happens. Also make sure to use the latest depthai, many bugs are ironed out compared to older versions.

Thanks,
Jaka

pfgnoobi

Hi again,
i´m trying to get a custom NN working, i´m struggeling implementing the morphology.open.
Does someone know how to fix this or where to look further?

custom nn for cv2.morphologyEx(thresh, cv2.MORPH_OPEN,cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (22, 22))):

#!/usr/bin/env python3

from pathlib import Path
import torch
from torch import nn
import kornia
import onnx
from onnxsim import simplify
import blobconverter
# Name of the model
name = 'threshold'
# Define the model class
class Model(nn.Module):

    def forward(self, image):
        # Define structuring element (kernel) for morphological operation
        kernel_size = (22, 22)
        # Create an empty tensor to represent the elliptical structuring element
        structuring_element = torch.zeros(*kernel_size)
        # Calculate the coordinates of the center of the ellipse
        center_x = kernel_size[1] // 2
        center_y = kernel_size[0] // 2
        # Set the values inside the ellipse to 1
        for i in range(kernel_size[0]):
            for j in range(kernel_size[1]):
                if ((j - center_x) / (kernel_size[1] // 2)) ** 2 + ((i - center_y) / (kernel_size[0] // 2)) ** 2 <= 1:
                    structuring_element[i, j] = 1
        # Perform morphology operation (opening) using Kornia
        blob = kornia.morphology.opening(image, structuring_element)
        return blob

# Define the expected input shape (dummy input)
shape = (1, 3, 300, 300)  # Example input shape
model = Model()
X = torch.ones(shape, dtype=torch.float32)
# Create output directory if it doesn't exist
path = Path("out/")
path.mkdir(parents=True, exist_ok=True)
# Export the model to ONNX format
onnx_path = str(path / (name + '.onnx'))
print(f"Exporting model to {onnx_path}")
torch.onnx.export(
    model,
    X,
    onnx_path,
    opset_version=12,
    do_constant_folding=True,
)
# Simplify the ONNX model
onnx_simplified_path = str(path / (name + '_simplified.onnx'))
print(f"Simplifying model to {onnx_simplified_path}")
onnx_model = onnx.load(onnx_path)
model_simp, check = simplify(onnx_model)
onnx.save(model_simp, onnx_simplified_path)
# Convert ONNX model to blob format
print("Converting ONNX model to blob format...")
blobconverter.from_onnx(
    model=onnx_simplified_path,
    data_type="FP16",
    shaves=6,
    use_cache=False,
    output_dir="../models",
    optimizer_params=[]
)
print("Conversion completed.")

error log:

Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 49, in <module>
    torch.onnx.export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 516, in export
    _export(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1613, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1139, in _model_to_graph
    graph = _optimize_graph(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 677, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py", line 1957, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset12.py", line 331, in unfold
    return opset9.unfold(g, input, dimension, const_size, const_step)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_opset9.py", line 3149, in unfold
    return symbolic_helper._unimplemented(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 612, in _unimplemented
    _onnx_unsupported(f"{op}, {msg}", value)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\symbolic_helper.py", line 623, in _onnx_unsupported
    raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of operator Unfold, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues  [Caused by the value '18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.] 
    (node defined in C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py(4495): pad
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(174): erosion
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\kornia\morphology\morphology.py(259): opening
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(34): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1501): _slow_forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(129): wrapper
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(138): forward
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1520): _call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\modules\module.py(1511): _wrapped_call_impl
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\jit\_trace.py(1296): _get_trace_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(915): _trace_and_get_graph_from_model
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1011): _create_jit_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1135): _model_to_graph
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(1613): _export
C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\onnx\utils.py(516): export
C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py(49): <module>
)

    Inputs:
        #0: image defined in (%image : Float(1, 3, 300, 300, strides=[270000, 90000, 300, 1], requires_grad=0, device=cpu) = prim::Param()
    )  (type 'Tensor')
        #1: 18837 defined in (%18837 : Long(8, strides=[1], device=cpu) = onnx::Cast[to=7](%18836), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
        #2: 18838 defined in (%18838 : Float(device=cpu) = onnx::Constant[value={10000}](), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')
    Outputs:
        #0: 18839 defined in (%18839 : Float(*, *, *, *, strides=[309123, 103041, 321, 1], requires_grad=0, device=cpu) = onnx::Pad[mode="constant"](%image, %18837, %18838), scope: __main__.Model:: # C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\torch\nn\functional.py:4495:0
    )  (type 'Tensor')

Process finished with exit code 1

Does this mean that the onnx export does not support Unfold or something with onnx: Pad. How would i get arround this? Would i need to ask Pytorch ONNX people for help here?

Thanks a lot for any clues 🙂

erik

Hi @pfgnoobi ,
For the 1st post, it seems it's quite evident from the error log:

nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=nms_kernels.shape[-1] // 2)
RuntimeError: expected padding to be a single integer value or a list of 2 values to match the convolution dimensions, but got padding=[]

So it seems you didn't supply padding.

For second issue, it seems like pytorch onnx export issue, so best to open a github issue on their github repo. I would also note that the nested loop when creating a model isn't feasible - you should use pytorch (vector) functions.
Thanks, Erik

pfgnoobi

erik
Hi @erik,
thanks for having a look.
i tried fixing the padding like this:

    padding = nms_kernels.shape[-1] // 2    
    nms_magnitude: Tensor = F.conv2d(magnitude, nms_kernels, padding=(padding, padding))

which got me a bit further, but then i get to following error: something with Atan not supported when converting onnx to blob. Do you have an idea on how to fix this? OpenVino Documentations says it supports atan, sin, cos ...
creating the model with kornia.canny uses angle: Tensor = torch.atan2(gy, gx). i tried to change this to sin, cos, ... but it will throw same error message with Sin,Acos ... instead

Simplifying model to out\threshold_simplified.onnx
Converting ONNX model to blob format...
Downloading ..\models\threshold_simplified_openvino_2022.1_6shave.blob...
{
    "exit_code": 1,
    "message": "Command failed with exit code 1, command: /opt/intel/openvino2022_1/tools/compile_tool/compile_tool -m /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.xml -o /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/threshold_simplified/FP16/threshold_simplified.blob -c /tmp/blobconverter/b7b6f29619d74e289a15dcbaf32ea741/myriad_compile_config.txt -d MYRIAD -ip U8",
    "stderr": "[ GENERAL_ERROR ] \n/home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu20/b/repos/openvino/src/plugins/intel_myriad/graph_transformer/src/frontend/frontend.cpp:592 Failed to compile layer \"/Atan_output_0\": unsupported layer type \"Atan\"\n",
    "stdout": "OpenVINO Runtime version ......... 2022.1.0\nBuild ........... 2022.1.0-7019-cdb9bec7210-releases/2022/1\nNetwork inputs:\n    image : u8 / [...]\nNetwork outputs:\n    magnitude/sink_port_0 : f16 / [...]\n"
}
Traceback (most recent call last):
  File "C:\Users\lh\Documents\DEV\OAK Projects\depthai-experiments\gen2-custom-models\generate_model\kornia_threashold.py", line 74, in <module>
    blobconverter.from_onnx(
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 421, in from_onnx
    return compile_blob(blob_name=Path(model_name).stem, req_data={"name": Path(model_name).stem}, req_files=files, data_type=data_type, **kwargs)
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\blobconverter\__init__.py", line 307, in compile_blob
    response.raise_for_status()
  File "C:\Users\lh\anaconda3\envs\depthai-experiments\lib\site-packages\requests\models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://blobconverter.luxonis.com/compile?version=2022.1&no_cache=True

thanks 🙂

erik

Hi @pfgnoobi .
Atan actually isn't supported by the VPU plugin, see their docs here.

pfgnoobi

erik
Hi @erik,
thanks for pointing me to the correct docs. Seems like i´m at a dead end wih canny.

Let´s say i would have a custom nn with all the image proccesing. How would i send back detections as imgDetections from that custom nn to the Objeckt Tracker Node. Or in more detail what would i need to add to the custom nn? If i just add another return value e.g a tensor with [label, confidence, xmin,ymin,xmax,ymax] how would i convert this to imgDetections before passing it to ObjectTracker Node on device?

Rethinking all of this does it even make sense to try doing this all on device to achive realtime tracking values? Or proceed further with just taking the depth image from the camera and do all the rest on Host?
What are your experienced thoughts on this?

big thanks