Difficulty trying to find the orientation of detected objects using YoloV5

FearlessMailbox

Heyo, i'm not sure if this forum the right place to ask questions like this, if not, let me know and i'll delete this post. But i'm working on something and running into some difficulties. I'm working on a sorting robot that needs to sort Dutch fryable snacks. I've trained a YoloV5 model, which works very well for detecting which snacks are where, but in order for me to use a robot arm to actually sort the snack, i will need a location, and how many degrees the snack is rotated.

Because a bounding box using YoloV5 is always recatangular, i needed a different way of detecting the orientation of the snacks. For that i used basic OpenCV functions to filter out the backgound, and create contours of the remaining snacks. The problem with this, is if snacks overlap, or are directly against eachother, it will detect 2 snacks as 1 big snack.

I'm trying to find better solutions to find the orientation of the snacks, i was wondering if any of you people know any tips to find a solution to this.

PS: If this is too vague, i've also created a PDF with pictures and more context.

jakaskerl

Hi FearlessMailbox
A quick thought, if yolo is able to detect the snacks as separate even when overlapping, maybe try rotating the image in steps from [0, 180] and check where each snack bbox has the smallest area --> this should be where the snack and bbox are parallel.

Thoughts?
Jaka

vital

FearlessMailbox Have you thought about doing two stages? 1. Find the snack and pass that detection to second NN trained on snack labels rotated in different positions 2. Find the rotation angle of the snack label

erik

Hi vital ,
I think the most optimal way would be to add another yolo head for the orientation/rotation value, based the detection head itself. Our ML lead mentioned that they added it to the roadmap, and will be looking into this model architecture/tutorial in the upcoming weeks.
For the 2stage pipeline - it's something similar that we have for head pose estimation: https://github.com/luxonis/depthai-experiments/tree/master/gen2-head-posture-detection#head-posture-detection

FearlessMailbox

Thx people, will definetely look into it.
@vital Yeah i've tried doing it in two stages, but i'll go experiment around some more. Thx for the reply tho

tinus

Haha, I'm working on the same project as you were last year. Did you find out a way to get the correct orientation?

edit: i used some opencv code to detect the orientation
def getOrientation(pts):

## [pca]

sz = len(pts)

data_pts = np.empty((sz, 2), dtype=np.float64)

for i in range(data_pts.shape[0]):

data_pts[i,0] = pts[i,0,0]

data_pts[i,1] = pts[i,0,1]

# Perform PCA analysis

mean = np.empty((0))

mean, eigenvectors = cv.PCACompute(data_pts, mean)

# Store the center of the object

cntr = (int(mean[0,0]), int(mean[0,1]))

angle = atan2(eigenvectors[0,1], eigenvectors[0,0]) # orientation in radians

angle = -int(np.rad2deg(angle)) - 90

return angle

def findAngle(img):

# Convert image to binary

_, bw = cv.threshold(img, 50, 255, cv.THRESH_BINARY | cv.THRESH_OTSU)

# Find all the contours in the thresholded image

_, contours, _ = cv.findContours(bw, cv.RETR_LIST, cv.CHAIN_APPROX_NONE)

for i, c in enumerate(contours):

# Calculate the area of each contour

area = cv.contourArea(c)

# Ignore contours that are too small or too large

if area < 3700 or 1000000 < area:

continue

# Find the orientation of each shape

return getOrientation(c, img)