setWarpTransformFourPoints giving wrong transformation

RicardoFranca

Hello Support Team,

I am currently implementing obj tracking in python. I'm following the RGB Rotate Warp example (https://docs.luxonis.com/projects/api/en/latest/samples/ImageManip/rgb_rotate_warp/#rgb-rotate-warp) but using the center points of 4 aruco-markers to perform 4 points wrap transform to the live camera view. However the results of the transformation are very unstable while using setWarpTransformFourPoints function.

Small variations in these central points are resulting in a wrong transformation. Take as example the bellow results. In the attached images we can see both original and transformed images. In the original one the selected aruco markers are highlighted and their central points are marked with a red dot.

Figure 1: When the central points are: [[21.5, 70.5], [541.5, 46.0], [558.25, 375.25], [39.25, 385.0]], the transformation is as expected.

Figure 2: for a small difference in these points: [[21.75, 70.5], [541.75, 46.0], [559.0, 374.75], [70.5, 384.0]],the transformation is incorrect (or not as expected).

The same problem is also happening in static images.

Other than the input sequence, are there any further condition for these points to proper transform the image? If not, what could be the problem in this application?

Please let me know if you need any further information!

Thank you for the help!

jakaskerl

Hi RicardoFranca
Could you add some minimal reproducible code. From what I have tested locally, this seems to work fine.

Thanks,
Jaka

RicardoFranca

# !/usr/bin/env python3

import depthai as dai
import cv2
import numpy as np
import cv2.aruco as aruco

# Create pipeline
pipeline = dai.Pipeline()

"""Main nodes"""
# Camera
colorCameraNode = pipeline.create(dai.node.ColorCamera)
# Properties
colorCameraNode.setPreviewSize(600, 400)
colorCameraNode.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
colorCameraNode.setInterleaved(False)
colorCameraNode.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
# Img Transf
imgTransfNode = pipeline.create(dai.node.ImageManip)
# Properties
imgTransfNode.setMaxOutputFrameSize(2000 * 1500 * 3)

"""LinkIn/Out nodes"""
# Camera
colorCameraOutNode = pipeline.create(dai.node.XLinkOut)
colorCameraOutNode.setStreamName("colorCameraOutQueue")
# Config In
imgTransfConfigInNode = pipeline.create(dai.node.XLinkIn)
imgTransfConfigInNode.setStreamName("imgTransfConfigQueue")
# Tranf Img out
imgTransfOutNode = pipeline.create(dai.node.XLinkOut)
imgTransfOutNode.setStreamName("imgTransfOutQueue")

"""Connections"""
# Camera Node to Camera Out
colorCameraNode.preview.link(colorCameraOutNode.input)
# Camera Node to Img Transf Node
colorCameraNode.preview.link(imgTransfNode.inputImage)
# Config to Img Transf Node
imgTransfConfigInNode.out.link(imgTransfNode.inputConfig)
# Img Transf Node to Img Out
imgTransfNode.out.link(imgTransfOutNode.input)

# Connect to device
device = dai.Device(pipeline)

# Create queues
colorCameraOutQueue = device.getOutputQueue(
    name="colorCameraOutQueue",
    maxSize=4
)
imgTransfConfigQueue = device.getInputQueue(
    name="imgTransfConfigQueue",
    maxSize=4
)
imgTransfOutQueue = device.getOutputQueue(
    name="imgTransfOutQueue",
    maxSize=4
)

aruco_dict = aruco.Dictionary_get(aruco.DICT_4X4_1000)
parameters = aruco.DetectorParameters_create()

# Variables initialization
imageCornersPoints = [
    [0, 0],
    [1, 0],
    [1, 1],
    [0, 1],
]
normalized = True
arucoCenterPoints = None
key = -1

while key != ord('q'):
  # Send an updated config key press
  cfg = dai.ImageManipConfig()
  points = imageCornersPoints if arucoCenterPoints is None else arucoCenterPoints
  point2fList = []
  for p in points:
    pt = dai.Point2f()
    pt.x, pt.y = p[0], p[1]
    point2fList.append(pt)
  cfg.setWarpTransformFourPoints(point2fList, normalized)
  imgTransfConfigQueue.send(cfg)

  for q in [colorCameraOutQueue, imgTransfOutQueue]:
    pkt = q.get()
    name = q.getName()
    frame = pkt.getCvFrame()
    if q == colorCameraOutQueue:
            """
               Aruco Markers detection process
               output: arucoCenterPoints = [P0,P1,P2,P3], where P0...3 are the central points of the arucomarkers (pixels).
               such as: 

                 P0  ->  P1
                   ^       v
                 P3  <-  P2

                since values not normalized > normalized = False
                """
    cv2.imshow(name, frame)

  key = cv2.waitKey(1)

jakaskerl

Hi RicardoFranca
You're right. Seems like sudden changes in center points positions confuse the manip. If both the camera and the points are stationary, the problem disappears.

Ill dig around some more to see if I can find the underlying problem.

Thanks,
Jaka

RicardoFranca

Thank you for the feedback @jakaskerl.
Could you find the underlying problem behind it?

Sincerely,
Ricardo

jakaskerl

Hi RicardoFranca
Sorry but I haven't had any time yet. Ill return if I find something of value. You are welcome to do the same.

Thanks,
Jaka

jakaskerl

Hi RicardoFranca
I tried to achieve the same thing but by calculating the fourth corner of the image. This way, you would always get a rectangle in the Manip warp function. Seems as though sometimes, when the shape is not simple (square/ rect), the warp messes up and distorts the whole image. I could not find out what the cause of this is, most likely something firmware related.

What i have added:

if q == colorCameraOutQueue:
        """
        Aruco Markers detection process
        output: arucoCenterPoints = [P0,P1,P2,P3], where P0...3 are the central points of the arucomarkers (pixels).
        """

        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        corners, ids, rejectedImgPoints = aruco.detectMarkers(gray, aruco_dict, parameters=parameters)

        # Create a dictionary for markers ids and their centers
        marker_centers = {}

        # Calculate the center points for the markers
        for i in range(len(corners)):
            c = corners[i][0]
            center = np.mean(c, axis=0)
            marker_centers[ids[i][0]] = center.tolist()

        # Check if the required ids are present
        if set([1, 2, 4]).issubset(set(marker_centers.keys())):
            # Calculate the fourth center assuming it's part of the same rectangle
            P1 = np.array(marker_centers[1])
            P2 = np.array(marker_centers[2])
            P3 = np.array(marker_centers[4])
            P4 = P1 + P3 - P2

            arucoCenterPoints = [marker_centers[1], marker_centers[2], marker_centers[4], P4.tolist()]

            # Retrieve frame dimensions
            height, width = frame.shape[:2]

            # Normalize the points
            arucoCenterPoints = [[point[0] / width, point[1] / height] for point in arucoCenterPoints]

the script seems to work better this way, but will cause problems (will not look as nice) if aruco marker plane is too different from the camera plane; best to keep them parallel.

Thoughts?
Jaka

RicardoFranca

Thank you @jakaskerl.
This would be exactly the next approach we would like to try and then debate over the results.

I'll keep you update if I have any further promising discover.

I appreciate your help.
Ricardo

jakaskerl

Hi RicardoFranca
I created an issue at depthai-core repo so you can follow that as well.

Thanks,
Jaka