Hello!

As I happily await for my new OAK-D board, I have many ideas and questions. I am an AI engineer that utilizes computer vision for real-life manufacturing and robotic applications, thus I try to leverage all the products I use in the most effective manner.

Q1: If I want to compile the latests openCV-main repo, and have it run on the OAK-D... How would I do so? I ask this because I currently do all my development on the Jetson Xavier devices, and I recompile the repo to utilize the Jetsons hardware and to provide me more opencv functionality compared to a pip-installed version.

Q2: Since the patent for SIFT has expired this year, openCV has transitioned the algo to the main branch. I think SIFT would be a fantastic application for the depthAI platform especially for the depth aspect.
Would you recommend running SIFT on the OAK-D or using the pointcloud information from OAK-D cameras and then do all processing with the Xavier? I am curious about the performance since this would be another fantastic application for pick-and-place robotics.

To run some code for benchmarkings, please see the following code example:

import numpy as np
import cv2
import time
from matplotlib import pyplot as plt

MIN_MATCH_COUNT = 10
FLANN_INDEX_KDTREE = 0

img1 = cv2.imread('box.png',0)          # queryImage
img2 = cv2.imread('box_in_scene.png',0) # trainImage

sift = cv2.SIFT_create()

while(1):
    # find the keypoints and descriptors with SIFT
    kp1, des1 = sift.detectAndCompute(img1,None)
    kp2, des2 = sift.detectAndCompute(img2,None)
    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
    search_params = dict(checks = 50)

    flann = cv2.FlannBasedMatcher(index_params, search_params)
    matches = flann.knnMatch(des1,des2,k=2)

    # store all the good matches as per Lowe's ratio test.
    good = []
    for m,n in matches:
        if m.distance < 0.7*n.distance:
            good.append(m)

    if len(good)>MIN_MATCH_COUNT:
        src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
        dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)

        M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
        matchesMask = mask.ravel().tolist()

        h,w = img1.shape
        pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
        dst = cv2.perspectiveTransform(pts,M)

        img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)

    else:
        print("Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT))
        matchesMask = None

    draw_params = dict(matchColor = (0,255,0), # draw matches in green color
                       singlePointColor = None,
                       matchesMask = matchesMask, # draw only inliers
                       flags = 2)

    img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)

    #cv2.imwrite('sift_final.jpg',img3)
    plt.imshow(img3, 'gray'),plt.show()

With that said, I have a huge array of different use cases for this in real-word scenerios. If you and the team would like me to test other applications of DepthAI or develop potentially new ones, please feel free to reach out. 🙂

Hi @mboof ,

Great questions - thanks!

  1. So the OpenCV AI Kit actually does not run OpenCV but acts as a performant camera that works cleanly in conjunction with OpenCV. The purpose is to act as a Spatial AI camera that performs hardware-accelerated functions before the data even shows up on a host computer, performed directly on image sensor data from the onboard camera modules. And in some cases, to allow no computer to be used at all. So for the list of hardware acceleration (and what is already done vs. what will be delivered with the KickStarter), see HERE. And for progress tracking of implementing these, see HERE. All this said, there is some work being done with G-API and OpenCL (more on that later) to allow OpenCV functions to be accelerated directly on OAK. But G-API for example will come later than the capabilities listed in those links.
  2. Great question. I actually don't immediately know if we have any SIFT support already ported directly on the platform. I'm going to check. That said, OpenCL can be used to compile code to be run on the SHAVES on the Myriad X. And then theoretically these should be runnable in our Gen2 system already as nodes in the pipeline.

    I'll circle back as well after doing some more searching SIFT.

Thanks,
Brandon

With that said, I have a huge array of different use cases for this in real-word scenerios. If you and the team would like me to test other applications of DepthAI or develop potentially new ones, please feel free to reach out. 🙂

And will do, thanks!

2 years later

Any updates on OpenCL or another mainstream accelerator API support, e.g. SYCL?

  • erik replied to this.