Custom Nodes and On-Device-Computing

ADP

Hi, I am new to Luxonis on device computing and would like to know if my idea is possible on an OAK-D Pro PoE.

The final functionality I want to archive is on-device Feature Point Tracking followed by Spatial Location Calculation for each of the features and finally a calculation of how much the feature positions differ from the last frame e.g. how far they have moved.

So while this isn't the most complicated on paper, there still are multiple calculations necessary and the feature position needs to be saved and linked together on-device. Is this possible, or what parts need to be simplified or done on the host side? Is there already some kind of node that calculates the difference/shift distance between features of consecutive frames?

jakaskerl

Hi ADP
Should be possible I think.

Use Feature Tracker Node to acquire the features and track them.
Pipe the output to a script node, extract the X,Y of each feature and create a small ROI around it.
Add the ROI to the spatial calculator config like done here
Pipe the spatials from the calculator via Xlink to the host and perform frame-to-frame difference calculation on those spatials.

Thoughts?

Thanks,
Jaka

ADP

Thanks, that would be the pipeline I had in mind, but one of my main concerns is/was that I would like the difference calculation also on-device. So as far as I understand it, I would have to save the feature ids and 3D positiosn of the last frame, link the features by id to the one of the current frame and then calculate the difference and send that to the host.
Would this also be possible, or will i get into memory, CPU or logical problems?

ADP

In my usecase, the whole picture will shift by a distance I want measure using feature tracking. So features will only be visible on a few consecutive frames, before there are new features.

ADP

As far as I understand the system, it should be possible by extending the pipeline like this, but I am pretty inexperienced in this manner.

Use Feature Tracker Node to acquire the features and track them.
Pipe the output to a script node, extract the X,Y of each feature and create a small ROI around it.
Add the ROI to the spatial calculator config like done here
Pipe the output to script node to combine/extract spacial position and feature id
script node with the two of these lists as inputs, the current frame is passed through for the next iteration of this node, the script calculates the delta distance for for all features that can be found in both lists, this distance list gets sent to the host

jakaskerl

Hi ADP
While this is entirely possible, I'm really not sure about the performance. Script node is not particularly fast, but as long as you are not processing entire image frames, this should be fine.

If you run into problems, you can also decrease the amount of features tracked.

Thanks,
Jaka