setWarpMesh after pipeline starts?

Huimin

May I know if there is a way to update the ImageManip node's mesh after the streaming starts? I would like to occasionally change the wrapping effect, based on user inputs. Thanks!

jakaskerl

Hi @Huimin
Something like this?

Thanks,
Jaka

Huimin

This requires a restart of the pipeline. which stops the all camera streams and imu data etc. Probably introduce a significant gap of sensing data. say 3-5 second.

Anyway to avoid that? Namely changing the mesh while the pipeline is running.

jakaskerl

Hi @Huimin
Not possible I'm afraid. The ImageManipConfig exposes some methods for warping which I think should be changeable on runtime, but they are matrices not meshes: https://docs.luxonis.com/projects/api/en/latest/components/messages/image_manip_config/#imagemanipconfig

Thanks,
Jaka

Huimin

Thanks @jakaskerl ,

I am thinking if Luxonis could consider to implement a new API to allow users to update the meshes at runtime?

As disparity map is one of the main products from the stereo cameras, being able to update meshes to make the camera stay calibrated is pretty desirable feature. In many use cases, shutdown the camera and restart is not a good idea.

Since we already have similar API for changing through matrix, hope the change to enable mesh-based update is not too difficult?

If it is no longer possible for RVC2 products. Could this by any chance feasible for the upcoming RVC4? @erik Would be great if you can shine some lights here too!

erik

Hi @Huimin ,
I just added feature request for dynamic recalibration suppor t on RVC2 as well. As it's quite low effort, we'll likely add support for it in the coming weeks (to develop).
Thanks, Erik

Huimin

@erik This is fantastic news! Thanks a ton 🙂

Huimin

Good day @erik , I am wondering if this feature is in active development? Thanks!

erik

Hi @Huimin , it is: luxonis/depthai-core1022

Huimin

Hi @eric-soderberg , thanks for the quick reply! However I think the added API is not the intention of my original post.

The API needs to be exposed is to update the mesh data of the ImageManip node. Not to change the device eeprom calibration.

Even if we updated the calibration, that does not translate to updating of custom mesh to the ImageManip node. Hope I made myself clear.

erik

@Huimin why are you using ImageManip for stereo rectification, when StereoDepth node does that for you?

Huimin

erik There are couple of reasons. The main reason is that we are using custom calibration model. The other is that we are mixing cameras of different resolutions. None of these are supported by the StereoDepth node. We like the flexibility that Imagemanip gives.

Huimin

erik

@Huimin I see. Could you create a feature request on depthai-core for runtime warp update, and specify the 2 reasons you mentioned?

Huimin

erik Ok, will do!

Huimin

luxonis/depthai-core1026

posted here