ROS - How to setup the tf to transform camera data into the robot's world frame?

Goulart

I'm trying to transform the data collected from the camera frame of reference "oak' to my robot's base_link. But, I cannot find any documentation explaining where the frames generated by depthai_description are physically located in the camera. My understanding is that I have to setup "oak_parent_frame" (default name); but, I don't know where the frame "oak" is located. Based on what is shown in Rviz2, this frame seems to be on top of the colour sensor. Is that correct? Can I assume an offset of zero between the "oak" frame and the front glass/face/plane of the camera?

Given the following simplified example to publish a pointcloud message:

// config pointcloud…
pc->initialConfig->setLengthUnit(dai::LengthUnit::METER);
pc->setTargetCoordinateSystem(dai::CameraBoardSocket::CAM_A);
.
.
// create pointcloud example message
sensor_msgs::msg::PointCloud2 cloud;
cloud.header.frame_id = "oak_rgb_camera_optical_frame";

Is the optical frame the correct choice?

Also, I have an OAK 4 D PRO which doesn't seem to be supported by the depthai_description. Can I use the non-pro config for the transforms (ie. "i_tf_device_model: OAK4-D")?

Oskar-Luxonis

Hey Goulart - oak_parent_frame is the parent/mounting frame you provide from your robot TF tree. The oak frame is the camera base_frame used by depthai_ros.

For OAK-D style devices, oak is tied to the RGB / CAM_A camera frame, not to the front glass or outer face of the enclosure. So I would not assume zero offset between oak and the front glass plane.

For your point cloud example:

pc->initialConfig->setLengthUnit(dai::LengthUnit::METER);
pc->initialConfig->setTargetCoordinateSystem(dai::CameraBoardSocket::CAM_A);

cloud.header.frame_id = "oak_rgb_camera_optical_frame";

Yes, oak_rgb_camera_optical_frame is the correct ROS frame if the point cloud is expressed in CAM_A / RGB camera coordinates.

For OAK4-D Pro: current depthai_ros maps OAK4-D-PRO to the OAK4-D description model internally, so using i_tf_device_model: OAK4-D is the expected choice.

Best,
Oskar

Goulart

Thanks for the help @Oskar-Luxonis

I think I'm missing something here because I don't know how to measure the offsets for oak_parent_frame. If oak is tied to the colour sensor, how do I figure out the offsets between the oak frame and any physical part of the camera?
For example, let's say I have an OAK4-D Pro mounted to a known point of my robot. And, I want the oak_parent_frame to be located at the centre of bottom plane of the OAK camera. What are the x,y,z offsets? Since I'm not sure if my description makes any sense, I generated a simple diagram in an attempt to illustrate the scenario.

Regards,
Goulart

Goulart

Just want to add: I can easily estimate these offsets with a caliper or tape measure. I'm just wondering if there is a proper/better way.

Oskar-Luxonis

Goulart - your diagram makes sense. oak_parent_frame can be any mounting/reference frame you choose; cam_pos_x/y/z define the translation from that frame to oak.

depthai_ros does not currently provide a built-in frame for “center of bottom plane”. The known geometry is for named points such as CAM_A, FRONT_CAM_A, VESA_A, VESA_B, etc. So if your mounting point is not one of those named points, measuring the offset from that point to the RGB / CAM_A optical center is the correct approach.

So yes, using a caliper/CAD/mechanical drawing is appropriate here. Then set cam_pos_x/y/z to that measured vector, expressed in the oak_parent_frame axes.

Best,
Oskar

Goulart

Thanks Oskar-Luxonis.

Got it.

I didn't realise you could use the VESA mounting holes for the point cloud. My bad.. I found some documentation about it in the PointCloud page. It would be good to add an explanation about the difference between VESA_A, VESA_B, etc.. I had to manually figure out based on the values I found in the depthai-boards repo.

I decided to not use depthai_descriptions. Since all I need is to link the camera data to my robot's world frame, I ended up using: pc->setTargetCoordinateSystem(dai::HousingCoordinateSystem::VESA_C);. Then, I added a link+joint for the VESA_C to my existing urdf.

Notes:

good to know that a point tied to the front glass of the camera ( FRONT_CAM_A ) exists. Quite convenient;
I would argue that there should be a frame (e.g. oak) tied to the housing of the camera in the depthai_descriptions. Or, maybe add a comment/documentation suggesting where the user can find examples of values for oak_parent_frame offsets (e.g. depthai-boards repo), which would be the VESA mounting points for the chosen camera model. I had no idea depthai-boards existed. I only found it because the repository has been updated recently.

Oskar-Luxonis

Goulart - Thanks for the detailed feedback. Your approach using HousingCoordinateSystem::VESA_C and connecting that frame directly to your robot URDF is valid.

One important detail: since the point cloud is now expressed in VESA_C, make sure PointCloud2.header.frame_id is the corresponding VESA_C frame from your URDF, not oak_rgb_camera_optical_frame.

Agreed that the housing-frame definitions and the link to depthai-boards should be easier to discover in the documentation. Will report.

Best,
Oskar