This is an issue that I've since resolved, but haven't seen particularly clear explanations of outside of a tangential github report.

At least two of the OAK-D-CM4 PoE models that I've purchased have had a wildly incorrect spec loaded for their stereo (mono) camera positions. The factory calibration of these cameras has been correct, but by default the stereo depth node will load the spec numbers, not the factory calibration numbers. Most of the others have not had this issue.
"specTranslation": {
"x": 7.0,
"y": 0.0,
"z": 0.0},
"toCameraSocket": 0,
"translation": {
"x": 1.2537951469421387,
"y": -0.016799528151750565,
"z": -0.25653964281082153
}

The main symptom of this issue is being unable to align the depth map with the color images, even when using unaltered example code.

Downloading and reading the calibration file from the camera will confirm if this is the exact issue (or if you need to do your own calibration, I guess)

Adding
stereoDepth.setDisparityToDepthUseSpecTranslation(False)
stereoDepth.setDepthAlignmentUseSpecTranslation(False)
to my pipeline setup resolved this issue for me without needing to recalibrate the camera.

Hi @BWixted , we apologize for the inconvenience. Could we get the MXID of the device?

19443010B11F772700
19443010C145772700
Both show the same incorrect specTranslation (x = 7.0) for the 1.3cm offset mono camera.