This is an issue that I've since resolved, but haven't seen particularly clear explanations of outside of a tangential github report.
At least two of the OAK-D-CM4 PoE models that I've purchased have had a wildly incorrect spec loaded for their stereo (mono) camera positions. The factory calibration of these cameras has been correct, but by default the stereo depth node will load the spec numbers, not the factory calibration numbers. Most of the others have not had this issue.
"specTranslation": {
"x": 7.0,
"y": 0.0,
"z": 0.0},
"toCameraSocket": 0,
"translation": {
"x": 1.2537951469421387,
"y": -0.016799528151750565,
"z": -0.25653964281082153
}
The main symptom of this issue is being unable to align the depth map with the color images, even when using unaltered example code.
Downloading and reading the calibration file from the camera will confirm if this is the exact issue (or if you need to do your own calibration, I guess)
Adding
stereoDepth.setDisparityToDepthUseSpecTranslation(False)
stereoDepth.setDepthAlignmentUseSpecTranslation(False)
to my pipeline setup resolved this issue for me without needing to recalibrate the camera.
