I am using the below snippet of code to convert Planar FP16 BGR data to cv::mat for display using imshow. I have verified that the type (planar16bgr) and size (416x416) are correct. However the image does not display correctly and appears to be even varying over time - though scene is static. I am lost on what I am doing wrong - any inputs are appreciated.

Example images

BGR values at certain pixels (100,100) and (415,415) in consecutive frames

Code Snippet

auto inRgb = ptQ->get<dai::ImgFrame>();  // Get Image Frame
auto img_type=inRgb->getType();  // Verified it evaluates to 27 - which is Planar FP16 BGR data
auto mat_in = inRgb->getData(); //Using getData(). getCvFrame() failed, complains about not having built with OpenCV


cv::Size size = cv::Size(inRgb->getWidth(), inRgb->getHeight());
cv::Mat mat = cv::Mat(size, CV_8UC3);
std::cout << "Width: " << size.width << std::endl;      // Verified to be 416
std::cout << "Height: " << size.height << std::endl;    // Verified to be 416
std::cout << "Size: " << mat_in.size() << std::endl;    // Verified to be 1038336 = 416*416*3*2

int off1=(415 * 416 + 415)*2+2; //offset to second plane
int off2=(415 * 416 + 415)*4+4; //offset to third plane

for (int x = 0; x < 416; ++x) {
   for (int y = 0; y < 416; ++y) {
            int idx=(x * 416 + y)*2;
            const uint16_t* fp16_b = (const uint16_t*) (mat_in.data() + idx);
            uint8_t b = (uint8_t) (fp16_ieee_to_fp32_value(fp16_b[0]) * 255.0f);
            const uint16_t* fp16_g = (const uint16_t*) (mat_in.data() + idx + off1);
            uint8_t g = (uint8_t) (fp16_ieee_to_fp32_value(fp16_g[0]) * 255.0f);
            const uint16_t* fp16_r = (const uint16_t*) (mat_in.data() + idx + off2);
            uint8_t r = (uint8_t) (fp16_ieee_to_fp32_value(fp16_r[0]) * 255.0f);
            mat.at<cv::Vec3b>(x, y) = cv::Vec3b(b,g,r);
        }
    }

cv::imshow("image", mat); 
cv::waitKey(10);

    Hi VijayVenkataraman
    Could you try these suggestions. I'm not too keen on c++ 🙂

    You're trying to convert the planar FP16 BGR data to a displayable format. Let's address potential issues step by step:

    1. Verify FP16 to FP32 Conversion: Ensure that the function fp16_ieee_to_fp32_value is correctly converting FP16 values to FP32.

    2. Planar Memory Layout:
      Planar data layout means that the full plane of one channel (e.g., blue) comes first, followed by the next channel (e.g., green), and finally the last channel (e.g., red).

          const size_t planeSize = 416 * 416 * 2; // Width * Height * 2 bytes for FP16
    const uint8_t* bluePlane = mat_in.data();
    const uint8_t* greenPlane = bluePlane + planeSize;
    const uint8_t* redPlane = greenPlane + planeSize;

    3. Pixel Iteration and Access:
      Ensure you're iterating in a row-major order.

         for (int y = 0; y < 416; ++y) {
       for (int x = 0; x < 416; ++x) {
           ...
           mat.at<cv::Vec3b>(y, x) = cv::Vec3b(b, g, r);
       }
   }

    4. Planar Data Extraction:
      Index into the planes directly.

         int idx = (y * 416 + x) * 2;
   const uint16_t* fp16_b = (const uint16_t*) (bluePlane + idx);
   const uint16_t* fp16_g = (const uint16_t*) (greenPlane + idx);
   const uint16_t* fp16_r = (const uint16_t*) (redPlane + idx);

    5. Clipping Values:
      Ensure pixel values are clipped within [0, 255].

         uint8_t b = std::clamp((uint8_t)(fp16_ieee_to_fp32_value(fp16_b[0]) * 255.0f), (uint8_t)0, (uint8_t)255);
   uint8_t g = std::clamp((uint8_t)(fp16_ieee_to_fp32_value(fp16_g[0]) * 255.0f), (uint8_t)0, (uint8_t)255);
   uint8_t r = std::clamp((uint8_t)(fp16_ieee_to_fp32_value(fp16_r[0]) * 255.0f), (uint8_t)0, (uint8_t)255);

    With these corrections, try visualizing the image again. If there's still an issue, inspect the values from fp16_ieee_to_fp32_value to ensure they're in the range [0, 1].

    Hope it helps or points you in the right direction.

    Thanks,
    Jaka

      6 days later

      Thanks for the pointers - the last check to see if the values were between 0 and 1 was the key. The values were not between 0 and 1. I removed the multiplication by 255 and the image looks good now.