Question regarding ColorCamera configuration/control

@erik Thanks for the pointer to the demo. I injected the approach into my use of the MobileNetSpatialDetectionNetwork node and it worked nicely, at least as far as I could tell. The total loading on the LeonOS and LeonRT was about the same as my original approach, but with reduced load on OS and more on RT. Not sure if that matters.

There is one thing that bothers me a bit. The demo approach in effect uses 'letter-boxing" which reduces size of any feature/object that might be detected. I don't know enough to know if that makes a difference for accuracy of the NN detection. Does it?

erik I recreated the demo you mentioned in my environment, with the exception of using a different blob. There was actually a difference between what is shown in the readme and what I see. The passthrough is in grayscale, not color as shown in the readme. Am I doing something wrong?

erik I realized I was not clear in my previous post. I copied the code from the demo you mentioned and changed exactly two lines of code to use a different blob (tho apparently, based on the labels, it is the same model already converted) for the NN. There are no mono camera nodes created. So in theory I should see a color image for the passthrough. But I do not.

I added some code to show the ImageManip.out and it too displays in grayscale. I checked the type and it was RGB888p.

I am confused and apparently have reached the limit of my knowledge about DepthAI and OpenCV. I am not sure what else to check.

erik Happy to supply the code. I derived my version from your GitHub repository here. Below find my code (cannot find a way to attach a file). I think I'm running DepthAI version 2.13.3 with some patches (I've forgotten how to get the version programmatically).

Here are the changes between the original in GitHub and my code (I inserted additional comments to show my changes):

• since I don't use the blob converter commented the import
• since I need to establish a different path for the blob I used, I import from pathlib
• for the NN blob path, I set the blob path to that created for the existing blob

#!/usr/bin/env python3

import cv2
import depthai as dai
# import blobconverter  # <-- REMOVED
import numpy as np
from pathlib import Path  # <-- ADDED

# MobilenetSSD label texts

labelMap = ["background", "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow",
            "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"]
nnBlobPath = str((Path(__file__).parent / Path('examples/models/mobilenet-ssd_openvino_2021.4_5shave.blob')).resolve().absolute())  # <-- added

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setInterleaved(False)
camRgb.setIspScale(1,5) # 4032x3040 -> 812x608

xoutIsp = pipeline.create(dai.node.XLinkOut)
xoutIsp.setStreamName("isp")
camRgb.isp.link(xoutIsp.input)

# Use ImageManip to resize to 300x300 and convert YUV420 -> RGB
manip = pipeline.create(dai.node.ImageManip)
manip.setMaxOutputFrameSize(270000) # 300x300x3
manip.initialConfig.setResizeThumbnail(300, 300)
manip.initialConfig.setFrameType(dai.RawImgFrame.Type.RGB888p) # needed for NN
camRgb.isp.link(manip.inputImage)

# NN to demonstrate how to run inference on full FOV frames
nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
nn.setConfidenceThreshold(0.5)
# nn.setBlobPath(str(blobconverter.from_zoo(name="mobilenet-ssd", shaves=6)))  # <-- REMOVED
nn.setBlobPath(nnBlobPath)  # <-- ADDED
manip.out.link(nn.input)

xoutNn = pipeline.create(dai.node.XLinkOut)
xoutNn.setStreamName("nn")
nn.out.link(xoutNn.input)

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
nn.passthrough.link(xoutRgb.input)

with dai.Device(pipeline) as device:
    qRgb = device.getOutputQueue(name='rgb')
    qNn = device.getOutputQueue(name='nn')
    qIsp = device.getOutputQueue(name='isp')

    def frameNorm(frame, bbox):
        normVals = np.full(len(bbox), frame.shape[0])
        normVals[::2] = frame.shape[1]
        return (np.clip(np.array(bbox), 0, 1) * normVals).astype(int)

    def displayFrame(name, frame, detections):
        color = (255, 0, 0)
        for detection in detections:
            bbox = frameNorm(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))
            cv2.putText(frame, labelMap[detection.label], (bbox[0] + 10, bbox[1] + 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, color)
            cv2.putText(frame, f"{int(detection.confidence * 100)}%", (bbox[0] + 10, bbox[1] + 40), cv2.FONT_HERSHEY_TRIPLEX, 0.5, color)
            cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), color, 2)
        cv2.imshow(name, frame)

    while True:
        if qNn.has():
            dets = qNn.get().detections
            frame = qRgb.get()
            f = frame.getCvFrame()
            displayFrame("rgb", f, dets)
        if qIsp.has():
            frame = qIsp.get()
            f = frame.getCvFrame()
            cv2.putText(f, str(f.shape), (20, 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255,255,255))
            cv2.imshow("isp", f)

        if cv2.waitKey(1) == ord('q'):
            break

erik Thanks for checking. I upgraded my PyCharm environment to 2.14.0. I still get the grayscale result. I decided to remove the NN, and thus the blob dependency, from the script in my previous post. The resulting code, included below, simply shows the ISP and the output of the ImageManip node, so it should be easy to run. In my environment, it still produces a grayscale image. I remain puzzled. I hope it is not too much of an imposition to ask that you run it. I suspect you will see a color image, but at least that will confirm I've got something wrong in my environment, tho I have no idea what.

#!/usr/bin/env python3

import cv2
import depthai as dai

# Create pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setInterleaved(False)
camRgb.setIspScale(1,5) # 4032x3040 -> 812x608

xoutIsp = pipeline.create(dai.node.XLinkOut)
xoutIsp.setStreamName("isp")
camRgb.isp.link(xoutIsp.input)

# Use ImageManip to resize to 300x300 and convert YUV420 -> RGB
manip = pipeline.create(dai.node.ImageManip)
manip.setMaxOutputFrameSize(270000) # 300x300x3
manip.initialConfig.setResizeThumbnail(300, 300)
manip.initialConfig.setFrameType(dai.RawImgFrame.Type.RGB888p) # needed for NN
camRgb.isp.link(manip.inputImage)

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutRgb.setStreamName("rgb")
manip.out.link(xoutRgb.input)

with dai.Device(pipeline) as device:
    qRgb = device.getOutputQueue(name='rgb')
    qIsp = device.getOutputQueue(name='isp')

    while True:
        if qRgb.has():
            frame = qRgb.get()
            f = frame.getCvFrame()
            cv2.imshow("rgb", f)

        if qIsp.has():
            frame = qIsp.get()
            f = frame.getCvFrame()
            cv2.imshow("isp", f)

        if cv2.waitKey(1) == ord('q'):
            break

Thanks.