Capture Image using RPi GPIO, then feed it to a NN in the OAK-1.

erik
Erik, thanks a lot for your informative reply.
Will keep them in consideration when working on the project

For (1), do you mean to capture the frame and send it to the host first, and then accordingly send it back to the device to either NN? If so, I have tried sending a frame back to the device, and then again back to the host to display the inference result, however, the output couldn't be displayed. Instead of having an image of (300x300x3), I had an output size of (270000, ). Any thoughts on how to solve this issue?
The pipeline looks like this: I implemented it to solve the issue that ImageManip cannot resize STILL images; so I created this pipeline to capture a still image, resize on the host, and send it back to device for NN inference
xinCaptureCommand --> camRGB -> outStillRGB
then, do the resizing on host, and we have
inResizedImage -> outResult

x = cv2.imread("blue.png")
pipeline = dai.Pipeline()

# Create input control node to acquire capture command
xinCaptureCommand = pipeline.create(dai.node.XLinkIn)
xinCaptureCommand.setStreamName("capture")

# Create Camera node and give its properties
camRGB = pipeline.create(dai.node.ColorCamera)
camRGB.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRGB.setStillSize(1080, 1080)
camRGB.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)

# Create output node for still images
outStillRGB = pipeline.create(dai.node.XLinkOut)
outStillRGB.setStreamName("rgbStill")

# Create input node to receive resized imsge from host
inResizedImage = pipeline.create(dai.node.XLinkIn)
inResizedImage.setStreamName("resizedImage")

# Create outResult output node for still images
outResult = pipeline.create(dai.node.XLinkOut)
outResult.setStreamName("outResult")

# Linking
xinCaptureCommand.out.link(camRGB.inputControl)
camRGB.still.link(outStillRGB.input)
inResizedImage.out.link(outResult.input)

# Connect to device and start the pipeline
with dai.Device(pipeline) as device:

    # Create queues
    stillQueue = device.getOutputQueue(name="rgbStill")
    captureInputQueue = device.getInputQueue("capture")
    sendResizedQueue = device.getInputQueue("resizedImage")
    outResultQ = device.getOutputQueue("outResult")

    cv2.imshow("x",x)

    while True:
        stillFrame = stillQueue.tryGet()
        if stillFrame is not None:
            print("still frame:", stillFrame.getHeight(), stillFrame.getWidth(), stillFrame.getType())        
            frame = stillFrame.getCvFrame()
            cv2.imshow("frame", frame)
           
            resized = imutils.resize(frame, width = 300)
            print("resized frame:" , resized.shape)
            sendResized = dai.ImgFrame()
            sendResized.setData(resized)
            #sendResized.setData(to_planar(resized, (300, 300)))
            sendResized.setHeight(300)
            sendResized.setWidth(300)
            #sendResized.setType("RGB888i")
            sendResizedQueue.send(sendResized)
           
           
        testFrame = outResultQ.tryGet()
        if testFrame is not None:
            print("frame W,H & type after receiving back again:", testFrame.getHeight(), testFrame.getWidth(), testFrame.getType())
            result = testFrame.getCvFrame()
            cv2.imshow("result", result)
            print("final result shape:", result.shape)

        # Send capture command from host to device
        key = cv2.waitKey(1)
        if key == ord("q"):
            break
           
        elif key == ord('c'):
            ctrl = dai.CameraControl()
            ctrl.setCaptureStill(True)
            captureInputQueue.send(ctrl)
            print("captured")

For (2), what to you mean by "other outputs"?

Best,
Hussain

erik
Hey Erik,
Thank you for your support !
I was able to do it
I implemented a code that captures an image based on a user keypress, send it to host and resize it, send it back to the device to do inference, and finally display the results.

I am attaching here a simple version code for reference in case anyone might need it.
Note that the code attached does NOT contain any NN node. It simply captures an image, resized it on host, sends it back to device (here you can do whatever you want with the resized image), and finally send results back to host

#!/usr/bin/env python3

import cv2
import depthai as dai
import imutils
import time
import numpy as np

xxx = cv2.imread("blue.png")

# Create the pipeline
pipeline = dai.Pipeline()


# HOST --> xinCaptureCommand --> camRGB --> outstillRGB --> HOST
# Resize image in host
# HOST --> inResizedImage --> outResizedImage --> HOST

# Create input control node to acquire capture command
xinCaptureCommand = pipeline.create(dai.node.XLinkIn)
xinCaptureCommand.setStreamName("capture")


# Create Camera node and give its properties
camRGB = pipeline.create(dai.node.ColorCamera)
camRGB.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRGB.setStillSize(1080, 1080)
camRGB.setPreviewSize(1080, 1080)
camRGB.setVideoSize(1080, 1080)
# camRGB.setInterleaved(False)
camRGB.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)

# Create output node for still images
outStillRGB = pipeline.create(dai.node.XLinkOut)
outStillRGB.setStreamName("rgbStill")


# Create input node to receive resized image from host
inResizedImage = pipeline.create(dai.node.XLinkIn)
inResizedImage.setStreamName("inResizedImage")

# Create output node for still images
outResizedImage = pipeline.create(dai.node.XLinkOut)
outResizedImage.setStreamName("outResizedImage")

# Link output of xinCaptureCommand to camera input control
xinCaptureCommand.out.link(camRGB.inputControl)

# Link output of camera to input of xlinkout to send to deivce
camRGB.still.link(outStillRGB.input)

# Link output of inResizedImage to input of outResizedImage
inResizedImage.out.link(outResizedImage.input)



# Connect to device and start the pipeline
with dai.Device(pipeline) as device:


    # Create input queue to device, that receives capture command
    captureInputQueue = device.getInputQueue("capture")
    
    
    # Create output queue that will get RGB frame (Output from device, and input to host)
    stillQueue = device.getOutputQueue(name="rgbStill")


    # Create input queue to device, that receives resized image 
    sendResizedQueue = device.getInputQueue("inResizedImage")
    
    # Create output queue to that will send output images to host
    outResizedImageQueue = device.getOutputQueue("outResizedImage")

    cv2.imshow("xxx",xxx)
    
    
    def to_planar(arr: np.ndarray, shape: tuple) -> np.ndarray:
        return cv2.resize(arr, shape).transpose(2,0,1).flatten()
        
        
    while True:
          
        stillFrame = stillQueue.tryGet()
        if stillFrame is not None:
            print("Captured!")
            print(stillFrame.getHeight(), stillFrame.getWidth(), stillFrame.getType())        
            frame = stillFrame.getCvFrame()
            cv2.imshow("frame", frame)
            
            resized = imutils.resize(frame, width = 300)
            print(resized.shape)
            
            # Create message and send resized frame
            sendResized = dai.ImgFrame()
            sendResized.setData(to_planar(resized, (300, 300)))            
            sendResized.setHeight(300)
            sendResized.setWidth(300)
            sendResized.setType(dai.ImgFrame.Type.BGR888p)
            sendResizedQueue.send(sendResized)
            
            


        finalFrame = outResizedImageQueue.tryGet()
        if finalFrame is not None:
            print(finalFrame.getHeight(), finalFrame.getWidth(), finalFrame.getType())
            f= finalFrame.getCvFrame()
            cv2.imshow("finalFrame", f)
            print(f.shape)
        

        # Send capture command from host to device
        key = cv2.waitKey(1)
        if key == ord("q"):
            break
            
        elif key == ord('c'):
            ctrl = dai.CameraControl()
            ctrl.setCaptureStill(True)
            captureInputQueue.send(ctrl)

erik
Currently, I have an issue with focusing. I am using ColorCamera STILL output.
Each time the user presses a button, a capture command is issued.
The problem is that, with the first 3-4 button presses, the image is out of focus.
The 5th or 6th press yields an image that is in focus.
It seems that the autofocus needs several frames to adjust itself.

1) Any thoughts how to solve this issue? Is there any property in the ColorCamera node that resolves the issue?

2) One solution came to my mind is to capture several frames (say 6 frames), and do the inference and processing on the 6th frame only (i.e. the first 5 frames will be used for focusing, and the 6th one for inference and processing). Do you think that this makes sense and would work on the DepthAI environment?

Best,
Hussain

erik
Sounds good and would make my life easier.
Will give it a try.