UVC mode

glitchyordis

Hi, I've tried to activate UVC mode with depthai v2.16 and v2.13 but got the following error:
AttributeError: 'depthai.Pipeline' object has no attribute 'createUVC'

Also, can we access the full RGB resolution, control autofocus and access depth perception while in uvc mode?

erik

Hello glitchyordis ,
For UVC, please follow this guide. Regarding other questions; I believe UVC node is currently limited to NV12 1080P frames. So you wouldn't be able to send depth frames or use 4k/12MP resolution. You should be able to control autofocus. Thoughts?
Thanks, Erik

glitchyordis

Hi erik ,

I see. I intend to use OAK-D PRO AF mounted on a mechanism, with a dekstop as my host. My use case is to detect an object, the mechanism with move towards the object and perform an inspection. We'll display the RGB feed using pysimplegui.

As we all know, we usually perform object detection eg using YOLO with small resolutions eg 640*640. So maybe I can initiate the OAK-D with THE_1080_P or something similar. However, I would like to take a 12MP (4032x3040) image once the mechanism has positioned the camera in front of the object.

The OAK's 12MP RGB view presented to the user via the GUI can be downscaled/resized (eg 1000*1000px to fit the GUI window). But this presented view should have all the objects in the original 12MP passed to the inspection algorithm so I can determine whether it is in focus.

But I've observed that the frame is lagging if we just take the ISP output to the host and present it. Cropping/resizing after taking the isp output to host did not help. "Preview" and "video" is side-cropped but do correct me if I've mistaken.

Based on most of the tutorial I've seen , we set the RGB resolution, eg. "cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)" at the start. Can we change this sensor resolution in the code automatically midway? ie

if depth == 10cm: # object was detected, mechanism moves camera towards the object
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12MP) # change resolution to take 12MP image
take a picture and save it for inspection
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P) # switch back to original resolution for object detection

Pls let me know if you have any suggestions.

erik

Hello glitchyordis ,
That's an interesting project! I would actually suggest downscaling the ISP and running object detection on that. A great demo of that can be found here. You could also add a logic in Script node to only take still image and send that to the host whenever it detects an object closer to 10cm (as specified in your comment). Thoughts?
Thanks, Erik

glitchyordis

erik
I've downscaled ISP with "camRgb.setIspScale(1,5)" for the object detection, but when I request a still, the still is not the full 12MP resolution? Any suggestion?

import cv2
import depthai as dai
pipeline = dai.Pipeline()

camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
camRgb.setIspScale(1,5)
camRgb.setInterleaved(False) 
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
camRgb.setFps(25)

stillEncoder = pipeline.create(dai.node.VideoEncoder)
stillMjpegOut = pipeline.create(dai.node.XLinkOut)
controlIn = pipeline.create(dai.node.XLinkIn)
xoutIsp = pipeline.create(dai.node.XLinkOut)
xoutIsp.setStreamName("isp")
controlIn.setStreamName('control')

stillMjpegOut.setStreamName('still')
stillEncoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)
camRgb.isp.link(xoutIsp.input)
camRgb.still.link(stillEncoder.input)
controlIn.out.link(camRgb.inputControl)
stillEncoder.bitstream.link(stillMjpegOut.input)

with dai.Device(pipeline) as device:

    stillQueue = device.getOutputQueue('still')
    controlQueue = device.getInputQueue('control')
    qIsp = device.getOutputQueue(name='isp')
    
    while True:

        stillFrames = stillQueue.tryGetAll()
        for stillFrame in stillFrames:
            # Decode JPEG
            frame = cv2.imdecode(stillFrame.getData(), cv2.IMREAD_UNCHANGED)
            # Display
            cv2.putText(frame, str(frame.shape), (20, 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255,255,255))
            cv2.imshow('still', frame)

        frame = qIsp.get()
        f = frame.getCvFrame()
        cv2.putText(f, str(f.shape), (20, 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255,255,255))
        cv2.imshow("isp", f)

        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        elif key == ord('c'):
            print("c was pressed")
            ctrl = dai.CameraControl()
            ctrl.setCaptureStill(True)
            controlQueue.send(ctrl)

erik

Hi glitchyordis ,
So the isp downscalling will also downscale the still. Still just takes a frame from isp output. You should rather resize the video/preview output and use that for the NN, and trigger still to get full 12MP.
Thanks ,Erik

glitchyordis

Hey erik

It is my understanding all modes apart from ISP and raw is cropped from the left/top side of ISP, instead of cropping from centre (image a rectangle's midpoint in x and y axis).

Is it possible to pass ISP to ImageManip to crop from centre before sending it to host?

erik

Hello glitchyordis ,
I believe the video is actually cropped in the middle, due to different aspect ratio as below. This is if you don't change video size.

If you change preview, it should actually crop in the middle of the video, depending on the aspect ratio (if same as video, the whole video will just be resized). If you set video resize, it will crop strangely currently instead of resizing, I think this is fixed in image_manip_refactor. Thoughts?
Thanks, Erik

glitchyordis

erik Oh, I'll give it a go later. Can we downscale ISP (so instead of cropping) in ImageManip before passing it to the host without using setIspScale in the first place (would like "still" to be in 12MP)?

erik

glitchyordis ImageManip does resizing, but the release version doesn't support YUV420 frames. So you would need to use image_manip_refactor branch (on depthai-python, after git checkout run python examples/install_requirements.py to get the correct lib) in order so resize the isp.
Thanks, Erik

glitchyordis

erik
I have run the install_requirements.py from image_manip_refactor branch. The ISP frame is show but this error pops up right away. Perhaps my installation is incorrect?

erik

Hi glitchyordis ,
Could you copy the code, so I can try to repro locally?
Thanks, Erik

glitchyordis

erik

import cv2
import depthai as dai

#Create pipeline
pipeline = dai.Pipeline()

cam = pipeline.create(dai.node.ColorCamera) 
cam.setInterleaved(False)

manip = pipeline.create(dai.node.ImageManip) 
manip.initialConfig.setResize(50,50)
xout_isp = pipeline.create(dai.node.XLinkOut)
xout_isp.setStreamName('isp')
xout_manip = pipeline.create(dai.node.XLinkOut)
xout_manip.setStreamName('out1')

cam.video.link(manip.inputImage)
cam.isp.link(xout_isp.input)
manip.out.link(xout_manip.input)


with dai.Device(pipeline) as device:
    #Output queue will be used to get the rgb frames from the output defined above
    q1 = device.getOutputQueue(name="out1", maxSize=4, blocking=False)


    while True:
        in1 = q1.tryGet()
        print(in1)
        if in1 is not None:
            cv2.imshow("Tile 1", in1.getCvFrame())


        if cv2.waitKey(1) == ord('q'):
            break

erik

Hi glitchyordis ,
video (NV12 format) isn't supported by ImageManip yet (see here). I have fixed your code so it now works:

import cv2
import depthai as dai

#Create pipeline
pipeline = dai.Pipeline()

cam = pipeline.create(dai.node.ColorCamera) 
cam.setInterleaved(False)

manip = pipeline.create(dai.node.ImageManip) 
manip.initialConfig.setResize(50,50)
cam.preview.link(manip.inputImage)

xout_manip = pipeline.create(dai.node.XLinkOut)
xout_manip.setStreamName('out1')
manip.out.link(xout_manip.input)

with dai.Device(pipeline) as device:
    #Output queue will be used to get the rgb frames from the output defined above
    q1 = device.getOutputQueue(name="out1", maxSize=4, blocking=False)
    while True:
        in1 = q1.tryGet()
        print(in1)
        if in1 is not None:
            cv2.imshow("Tile 1", in1.getCvFrame())
        if cv2.waitKey(1) == ord('q'):
            break