Hi, I've tried to activate UVC mode with depthai v2.16 and v2.13 but got the following error:
AttributeError: 'depthai.Pipeline' object has no attribute 'createUVC'

Also, can we access the full RGB resolution, control autofocus and access depth perception while in uvc mode?

  • erik replied to this.

    Hello glitchyordis ,
    For UVC, please follow this guide. Regarding other questions; I believe UVC node is currently limited to NV12 1080P frames. So you wouldn't be able to send depth frames or use 4k/12MP resolution. You should be able to control autofocus. Thoughts?
    Thanks, Erik

      Hi erik ,

      I see. I intend to use OAK-D PRO AF mounted on a mechanism, with a dekstop as my host. My use case is to detect an object, the mechanism with move towards the object and perform an inspection. We'll display the RGB feed using pysimplegui.

      As we all know, we usually perform object detection eg using YOLO with small resolutions eg 640*640. So maybe I can initiate the OAK-D with THE_1080_P or something similar. However, I would like to take a 12MP (4032x3040) image once the mechanism has positioned the camera in front of the object.

      The OAK's 12MP RGB view presented to the user via the GUI can be downscaled/resized (eg 1000*1000px to fit the GUI window). But this presented view should have all the objects in the original 12MP passed to the inspection algorithm so I can determine whether it is in focus.

      But I've observed that the frame is lagging if we just take the ISP output to the host and present it. Cropping/resizing after taking the isp output to host did not help. "Preview" and "video" is side-cropped but do correct me if I've mistaken.

      Based on most of the tutorial I've seen , we set the RGB resolution, eg. "cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)" at the start. Can we change this sensor resolution in the code automatically midway? ie

      if depth == 10cm: # object was detected, mechanism moves camera towards the object
      cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12MP) # change resolution to take 12MP image
      take a picture and save it for inspection
      cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P) # switch back to original resolution for object detection

      Pls let me know if you have any suggestions.

      • erik replied to this.

        Hello glitchyordis ,
        That's an interesting project! I would actually suggest downscaling the ISP and running object detection on that. A great demo of that can be found here. You could also add a logic in Script node to only take still image and send that to the host whenever it detects an object closer to 10cm (as specified in your comment). Thoughts?
        Thanks, Erik

          6 days later

          erik
          I've downscaled ISP with "camRgb.setIspScale(1,5)" for the object detection, but when I request a still, the still is not the full 12MP resolution? Any suggestion?

          import cv2
          import depthai as dai
          pipeline = dai.Pipeline()
          
          camRgb = pipeline.create(dai.node.ColorCamera)
          camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
          camRgb.setIspScale(1,5)
          camRgb.setInterleaved(False) 
          camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
          camRgb.setFps(25)
          
          stillEncoder = pipeline.create(dai.node.VideoEncoder)
          stillMjpegOut = pipeline.create(dai.node.XLinkOut)
          controlIn = pipeline.create(dai.node.XLinkIn)
          xoutIsp = pipeline.create(dai.node.XLinkOut)
          xoutIsp.setStreamName("isp")
          controlIn.setStreamName('control')
          
          stillMjpegOut.setStreamName('still')
          stillEncoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)
          camRgb.isp.link(xoutIsp.input)
          camRgb.still.link(stillEncoder.input)
          controlIn.out.link(camRgb.inputControl)
          stillEncoder.bitstream.link(stillMjpegOut.input)
          
          with dai.Device(pipeline) as device:
          
              stillQueue = device.getOutputQueue('still')
              controlQueue = device.getInputQueue('control')
              qIsp = device.getOutputQueue(name='isp')
              
              while True:
          
                  stillFrames = stillQueue.tryGetAll()
                  for stillFrame in stillFrames:
                      # Decode JPEG
                      frame = cv2.imdecode(stillFrame.getData(), cv2.IMREAD_UNCHANGED)
                      # Display
                      cv2.putText(frame, str(frame.shape), (20, 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255,255,255))
                      cv2.imshow('still', frame)
          
                  frame = qIsp.get()
                  f = frame.getCvFrame()
                  cv2.putText(f, str(f.shape), (20, 20), cv2.FONT_HERSHEY_TRIPLEX, 0.5, (255,255,255))
                  cv2.imshow("isp", f)
          
                  key = cv2.waitKey(1)
                  if key == ord('q'):
                      break
                  elif key == ord('c'):
                      print("c was pressed")
                      ctrl = dai.CameraControl()
                      ctrl.setCaptureStill(True)
                      controlQueue.send(ctrl)
          • erik replied to this.

            Hi glitchyordis ,
            So the isp downscalling will also downscale the still. Still just takes a frame from isp output. You should rather resize the video/preview output and use that for the NN, and trigger still to get full 12MP.
            Thanks ,Erik

              Hey erik

              It is my understanding all modes apart from ISP and raw is cropped from the left/top side of ISP, instead of cropping from centre (image a rectangle's midpoint in x and y axis).

              Is it possible to pass ISP to ImageManip to crop from centre before sending it to host?

              • erik replied to this.

                Hello glitchyordis ,
                I believe the video is actually cropped in the middle, due to different aspect ratio as below. This is if you don't change video size.

                If you change preview, it should actually crop in the middle of the video, depending on the aspect ratio (if same as video, the whole video will just be resized). If you set video resize, it will crop strangely currently instead of resizing, I think this is fixed in image_manip_refactor. Thoughts?
                Thanks, Erik

                  erik Oh, I'll give it a go later. Can we downscale ISP (so instead of cropping) in ImageManip before passing it to the host without using setIspScale in the first place (would like "still" to be in 12MP)?

                  • erik replied to this.

                    glitchyordis ImageManip does resizing, but the release version doesn't support YUV420 frames. So you would need to use image_manip_refactor branch (on depthai-python, after git checkout run python examples/install_requirements.py to get the correct lib) in order so resize the isp.
                    Thanks, Erik

                      erik
                      I have run the install_requirements.py from image_manip_refactor branch. The ISP frame is show but this error pops up right away. Perhaps my installation is incorrect?

                      • erik replied to this.

                        erik

                        import cv2
                        import depthai as dai
                        
                        #Create pipeline
                        pipeline = dai.Pipeline()
                        
                        cam = pipeline.create(dai.node.ColorCamera) 
                        cam.setInterleaved(False)
                        
                        manip = pipeline.create(dai.node.ImageManip) 
                        manip.initialConfig.setResize(50,50)
                        xout_isp = pipeline.create(dai.node.XLinkOut)
                        xout_isp.setStreamName('isp')
                        xout_manip = pipeline.create(dai.node.XLinkOut)
                        xout_manip.setStreamName('out1')
                        
                        cam.video.link(manip.inputImage)
                        cam.isp.link(xout_isp.input)
                        manip.out.link(xout_manip.input)
                        
                        
                        with dai.Device(pipeline) as device:
                            #Output queue will be used to get the rgb frames from the output defined above
                            q1 = device.getOutputQueue(name="out1", maxSize=4, blocking=False)
                        
                        
                            while True:
                                in1 = q1.tryGet()
                                print(in1)
                                if in1 is not None:
                                    cv2.imshow("Tile 1", in1.getCvFrame())
                        
                        
                                if cv2.waitKey(1) == ord('q'):
                                    break
                        • erik replied to this.

                          Hi glitchyordis ,
                          video (NV12 format) isn't supported by ImageManip yet (see here). I have fixed your code so it now works:

                          import cv2
                          import depthai as dai
                          
                          #Create pipeline
                          pipeline = dai.Pipeline()
                          
                          cam = pipeline.create(dai.node.ColorCamera) 
                          cam.setInterleaved(False)
                          
                          manip = pipeline.create(dai.node.ImageManip) 
                          manip.initialConfig.setResize(50,50)
                          cam.preview.link(manip.inputImage)
                          
                          xout_manip = pipeline.create(dai.node.XLinkOut)
                          xout_manip.setStreamName('out1')
                          manip.out.link(xout_manip.input)
                          
                          with dai.Device(pipeline) as device:
                              #Output queue will be used to get the rgb frames from the output defined above
                              q1 = device.getOutputQueue(name="out1", maxSize=4, blocking=False)
                              while True:
                                  in1 = q1.tryGet()
                                  print(in1)
                                  if in1 is not None:
                                      cv2.imshow("Tile 1", in1.getCvFrame())
                                  if cv2.waitKey(1) == ord('q'):
                                      break