We're having some trouble with setting focus on the OAK-FFC IMX582 sensor that works consistently across all of the boards we have. These cameras are flashed with a pipeline that uses a script to crop the feed around a detected face. Our current focus logic sets the lens position in the script based on the size of the detected face in the image (similar to the depth-driven focus demo, but using bounding box size instead of depth as we are working with a single sensor). This seems to work for most sensors, however, there are a small number of sensors produce a blurry at the calculated lens position. We can adjust the initial lens position in our equation for these cameras to improve focus, however it would be impractical for us to have to re-flash these cameras until we find the right initial lens position for each camera.

We've tried using autofocus as well, however it seems that once the sensor settles on a lens position, it will not adjust if the person moves closer/further from the camera.

 

We've also tried triggering the autofocus from the script, but that doesn't seem to do anything. Another thing we tried was to set the auto focus region in the script to be the bounding box of the detected face, but this seems to continually trigger the autofocus, which causes the camera to constantly adjust the lens position and never be in focus.

Are there any suggestions or recommendations on how we can handle focus better on these sensors where the best lens position may vary between sensors?

    tkoon We've tried using autofocus as well, however it seems that once the sensor settles on a lens position, it will not adjust if the person moves closer/further from the camera.

    Maybe the control is not setup correctly in the pipeline?

    Have you tried setting different AF modes? https://docs.luxonis.com/projects/api/en/latest/references/python/?highlight=autofocus#depthai.CameraControl.setAutoFocusMode

    enum class AutoFocusMode : uint8_t {
    /**
    * Autofocus disabled. Suitable for manual focus
    */
    OFF = 0,
    /**
    * Basic automatic focus mode. In this mode, the lens does not move unless the autofocus trigger action is called.
    */
    AUTO,
    /**
    * Close-up focusing mode - this mode is optimized for focusing on objects very close to the camera.
    */
    MACRO,
    /**
    * In this mode, the AF algorithm modifies the lens position continually to attempt to provide a constantly-in-focus image stream.
    * The focusing behavior should be suitable for good quality video recording; typically this means slower focus movement and no overshoots.
    */
    CONTINUOUS_VIDEO,
    /**
    * In this mode, the AF algorithm modifies the lens position continually to attempt to provide a constantly-in-focus image stream.
    * The focusing behavior should be suitable for still image capture; typically this means focusing as fast as possible
    */
    CONTINUOUS_PICTURE,
    /**
    * Extended depth of field (digital focus) mode. The camera device will produce images with an extended depth of field automatically.
    * AF triggers are ignored.
    */
    EDOF,
    };

    Thanks,
    Jaka

        jakaskerl

        We've tried using all of the different autofocus modes, but with no success. Here's the part of our pipeline that sets up the initial control to use autofocus:

        cam.initialControl.setAutoFocusMode(dai.CameraControl.AutoFocusMode.CONTINUOUS_VIDEO)
        cam.initialControl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.AUTO)
        cam.initialControl.setSceneMode(dai.CameraControl.SceneMode.FACE_PRIORITY)
        cam.initialControl.setAutoExposureEnable()
        cam.initialControl.setSharpness(0)
        cam.initialControl.setLumaDenoise(0)
        cam.initialControl.setChromaDenoise(4)

        It seems like no matter what autofocus mode we choose, once the camera settles on a lens position, it does not change as the person moves further/closer. Ideally, we would like to dynamically set the region for the AF so that it focuses on the bounding box of the detected face. I'm guessing that would likely improve the AF logic rather than focusing on a wider area by default. However, calling setAutoFocusRegion() in the script doesn't seem to behave as we expected it to.

        We are setting the Auto Exposure region in the script using setAutoExposureRegion(), and that is working well for us.

        FWIW, these cameras are running on UVC, which is why we are relying on the script for control of things like auto exposure region/potentially auto focus logic.

          Hi @tkoon
          Would you mind creating a MRE so I can test it myself to see where the issue is?

          Thanks,
          Jaka

            jakaskerl

            Using the following code, setting the AF region in the script causes the lens to be constantly re-focusing, while only setting the region one time. Ideally, I would like to update the region as the person in the frame moves.

            import depthai as dai
import cv2

pipeline = dai.Pipeline()
cam = pipeline.create(dai.node.ColorCamera)
cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_48_MP)
cam.initialControl.setAutoFocusMode(dai.CameraControl.AutoFocusMode.CONTINUOUS_VIDEO)  # Changed from AUTO

script = pipeline.create(dai.node.Script)
script.outputs['ctrl'].link(cam.inputControl)
script.setScript(
    '''
    ctrl = CameraControl()
    set_region = True
    while True:
        if set_region:
            ctrl.setAutoFocusRegion(300, 300, 300, 300)
            set_region = False
        node.io['ctrl'].send(ctrl)    
    ''')

xout = pipeline.create(dai.node.XLinkOut)
xout.setStreamName("video")
cam.video.link(xout.input)

with dai.Device(pipeline) as device:
    device.setLogLevel(dai.LogLevel.OFF)

    qVideo = device.getOutputQueue(name='video')

    while True:
        if qVideo.has():
            feed = qVideo.get()
            frame = feed.getCvFrame()
            frame = cv2.resize(frame, (1440, 810))
            cv2.putText(frame, f'{feed.getLensPosition()}', (40, 70), cv2.FONT_HERSHEY_SIMPLEX, 3, (0, 255, 0), 7, 2)
            cv2.imshow('Video', frame)

            key = cv2.waitKey(1)
            if key == ord('q'):
                break

                So part of the issue, I think, is in the script calling node.io['ctrl'].send(ctrl) for each iteration in the while loop. This is what is causing the lens to constantly refocus I believe. So, rather than calling it every iteration in the loop, I'm thinking the only way to get this to work properly is to only set the CameraControl if the person moves past a given threshold outside the current AF region. It's not the ideal solution, however, because the lens still will go out way out of focus while trying to adjust to the new region (lens position goes to 97 when region is set, then adjusts to correct lens position). Ideally, the lens would stay at it's current position and adjust to the correct position for the new region.

                I've tried limiting setting cam.initialControl.setAutoFocusLensRange(105, 145) to try and fix the lens dropping < 100 when setting the region, but I was getting unexpected behavior when trying it out. The lens position actually dropped lower (< 90) before settling in the set range.

                tkoon ctrl.setAutoFocusRegion(300, 300, 300, 300)

                The region is off.

                tkoon node.io['ctrl'].send(ctrl)

                ^^^^ this should be on the same level as setRegion. Otherwise it gets triggered sending empty info.

                Thanks,
                Jaka

                      jakaskerl ^^^^ this should be on the same level as setRegion. Otherwise it gets triggered sending empty info.

                      I realized this in my next post. I think my solution will have to be adjusting the region as the person moves a given threshold outside of the current region. I think it may be unavoidable to set the AF region without the lens position first moving to a position that makes the image blurry while re-focusing.

                          tkoon think it may be unavoidable to set the AF region without the lens position first moving to a position that makes the image blurry while re-focusing.

                          That's AF working. I think the algorithm first resets the lens position. If you were to manually set the position, it wouldn't do that I believe.

                          Thanks,
                          Jaka