Needing a workaround for a problem that's crashing my OAK camera app.

ChristopherNichols

Problem Description🇦

What works:

I have code similar to the "hello world" example. I run the script, e.g. as python3 main.py. It works perfectly. The camera characteristics are printed out, my camera displays the video feed, and when II press "q" to terminate the infinite loop I and am able to obtain the last frame and process the image.

What fails:

When I include the same code in a very simple, basic web app using Flask, it crashes. The code runs up to the point where it tries to show the video output preview window (cv2.imshow("preview", frame). The window does not appear. I get an error message starting with:

Assertion failed: (NSViewIsCurrentlyBuildingLayerTreeForDisplay() != currentlyBuildingLayerTree), function NSViewSetCurrentlyBuildingLayerTreeForDisplay, file NSView.m, line 13477.

From what I have searched, the problem may be that Flask is single-threaded and that the window needs a separate thread. Im not sure though. And I am not familiar with thread programming.

I am running on MacOS, using Python 3.11, with an OAK 1 Lite camera.. Is there a solution to this problem? Any advice is appreciated.

jakaskerl

Hi ChristopherNichols
I'm going to need some minimal code to figure out what might be happening.

Thanks,
Jaka

ChristopherNichols

Hi Jaka,

I stripped down the code to just what's necessary to demo the problem. I've uploaded 4 files. You need to run them in an environment with the depth ai dependencies and pip install Flask.

First demonstrate the working case using the files demo_main.py and demo_camera.py by running python3 demo_main.py in the terminal. You should get the length of the last frame after you click "q".

Second demonstrate the case that is failing using the files demo_web.py, demo_camera.py, and demo_temlpate.htm. You need to put the template inside a folder called templates. After running python3 demo_web.py in the terminal you need to open a browser and type in localhost:8080 to start the web application. The result is the output of the error that occurs.

Thanks,

Christopher

ChristopherNichols

Apologies. Here are the 4 files. Oh, it says uploading files of this type are not allowed. I have tried renaming them as txt, and also zipping them them together but that does not work, so I am sending images, and the demo_camera file in its entirety.

Thanks,

Christopher

See code for demo_camera.py below🇦

# LIBRARIES

import numpy as np

import cv2

import depthai

import blobconverter

#------------------------------------------------------------------------------

# PARAMETERS

#------------------------------------------------------------------------------

# CODE

#------------------------------------------------------------------------------

# Normalise the frame/image data.

# Neural network inference output bounding box coordinates are

# represented as floats from <0..1> range. The images must be scaled relative

# to frame width/height

# (e.g. if image has 200px width and the nn returned x_min coordinate of 0.2,

# this means the actual (normalised) x_min coordinate is 40px),

# i.e 200 * 0.2, or a fifth.

def normaliseFrame(frame, bbox):

normVals = np.full(len(bbox), frame.shape[0])

normVals[::2] = frame.shape[1]

return (np.clip(np.array(bbox), 0, 1) \* normVals).astype(int)

#------------------------------------------------------------------------------

# # Get the last frame from the camera after the user presses 'q'

def getLastFrame():


try:

    # Create pipeline

    

    pipeline = depthai.Pipeline()

    

    # Assemble the pipeline and its components:

    

    # Create camera node

    cam_rgb = pipeline.create(depthai.node.ColorCamera)

    cam_rgb.setPreviewSize(300, 300)

    cam_rgb.setInterleaved(False)

    

    # Create neural network node

    detection_nn = pipeline.create(depthai.node.MobileNetDetectionNetwork)

    

    # Set path of the blob (NN model). The blob converter converts and downloads

    # the model

    detection_nn.setBlobPath(blobconverter.from_zoo(name='mobilenet-ssd', shaves=6))

    

    # Set the confidence threshold to filter out incorrect results

    detection_nn.setConfidenceThreshold(0.5)

    

    # Connect a color camera preview output to the neural network input

    cam_rgb.preview.link(detection_nn.input)

    

    # Allow the host to receive camera output (frames) via XLink

    # Create an XLinkOut node

    xlink_out_cam = pipeline.create(depthai.node.XLinkOut)

    xlink_out_cam.setStreamName("rgb")

    cam_rgb.preview.link(xlink_out_cam.input)

    

    # Allow the host to receive neural network output (detections) via XLink

    # Create an XLinkOut node

    xlink_out_nn = pipeline.create(depthai.node.XLinkOut)

    xlink_out_nn.setStreamName("nn")

    detection_nn.out.link(xlink_out_nn.input)

    

    # Set up the camera

    last_frame = None

    

    with depthai.Device(pipeline) as device:

        

        # Display camera features

        print('Connected cameras:', device.getConnectedCameraFeatures())

        

        # Display out usb speed

        print('Usb speed:', device.getUsbSpeed().name)

        # Bootloader version

        if device.getBootloaderVersion() is not None:

            print('Bootloader version:', device.getBootloaderVersion())

        # Device name

        print('Device name:', device.getDeviceName())

    

    

        # Access camera and neural network output on the host side

    

        print("ACCESSING CAMERA AND NN OUTPUT")

        

        # Queue for receiving camera out over XLink

        que_cam_rgb = device.getOutputQueue("rgb") #(name="rgb", maxSize=1, blocking=False)

    

        # Queue for receiving neursl network output over XLink

        que_nn = device.getOutputQueue("nn") #(name="nn", maxSize=1, blocking=False)

     

        # Create queue consumers 

        frame = None

        

        detections = []

        

        # Continue looping until user presses "q"

        loop = True

        

        iterations = 0

        

    

        while loop:

                

            # Fetch the latest output from the camera node and the nn node from 

            # the respective queues

            

            # tryGet method returns either the latest result or None if the queue is empty.

            

            # Results from the camera and neural network are of type numpy ndarray,

            # specificically 1D array in this case. Both sets of output require

            # transformation for them to be useful.  

            # display.

            # Frame normalisation is one of the required transformations

        

            # Normalise the frame/image data.    

    

            # Receive a frame from the camera.  

            # This is a blocking call, i.e. it will wait until new data has arrived

            in_cam_rgb = que_cam_rgb.tryGet()

            

            if in_cam_rgb is not None:

                 frame = in_cam_rgb.getCvFrame()

                                 

            # MobileNetSSD output fields:

                # image_id

                # label 

                # confidence

                # x_min, 

                # y_min,

                # x_max, 

                # y_max 

    

            # Receive the output from the neural network

            in_nn = que_nn.tryGet()

            

            if in_nn is not None:

                detections = in_nn.detections

    

            # By accessing the detections array, we receive the detected 

            # objects that allow us to access these fields            

            # Display the results

    

            if frame is not None:

                  

                for detection in detections:

                    bbox = normaliseFrame(frame, (detection.xmin, detection.ymin, detection.xmax, detection.ymax))

                    

                    cv2.rectangle(frame, (bbox[0], bbox[1]), (bbox[2], bbox[3]), (255, 0, 0), 2)

                    

                    cv2.imshow("preview", frame)    

                    

                    last_frame = frame

    

                    # Count the number of iterations. 

                    iterations += 1

            

            

            print("Iteration = " + str(iterations))

            

            if cv2.waitKey(1) == ord("q"):

                

                break

            

                loop = False

            

        return last_frame           

except Exception as e:

    return e

#------------------------------------------------------------------------------

jakaskerl

Hi ChristopherNichols
I'd recommend deleting the whole depthai code and just replacing it with cv2 imshow(some random local image). This really seems like a cv-flask issue.

GPT4:

The error you're encountering is specific to macOS and appears to be related to the threading model used by the Cocoa API, which OpenCV uses for its GUI functionalities on macOS. This error generally occurs when you're trying to update the UI from a thread that is not the main thread.

In a Flask application, routes are handled in separate threads by default. This means that when you click a button on your web interface, the corresponding Python code is not executing in the main thread.

Solutions

1. Using `cv2.imshow()` in the Main Thread

You can use a message queue to communicate between the Flask thread and the main thread. The main thread checks this queue and opens the image if a message is found.

2. Web-based Image Display

Instead of using cv2.imshow() to display the image, you could convert the image to a format that can be displayed in the web browser. For example, you could convert the image to JPEG or PNG format and then send it as a response to be displayed in the web interface.

Example Code

from flask import Flask, Response
import cv2

app = Flask(__name__)

@app.route('/image')
def image():
    img = cv2.imread('image.jpg')
    _, buffer = cv2.imencode('.jpg', img)
    return Response(buffer.tobytes(), mimetype='image/jpeg')

3. Using JavaScript for Real-time Updates

For a more real-time solution, you can use JavaScript on the client side to periodically fetch the image from the server.

Hope this helps.

Thanks,
Jaka

ChristopherNichols

Hi Jaka,

Thanks a lot for the suggestions.

Regards,

Christopher