Standalone mode oak ffc 4p does not work

erik · Apr 14, 2023

Hi Maimilian ,
Does the serial communication work in non-standalone mode? Could you provide minimal repro code?
Thanks, Erik

Maimilian · Apr 14, 2023

erik Hi, yes, it works perfectly in non-standalone mode. Here is my code:

from pathlib import Path

import cv2

import depthai as dai

import numpy as np

import time

picWidth = 1280

picHeight = 800

\# Get argument first

yolo_v5_path = str((Path(**file**).parent / Path('ai_v2/result/best_openvino_2022.1_6shave.blob')).resolve().absolute())

GPIO = dai.BoardConfig.GPIO

\# Define outputs for serial communication

uartNum = 0

uarts = {

0: { # Suitable for the custom TG board

'txPin' : 15,

'rxPin' : 16,

'txPinMode' : GPIO.ALT_MODE_2,

'rxPinMode' : GPIO.ALT_MODE_2,

}


}

\# Start defining a pipeline

pipeline = dai.Pipeline()

pipeline.setOpenVINOVersion(version=dai.OpenVINO.Version.VERSION_2022_1)

pyserial = """

import time

import serial

#open uart port and set baudrate

ser = serial.Serial("/dev/ttyS0", baudrate=115200)

frame = None

bboxes = []

side = "f" # f = front und b = back

while True:

in_nn = node.io['detections'].get()

if in_nn is not None:

    bboxes = in_nn.detections



if bboxes is not None:

    # if the frame is available, draw bounding boxes on it and show the frame

    size = 1000

    

    firstYellow = True

    firstBlue   = True

    firstBall   = True

    yellowValues = [-1,-1,-1,-1]

    blueValues   = [-1,-1,-1,-1]

    ballValues   = [-1,-1,-1,-1]

    

    for bbox in bboxes:

    

        # denormalize bounding box

        x1 = int(bbox.xmin \* size)

        x2 = int(bbox.xmax \* size)

        y1 = int(bbox.ymin \* size)

        y2 = int(bbox.ymax \* size)

        

        check = False

        if bbox.label == 0 and firstYellow is True:

            check = True

        if bbox.label == 1 and firstBlue is True:

            check = True

        if bbox.label == 2 and firstBall is True:

            check = True

        

        if check:

        

            coordinates = list()

            coordinates.append(x1)

            coordinates.append(x2)

            coordinates.append(y1)

            coordinates.append(y2)

            

            for coordinate in coordinates:

                index = coordinates.index(coordinate)

                if coordinate < 0:

                    coordinates[index] = 0

                if coordinate > size:

                    coordinates[index] = size

            

            if bbox.label == 0:

                yellowValues = coordinates

                firstYellow = False

            

            if bbox.label == 1:

                blueValues = coordinates

                firstBlue = False

            

            if bbox.label == 2:

                ballValues = coordinates

                firstBall = False

       

    serString = side + '/ye:' + str(yellowValues[0]) + ',' + str(yellowValues[1]) + ',' + str(yellowValues[2]) + ',' + str(yellowValues[3]) + '/bl:' + str(blueValues[0]) + ',' + str(blueValues[1]) + ',' + str(blueValues[2]) + ',' + str(blueValues[3]) + '/ba:' + str(ballValues[0]) + ',' + str(ballValues[1]) + ',' + str(ballValues[2]) + ',' + str(ballValues[3])  + '\\z'

    

    ser.write(serString.encode())

    

    if side == "f":

        side = "b"

    else:

        side = "f"


"""

\# Define script for output

script = pipeline.create(dai.node.Script)

script.setProcessor(dai.ProcessorType.LEON_CSS)

script.setScript(pyserial)

\# Define manip for scaling

manip_front = pipeline.create(dai.node.ImageManip)

manip_front.initialConfig.setResizeThumbnail(416,416,0,0,0)

manip_back = pipeline.create(dai.node.ImageManip)

manip_back.initialConfig.setResizeThumbnail(416,416,0,0,0)

\# Define sources - color cameras

cam_front = pipeline.create(dai.node.ColorCamera)

cam_back = pipeline.create(dai.node.ColorCamera)

cam_front.setBoardSocket(dai.CameraBoardSocket.CAM_A)

cam_back.setBoardSocket(dai.CameraBoardSocket.CAM_B)

cam_front.setPreviewSize(picWidth, picHeight)

cam_back.setPreviewSize(picWidth, picHeight)

cam_front.setInterleaved(False)

cam_back.setInterleaved(False)

cam_front.setFps(9)                # 9 sind optimal

cam_back.setFps(9)

cam_front.setResolution(dai.ColorCameraProperties.SensorResolution.THE_800_P)

cam_back.setResolution(dai.ColorCameraProperties.SensorResolution.THE_800_P)

cam_front.initialControl.setManualFocus(255)

cam_back.initialControl.setManualFocus(255)

\# network specific settings

detectionNetwork = pipeline.createYoloDetectionNetwork()

detectionNetwork.setConfidenceThreshold(0.5)                    # unter Umständen umstellen

detectionNetwork.setIouThreshold(0.5)

detectionNetwork.setNumClasses(3)

detectionNetwork.setCoordinateSize(4)

anchors = np.array([10, 13, 16, 30, 33, 23, 30, 61, 62, 45, 59, 119, 116, 90, 156, 198, 373, 326])

detectionNetwork.setAnchors(anchors)

anchorMasks52 = np.array([0, 1, 2])

anchorMasks26 = np.array([3, 4, 5])

anchorMasks13 = np.array([6, 7, 8])

anchorMasks = {

"side52": anchorMasks52,

"side26": anchorMasks26,

"side13": anchorMasks13


}

detectionNetwork.setAnchorMasks(anchorMasks)

detectionNetwork.setBlobPath(yolo_v5_path)

detectionNetwork.setNumNCEPerInferenceThread(1)

detectionNetwork.setNumInferenceThreads(2)

detectionNetwork.input.setBlocking(False)

cam_front.preview.link(manip_front.inputImage)

cam_back.preview.link(manip_back.inputImage)

manip_front.out.link(detectionNetwork.input)

manip_back.out.link(detectionNetwork.input)

\# Create output

detectionNetwork.out.link(script.inputs["detections"])

\# Create BoardConfig and enable UART

config = dai.Device.Config()

\# UART TX

config.board.gpio[uarts[uartNum]['txPin']] = GPIO(GPIO.OUTPUT, uarts[uartNum]['txPinMode'])

\# UART RX

config.board.gpio[uarts[uartNum]['rxPin']] = GPIO(GPIO.INPUT, uarts[uartNum]['rxPinMode'])

\# Enable UART{uartNum}

config.board.uart[uartNum] = dai.BoardConfig.UART()

config.version = config.version.VERSION_2021_4

\# Flash the pipeline

(f, bl) = dai.DeviceBootloader.getFirstAvailableDevice()

bootloader = dai.DeviceBootloader(bl)

progress = lambda p: print(f'Flashing progress: {p \* 100:.1f}%')

bootloader.flash(progress, pipeline)

erik · Apr 14, 2023

Hi Maimilian , This is far from minimal reproducible code.

Maimilian · Apr 14, 2023

erik oh sorry i missed it in my hurry

Here's a new version:


from pathlib import Path

import cv2

import depthai as dai

import numpy as np

\# Get argument first

yolo_v5_path = str((Path(__file__).parent / Path('ai_v2/result/best_openvino_2022.1_6shave.blob')).resolve().absolute())

GPIO = dai.BoardConfig.GPIO

\# Define outputs for serial communication

uartNum = 0

uarts = {

    0: {  # Suitable for the custom TG board

        'txPin' : 15,

        'rxPin' : 16,

        'txPinMode' : GPIO.ALT_MODE_2,

        'rxPinMode' : GPIO.ALT_MODE_2,

    }

}

\# Start defining a pipeline

pipeline = dai.Pipeline()

pipeline.setOpenVINOVersion(version=dai.OpenVINO.Version.VERSION_2022_1)

pyserial = """

    import serial

    ser = serial.Serial("/dev/ttyS0", baudrate=115200)

    bboxes = []

    


    while True:

        in_nn = node.io['detections'].get()

        if in_nn is not None:

            bboxes = in_nn.detections

        


        if bboxes is not None:

            yellowValues = [-1,-1,-1,-1]

            blueValues   = [-1,-1,-1,-1]

            ballValues   = [-1,-1,-1,-1]

            


            for bbox in bboxes:

            


                # denormalize bounding box

                x1 = int(bbox.xmin \* size)

                x2 = int(bbox.xmax \* size)

                y1 = int(bbox.ymin \* size)

                y2 = int(bbox.ymax \* size)

                


                if check:

                


                    coordinates = list()

                    coordinates.append(x1)

                    coordinates.append(x2)

                    coordinates.append(y1)

                 


               


            serString = 'str(ballValues[0])'

            


            ser.write(serString.encode())

"""

\# Define script for output

script = pipeline.create(dai.node.Script)

script.setProcessor(dai.ProcessorType.LEON_CSS)

script.setScript(pyserial)

\# Create output

detectionNetwork.out.link(script.inputs["detections"])

config = dai.Device.Config()

config.board.gpio[uarts[uartNum]['txPin']] = GPIO(GPIO.OUTPUT, uarts[uartNum]['txPinMode'])

config.board.gpio[uarts[uartNum]['rxPin']] = GPIO(GPIO.INPUT, uarts[uartNum]['rxPinMode'])

config.board.uart[uartNum] = dai.BoardConfig.UART()

config.version = config.version.VERSION_2021_4

\# Flash the pipeline

(f, bl) = dai.DeviceBootloader.getFirstAvailableDevice()

bootloader = dai.DeviceBootloader(bl)

progress = lambda p: print(f'Flashing progress: {p \* 100:.1f}%')

bootloader.flash(progress, pipeline)



DetectionNetwork is my network with the blobpath.

erik · Apr 14, 2023

Maimilian are you using latest depthai (2.21.2)? There were some fixes recently (from changelog: FW: fix for UART0 / '/dev/ttyS0' init failure in Script node).

Maimilian · Apr 14, 2023

erik yes I do. It works also in non standalone mode…

Maimilian · Apr 14, 2023

erik maybe I have to flash the config part? Because it is not part of the pipeline is it?

erik · Apr 14, 2023

Maimilian Could you also try flashing latest bootloader first?

Maimilian · Apr 14, 2023

erik I have done this via the device manager. It is installed version 0.0.24. I think that's the newest version (this says the device manager too). Then I tried to flash it. The output in the terminal then says: progress 0.0% (something like that) and then it finishes.

Maimilian · Apr 15, 2023

erik Maybe there's also a problem with the current and latest bootloader version. I think so because when I flashed with the version 0.0.18 instead of the latest version 0.0.24 it flashed normally (about 2 minutes with a progress). However, even there standalone mode has not worked yet. Nevertheless, is there a way to downgrade bootloader version?

Furthermore I read about the parameter compress in bootloader.flash(). Should this be true?

erik · Apr 15, 2023

Maimilian I can confirm it does work in normal mode, but not in flashed mode with latest bootlaoder. Our firmware team will be looking into this shortly. You can downgrade bootloader by installing older depthai version (eg. 2.19.1).

erik · Apr 15, 2023

Maimilian very likely you have only 16MB of memory, and if you don't compress the firmware (bootlaoder.flash(progress, pipeline, compress=True)) it will fail. Try the following:

(ok, err) = bootloader.flash(progress, pipeline)
print(f"Flashing success: {ok}, error: {err}")

And then change to:

(ok, err) = bootloader.flash(progress, pipeline, compress=True)

erik · Apr 15, 2023

Also note that with 16MB memory, I don't think you'll be able to flash any kind of neural network to the device. The depthai itself takes like 10MB-15MB (when compressed), so you will have a small margin. It might also be that only the USB Bootlaoder doesn't find eMMC flash (usually present on SOMs). I opened Feature Request here.

Maimilian · Apr 15, 2023

erik Thanks for answering. The flash progress was succesful. However, I need to flash my ai model on the board. It is 14.1MB large.

erik · Apr 15, 2023

Maimilian For that, I believe it would be best to open feature request on depthai-core, that even USB bootloader can access eMMC memory (currently, only NETWORK bootlaoder can). I also don't think flashing BoardConfig (needed for UART) is available, perhaps creating separate feature request for that would be useful as well.

Maimilian · Apr 15, 2023

erik Is there any chance that flashing config for uart will be available in the next two weeks?

Maimilian · Apr 15, 2023

erik where should I create a feature request?

Maimilian · Apr 15, 2023

erik And another question: is it possible to use network bootloader with the oak ffc 4p?

Maimilian · Apr 15, 2023

erik Or is there any other way to work without a Computer?