Running the QR code detection model in Standalone mode

zepman

I have the OAK-D Pro PoE and have this simple test pipeline that uses the qr_code_detection_384x384 NN model, and uses a script node to send an output message via MQTT.

import depthai as dai
import blobconverter

pipeline = dai.Pipeline()

# Define camera node
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setPreviewSize(384, 384)
camRgb.setInterleaved(False)
camRgb.initialControl.setManualFocus(145)
camRgb.setFps(10)

# Define a neural network that will detect qr codes based on the source frames
nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
nn.setBlobPath(blobconverter.from_zoo(name="qr_code_detection_384x384", zoo_type="depthai", shaves=6))

# Create a script node that publishes to an MQTT server
script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

# Change the IP to your MQTT broker!
MQTT_BROKER = "192.168.0.38"
MQTT_BROKER_PORT = 1883
MQTT_TOPIC = "oak_d/detection"
script_text = f"""
import time

mqttc = Client()
node.warn('Connecting to MQTT broker...')
mqttc.connect("{MQTT_BROKER}", {MQTT_BROKER_PORT}, 120)
node.warn('Successfully connected to MQTT broker!')

mqttc.loop_start()
cnt = 0
while True:
    cnt+=1
    if cnt > 50000:
        node.warn(str(cnt))
        cnt = 0
        detections = node.io['detections'].get().detections
        if len(detections) > 0:
            msg = 'qr code detected'
        else:
            msg = 'no qr code detected'

        node.warn('sending: ' + msg)
        (ok, id) = mqttc.publish("{MQTT_TOPIC}", msg, qos=2)
        if (ok == 0):
            node.warn('published successfully, id: '+ str(id))

"""
with open("utils/paho-mqtt.py", "r") as f:
    paho_script = f.read()
    script.setScript(f"{paho_script}\n{script_text}")

# Link camera to NN input
camRgb.preview.link(nn.input)
# Link NN output to script input
nn.out.link(script.inputs['detections'])

This pipeline works fine in host mode with the following piece of code. Using MQTT Explorer, I can see messages being published continuously.

## Host method
import time
with dai.Device(pipeline) as device:
    print('Connected to OAK')
    while not device.isClosed():
        time.sleep(1)

However, the same pipeline doesn't work in standalone mode. I've used the following piece of code to flash the above pipeline to device.

## Standalone method
(f, deviceInfo) = dai.DeviceBootloader.getFirstAvailableDevice()
bootloader = dai.DeviceBootloader(deviceInfo)
print_progress = lambda p : print(f'Flashing progress: {p*100:.1f}%')

# Flash pipeline to device directly
success, msg = bootloader.flash(print_progress, pipeline, applicationName="QRCode")
msg = f"Flashing was successful." if success else f"Flashing failed. Error: {msg}"
print(msg)

The flashing finishes successfully. After about a minute, I hear a click from the camera, indicating that it has rebooted. However, there are no messages being published. The camera seems to be rebooting repeatedly as I hear a click from the camera after every minute has passed. So, it looks like the application is crashing.

This might have something to do with the qr_code_detection_384x384 NN model as the pipeline runs fine (in both host and standalone model). Things are fine when I replace that model with a different model ("mobilenet-ssd", and "mask_detection_300x300", for example, which I have tried).

I know how to enable logging when using host mode, but is there any way of getting/accessing logging information in standalone mode? Is there something that needs to be done differently to get models like "qr_code_detection_384x384" working in standalone mode?

jakaskerl

Hi zepman
You can do logging on emmc storage as it is done here: Important lines are 8-13.

Hope this helps,
Jaka

zepman

jakaskerl

Thanks for that. From what I understand, the OAK-D Pro PoE has no emmc storage, so running the example code as-is gives me an error. But I was able to modify that logging code so it writes to flash memory instead: in the cfg settings, changing logPath to '/' and emmc to False, and changing the memory argument to AUTO when flashing. This made it work for me as I could then see the files via HTTP.

I incorporated the logging bits into my original pipeline code and it worked fine then too…except when I used the qr_code_detection_384x384 model. Since the pipeline seems to be crashing, the file server too is inaccessible.

Is there a way to make the log file written to the flash memory persistent? This would allow me to inspect log files created by previous pipeline runs by running another pipeline that would only contain the script server node.

jakaskerl

Hi zepman
That's weird, are you sure you are actually writing to flash? The memory should AFAIK be persistent between reboots.
Could you add a MRE as well?

Thanks,
Jaka

zepman

jakaskerl

Thanks for your prompt replies!

Here's a trimmed-down version of the logging code you posted earlier.

import time
import random
import depthai as dai

## Define pipeline

pipeline = dai.Pipeline()

cfg = dai.BoardConfig()
cfg.emmc = False
cfg.logPath = f'/depthai_log_file{random.randrange(0, 100)}.txt'
config = dai.Device.Config()
config.board = cfg
pipeline.setBoardConfig(cfg)

# Only two nodes - both scripts, one simply does logging, another creates a file server
scriptSave = pipeline.create(dai.node.Script)
scriptServer = pipeline.create(dai.node.Script)

# Note: all scripts accessing the media storage must run on LEON_CSS
scriptSave.setProcessor(dai.ProcessorType.LEON_CSS)
scriptSave.setScript("""
import time
counter = 0
while True:
    node.warn(f'counter: {counter}')
    counter += 1
    time.sleep(3)
""")

scriptServer.setProcessor(dai.ProcessorType.LEON_CSS)
scriptServer.setScript("""
import http.server
import socketserver
import socket
import fcntl
import struct
import os
import time
def get_ip_address(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    return socket.inet_ntoa(fcntl.ioctl(
        s.fileno(),
        -1071617759,  # SIOCGIFADDR
        struct.pack('256s', ifname[:15].encode())
    )[20:24])
# Note: `chdir` here will prevent unmount, this should be improved!
time.sleep(1)
os.chdir('/')
PORT = 80
Handler = http.server.SimpleHTTPRequestHandler
with socketserver.TCPServer(("", PORT), Handler) as httpd:
    ip = get_ip_address('re0')
    node.warn(f'===== HTTP file server accessible at: http://{ip}')
    httpd.serve_forever()
""")

## Run pipeline

with dai.Device(pipeline) as device:
    while not device.isClosed():
        time.sleep(1)

When I run the above code, I see the log file created on the file server:

If I stop execution and run the code again, I only see the new log file created; the file created by the previous run is gone:

I might be missing something or doing something silly here, but based on the above, it looks like files don't persist between runs/reboots. This is the same even in standalone mode i.e., if I modify the last part of the code to the following.

...
...
## Run pipeline

(f, bl) = dai.DeviceBootloader.getFirstAvailableDevice()
bootloader = dai.DeviceBootloader(bl)
progress = lambda p : print(f'Flashing progress: {p*100:.1f}%')
(r, errmsg) = bootloader.flash(progress, pipeline, memory=dai.DeviceBootloader.Memory.FLASH, applicationName="TestLogging")
if r: print("Flash OK")
else: print("Flash ERROR:", errmsg)

jakaskerl

Hi zepman
Seems this is the case here yes. The files are written to device RAM which clears when you reboot the device. Only possible persistent storage locations are on SD card and emmc.

This is a bit annoying really.

I propose this:

Have the device send errors/logs via tcp:

Host side server:

import socket

def start_tcp_server():
    HOST = '0.0.0.0'  # Listen on all interfaces
    PORT = 65432      # Port to listen on

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind((HOST, PORT))
        s.listen()
        print('Listening for logs...')
        while True:  # Keep the server running
            conn, addr = s.accept()
            with conn:
                print('Connected by', addr)
                while True:
                    data = conn.recv(1024)
                    if not data:
                        break
                    print(data.decode('utf-8'))

start_tcp_server()

modify following code to the device to send data:

scriptSave.setScript("""
import time
import socket
counter = 0
                     
def send_log_to_host(log_message):
    HOST = '169.254.123.26'  # The server's IP address
    PORT = 65432                # The port used by the server
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((HOST, PORT))
        s.sendall(log_message.encode('utf-8'))

while True:
    node.warn(f'counter: {counter}')
    # Example usage
    try:
        # Your OAK-D operations here
        # ...
        send_log_to_host(f'counter: {counter}')
    except Exception as e:
        send_log_to_host(f"Error: {e}")
    counter += 1
    time.sleep(3)
""")

This should send info through but you must be smart where to put the send_log_to_host function to pinpoint your error. NOTE that this won't work if the error happens before a TCP connection is established.

If that fails, I might be able to run it locally with EMMC capable device to see what the problem is. You are also able to help by removing functions bit by bit to find the culprit.

Thanks,
Jaka

zepman

jakaskerl

Thanks a lot for that.

I tried your code in my pipeline and the TCP communication works fine in host-connected mode. However, for some reason, when I flash the pipeline to the device, I see no messages coming through. Don't know if it would have something to do with my network settings, but strange that it does work fine in host-connected mode. Anyway, this would be a separate issue (and I might start a separate thread if necessary).

I modified the send_log_to_host function in the script so it publishes to MQTT instead of using TCP. This works in both host-connected and standalone mode. However, I am unclear now about the # Your OAK-D operations here bit in the script. The main operation in my case is QR code detection using the qr_code_detection_384x384 mode so it needs to be implemented in an NN node which is what seems to be causing a problem.
The following minimal pipeline uses the script node to send out logs (via MQTT but could very well be via TCP, like in your original code), but also has my NN node. To keep it minimal, there is no camera node, and the two nodes aren't even linked. It doesn't matter; the pipeline runs fine in host-connected mode and I can see the logs published. However, in standalone mode, I see nothing, which indicates that the device is crashing (based on it clicking every minute) even before the logs can be sent out. The sequence of the nodes in the pipeline also doesn't matter, as the crash happens no matter what order the nodes are created in.

import depthai as dai
import blobconverter

## Define pipeline

pipeline = dai.Pipeline()

cfg = dai.BoardConfig()
cfg.emmc = False
cfg.logPath = f'/depthai_log_file.txt'
config = dai.Device.Config()
config.board = cfg
pipeline.setBoardConfig(cfg)

# Add a script node to send the logs out
scriptSave = pipeline.create(dai.node.Script)
scriptSave.setProcessor(dai.ProcessorType.LEON_CSS)
script_text = """
import time
counter = 0

MQTT_BROKER = "192.168.0.38"
MQTT_BROKER_PORT = 1883
MQTT_TOPIC = "oak_d/log"

mqttc = Client()
node.warn('Connecting to MQTT broker...')
mqttc.connect(MQTT_BROKER, MQTT_BROKER_PORT, 120)
node.warn('Successfully connected to MQTT broker!')
mqttc.loop_start()

def send_log_to_host(log_message):
    (ok, id) = mqttc.publish(MQTT_TOPIC, log_message, qos=2)

while True:
    node.warn(f'counter: {counter}')
    # Example usage
    try:
        # Your OAK-D operations here
        # ...
        byteArray = None
        with open('/depthai_log_file.txt', 'r') as f:
            msg = f.read()
            byteArray = bytes(msg, encoding='utf8')
        if byteArray:
            send_log_to_host(byteArray)
    except Exception as e:
        send_log_to_host(f"Error: {e}")
    counter += 1
    time.sleep(3)
"""

with open("utils/paho-mqtt.py", "r") as f:
    paho_script = f.read()
    scriptSave.setScript(f"{paho_script}\n{script_text}")

# Add an NN node - no need to link it
nn = pipeline.create(dai.node.MobileNetDetectionNetwork)
nn.setConfidenceThreshold(0.7)
nn.setBlobPath(blobconverter.from_zoo(name="qr_code_detection_384x384", zoo_type="depthai")) # doesn't run in standalone mode

The culprit is the last line, as there is no issue if I use a different model:

nn.setBlobPath(blobconverter.from_zoo(name="mobilenet-ssd", shaves=6)) # runs fine in standalone mode

Don't know how reproducible it would be on a different device, but really appreciate all the help!