Automatic startup

Simone

good morning, I wanted to ask if there was a possibility to structure a project in the following way: I would need to load directly inside the cameras e.g. a generic people recognition pattern and that as soon as the OAK D PoE camera receives power it starts to transmit its data to a server the pc, whether it is then on or not is not important. (All systems will have static iPs)

I tried to follow the guide on depthai-experiments TCP etc but it only works if I keep both .py host and oak started, but precisely I would need to compile the camera software only once is that possible?
in the sense that even if I disconnect the camera and then reconnect it, does it remain in memory or is it necessary to always run oak.py again?

Thanks
For example I should convert this code and adapt it to an oak.py to make it flashable on a PoELuxonis camera, like luxonis/depthai-experimentstree/662ed31b6d375f9106c00824052be669bb761b50/gen2-poe-tcp-streaming/poe-host-yolo
`#INIZIALIZZAZIONE MODELLO E PIPELINE
self.nnBlobPath = setting.model
self.labelMap = ["person"]
self.syncNN = True
self.pipeline = dai.Pipeline()

Define sources and outputs

self.camRgb = self.pipeline.create(dai.node.ColorCamera)
self.objectTracker = self.pipeline.create(dai.node.ObjectTracker)
self.spatialDetectionNetwork = self.pipeline.create(dai.node.MobileNetSpatialDetectionNetwork)
self.monoLeft = self.pipeline.create(dai.node.MonoCamera)
self.monoRight = self.pipeline.create(dai.node.MonoCamera)
self.stereo = self.pipeline.create(dai.node.StereoDepth)

Create X-Outlink

self.xoutRgb = self.pipeline.create(dai.node.XLinkOut)
self.xoutNN = self.pipeline.create(dai.node.XLinkOut)
self.xoutDepth = self.pipeline.create(dai.node.XLinkOut)

    self.objectTracker.setDetectionLabelsToTrack([15])  # track only person
    # possible tracking types: ZERO_TERM_COLOR_HISTOGRAM, ZERO_TERM_IMAGELESS, SHORT_TERM_IMAGELESS, SHORT_TERM_KCF
    self.objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
    # take the smallest ID when new object is tracked, possible options: SMALLEST_ID, UNIQUE_ID
    self.objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.SMALLEST_ID)

    # Setting Stream Name
    self.xoutRgb.setStreamName("rgb")
    self.xoutNN.setStreamName("detections")
    self.xoutDepth.setStreamName("depth")
    # Setting Properties
    self.camRgb.setPreviewSize(setting.x, setting.y)
    self.camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    self.camRgb.setInterleaved(False)
    self.camRgb.setFps(40)
    self.camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
    # Setting stereo resolution
    self.monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    self.monoLeft.setCamera("left")
    self.monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    self.monoRight.setCamera("right")

    #self.camRgb.setIspScale(1, 3) # You don't need to downscale (4k -> 720P) video frames
    #Squeeze the frame
    self.camRgb.setPreviewKeepAspectRatio(False)
    # Setting node configs
    self.stereo.initialConfig.setConfidenceThreshold(setting.sCT)
    # Align depth map to the perspective of RGB camera, on which inference is done
    self.stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
    self.stereo.setSubpixel(False)
    self.stereo.setOutputSize(self.monoLeft.getResolutionWidth(), self.monoLeft.getResolutionHeight())
    # Setting parameters for Spatial Detection
    self.spatialDetectionNetwork.setBlobPath(self.nnBlobPath)
    self.spatialDetectionNetwork.setConfidenceThreshold(setting.CO)
    self.spatialDetectionNetwork.input.setBlocking(False)
    self.spatialDetectionNetwork.setBoundingBoxScaleFactor(0.9)
    self.spatialDetectionNetwork.setDepthLowerThreshold(setting.minDet)
    self.spatialDetectionNetwork.setDepthUpperThreshold(setting.maxDet) #distanza di riconoscimento
    # Linking
    self.monoLeft.out.link(self.stereo.left)
    self.monoRight.out.link(self.stereo.right)
    self.camRgb.preview.link(self.spatialDetectionNetwork.input)
    self.objectTracker.passthroughTrackerFrame.link(self.xoutRgb.input)
    self.objectTracker.out.link(self.xoutNN.input)

    if self.syncNN:
        self.spatialDetectionNetwork.passthrough.link(self.objectTracker.inputTrackerFrame)
    else:
        self.camRgb.preview.link(self.objectTracker.inputTrackerFrame)

    self.spatialDetectionNetwork.passthrough.link(self.objectTracker.inputDetectionFrame)
    self.spatialDetectionNetwork.out.link(self.objectTracker.inputDetections)        
    self.stereo.depth.link(self.spatialDetectionNetwork.inputDepth)

Simone

Simone
my new code here:
`import depthai as dai
import time
import os
import json
import blobconverter

FLASH_APP = False

pipeline = dai.Pipeline()

Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)
objectTracker = pipeline.create(dai.node.ObjectTracker)
spatialDetectionNetwork = pipeline.create(dai.node.MobileNetSpatialDetectionNetwork)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)

Create X-Outlink

xoutRgb = pipeline.create(dai.node.XLinkOut)
xoutNN = pipeline.create(dai.node.XLinkOut)
xoutDepth = pipeline.create(dai.node.XLinkOut)

Setting Stream Name

xoutRgb.setStreamName("rgb")
xoutNN.setStreamName("detections")
xoutDepth.setStreamName("depth")

objectTracker.setDetectionLabelsToTrack([15])
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.SMALLEST_ID)

Setting Properties

camRgb.setPreviewSize(setting.x, setting.y)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
camRgb.setFps(40)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
camRgb.setPreviewKeepAspectRatio(False)

Setting stereo resolution

monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setCamera("left")
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setCamera("right")

stereo.initialConfig.setConfidenceThreshold(setting.sCT)

stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
stereo.setSubpixel(False)
stereo.setOutputSize(monoLeft.getResolutionWidth(), monoLeft.getResolutionHeight())

Network specific settings

spatialDetectionNetwork.setBlobPath(setting.model)
spatialDetectionNetwork.setConfidenceThreshold(setting.CO)
spatialDetectionNetwork.input.setBlocking(False)
spatialDetectionNetwork.setBoundingBoxScaleFactor(0.9)
spatialDetectionNetwork.setDepthLowerThreshold(setting.minDet)
spatialDetectionNetwork.setDepthUpperThreshold(setting.maxDet) #distanza di riconoscimento

Linking

monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
camRgb.preview.link(spatialDetectionNetwork.input)
objectTracker.passthroughTrackerFrame.link(xoutRgb.input)
objectTracker.out.link(xoutNN.input)

Create Script node that will handle TCP communication

script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

spatialDetectionNetwork.passthrough.link(script.inputs['frame_in'])
spatialDetectionNetwork.out.link(script.inputs['detection_in'])
stereo.depth.link(script.inputs['stereo_in'])

script.setScript("""
import socket
import time
import threading
node.warn("Server up")

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", 5000))
server.listen()

while True:
conn, client = server.accept()
node.warn(f"Connected to client IP: {client}")

try:
    while True:
        detections = node.io["detection_in"].get().detections
        node.warn(detections)
        img = node.io["frame_in"].get()
        node.warn('Received frame + dets')
        img_data = img.getData()
        ts = img.getTimestamp()
        det_arr = []
        for det in detections:
            det_arr.append(f"{det.label};{(det.spatialCoordinates.z/1000):.1f};{det.xmin:.4f};{det.ymin:.4f};{det.xmax:.4f};{det.ymax:.4f}")
        det_str = "|".join(det_arr)

        header = f"IMG {ts.total_seconds()} {len(img_data)} {len(det_str)}".ljust(32)
        node.warn(f'>{header}<')
        conn.send(bytes(header, encoding='ascii'))
        if 0 < len(det_arr):
            conn.send(bytes(det_str, encoding='ascii'))
        conn.send(img_data)
except Exception as e:
    node.warn("Client disconnected")

""")

if FLASH_APP: # flash the app for Standalone mode
(f, bl) = dai.DeviceBootloader.getFirstAvailableDevice()
bootloader = dai.DeviceBootloader(bl)
progress = lambda p : print(f'Flashing progress: {p*100:.1f}%')
bootloader.flash(progress, pipeline)
else: # Run the app in peripheral mode
with dai.Device(pipeline) as device:
print("Connected")
while True:
time.sleep(1)
`

I tried to adapt it but it crashes at the point after it gets the connection from host.py as soon as it tries to read detection it doesn't go any further and the try catch doesn't print any error

jakaskerl

Simone good morning, I wanted to ask if there was a possibility to structure a project in the following way: I would need to load directly inside the cameras e.g. a generic people recognition pattern and that as soon as the OAK D PoE camera receives power it starts to transmit its data to a server the pc, whether it is then on or not is not important. (All systems will have static iPs)

Possible if you flash the pipeline on the device.

The second script still has the links to output queues. You need to remove everything to do with XLinkOUT/IN. You can run the app before flashing it by just making an empty with dai.Device(): while True loop. This should print at least something.
Again, check that all links are valid.

Thanks,
Jaka

Simone

jakaskerl
I deleted those Xin/OUT lines, and yes I already had that empty loop you recommended running but still it keeps getting stuck here “detections = node.io[”detection_in"].get().detections”


# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
objectTracker = pipeline.create(dai.node.ObjectTracker)
spatialDetectionNetwork = pipeline.create(dai.node.MobileNetSpatialDetectionNetwork)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)


objectTracker.setDetectionLabelsToTrack([15])
objectTracker.setTrackerType(dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM)
objectTracker.setTrackerIdAssignmentPolicy(dai.TrackerIdAssignmentPolicy.SMALLEST_ID)

# Setting Properties
camRgb.setPreviewSize(setting.x, setting.y)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
camRgb.setFps(40)
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
camRgb.setPreviewKeepAspectRatio(False)
# Setting stereo resolution
monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoLeft.setCamera("left")
monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
monoRight.setCamera("right")

stereo.initialConfig.setConfidenceThreshold(setting.sCT)

stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
stereo.setSubpixel(False)
stereo.setOutputSize(monoLeft.getResolutionWidth(), monoLeft.getResolutionHeight())
# Network specific settings
spatialDetectionNetwork.setBlobPath(setting.model)
spatialDetectionNetwork.setConfidenceThreshold(setting.CO)
spatialDetectionNetwork.input.setBlocking(False)
spatialDetectionNetwork.setBoundingBoxScaleFactor(0.9)
spatialDetectionNetwork.setDepthLowerThreshold(setting.minDet)
spatialDetectionNetwork.setDepthUpperThreshold(setting.maxDet) #distanza di riconoscimento

# Linking
monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
camRgb.preview.link(spatialDetectionNetwork.input)

# Create Script node that will handle TCP communication
script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

spatialDetectionNetwork.passthrough.link(script.inputs['frame_in'])
spatialDetectionNetwork.out.link(script.inputs['detection_in'])        
stereo.depth.link(script.inputs['stereo_in'])

script.setScript("""
import socket
import time
import threading
node.warn("Server up")


server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", 5000))
server.listen()

while True:
    conn, client = server.accept()
    node.warn(f"Connected to client IP: {client}")

    try:
        while True:
            detections = node.io["detection_in"].get().detections (problem here)
            node.warn(detections)
            img = node.io["frame_in"].get()
            node.warn('Received frame + dets')
            img_data = img.getData()
            ts = img.getTimestamp()
            det_arr = []
            for det in detections:
                det_arr.append(f"{det.label};{(det.spatialCoordinates.z/1000):.1f};{det.xmin:.4f};{det.ymin:.4f};{det.xmax:.4f};{det.ymax:.4f}")
            det_str = "|".join(det_arr)

            header = f"IMG {ts.total_seconds()} {len(img_data)} {len(det_str)}".ljust(32)
            node.warn(f'>{header}<')
            conn.send(bytes(header, encoding='ascii'))
            if 0 < len(det_arr):
                conn.send(bytes(det_str, encoding='ascii'))
            conn.send(img_data)
    except Exception as e:
        node.warn("Client disconnected")
""")

I am using this blob -> mobilenet-ssd_openvino_2021.4_5shave.blob

Simone

I managed to get it to start (following the linked code), modifying and readjusting so that I can also have the distance of the object recognized, but now I am facing another problem, after a total of about 50 seconds(since I run host.py) it seems that it fills up the RAM? or at any rate the CPU of the camera (OAK PoE) cannot handle the load and it is as if it resets, in fact after a few seconds it redoes the “click” and starts again.
By lowering the fps I have achieved almost 60s but still little I need an “almost infinite autonomy” at least as long as the camera is attached

`import depthai as dai
import time
import blobconverter

FLASH_APP = True

pipeline = dai.Pipeline()

Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)
#spatialDetectionNetwork = pipeline.create(dai.node.MobileNetSpatialDetectionNetwork)
spatialDetectionNetwork = pipeline.create(dai.node.YoloSpatialDetectionNetwork)
monoLeft = pipeline.create(dai.node.MonoCamera)
monoRight = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)

Properties

camRgb.setPreviewSize(640, 352)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setInterleaved(False)
camRgb.setFps(40)
camRgb.setPreviewKeepAspectRatio(False)

#stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
stereo.setOutputSize(monoLeft.getResolutionWidth(), monoLeft.getResolutionHeight())
stereo.setSubpixel(True)

#nnBlobPath = "mobilenet-ssd_openvino_2021.4_5shave.blob"

Network specific settings

spatialDetectionNetwork.setBlobPath(blobconverter.from_zoo('yolov8n_coco_640x352', zoo_type='depthai'))
#spatialDetectionNetwork.setBlobPath(nnBlobPath)
spatialDetectionNetwork.setConfidenceThreshold(0.5)
spatialDetectionNetwork.input.setBlocking(False)
spatialDetectionNetwork.setBoundingBoxScaleFactor(0.5)
spatialDetectionNetwork.setDepthLowerThreshold(100)
spatialDetectionNetwork.setDepthUpperThreshold(5000)

Yolo specific parameters

spatialDetectionNetwork.setNumClasses(80)
spatialDetectionNetwork.setCoordinateSize(4)
spatialDetectionNetwork.setAnchors([10,14, 23,27, 37,58, 81,82, 135,169, 344,319])
spatialDetectionNetwork.setAnchorMasks({ "side26": [1,2,3], "side13": [3,4,5] })
spatialDetectionNetwork.setIouThreshold(0.5)

Linking

monoLeft.out.link(stereo.left)
monoRight.out.link(stereo.right)
camRgb.preview.link(spatialDetectionNetwork.input)

Create Script node that will handle TCP communication

script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

Link outputs (RGB stream, NN output) to the Script node

stereo.depth.link(spatialDetectionNetwork.inputDepth)
spatialDetectionNetwork.passthrough.link(script.inputs['frame_in'])
spatialDetectionNetwork.out.link(script.inputs['detection_in'])

script.setScript("""
import socket
import time
import threading
node.warn("Server up")

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("0.0.0.0", 5000))
server.listen()

while True:
conn, client = server.accept()
node.warn(f"Connected to client IP: {client}")

try:
    while True:
        detections = node.io["detection_in"].get().detections
        img = node.io["frame_in"].get()
        node.warn('Received frame + dets')
        img_data = img.getData()
        ts = img.getTimestamp()

        det_arr = []
        for det in detections:
            det_arr.append(f"{det.label};{(det.confidence*100):.1f};{det.spatialCoordinates.z};{det.xmin:.4f};{det.ymin:.4f};{det.xmax:.4f};{det.ymax:.4f}")
        det_str = "|".join(det_arr)

        header = f"IMG {ts.total_seconds()} {len(img_data)} {len(det_str)}".ljust(32)
        node.warn(f'>{header}<')
        conn.send(bytes(header, encoding='ascii'))
        if 0 < len(det_arr):
            conn.send(bytes(det_str, encoding='ascii'))
        conn.send(img_data)
except Exception as e:
    node.warn("Client disconnected")

""")

jakaskerl

Simone
Hey, try using the latest develop branch and install that depthai version. Found a bug that would overflow the memory if messages were printed during runtime.

Thanks,
Jaka

Simone

Hi jakaskerl, my version of depthai is -> 2.29.0.0, How can I install the version you say? have you by any chance tried my code if successful or after a while it crashes for you too?

jakaskerl

Hi Simone
You can install it with python examples/install_requirements.py after you have switched to develop branch (git checkout develop git pull --recurse-submodules.

Thanks,
Jaka

Simone

Hi jakaskerl Yes it is true it seems to work now, thank you, I suggest to put an example or readme in the “depthai-experiments” section/gen2-poe-tcp-streaming” as it might be useful to other users as well.
thank you again

Simone

jakaskerl Sorry to bother you again, maybe it's less related to the post, but I noticed that my cameras (OAK D PoE) when I connect them to the network before they turn on it takes about 40s (they turn on by themselves because I managed to flash the pipeline) however about after that time they make a small audible “click” and then they start.

is this time somehow optimizable/reducible?
thanks in advance

Simone

jakaskerl

Simone
Set watchdogInitialDelayMs to something like 10000:
https://docs.luxonis.com/software/depthai-components/device/#Device-Watchdog

Thanks,
Jaka