does DepthAI pipeline have conflict with creating socket server (in Python)?

KwinT

Hello team,

When I use Oak camera to run a YOLO object detection model solely, it works fine untill I create a socket server and client to send and receive the data i have in my YOLO model. However, when the server is integrated, it get stuck at this line "client_socket,client_address=s.accept()" where the camera stopped capturing and turned black. I tried to bring this line out of the pipeline loop but it did not solve the problem. Therefore, I want to know if the DepthAI pipeline has any conflict with creating socket server in python, or at least could you tell me some possible problems that make the camera stops working when that line is activated. For your info, I break the YOLO and server part into 2 files and test them and they work fine separately. I also integrated socket server to other models that provided by OAK camera but did not work either.

here are parts of the server and YOLO integration:

and here is the client side:

jakaskerl

Hi KwinT
The depthai library should have no conflict with the socket server. You can actually run the server inside the script node of the device ->https://github.com/luxonis/depthai-experiments/blob/master/gen2-poe-tcp-streaming/oak.py

Check this tcp streaming experiment.

Not sure where the problem in your code is, if you have problems, please paste the code inside the code block so I can debug it locally.

Thanks,
Jaka

KwinT

Hi Jaka and team,

I tried your suggestion and got ConnectionRefusedError: [Errno 111] Connection refused at line sock.connect((OAKIP, 1080)) in host.py file. Below are my terminals datastreamserver.py is oak.py and datastreanclient.py is host.py

I did change to my ip address as well as my port number and I used 2 different local addresses to connect my linux machine to but it still did not work.

KwinT

one of the reasons I was thinking as a possible trouble that make the camera stop working in my earlier post might because I did not use MJPEG so it got stuck. I was hoping that if I could run the TCP streaming files that you suggested then I can use it. You mentioned that I could send my code inside the code block but I dont see where it is and also I cannot send my yolo model here because I got a message "uploading files of the type is not allowed". I tried what you suggested in this post https://discuss.luxonis.com/d/2395-oak-d but the terminal returned

Thank you for your help!

jakaskerl

Hi KwinT
You can create a code block by enclosing your code in tripple ` on each side (same character as for the one line code).

Thanks,
Jaka

KwinT

hi Jaka,
I run "pip install ultralytics" to get yolo package first, then here is my server file:

import cv2
import depthai as dai
import socket
import json
from ultralytics import YOLO

\# Create YOLO model
model = YOLO('yolov8n-seg.pt')

\# Create a DepthAI pipeline
pipeline = dai.Pipeline()

\# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
xoutVideo = pipeline.create(dai.node.XLinkOut)

xoutVideo.setStreamName("video")

\# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setVideoSize(640,480)

xoutVideo.input.setBlocking(False)
xoutVideo.input.setQueueSize(1)

\# Linking
camRgb.video.link(xoutVideo.input)

\# Create Socket
server_port=1080
host = socket.gethostname()
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('',server_port))
print("socket successfully binding")
s.listen(5)  # LISTEN FOR 5 INCOMING CONNECTION
print("socket successfully created finally")

def server(s):
    while True:
        print("client connection")
        try:
            client_socket,client_address=s.accept()
            print("got connection from ", client_address)
        except socket.timeout:
            print("no client connection")
    

\# Connect to device and start pipeline
with dai.Device(pipeline) as device:
    video = device.getOutputQueue(name="video", maxSize=1, blocking=False)

    while True:
        
        videoIn = video.get()

        # Get BGR frame from NV12 encoded video frame to show with OpenCV
        frame = videoIn.getCvFrame()

        # Perform YOLOv8 inference on the frame
        results = model(frame)

        # Get annotated frame with YOLOv8 predictions
        annotated_frame = results[0].plot()

        # All data needed to be transferred 
        result=results[0]
        output=[]

        for box in result.boxes:
            # coordinates of the bounding box
            x1,y1,x2,y2=[round(x) for x in box.xyxy[0].tolist()]
            # obj id 
            class_id=box.cls[0].item()
            id=result.names[class_id]
            output.append([x1,y1,x2,y2, id])
            print(output)

            server(s)
            
            server.client_socket.send(id.encode())

            
        # Display the annotated frame
        cv2.imshow("YOLOv8 Inference", annotated_frame)

        if cv2.waitKey(1) == ord('q'):
            s.close()
            print("socket closed")
            break

cv2.destroyAllWindows()

'''

and here is my client file:

''' import socket
import json

s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_port=1080

host= socket.gethostname()

\# receive data from the server

s.connect((host,server_port))

print("client is connecting")
\#msg=s.recv(1024)
name=s.recv(4096).decode()

while len(name)>1:



print("client closed")
s.close()

jakaskerl

Hi KwinT
Try this:

Updated Server Code

import cv2
import depthai as dai
import socket
import json
from ultralytics import YOLO

# Create YOLO model
model = YOLO('yolov8n-seg.pt')

# Create a DepthAI pipeline
pipeline = dai.Pipeline()

# Define source and output
camRgb = pipeline.create(dai.node.ColorCamera)
xoutVideo = pipeline.create(dai.node.XLinkOut)
xoutVideo.setStreamName("video")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setVideoSize(640, 480)

# Linking
camRgb.video.link(xoutVideo.input)

# Create Socket
server_port = 1080
host = socket.gethostname()
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('', server_port))
server_socket.listen(5)
print("Socket is listening")

# Accept client connection
client_socket, client_address = server_socket.accept()
print(f"Got connection from {client_address}")

# Connect to device and start pipeline
with dai.Device(pipeline) as device:
    video_queue = device.getOutputQueue(name="video", maxSize=1, blocking=False)

    while True:
        video_frame = video_queue.get()

        # Get BGR frame from NV12 encoded video frame to show with OpenCV
        frame = video_frame.getCvFrame()

        # Perform YOLOv8 inference on the frame
        results = model(frame)

        # Get annotated frame with YOLOv8 predictions
        annotated_frame = results.show()

        # All data needed to be transferred
        output = []
        for box in results.xyxy[0]:
            x1, y1, x2, y2, _, class_id = [round(x) for x in box.tolist()]
            class_name = results.names[int(class_id)]
            output.append([x1, y1, x2, y2, class_name])
            print(output)

            # Send detection results to the client
            client_socket.send(json.dumps(output).encode())

        # Display the annotated frame
        cv2.imshow("YOLOv8 Inference", annotated_frame)

        if cv2.waitKey(1) == ord('q'):
            break

    client_socket.close()
    server_socket.close()
    print("Sockets closed")

cv2.destroyAllWindows()

Changes Made:

Set the camera board socket to RGB: Adjusted camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A) to camRgb.setBoardSocket(dai.CameraBoardSocket.RGB) for the correct socket.
Use the correct YOLO method: Changed the method to display the annotated frame to results.show() which is the assumed correct method from the YOLO library.
JSON encoding: Encoded the output list to a JSON string before sending it over the socket for easier parsing on the client side.
Socket handling: The server now accepts the connection before entering the main loop, and properly closes sockets upon exit.
Removed recursive server calls: Removed the unnecessary server(s) calls inside the while loop.

Updated Client Code

import socket
import json

# Set up the client socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

server_port = 1080
host = socket.gethostname()

# Connect to the server
s.connect((host, server_port))
print("Connected to the server")

try:
    while True:
        # Receive data from the server
        data = s.recv(4096)
        if not data:
            break
        detections = json.loads(data.decode())
        print(detections)
finally:
    print("Closing the client socket")
    s.close()

Changes Made:

Consistent data handling: Implemented a JSON-based protocol for the client to receive and decode the data.
Exception handling: Added try-finally block to ensure the client socket is always closed properly.

Thanks,
Jaka

KwinT

it got some errors but i made some changes into your files and it works now. thank you so much!