Hello team,

When I use Oak camera to run a YOLO object detection model solely, it works fine untill I create a socket server and client to send and receive the data i have in my YOLO model. However, when the server is integrated, it get stuck at this line "client_socket,client_address=s.accept()" where the camera stopped capturing and turned black. I tried to bring this line out of the pipeline loop but it did not solve the problem. Therefore, I want to know if the DepthAI pipeline has any conflict with creating socket server in python, or at least could you tell me some possible problems that make the camera stops working when that line is activated. For your info, I break the YOLO and server part into 2 files and test them and they work fine separately. I also integrated socket server to other models that provided by OAK camera but did not work either.

here are parts of the server and YOLO integration:

and here is the client side:

  • jakaskerl replied to this.
  • Hi KwinT
    Try this:

    Updated Server Code

    import cv2
    import depthai as dai
    import socket
    import json
    from ultralytics import YOLO
    
    # Create YOLO model
    model = YOLO('yolov8n-seg.pt')
    
    # Create a DepthAI pipeline
    pipeline = dai.Pipeline()
    
    # Define source and output
    camRgb = pipeline.create(dai.node.ColorCamera)
    xoutVideo = pipeline.create(dai.node.XLinkOut)
    xoutVideo.setStreamName("video")
    
    # Properties
    camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
    camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    camRgb.setVideoSize(640, 480)
    
    # Linking
    camRgb.video.link(xoutVideo.input)
    
    # Create Socket
    server_port = 1080
    host = socket.gethostname()
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.bind(('', server_port))
    server_socket.listen(5)
    print("Socket is listening")
    
    # Accept client connection
    client_socket, client_address = server_socket.accept()
    print(f"Got connection from {client_address}")
    
    # Connect to device and start pipeline
    with dai.Device(pipeline) as device:
        video_queue = device.getOutputQueue(name="video", maxSize=1, blocking=False)
    
        while True:
            video_frame = video_queue.get()
    
            # Get BGR frame from NV12 encoded video frame to show with OpenCV
            frame = video_frame.getCvFrame()
    
            # Perform YOLOv8 inference on the frame
            results = model(frame)
    
            # Get annotated frame with YOLOv8 predictions
            annotated_frame = results.show()
    
            # All data needed to be transferred
            output = []
            for box in results.xyxy[0]:
                x1, y1, x2, y2, _, class_id = [round(x) for x in box.tolist()]
                class_name = results.names[int(class_id)]
                output.append([x1, y1, x2, y2, class_name])
                print(output)
    
                # Send detection results to the client
                client_socket.send(json.dumps(output).encode())
    
            # Display the annotated frame
            cv2.imshow("YOLOv8 Inference", annotated_frame)
    
            if cv2.waitKey(1) == ord('q'):
                break
    
        client_socket.close()
        server_socket.close()
        print("Sockets closed")
    
    cv2.destroyAllWindows()

    Changes Made:

    1. Set the camera board socket to RGB: Adjusted camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A) to camRgb.setBoardSocket(dai.CameraBoardSocket.RGB) for the correct socket.
    2. Use the correct YOLO method: Changed the method to display the annotated frame to results.show() which is the assumed correct method from the YOLO library.
    3. JSON encoding: Encoded the output list to a JSON string before sending it over the socket for easier parsing on the client side.
    4. Socket handling: The server now accepts the connection before entering the main loop, and properly closes sockets upon exit.
    5. Removed recursive server calls: Removed the unnecessary server(s) calls inside the while loop.

    Updated Client Code

    import socket
    import json
    
    # Set up the client socket
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
    server_port = 1080
    host = socket.gethostname()
    
    # Connect to the server
    s.connect((host, server_port))
    print("Connected to the server")
    
    try:
        while True:
            # Receive data from the server
            data = s.recv(4096)
            if not data:
                break
            detections = json.loads(data.decode())
            print(detections)
    finally:
        print("Closing the client socket")
        s.close()

    Changes Made:

    1. Consistent data handling: Implemented a JSON-based protocol for the client to receive and decode the data.
    2. Exception handling: Added try-finally block to ensure the client socket is always closed properly.

    Thanks,
    Jaka

    Hi Jaka and team,

    I tried your suggestion and got ConnectionRefusedError: [Errno 111] Connection refused at line sock.connect((OAKIP, 1080)) in host.py file. Below are my terminals datastreamserver.py is oak.py and datastreanclient.py is host.py

    I did change to my ip address as well as my port number and I used 2 different local addresses to connect my linux machine to but it still did not work.

    one of the reasons I was thinking as a possible trouble that make the camera stop working in my earlier post might because I did not use MJPEG so it got stuck. I was hoping that if I could run the TCP streaming files that you suggested then I can use it. You mentioned that I could send my code inside the code block but I dont see where it is and also I cannot send my yolo model here because I got a message "uploading files of the type is not allowed". I tried what you suggested in this post https://discuss.luxonis.com/d/2395-oak-d but the terminal returned

    Thank you for your help!

      Hi KwinT
      You can create a code block by enclosing your code in tripple ` on each side (same character as for the one line code).

      Thanks,
      Jaka

      12 days later

      hi Jaka,
      I run "pip install ultralytics" to get yolo package first, then here is my server file:

      import cv2
      import depthai as dai
      import socket
      import json
      from ultralytics import YOLO
      
      \# Create YOLO model
      model = YOLO('yolov8n-seg.pt')
      
      \# Create a DepthAI pipeline
      pipeline = dai.Pipeline()
      
      \# Define source and output
      camRgb = pipeline.create(dai.node.ColorCamera)
      xoutVideo = pipeline.create(dai.node.XLinkOut)
      
      xoutVideo.setStreamName("video")
      
      \# Properties
      camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
      camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
      camRgb.setVideoSize(640,480)
      
      xoutVideo.input.setBlocking(False)
      xoutVideo.input.setQueueSize(1)
      
      \# Linking
      camRgb.video.link(xoutVideo.input)
      
      \# Create Socket
      server_port=1080
      host = socket.gethostname()
      s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      s.bind(('',server_port))
      print("socket successfully binding")
      s.listen(5)  # LISTEN FOR 5 INCOMING CONNECTION
      print("socket successfully created finally")
      
      def server(s):
          while True:
              print("client connection")
              try:
                  client_socket,client_address=s.accept()
                  print("got connection from ", client_address)
              except socket.timeout:
                  print("no client connection")
          
      
      \# Connect to device and start pipeline
      with dai.Device(pipeline) as device:
          video = device.getOutputQueue(name="video", maxSize=1, blocking=False)
      
          while True:
              
              videoIn = video.get()
      
              # Get BGR frame from NV12 encoded video frame to show with OpenCV
              frame = videoIn.getCvFrame()
      
              # Perform YOLOv8 inference on the frame
              results = model(frame)
      
              # Get annotated frame with YOLOv8 predictions
              annotated_frame = results[0].plot()
      
              # All data needed to be transferred 
              result=results[0]
              output=[]
      
              for box in result.boxes:
                  # coordinates of the bounding box
                  x1,y1,x2,y2=[round(x) for x in box.xyxy[0].tolist()]
                  # obj id 
                  class_id=box.cls[0].item()
                  id=result.names[class_id]
                  output.append([x1,y1,x2,y2, id])
                  print(output)
      
                  server(s)
                  
                  server.client_socket.send(id.encode())
      
                  
              # Display the annotated frame
              cv2.imshow("YOLOv8 Inference", annotated_frame)
      
              if cv2.waitKey(1) == ord('q'):
                  s.close()
                  print("socket closed")
                  break
      
      cv2.destroyAllWindows()
      
      '''
      
      and here is my client file:
      
      ''' import socket
      import json
      
      s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      
      server_port=1080
      
      host= socket.gethostname()
      
      \# receive data from the server
      
      s.connect((host,server_port))
      
      print("client is connecting")
      \#msg=s.recv(1024)
      name=s.recv(4096).decode()
      
      while len(name)>1:
      
      
      
      print("client closed")
      s.close()

        Hi KwinT
        Try this:

        Updated Server Code

        import cv2
        import depthai as dai
        import socket
        import json
        from ultralytics import YOLO
        
        # Create YOLO model
        model = YOLO('yolov8n-seg.pt')
        
        # Create a DepthAI pipeline
        pipeline = dai.Pipeline()
        
        # Define source and output
        camRgb = pipeline.create(dai.node.ColorCamera)
        xoutVideo = pipeline.create(dai.node.XLinkOut)
        xoutVideo.setStreamName("video")
        
        # Properties
        camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
        camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
        camRgb.setVideoSize(640, 480)
        
        # Linking
        camRgb.video.link(xoutVideo.input)
        
        # Create Socket
        server_port = 1080
        host = socket.gethostname()
        server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        server_socket.bind(('', server_port))
        server_socket.listen(5)
        print("Socket is listening")
        
        # Accept client connection
        client_socket, client_address = server_socket.accept()
        print(f"Got connection from {client_address}")
        
        # Connect to device and start pipeline
        with dai.Device(pipeline) as device:
            video_queue = device.getOutputQueue(name="video", maxSize=1, blocking=False)
        
            while True:
                video_frame = video_queue.get()
        
                # Get BGR frame from NV12 encoded video frame to show with OpenCV
                frame = video_frame.getCvFrame()
        
                # Perform YOLOv8 inference on the frame
                results = model(frame)
        
                # Get annotated frame with YOLOv8 predictions
                annotated_frame = results.show()
        
                # All data needed to be transferred
                output = []
                for box in results.xyxy[0]:
                    x1, y1, x2, y2, _, class_id = [round(x) for x in box.tolist()]
                    class_name = results.names[int(class_id)]
                    output.append([x1, y1, x2, y2, class_name])
                    print(output)
        
                    # Send detection results to the client
                    client_socket.send(json.dumps(output).encode())
        
                # Display the annotated frame
                cv2.imshow("YOLOv8 Inference", annotated_frame)
        
                if cv2.waitKey(1) == ord('q'):
                    break
        
            client_socket.close()
            server_socket.close()
            print("Sockets closed")
        
        cv2.destroyAllWindows()

        Changes Made:

        1. Set the camera board socket to RGB: Adjusted camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A) to camRgb.setBoardSocket(dai.CameraBoardSocket.RGB) for the correct socket.
        2. Use the correct YOLO method: Changed the method to display the annotated frame to results.show() which is the assumed correct method from the YOLO library.
        3. JSON encoding: Encoded the output list to a JSON string before sending it over the socket for easier parsing on the client side.
        4. Socket handling: The server now accepts the connection before entering the main loop, and properly closes sockets upon exit.
        5. Removed recursive server calls: Removed the unnecessary server(s) calls inside the while loop.

        Updated Client Code

        import socket
        import json
        
        # Set up the client socket
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        
        server_port = 1080
        host = socket.gethostname()
        
        # Connect to the server
        s.connect((host, server_port))
        print("Connected to the server")
        
        try:
            while True:
                # Receive data from the server
                data = s.recv(4096)
                if not data:
                    break
                detections = json.loads(data.decode())
                print(detections)
        finally:
            print("Closing the client socket")
            s.close()

        Changes Made:

        1. Consistent data handling: Implemented a JSON-based protocol for the client to receive and decode the data.
        2. Exception handling: Added try-finally block to ensure the client socket is always closed properly.

        Thanks,
        Jaka

        it got some errors but i made some changes into your files and it works now. thank you so much!