Hi, I am attempting to create a pipeline that outputs a pointcloud and an image when a HTTP request is sent to a server hosted in the Script node of the camera pipeline. Due to the camera operating on a somewhat unstable network, I am trying to run the camera in 'Standalone' mode. I get the pipeline up and running properly, but I face the following issues: 1) When a request is sent, the retrieved image is no real-time and has \~5sec latency. How can this be mitigated and images retrieved closer to real-time? 2) The image and pipeline seem to not be properly synced, I had to remove the sync node in order to use the Script node. Is there any known way to link them up to receive synced pointclouds and images in the Script node? The code I used is as follows: Setup camera and pipeline in standalone mode `import depthai as dai` `import time` `from datetime import timedelta` `def create_pipeline(camera_ip, camera_port, script_path): ` `pipeline = dai.Pipeline()` `RGB_SOCKET = dai.CameraBoardSocket.CAM_C ` `TOF_SOCKET = dai.CameraBoardSocket.CAM_A` ` # Create nodes ` `camRgb = pipeline.create(dai.node.ColorCamera) ` `tof = pipeline.create(dai.node.ToF) ` `camTof = pipeline.create(dai.node.Camera) ` `align = pipeline.create(dai.node.ImageAlign) ` `pointcloud = pipeline.create(dai.node.PointCloud) ` `script = pipeline.create(dai.node.Script)` ` # Configure nodes ` `camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_800_P) camRgb.setBoardSocket(RGB_SOCKET) ` `camRgb.setIspScale(1,2)` `FPS = 1 ` `camRgb.setFps(FPS)` ` # ToF settings ` `camTof.setFps(FPS) ` `camTof.setImageOrientation(dai.CameraImageOrientation.ROTATE_180_DEG) camTof.setBoardSocket(TOF_SOCKET)` ` # Higher number => faster processing. 1 shave core can do 30FPS. ` `tof.setNumShaves(1)` ` # Median filter, kernel size 3x3 ` `tof.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_3x3)` `tofConfig = tof.initialConfig.get() # Temporal filter averages shuffle/non-shuffle frequencies tofConfig.enablePhaseShuffleTemporalFilter = True # Phase unwrapping, for longer range. tofConfig.phaseUnwrappingLevel = 2 ` `tofConfig.phaseUnwrapErrorThreshold = 500 ` `tof.initialConfig.set(tofConfig) ` `pointcloud.initialConfig.setSparse(False)` `script.setProcessor(dai.ProcessorType.LEON_CSS) ` `with open(script_path, "r") as f: ` ` script_content = f.read() ` `script_content = script_content.replace("IP_PLACEHOLDER",camera_ip).replace("PORT_PLACEHOLDER",str(camera_port)) ` `print(f"Script Content: {script_content}") ` `script.setScript(script_content) ` `# Linking ` `print("Linking nodes...") ` `camRgb.isp.link(script.inputs["rgb"]) ` `camRgb.isp.link(align.inputAlignTo) ` `camTof.raw.link(tof.input) ` `tof.depth.link(align.input) ` `align.outputAligned.link(pointcloud.inputDepth) pointcloud.outputPointCloud.link(script.inputs["pcl"])` ` script.outputs['out'].link(camRgb.inputControl) ` `print("Nodes linked") return pipeline` `if __name__ == "__main__": ` `camera_ip = '192.168.100.100' ` `camera_port = '8080' ` `script_path = "./http_server_script.py"` `pipeline = create_pipeline(camera_ip, camera_port, script_path) ` `devices = dai.DeviceBootloader.getAllAvailableDevices() ` `print(devices) ` `for device in devices: ` ` if device.name == camera_ip: ` ` bootloader = dai.DeviceBootloader(device) ` `progress = lambda p : print(f'Flashing progress: {p*100:.1f}%') ` `bootloader.flash(progress, pipeline)` HTTP script for Script node: `from http.server import BaseHTTPRequestHandler` `import socketserver` `import socket` `import fcntl` `import struct` `ctrl = CameraControl() # Assuming CameraControl can handle point cloudsctrl.setCaptureStill(True)` `def get_ip_address(ifname): ` ` s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) ` ` return socket.inet_ntoa(fcntl.ioctl( s.fileno(), -1071617759, # SIOCGIFADDR struct.pack('256s', ifname[:15].encode()) )[20:24])` `class HTTPHandler(BaseHTTPRequestHandler): ` ` def do_GET(self): ` ` if self.path == '/': ` ` self.send_response(200) ` ` self.end_headers() ` ` self.wfile.write(b'[DepthAI] Hello, world!Click here for an imageClick here for a pointcloud') ` ` elif self.path == '/img': ` ` # Capture the image from the camera ` ` node.io['out'].send(ctrl) ` ` image = node.io['rgb'].get() ` ` self.send_response(200) ` ` self.send_header('Content-Type', 'image/jpeg') ` ` self.send_header('Content-Length', str(len(image.getData()))) ` ` self.end_headers() ` ` print(f"Image size: {len(image.getData())}") ` ` self.wfile.write(image.getData()) ` ` elif self.path == '/pointcloud': ` ` # Capture the point cloud data` ` node.io['out'].send(ctrl) ` ` pointcloud = node.io['pcl'].get() ` ` self.send_response(200) ` ` self.send_header('Content-Type', 'application/octet-stream') # Binary file self.send_header('Content-Length', str(len(pointcloud.getData()))) self.end_headers() ` ` self.wfile.write(pointcloud.getData())` ` else: ` ` self.send_response(404) ` ` self.end_headers() ` ` self.wfile.write(b'Url not found...')` `with socketserver.TCPServer(("IP_PLACEHOLDER", PORT_PLACEHOLDER), HTTPHandler) as httpd:` ` node.warn(f"Serving at {get_ip_address('re0')}:{PORT_PLACEHOLDER}") ` ` httpd.serve_forever()`

> @"gherodes"#p26054 When a request is sent, the retrieved image is no real-time and has ~5sec latency. How can this be mitigated and images retrieved closer to real-time? Make sure you clear the queues constantly or set queue sizes to 1 to avoid reading old frames. Each node input has a queue size and blocking toggle. Eg: ``` script.inputs["rgb"].setBlocking(False) script.inputs["rgb"].setQueueSize(1) ``` Sync them based on timestamps. If they differ too much, make sure you discard the frames. Thanks, Jaka

Hi Jaka, Thanks for the fast reply, I applied your suggestions and it seems to have both synced up the frame and pointcloud aswell as now it retrieves them relatively real-time. If I wanted to add logging using the SystemLogger node, should I just link it up to the Script Node identically to the previous nodes? Thanks, Georg

Hi again Jaka, I have encountered a new issue with the pipeline above after implementing the suggested fixes: The camera appears to be restarting exactly every 6min 55sec. I confirmed this by querying the HTTP server for an image-pointcloud pair with a 1sec interval and retrieving the timestamps of the images. Initially I suspected there might have been an issue with the PoE power delivery so I tested with multiple PoE switches and injectors (90W Dahua PoE switch, 240W Planet Industrial PoE switch, 24W Ubiquity PoE injector) and saw the same behaviour on all these devices. I further tested this on 2 separate OAK-D Short Range PoE with ToF cameras in separate systems (separate switches/injectors/etc) to rule out faults in any of the hardware. Finally I suspect this is a RAM/memory issue. With the FPS of the pipeline set to 10, the crashes occurred approximately every 7 minutes, after lowering the FPS to 1 the pipeline has thus far not crashed for approx 40minutes. As you can see, it appears that some sort of cache or buffer is building up and not being cleared. Do you have any suggestions on how to clear any buffers that might be building up in this pipeline in standalone mode? I am implementing some more logging functionality and hope to be able to provide more logs/RAM usage shortly. Kind regards, Georg

@"gherodes"#p26079 > @"gherodes"#p26079 If I wanted to add logging using the SystemLogger node, should I just link it up to the Script Node identically to the previous nodes? Not sure if needed. The logger should have timestamped data. You probably don't need to perfectly sync the logs to the frames. > @"gherodes"#p26081 Do you have any suggestions on how to clear any buffers that might be building up in this pipeline in standalone mode? Can you create a mre for both side, so I can check locally with tested POE switch? You can mail the code if you want.

Hi Jakaskerl, For the tests I ran the 2 scripts above on the camera (along with the suggestions you provided in your first comment applied to both RGB and PCL inputs to the Script node). It runs on an OAK-D SR ToF camera with bootloader version 0.0.28 and `depthai-python` version 2.28.0. To retrieve images from the camera the script attached to this comment is used. `import requests` `import cv2` `import numpy as np` `import open3d as o3d` `import time` `def yuv420_to_bgr(yuv_bytes, width, height):` ` # Size of Y, U, and V planes` ` y_size = width * height` ` uv_size = (width // 2) * (height // 2)` ` # Split the YUV data into Y, U, and V planes` ` y_plane = np.frombuffer(yuv_bytes[0:y_size], dtype=np.uint8).reshape(` ` (height, width)` ` )` ` u_plane = np.frombuffer(` ` yuv_bytes[y_size : y_size + uv_size], dtype=np.uint8` ` ).reshape((height // 2, width // 2))` ` v_plane = np.frombuffer(yuv_bytes[y_size + uv_size :], dtype=np.uint8).reshape(` ` (height // 2, width // 2)` ` )` ` # Upsample the U and V planes to the size of Y plane` ` u_plane_up = cv2.resize(u_plane, (width, height), interpolation=cv2.INTER_LINEAR)` ` v_plane_up = cv2.resize(v_plane, (width, height), interpolation=cv2.INTER_LINEAR)` ` # Stack the Y, U, and V planes into a YUV image` ` yuv_img = np.stack((y_plane, u_plane_up, v_plane_up), axis=-1)` ` # Convert YUV to RGB using OpenCV` ` bgr_img = cv2.cvtColor(yuv_img, cv2.COLOR_YUV2BGR)` ` return bgr_img` `def retrieve_image_and_pointcloud(server_url):` ` # Define the endpoints for image and pointcloud` ` image_url = f"{server_url}/img"` ` pointcloud_url = f"{server_url}/pointcloud"` ` ping_url = f"{server_url}/ping"` ` # Retrieve the image` ` # print(f"Requesting image from {image_url}")` ` t0 = time.time()` ` image_response = requests.get(image_url)` ` t1 = time.time()` ` print(f"Time to retrieve image: {t1 - t0:.4f} seconds")` ` t2 = time.time()` ` pointcloud_response = requests.get(pointcloud_url)` ` t3 = time.time()` ` print(f"Time to retrieve pointcloud: {t3 - t2:.4f} seconds")` ` t4 = time.time()` ` ping_response = requests.get(ping_url)` ` t5 = time.time()` ` print(f"Time to ping server: {t5 - t4:.4f} seconds")` ` print(f"Total retrieval time: {t5 - t0:.4f} seconds")` ` if image_response.status_code == 200:` ` height = 400` ` width = 640` ` yuv_bytes = image_response.content` ` # Convert YUV420 to RGB` ` bgr_image_array = yuv420_to_bgr(yuv_bytes, width, height)` ` cv2.imwrite("image.jpg", bgr_image_array)` ` else:` ` print(f"Failed to retrieve image. Status code: {image_response.status_code}")` ` if pointcloud_response.status_code == 200:` ` # Save the point cloud as a PLY file` ` pcl_buffer = np.frombuffer(pointcloud_response.content, np.float32)` ` pcl_np = np.reshape(pcl_buffer, (-1, 3))` ` print(f"Point cloud shape: {pcl_np.shape}")` ` colors_np = (` ` np.reshape(cv2.cvtColor(bgr_image_array, cv2.COLOR_BGR2RGB), (-1, 3))` ` / 255.0` ` )` ` # Save the point cloud as a PLY file` ` pcl = o3d.geometry.PointCloud()` ` pcl.points = o3d.utility.Vector3dVector(pcl_np)` ` pcl.colors = o3d.utility.Vector3dVector(colors_np)` ` o3d.io.write_point_cloud("pointcloud.ply", pcl)` `if __name__ == "__main__":` ` # Example usage` ` server_url = "http://192.168.100.100:8080" # Replace this with the actual server address` ` while True:` ` t0 = time.time()` ` retrieve_image_and_pointcloud(server_url)` ` time.sleep(1)` ` t1 = time.time()` ` print("-----" * 10)` I hope this helps to reproduce the error, in the meanwhile I have observed that with FPS set to 1 the camera restarts after approximately 3 hours, while with the FPS at 3 the camera restarts after 23 minutes. Kind regards, Georg

@"gherodes"#p26154 Looking into it. I'll let you know when I have more info.

@"jakaskerl"#p26180 Hi, Any luck reproducing the issue? I can send some additional information on my dev environment if required. Georg

@"gherodes"#p26347 I could not reproduce the issue even on 10FPS running for 3h. I ran it in peripheral mode (not standalone). Does the issue only arise in standalone? Thanks Jaka

@"jakaskerl"#p26358 Yes, it only appears in Standalone mode with the scripts provided above. As mentioned in the original post, the network that the camera is running on is quite unstable so running in peripheral mode causes the host-camera connection to be lost frequently, making peripheral mode un-usable for my usecase. Georg

@"gherodes"#p26360 Ok. What is the bandwidth of the network. How common are the connection losses - could it be related to crashes - do they coincide? Thanks, Jaka

Retrieving synced pointcloud and image in real-time via Script node

gherodes

jakaskerl

Hi Jaka,
I tried out the script you provided and indeed the camera did not crash for a long time (7+h). However after logging the timestamps of the images and pointclouds I noticed that a) there is more discrepancy between the image and pointcloud timestamps than previously and b) there is up to 3 sec of latency. (Both the same issues as in the original post of this thread). Interestingly, I started seeing the crashing occur again after I added the following lines in an attempt to fix the timestamping and latency issue:

script.inputs["pcl"].setBlocking(False)
script.inputs["pcl"].setQueueSize(1)

The crashes again occurred consistently every _7min.

Any way you can add these lines to your test script and see if you can reproduce the error?

Kind regards
Georg

jakaskerl

gherodes
Managed to repro the error and am looking into it. Seems like a FW issue at first.

Thanks,
Jaka

gherodes

Hi Jaka,

Any new info on this issue?

Georg

jakaskerl

gherodes
FW issue, have forwarded to the dev team, but they are currently focusing on other tasks so it is not high prio.

I'll notify you when we fix it.

Thanks,
Jaka

jakaskerl

gherodes
Try on the latest develop branch.
git checkout develop
git pull --recurse-submodules
python3 examples/install_requirements.py

Then reflash the device. LMK if it fixes the issue.

Thanks,
Jaka

gherodes

Hi Jaka,

I have followed the instructions and reflashed the device. It has now been running 24h without crashing so I guess we can say the issue is fixed!

Thank you for your help!

Georg

« Previous Page