OAK-D running on Raspberry Pi 4 or Windows 10
When I call packet.getData() on a 'depth' stream the system crashes. I initially encountered this on a Raspberry Pi (4), and thought it might be the limitations of the Raspberry, but I built depthai on a Windows 10 machine and hit the same error. The Raspberry crashed (sub-system error) with no error information, but the Windows system did catch the errors. Calling getStringData() works (but not the most helpful data). The getData() call works fine for previewout and disparity_color streams. The depthai_demo.py program runs fine, even calling the depth stream.
I am trying to write a program that will merge the depth and RGB information and output a point cloud (ply file). Attached are the log file and code.
Thanks in advance.
Log file _________________________________________________________________________________
PS C:\Windows\system32\depthai-tutorials-practice\1-depth> python3 depth.py
XLink initialized.
Sending internal device firmware
Successfully connected to device.
Usb speed : Unknown
Mx serial id : UNKNOWN
Loading config file
Attempting to open stream config_d2h
watchdog started
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
EEPROM data: valid (v5)
Board name : BW1098OBC
Board rev : R0M0E0
HFOV L/R : 71.86 deg
HFOV RGB : 68.7938 deg
L-R distance : 7.5 cm
L-RGB distance : 3.75 cm
L/R swapped : yes
L/R crop region: center
Rectification Rotation R1 (left):
0.999788, -0.006367, 0.019576,
0.006242, 0.999960, 0.006473,
-0.019617, -0.006350, 0.999787,
Rectification Rotation R2 (right):
0.999843, -0.005849, -0.016727,
0.005741, 0.999963, -0.006460,
0.016764, 0.006363, 0.999839,
Calibration intrinsic matrix M1 (left):
862.272522, 0.000000, 640.404114,
0.000000, 862.898254, 407.731903,
0.000000, 0.000000, 1.000000,
Calibration intrinsic matrix M2 (right):
856.477661, 0.000000, 643.983032,
0.000000, 857.205261, 405.526367,
0.000000, 0.000000, 1.000000,
Calibration rotation matrix R:
0.999338, -0.000732, 0.036371,
0.000269, 0.999919, 0.012720,
-0.036377, -0.012702, 0.999257,
Calibration translation matrix T:
-7.477144,
0.043738,
0.125089,
Calibration Distortion Coeff d1 (Left):
-4.277044, 14.388677, -0.001378, -0.000263, -12.375067, -4.343279, 14.632688,
-12.588841, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000,
Calibration Distortion Coeff d2 (Right):
21.936172, -38.558819, -0.001881, -0.000786, 283.404510, 21.964676, -40.724358,
288.684937, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000,
./mobilenet-ssd/mobilenet-ssd.json
depthai: Calibration file is not specified, will use default setting;
config_h2d json:
{"_board":{"calib_data":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],"mesh_left":[0.0],"mesh_right":[0.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"previewout"},{"name":"metaout"},{"data_type":"uint16","name":"depth"},{"name":"disparity"}]},"ai":{"NCEs":1,"NN_config":{"NN_family":"mobilenet","confidence_threshold":0.5,"output_format":"detection"},"blob0_size":14489664,"blob1_size":0,"calc_dist_to_bb":false,"camera_input":"rgb","cmx_slices":7,"keep_aspect_ratio":true,"num_stages":1,"shaves":7},"app":{"enable_reconfig":true,"sync_sequence_numbers":false,"sync_video_meta_streams":false,"usb_chunk_KiB":64},"board":{"clear-eeprom":false,"left_fov_deg":69.0,"left_to_rgb_distance_m":0.0,"left_to_right_distance_m":0.03500000014901161,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":69.0,"stereo_center_crop":false,"store-to-eeprom":false,"swap-left-and-right-cameras":false},"camera":{"mono":{"fps":30.0,"resolution_h":720,"resolution_w":1280},"rgb":{"fps":30.0,"resolution_h":1080,"resolution_w":1920}},"depth":{"depth_limit_mm":10000,"lr_check":false,"median_kernel_size":7,"padding_factor":0.30000001192092896,"warp_rectify":{"edge_fill_color":-1,"mirror_frame":true,"use_mesh":false}},"ot":{"confidence_threshold":0.5,"max_tracklets":20}}
size of input string json_config_obj to config_h2d is ->1672
size of json_config_obj that is expected to be sent to config_h2d is ->1048576
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1048576 bytes to config_h2d
!!! XLink write successful: config_h2d (1048576)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Creating observer stream host_capture: ...
Attempting to open stream host_capture
Successfully opened stream host_capture with ID #0!
Creating observer stream host_capture: DONE.
Read: 14489664
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14489664 bytes to inBlob
!!! XLink write successful: inBlob (14489664)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file ./mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #1!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
Input layer :
Name: data
Index: 0
Element type: uint8
Element size: 1byte
Offset: 0 byte
Dimensions: [Batch : 1, Channel : 3, Height : 300, Width : 300]
Output layer :
Name: detection_out
Index: 0
Element type: float16
Element size: 2 bytes
Offset: 0 byte
Dimensions: [Batch : 1, Channel : 1, Height : 100, Width : 7]
CNN to depth bounding-box mapping: start(318, 38), max_size(643, 643)
Host stream start:depth
Opening stream for read: depth
Attempting to open stream depth
Successfully opened stream depth with ID #1!
Starting thread for stream: depth
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Started thread for stream: depth
Successfully opened stream metaout with ID #3!
Starting thread for stream: metaout
Host stream start:previewout
Opening stream for read: previewout
Attempting to open stream previewout
Started thread for stream: metaout
Successfully opened stream previewout with ID #4!
Starting thread for stream: previewout
Opening stream for read: disparity
Attempting to open stream disparity
Started thread for stream: previewout
Successfully opened stream disparity with ID #2!
Starting thread for stream: disparity
depthai: INIT OK!
Started thread for stream: disparity
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
Processing disparity_color
Shape (720, 1280, 3)
Processing depth
Getting data
In try
W[35mE: [global] [ 0] [] addEvent:264 iCondition failed: event->header.flags.bitField.ack != 1n[0m
Usb_WritePipe: System err 22
[35mE: [global] [ 0] [] addEvent:264 Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [ 0] [] addEventWithPerf:276 [35mE: [global] [ 0] [] addEventWithPerf:276 addEvent(event) method call failed with an error: 3 addEvent(event) method call failed with an error: 3[0m
[0m
[35mE: [global] [ 0] [] XLinkReadData:156 [35mE: [global] [ 0] [] XLinkReadData:156 Condition failed: (addEventWithPerf(&event, &opTime))Condition failed: (addEventWithPerf(&event, &opTime))[0m
[0m
Device get data failed: 7
Closing stream depth: ...
WinUsb_WritePipe failed with error:=22
Device get data failed: 7
[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_WRITE_RESP
Closing stream disparity: ...
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
WinClosing stream depth: DONE.
Usb_WritePipe: System err 22
Thread for depth finished.
WinUsb_WritePipe failed with error:=22
[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
Closing stream disparity: DONE.
WinUsb_WritePipe: SystemThread for disparity finished.
err 22
WinUsb_WritePipe failed with error:=22
[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
[35mE: [global] [ 0] [] addEvent:264 Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [ 0] [] addEventWithPerf:276 addEvent(event) method call failed with an error: 3[0m
WinUsb_WritePipe: System err 22
[35mE: [global] [ 0] [] XLinkReadData:156
WinUsb_WritePipe failed with error:=22
Condition failed: (addEventWithPerf(&event, &opTime))[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_READ_REL_REQ
[0m
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
Device get data failed: 7
Closing stream metaout: ...
Closing stream metaout: DONE.
Thread for metaout finished.
WinUsb_WritePipe: System err 22
WinUsb_WritePipe failed with error:=22
[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
[35mE: [global] [ 0] [] addEvent:264 Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [ 0] [] addEventWithPerf:276 addEvent(event) method call failed with an error: 3[0m
[35mE: [global] [ 0] [] XLinkReadData:156 Condition failed: (addEventWithPerf(&event, &opTime))[0m
Device get data failed: 7
Closing stream previewout: ...
Closing stream previewout: DONE.
Thread for previewout finished.
WinUsb_WritePipe: System err 22
WinUsb_WritePipe failed with error:=22
[35mE: [global] [ 0] [Scheduler00Thr] dispatcherEventSend:53 Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [ 0] [Scheduler00Thr] sendEvents:1027 Event sending failed[0m
PS C:\Windows\system32\depthai-tutorials-practice\1-depth>
Code ---------------------------------------------------------------------
from pathlib import Path
import numpy as np
import cv2
import depthai
from datetime import datetime
import inspect
def make_rgbd(rgb, depth):
# make sure are same size
print("RGB", rgb.shape())
print("Depth", depth.shape())
if rgb.shape() != depth.shape():
rgb = cv2.resize(depth.shape())
print("Resize", rgb.shape())
rgbd = np.hstack([rgb, depth])
return rgbd
def make_rgb(data):
'''
Convert packet data to frame
'''
data0 = data[0,:,:]
data1 = data[1,:,:]
data2 = data[2,:,:]
frame = cv2.merge([data0, data1, data2])
print(frame.shape)
return frame
def make_file_name():
file_name = '/home/pi/Pictures/' + format(datetime.now(), '%Y-%m-%d_%H%M') + '.ply'
return file_name
def save_point_cloud(rgb_frame, depth, file_name=''):
rgbd = make_rgbd(rgb_frame, depth_frame)
if file_name == '':
file_name = make_file_name()
ply_header = '''ply
format ascii 1.0
element vertex %(vert_num)d
property float x
property float y
property float z
property uchar red
property uchar green
property uchar blue
end_header
'''
with open(filename, 'w') as f:
print("Save", filename)
f.write(ply_header %dict(vert_num=len(vertices)))
np.savetxt(f,vertices,'%f %f %f %d %d %d')
def pre_display(frame, detections, name):
print('Processing', name)
img_h = frame.shape[0]
img_w = frame.shape[1]
for detection in detections:
pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)
cv2.rectangle(frame, pt1, pt2, (0, 0, 255), 2)
cv2.imshow(name, frame)
def disp_display(depth_frame, detections, name):
print('Processing', name)
print("Shape", depth_frame.shape)
img_h = depth_frame.shape[0]
img_w = depth_frame.shape[1]
for detection in detections:
pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)
cv2.rectangle(depth_frame, pt1, pt2, (0, 0, 255), 2)
cv2.imshow(name, depth_frame)
def depth_display(packet, detections, name):
print('Processing', name)
print("Getting data")
#print("Size", packet.sizeof())
depth_frame = []
try:
print("In try")
depth_frame = packet.getData()
except Exception as e:
print("Exception", e)
print(inspect.getmembers(packet))
return
print(depth_frame)
#print(depth_frame.shape)
#img_h = depth_frame.shape[0]
#img_w = depth_frame.shape[1]
'''
for detection in detections:
pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)
cv2.rectangle(depth_frame, pt1, pt2, (0, 0, 255), 2)
'''
#cv2.imshow(name, depth_frame)
def print_meta(packet):
if packet == None:
return
stream_name = packet.stream_name
print(stream_name)
meta = packet.getMetadata()
if meta is None:
print("No meta data")
return
else:
source = meta.getCameraName()
if stream_name.startswith('disparity') or stream_name.startswith('depth'):
source += '(rectif)'
print(" {:.6f}".format(meta.getTimestamp()), meta.getSequenceNum(), source, end='')
print('', meta.getFrameWidth(), meta.getFrameHeight(), meta.getFrameBytesPP(), end='')
print()
return
device = depthai.Device('', False)
pipeline = device.create_pipeline(config={
'streams': ['previewout', 'metaout','depth', 'disparity_color'],
'ai': {
"blob_file":"./mobilenet-ssd/mobilenet-ssd.blob",
"blob_file_config":"./mobilenet-ssd/mobilenet-ssd.json"
}
})
if pipeline is None:
raise RuntimeError('Pipeline creation failed!')
detections = []
rgb_frame = []
depth_frame = []
while True:
nnet_packets, data_packets = pipeline.get_available_nnet_and_data_packets()
for nnet_packet in nnet_packets:
detections = list(nnet_packet.getDetectedObjects())
for packet in data_packets:
#print_meta(packet)
if packet.stream_name == 'previewout':
rgb_frame = make_rgb(packet.getData())
pre_display(rgb_frame, detections, packet.stream_name)
if packet.stream_name == 'disparity_color':
disp_display(packet.getData(), detections, packet.stream_name)
if packet.stream_name == 'depth':
depth_display(packet, detections, packet.stream_name)
key = cv2.waitKey(1)
if key == ord('q'):
break
elif key == ord('p'):
save_point_cloud(rgb_frame, depth_frame, file_name)
print("Delete Pipeline")
del pipeline
print("Delete Device")
del device