depth stream - packet.getData

webbhm

OAK-D running on Raspberry Pi 4 or Windows 10
When I call packet.getData() on a 'depth' stream the system crashes. I initially encountered this on a Raspberry Pi (4), and thought it might be the limitations of the Raspberry, but I built depthai on a Windows 10 machine and hit the same error. The Raspberry crashed (sub-system error) with no error information, but the Windows system did catch the errors. Calling getStringData() works (but not the most helpful data). The getData() call works fine for previewout and disparity_color streams. The depthai_demo.py program runs fine, even calling the depth stream.
I am trying to write a program that will merge the depth and RGB information and output a point cloud (ply file). Attached are the log file and code.
Thanks in advance.
Log file _________________________________________________________________________________

PS C:\Windows\system32\depthai-tutorials-practice\1-depth> python3 depth.py
XLink initialized.
Sending internal device firmware
Successfully connected to device.
Usb speed : Unknown
Mx serial id : UNKNOWN
Loading config file
Attempting to open stream config_d2h
watchdog started
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
EEPROM data: valid (v5)
  Board name     : BW1098OBC
  Board rev      : R0M0E0
  HFOV L/R       : 71.86 deg
  HFOV RGB       : 68.7938 deg
  L-R   distance : 7.5 cm
  L-RGB distance : 3.75 cm
  L/R swapped    : yes
  L/R crop region: center
  Rectification Rotation R1 (left):
    0.999788,   -0.006367,    0.019576,
    0.006242,    0.999960,    0.006473,
   -0.019617,   -0.006350,    0.999787,
  Rectification Rotation R2 (right):
    0.999843,   -0.005849,   -0.016727,
    0.005741,    0.999963,   -0.006460,
    0.016764,    0.006363,    0.999839,
  Calibration intrinsic matrix M1 (left):
  862.272522,    0.000000,  640.404114,
    0.000000,  862.898254,  407.731903,
    0.000000,    0.000000,    1.000000,
  Calibration intrinsic matrix M2 (right):
  856.477661,    0.000000,  643.983032,
    0.000000,  857.205261,  405.526367,
    0.000000,    0.000000,    1.000000,
  Calibration rotation matrix R:
    0.999338,   -0.000732,    0.036371,
    0.000269,    0.999919,    0.012720,
   -0.036377,   -0.012702,    0.999257,
  Calibration translation matrix T:
   -7.477144,
    0.043738,
    0.125089,
  Calibration Distortion Coeff d1 (Left):
   -4.277044,   14.388677,   -0.001378,   -0.000263,  -12.375067,   -4.343279,   14.632688,
  -12.588841,    0.000000,    0.000000,    0.000000,    0.000000,    0.000000,    0.000000,
  Calibration Distortion Coeff d2 (Right):
   21.936172,  -38.558819,   -0.001881,   -0.000786,  283.404510,   21.964676,  -40.724358,
  288.684937,    0.000000,    0.000000,    0.000000,    0.000000,    0.000000,    0.000000,
./mobilenet-ssd/mobilenet-ssd.json
depthai: Calibration file is not specified, will use default setting;
config_h2d json:
{"_board":{"calib_data":[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],"mesh_left":[0.0],"mesh_right":[0.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"previewout"},{"name":"metaout"},{"data_type":"uint16","name":"depth"},{"name":"disparity"}]},"ai":{"NCEs":1,"NN_config":{"NN_family":"mobilenet","confidence_threshold":0.5,"output_format":"detection"},"blob0_size":14489664,"blob1_size":0,"calc_dist_to_bb":false,"camera_input":"rgb","cmx_slices":7,"keep_aspect_ratio":true,"num_stages":1,"shaves":7},"app":{"enable_reconfig":true,"sync_sequence_numbers":false,"sync_video_meta_streams":false,"usb_chunk_KiB":64},"board":{"clear-eeprom":false,"left_fov_deg":69.0,"left_to_rgb_distance_m":0.0,"left_to_right_distance_m":0.03500000014901161,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":69.0,"stereo_center_crop":false,"store-to-eeprom":false,"swap-left-and-right-cameras":false},"camera":{"mono":{"fps":30.0,"resolution_h":720,"resolution_w":1280},"rgb":{"fps":30.0,"resolution_h":1080,"resolution_w":1920}},"depth":{"depth_limit_mm":10000,"lr_check":false,"median_kernel_size":7,"padding_factor":0.30000001192092896,"warp_rectify":{"edge_fill_color":-1,"mirror_frame":true,"use_mesh":false}},"ot":{"confidence_threshold":0.5,"max_tracklets":20}}
size of input string json_config_obj to config_h2d is ->1672
size of json_config_obj that is expected to be sent to config_h2d is ->1048576
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1048576 bytes to config_h2d
!!! XLink write successful: config_h2d (1048576)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Creating observer stream host_capture: ...
Attempting to open stream host_capture
Successfully opened stream host_capture with ID #0!
Creating observer stream host_capture: DONE.
Read: 14489664
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14489664 bytes to inBlob
!!! XLink write successful: inBlob (14489664)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file ./mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #1!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
Input layer :
Name: data
Index: 0
Element type: uint8
Element size:  1byte
Offset: 0 byte
Dimensions: [Batch : 1, Channel : 3, Height : 300, Width : 300]

Output layer :
Name: detection_out
Index: 0
Element type: float16
Element size:  2 bytes
Offset: 0 byte
Dimensions: [Batch : 1, Channel : 1, Height : 100, Width : 7]

CNN to depth bounding-box mapping: start(318, 38), max_size(643, 643)
Host stream start:depth
Opening stream for read: depth
Attempting to open stream depth
Successfully opened stream depth with ID #1!
Starting thread for stream: depth
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Started thread for stream: depth
Successfully opened stream metaout with ID #3!
Starting thread for stream: metaout
Host stream start:previewout
Opening stream for read: previewout
Attempting to open stream previewout
Started thread for stream: metaout
Successfully opened stream previewout with ID #4!
Starting thread for stream: previewout
Opening stream for read: disparity
Attempting to open stream disparity
Started thread for stream: previewout
Successfully opened stream disparity with ID #2!
Starting thread for stream: disparity
depthai: INIT OK!
Started thread for stream: disparity
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
(300, 300, 3)
Processing previewout
Processing disparity_color
Shape (720, 1280, 3)
Processing depth
Getting data
In try
W[35mE: [global] [         0] [] addEvent:264  iCondition failed: event->header.flags.bitField.ack != 1n[0m
Usb_WritePipe: System err 22
[35mE: [global] [         0] [] addEvent:264   Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [         0] [] addEventWithPerf:276   [35mE: [global] [         0] [] addEventWithPerf:276  addEvent(event) method call failed with an error: 3 addEvent(event) method call failed with an error: 3[0m
[0m
[35mE: [global] [         0] [] XLinkReadData:156      [35mE: [global] [         0] [] XLinkReadData:156    Condition failed: (addEventWithPerf(&event, &opTime))Condition failed: (addEventWithPerf(&event, &opTime))[0m
[0m
Device get data failed: 7
Closing stream depth: ...

WinUsb_WritePipe failed with error:=22
Device get data failed: 7
[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53   Write failed (header) (err -2) | event XLINK_WRITE_RESP
Closing stream disparity: ...
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
WinClosing stream depth: DONE.
Usb_WritePipe: System err 22
Thread for depth finished.

WinUsb_WritePipe failed with error:=22
[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53   Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
Closing stream disparity: DONE.
WinUsb_WritePipe: SystemThread for disparity finished.
 err 22

WinUsb_WritePipe failed with error:=22
[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53   Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
[35mE: [global] [         0] [] addEvent:264   Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [         0] [] addEventWithPerf:276    addEvent(event) method call failed with an error: 3[0m
WinUsb_WritePipe: System err 22
[35mE: [global] [         0] [] XLinkReadData:156
WinUsb_WritePipe failed with error:=22
Condition failed: (addEventWithPerf(&event, &opTime))[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53    Write failed (header) (err -2) | event XLINK_READ_REL_REQ
[0m
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
Device get data failed: 7
Closing stream metaout: ...
Closing stream metaout: DONE.
Thread for metaout finished.
WinUsb_WritePipe: System err 22

WinUsb_WritePipe failed with error:=22
[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53   Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
[35mE: [global] [         0] [] addEvent:264   Condition failed: event->header.flags.bitField.ack != 1[0m
[35mE: [global] [         0] [] addEventWithPerf:276    addEvent(event) method call failed with an error: 3[0m
[35mE: [global] [         0] [] XLinkReadData:156      Condition failed: (addEventWithPerf(&event, &opTime))[0m
Device get data failed: 7
Closing stream previewout: ...
Closing stream previewout: DONE.
Thread for previewout finished.
WinUsb_WritePipe: System err 22

WinUsb_WritePipe failed with error:=22
[35mE: [global] [         0] [Scheduler00Thr] dispatcherEventSend:53   Write failed (header) (err -2) | event XLINK_CLOSE_STREAM_REQ
[0m
[35mE: [xLink] [         0] [Scheduler00Thr] sendEvents:1027   Event sending failed[0m
PS C:\Windows\system32\depthai-tutorials-practice\1-depth>

Code ---------------------------------------------------------------------

from pathlib import Path
import numpy as np
import cv2
import depthai
from datetime import datetime
import inspect

def make_rgbd(rgb, depth):
    # make sure are same size
    print("RGB", rgb.shape())
    print("Depth", depth.shape())
    if rgb.shape() != depth.shape():
        rgb = cv2.resize(depth.shape())
        print("Resize", rgb.shape())
    rgbd = np.hstack([rgb, depth])
    return rgbd

def make_rgb(data):
    '''
    Convert packet data to frame
    '''
    data0 = data[0,:,:]
    data1 = data[1,:,:]
    data2 = data[2,:,:]
    frame = cv2.merge([data0, data1, data2])
    print(frame.shape)
    
    return frame

def make_file_name():
    file_name = '/home/pi/Pictures/' + format(datetime.now(), '%Y-%m-%d_%H%M') + '.ply'
    
    return file_name
    
def save_point_cloud(rgb_frame, depth, file_name=''):
    rgbd = make_rgbd(rgb_frame, depth_frame)
    if file_name == '':
        file_name = make_file_name()
    
    ply_header = '''ply
        format ascii 1.0
        element vertex %(vert_num)d
        property float x
        property float y
        property float z
        property uchar red
        property uchar green
        property uchar blue
        end_header
        '''
    with open(filename, 'w') as f:
        print("Save", filename)
        f.write(ply_header %dict(vert_num=len(vertices)))
        np.savetxt(f,vertices,'%f %f %f %d %d %d')    
    

def pre_display(frame, detections, name):
            print('Processing', name)
            
            img_h = frame.shape[0]
            img_w = frame.shape[1]
            
            for detection in detections:
                pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
                pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)

                cv2.rectangle(frame, pt1, pt2, (0, 0, 255), 2)

            cv2.imshow(name, frame)
    
def disp_display(depth_frame, detections, name):
            print('Processing', name)
            print("Shape", depth_frame.shape)
            img_h = depth_frame.shape[0]
            img_w = depth_frame.shape[1]
            
            for detection in detections:
                pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
                pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)

                cv2.rectangle(depth_frame, pt1, pt2, (0, 0, 255), 2)
             
            cv2.imshow(name, depth_frame)
    
def depth_display(packet, detections, name):
            print('Processing', name)
            print("Getting data")
            #print("Size", packet.sizeof())
            depth_frame = []
            try:
                print("In try")
                depth_frame = packet.getData()
            except Exception as e:
                print("Exception", e)
                print(inspect.getmembers(packet))
                return
            print(depth_frame)
            #print(depth_frame.shape)
            
            #img_h = depth_frame.shape[0]
            #img_w = depth_frame.shape[1]
            '''
            for detection in detections:
                pt1 = int(detection.x_min * img_w), int(detection.y_min * img_h)
                pt2 = int(detection.x_max * img_w), int(detection.y_max * img_h)

                cv2.rectangle(depth_frame, pt1, pt2, (0, 0, 255), 2)
            ''' 
            #cv2.imshow(name, depth_frame)
            
def print_meta(packet):
    if packet == None:
        return
    stream_name = packet.stream_name
    print(stream_name)
    meta = packet.getMetadata()
    if meta is None:
        print("No meta data")
        return
    else:
        source = meta.getCameraName()
        if stream_name.startswith('disparity') or stream_name.startswith('depth'):
            source += '(rectif)'
        print(" {:.6f}".format(meta.getTimestamp()), meta.getSequenceNum(), source, end='')
        print('', meta.getFrameWidth(), meta.getFrameHeight(), meta.getFrameBytesPP(), end='')
    print()
    return    
            
device = depthai.Device('', False)

pipeline = device.create_pipeline(config={
    'streams': ['previewout', 'metaout','depth', 'disparity_color'],
    'ai': {
        "blob_file":"./mobilenet-ssd/mobilenet-ssd.blob",
        "blob_file_config":"./mobilenet-ssd/mobilenet-ssd.json"
    }
})

if pipeline is None:
    raise RuntimeError('Pipeline creation failed!')

detections = []
rgb_frame = []
depth_frame = []

while True:
    nnet_packets, data_packets = pipeline.get_available_nnet_and_data_packets()
    
    for nnet_packet in nnet_packets:
        detections = list(nnet_packet.getDetectedObjects())
        
    for packet in data_packets:
        #print_meta(packet)
        if packet.stream_name == 'previewout':
            rgb_frame = make_rgb(packet.getData())
            pre_display(rgb_frame, detections, packet.stream_name)
        if packet.stream_name == 'disparity_color':
            disp_display(packet.getData(), detections, packet.stream_name)
        if packet.stream_name == 'depth':
            depth_display(packet, detections, packet.stream_name)
                        
    key = cv2.waitKey(1)
    if key == ord('q'):
        break
    elif key == ord('p'):
        save_point_cloud(rgb_frame, depth_frame, file_name)
    


print("Delete Pipeline")
del pipeline
print("Delete Device")
del device

Brandon

Hi webbhm ,

Sorry about the trouble here. So I am still parsing the error messages here. My suspicion is the problem is because in Gen1 we require a neural network to be loaded, even if unused. In Gen2 this requirement was removed. I will look more at the error messages though - that's just a guess.

In the meantime I did want to chime in on the point-cloud bit as I think we have (Gen2) example code that could likely get you there faster:
https://github.com/luxonis/depthai-experiments/tree/master/gen2-camera-demo#gen2-camera-demo

So this actually allows you to do point-cloud without having to write additional code.

This does not do RGB combination into a pointcloud though. We do have an RGB pointcloud example for Gen1 here:
https://github.com/luxonis/depthai-experiments/tree/master/pcl-projection-rgb

Thoughts?

Thanks,
Brandon

Brandon

Not sure what is causing the error yet (but it doesn't seem to be the neural blob thing I mentioned). Would you mind taking a look at those examples to see if you can accomplish what you are looking for based on them?

Thanks again,
Brandon

webbhm

Was about to say that I didn't think it was the blob, as the metaout is giving identification information.
I had seen those examples, but was avoiding Gen2 till I had figured out Gen1, looks like I am going to have to advance my learning curve (and add a few libraries)! What you have in the experiment is very close to what I am wanting, it will be trivial to add a key capture and output to a ply file. If I can get Gen2 working then I will leave Gen1 behind.

Thanks again.
HW

Brandon

Hi webbhm ,

Yes, so yes I think Sachin actually has an example to save to a PLY as well. Let me ping him and see what he has.

Thanks,
Brandon

luxonis-Sachin

Brandon
@webbhm Here is the link which saves ply files. this example uses gen2 but API for camera calibration parameters is not added yet so it will use default.
To save the ply press p key when one of the opencv windows is in focus.

webbhm

@luxonis-Sachin, @Brandon
I had not gotten as sophisticated, but did the same key trick with the save functions in projector_3d. Open3d definitely makes things easy, though installing it is a pain. Gave up on building it for Raspberry do the issues with building make, and then finding that Open3d doesn't display on the arm architecture, also the set-up would not be portable for the outdoor work I want to do with this. Windows 10 was a pain till I loaded Thonny, set Python to 3.8, and then let Thonny load all the library packages. I got to liking Thonny on the Raspberry, and it definitely makes things easy on Windows.
Thanks for all the help. Will cut & paste Sachin's code into what I have. This looks like 99% of what I need.
I am doing citizen science with Friends of Lake Monroe. Trying to document shoreline erosion and thinking that some point clouds would give us good baseline information for evaluating future changes. Oak-D and depthai seem to be a good combination.

luxonis-Sachin

webbhm if you want something external then I would suggest using VTK here. But it might need a little more work on your side to get it working.