segfault when using with multiprocessing queue

PepperoniPingu

Hi!

After a couple of minutes of connecting and disconnecting an OAK-1 in a python loop I get this error:

Stack trace (most recent call last) in thread 3038424:
#31 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b22b83, in
#30 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b1aa31, in
#29 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b12ba8, in _PyEval_EvalFrameDefault
#28 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b22b83, in
#27 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b1a780, in _PyFunction_Vectorcall
#26 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0cf23, in
#25 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b12b29, in _PyEval_EvalFrameDefault
#24 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b15e67, in _PyObject_MakeTpCall
#23 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b160da, in
#22 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b2153b, in
#21 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b158e5, in _PyObject_FastCallDictTstate
#20 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b1a780, in _PyFunction_Vectorcall
#19 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0cf23, in
#18 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0e584, in _PyEval_EvalFrameDefault
#17 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b18e93, in
#16 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b53398, in PyWrapper_New
#15 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36afcd77, in _PyObject_GC_New
#14 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36afce9c, in
#13 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b7e5aa, in
#12 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0563b, in
#11 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b06aa1, in
#10 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b21278, in
#9 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b15d89, in
#8 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0a9d7, in
#7 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0abb6, in
#6 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b25544, in
#5 Object "/lib64/libpython3.9.so.1.0", at 0x7fee36b0aae6, in
#4 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7fee26956232, in
#3 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7fee269cf029, in
#2 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7fee26c24cab, in dai:😃ataOutputQueue::~DataOutputQueue()
#1 Object "/lib64/libstdc++.so.6", at 0x7fee264e3256, in std::thread::join()
#0 Object "/lib64/libc.so.6", at 0x7fee368ae4f4, in
Segmentation fault (Address not mapped to object [0x7fee05df7990])

Here is the code that is causing the error PepperoniPingu/depthai-segfault-mre . Just let it run for a minute and you should see this error.

The oak is not doing anything complex, it's just forwarding the video stream. It seems that the depthai output queue is interfering with my multiprocessing queue. If I remove either one, the problem disappears.

My setup:
- Fedora Linux 39
- Python 3.9
- Depthai 2.24, had the same problem on 2.19

jakaskerl

Hi @PepperoniPingu
Couldn't reproduce the issue on Mac. Couldn't test on fedora since I don't have access to it. Any chance you can try on a different host? Maybe run the code with DEPTHAI_DEBUG=1 env variable.

Thanks
Jaka

PepperoniPingu

Thanks for the reply @jakaskerl

The issue was reproduced on a Raspberry Pi 4 with Raspberry Pi OS Bookworm and Python 3.11

Here is the log from my fedora desktop with the debug flag set:

[2024-03-04 09:15:33.003] [host] [debug] Device - OpenVINO version: universal
[19443010D13B291300] [1.7] [1709540133.003] [host] [debug] Device - BoardConfig: {"camera":[],"emmc":null,"gpio":[],"imu":null,"logDevicePrints":true,"logPath":null,"logSizeMax":null,"logVerbosity":null,"network":{"mtu":0,"xlinkTcpNoDelay":true},"nonExclusiveMode":false,"pcieInternalClock":null,"sysctl":[],"uart":[],"usb":{"flashBootedPid":63037,"flashBootedVid":999,"manufacturer":"","maxSpeed":4,"pid":63035,"productName":"","vid":999},"usb3PhyInternalClock":null,"uvc":null,"watchdogInitialDelayMs":null,"watchdogTimeoutMs":null}
libnop:
0000: b9 12 b9 07 81 e7 03 81 3b f6 81 e7 03 81 3d f6 04 bd 00 bd 00 b9 02 00 01 ba 00 be be bb 00 bb
0020: 00 be be be be be be 01 00 bb 00 be be
[2024-03-04 09:15:34.279] [depthai] [debug] Searching for booted device: DeviceInfo(name=1.7, mxid=19443010D13B291300, X_LINK_BOOTED, X_LINK_USB_VSC, X_LINK_MYRIAD_X, X_LINK_SUCCESS), name used as hint only
[19443010D13B291300] [1.7] [0.908] [system] [warning] PRINT:LeonCss: [OK]: INIT_NOC: noc probes timer create
[OK]: INIT_NOC: noc probes timer start
[OK]: INIT_NOC: ddr bandwith profiling!
BootloaderConfig.options1 checksum doesn't match. Is: 0x10304419 should be: 0x607F3BC1
GPIO boot mode 0x16, interface USBD
Setting aons(0..4) back to boot from flash (offset = 0)
====ENABLE WATCHDOG====1
initial keepalive, countdown: 10
PLL0: 700000 AUX_IO0: 24000 AUX_IO1: 24000 MCFG: 24000 MECFG: 24000
Board init ret 3
DepthAI Firmware - version: a95f582a61ec9bdbd0f72dec84822455872ffaf7
eeprom configuration version: 55AA0007
Reading VERSION 7 --- ->
eeprom configuration load from user area, status: 0
Reading VERSION 7 --- ->
Didn't find any boards matching the eeprom data
Board options: 00000000
--> brdInit ...
brdInitAuxDevices: Error: SC = 27: io_initialize expander_cam_gpios_1 [OK]
spi_N25Q_init: Flash JEDEC ID: ff ff ff
Invalid Flash JEDEC ID... No NOR available
Could not register the spi device
inited hal ---
def: 53 46 45 34 10 06 08 34 10 53 14 45 46 08 06
val: 53 46 45 34 10 06 08 34 10 53 14 45 46 08 06
Opening bus for IR driver: 1
Failed to probe IR driver LM3644
Opening bus for IR driver: 2
Failed to probe IR driver LM3644
Opening bus for IR driver: 3
Failed to probe IR driver LM3644
Closing EEPROm!
MyriaX board configuration
pll0 frequency: 700000, ref0 frequency: 24000
Is booted from flash by bootloader: 0
Networking not available...
=== Enumerating on socket: Cam_A / RGB / Center
skipped OV7251
skipped AR0234
skipped S5K33D

Registered camera A12N02A (imx378) as /dev/Camera_0
camera socket: 0, name: color
config - w: 1920, h: 1080, type: COLOR
config - w: 3840, h: 2160, type: COLOR
config - w: 4056, h: 3040, type: COLOR
config - w: 1352, h: 1012, type: COLOR
config - w: 2024, h: 1520, type: COLOR
Adding socket 0: cam 2. Sen name: IMX378
=== Enumerating on socket: Cam_B / Left
skipped IMX214
skipped AR0234
skipped S5K33D
skipped IMX378
skipped IMX296
skipped IMX462
skipped IMX283
skipped OV5645
=== Enumerating on socket: Cam_C / Right
skipped IMX214
skipped AR0234
skipped S5K33D
skipped IMX378
skipped IMX296
skipped IMX462
skipped IMX283
skipped OV5645
Didn't find any boards matching the eeprom data
Initializing XLink...
UsbPumpVscAppI_Event: 5 VSC2_EVENT_ATTACH
UsbPumpVscAppI_Event: 4 VSC2_EVENT_RESET
UsbPumpVscAppI_Event: 4 VSC2_EVENT_RESET
initial keepalive, countdown: 9
UsbPumpVscAppI_Event: 0 VSC2_EVENT_INTERFACE_UP
Done!
Usb connection speed: Super - USB 3.0
Temperature: Driver registered.
[19443010D13B291300] [1.7] [1709540135.205] [host] [debug] Schema dump: {"connections":[{"node1Id":0,"node1Output":"video","node1OutputGroup":"","node2Id":1,"node2Input":"in","node2InputGroup":""}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"sippBufferSize":18432,"sippDmaBufferSize":16384,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":[[["","video"],{"blocking":false,"group":"","id":9,"name":"video","queueSize":8,"type":0,"waitForMessage":false}],[["","still"],{"blocking":false,"group":"","id":7,"name":"still","queueSize":8,"type":0,"waitForMessage":false}],[["","isp"],{"blocking":false,"group":"","id":6,"name":"isp","queueSize":8,"type":0,"waitForMessage":false}],[["","preview"],{"blocking":false,"group":"","id":8,"name":"preview","queueSize":8,"type":0,"waitForMessage":false}],[["","raw"],{"blocking":false,"group":"","id":5,"name":"raw","queueSize":8,"type":0,"waitForMessage":false}],[["","frameEvent"],{"blocking":false,"group":"","id":4,"name":"frameEvent","queueSize":8,"type":0,"waitForMessage":false}],[["","inputConfig"],{"blocking":false,"group":"","id":3,"name":"inputConfig","queueSize":8,"type":3,"waitForMessage":false}],[["","inputControl"],{"blocking":true,"group":"","id":2,"name":"inputControl","queueSize":8,"type":3,"waitForMessage":false}]],"name":"ColorCamera","properties":[185,26,185,30,0,3,0,0,0,185,3,0,0,0,185,5,0,0,0,0,0,185,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,185,3,0,0,0,185,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,255,189,0,255,0,1,0,129,44,1,129,44,1,255,255,255,255,1,136,0,0,240,65,0,136,0,0,128,191,136,0,0,128,191,1,185,4,0,0,0,0,3,3,4,4,4,190]}],[1,{"id":1,"ioInfo":[[["","in"],{"blocking":false,"group":"","id":1,"name":"in","queueSize":8,"type":3,"waitForMessage":true}]],"name":"XLinkOut","properties":[185,3,136,0,0,128,191,189,5,118,105,100,101,111,0]}]]}
[19443010D13B291300] [1.7] [1709540135.205] [host] [debug] Asset map dump: {"map":{}}
[19443010D13B291300] [1.7] [0.912] [system] [info] Memory Usage - DDR: 0.12 / 333.46 MiB, CMX: 2.04 / 2.50 MiB, LeonOS Heap: 7.50 / 82.56 MiB, LeonRT Heap: 2.89 / 40.68 MiB / NOC ddr: 25 MB/s
[19443010D13B291300] [1.7] [0.912] [system] [info] Temperatures - Average: 35.47C, CSS: 36.77C, MSS 34.64C, UPA: 35.35C, DSS: 35.11C
[19443010D13B291300] [1.7] [0.912] [system] [info] Cpu Usage - LeonOS 70.25%, LeonRT: 1.66%
[19443010D13B291300] [1.7] [0.919] [system] [warning] PRINT:LeonCss: Temperature: Initialized driver.
Temperature: Sensor opened: CSS.
Temperature: Sensor opened: MSS.
Temperature: Sensor opened: UPA.
Temperature: Sensor opened: DSS.
I: [Timesync] [ 4880155] [XLin] startSync:186 Timesync | Callback not set
[19443010D13B291300] [1.7] [0.929] [system] [info] SIPP (Signal Image Processing Pipeline) internal buffer size '18432'B, DMA buffer size: '16384'B
connected
[19443010D13B291300] [1.7] [0.965] [system] [info] ImageManip internal buffer size '366080'B, shave buffer size '34816'B
[19443010D13B291300] [1.7] [0.965] [system] [info] ColorCamera allocated resources: no shaves; cmx slices: [10-15]
ImageManip allocated resources: shaves: [15-15] no cmx slices.
[19443010D13B291300] [1.7] [0.974] [system] [warning] PRINT:LeonCss: [1900-01-00 00:00:00.967] [info] Time taken to build the pipeline: 39ms
== FSYNC enabled for cam mask 0x0
CAM ID: 0, width: 3840, height: 2160, orientation: 3
getDefaultDtp: camType:2, color:1, orientation:3
DTP: ext_dtp_database
Loaded DTP, handle 0x841f7228
== SW-SYNC: 0, cam mask 0x1
!!! Master Slave config is: single_master_slave !!!
Starting camera 0, server 0x841f7228
[E] app_guzzi_command_callback():173: command->id:1
[E] app_guzzi_command_callback():193: command "1 0" sent
[19443010D13B291300] [1.7] [0.975] [system] [warning] PRINT:LeonMss: sippPalThreadCreate: Thread /SIPP created
Initing ImgPreproc system!
Pre-proc Enc mem used: 138240 bytes. Available: 366080
LRT - build pipeline call
[19443010D13B291300] [1.7] [0.985] [system] [warning] PRINT:LeonCss: [E] iq_debug_create():161: iq_debug address 0x8886c7c0
[E] hai_cm_driver_load_dtp():852: Features for camera A12N02A (imx378) are received
[E] set_dtp_ids():396: //VIV HAL: Undefined VCM DTP ID 0
[E] set_dtp_ids():405: //VIV HAL: Undefined NVM DTP ID 0
[E] set_dtp_ids():414: //VIV HAL: Undefined lights DTP ID 0
[E] camera_control_start():360: Camera_id = 0 started.
[19443010D13B291300] [1.7] [0.996] [system] [warning] PRINT:LeonCss: ============================ Configuring IMX378 with mode 0
[19443010D13B291300] [1.7] [1.007] [system] [warning] PRINT:LeonCss: ========== IMX378 /dev/i2c.Camera_0_sen, FSIN enable 0, output 0
[E] vpipe_conv_config():1465: Exit Ok
[E] callback():133: Camera CB START_DONE event.
[19443010D13B291300] [1.7] [1.018] [system] [warning] PRINT:LeonCss: inc_camera_process set exposure and gain
AF_TRIGGER on camera 0
[E] app_guzzi_command_callback():173: command->id:5
[E] camera_control_focus_trigger():604: Focus trigger succeeded camera_id = 0.
[E] app_guzzi_command_callback():218: command "5 0" sent
Starting Guzzi command handling loop...
[19443010D13B291300] [1.7] [1.018] [system] [warning] PRINT:LeonMss: MIPI DPHY 4 error, stat 0x30
[19443010D13B291300] [1.7] [1.073] [system] [warning] PRINT:LeonMss: finished sippPrePrepareLinesFreeRT
Stack trace (most recent call last) in thread 3108616:
#31 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d160da, in
#30 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d2153b, in
#29 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d1595c, in _PyObject_FastCallDictTstate
#28 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d1a780, in _PyFunction_Vectorcall
#27 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0cf23, in
#26 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0f140, in _PyEval_EvalFrameDefault
#25 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d22b83, in
#24 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d1a780, in _PyFunction_Vectorcall
#23 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0cf23, in
#22 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d12ba8, in _PyEval_EvalFrameDefault
#21 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d1aa31, in
#20 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d130f5, in _PyEval_EvalFrameDefault
#19 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d15e67, in _PyObject_MakeTpCall
#18 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d1601a, in
#17 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6db76d3, in
#16 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6cfeb57, in PyDict_New
#15 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6cfcd77, in _PyObject_GC_New
#14 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6cfce9c, in
#13 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d7e5aa, in
#12 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0563b, in
#11 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d06aa1, in
#10 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d21278, in
#9 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d15d89, in
#8 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0a9d7, in
#7 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0abb6, in
#6 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d25544, in
#5 Object "/lib64/libpython3.9.so.1.0", at 0x7f13d6d0aae6, in
#4 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7f13c7956232, in
#3 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7f13c79cf029, in
#2 Object "/home/hugofrisk/Documents/depthai-segfault-mre/venv/lib64/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so", at 0x7f13c7c24cab, in dai:😃ataOutputQueue::~DataOutputQueue()
#1 Object "/lib64/libstdc++.so.6", at 0x7f13c74e3256, in std::thread::join()
#0 Object "/lib64/libc.so.6", at 0x7f13d6aae4f4, in
Segmentation fault (Address not mapped to object [0x7f13b4f41990])
[19443010D13B291300] [1.7] [1.092] [XLinkOut(1)] [info] Increasing 'video' stream size from 5242880B to 12492800B
[19443010D13B291300] [1.7] [1.546] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 8
[19443010D13B291300] [1.7] [1.913] [system] [info] Memory Usage - DDR: 113.90 / 333.46 MiB, CMX: 2.46 / 2.50 MiB, LeonOS Heap: 21.67 / 82.56 MiB, LeonRT Heap: 3.52 / 40.68 MiB / NOC ddr: 904 MB/s
[19443010D13B291300] [1.7] [1.913] [system] [info] Temperatures - Average: 37.88C, CSS: 39.58C, MSS 37.71C, UPA: 37.01C, DSS: 37.24C
[19443010D13B291300] [1.7] [1.913] [system] [info] Cpu Usage - LeonOS 42.65%, LeonRT: 11.98%
[19443010D13B291300] [1.7] [1709540136.356] [host] [debug] Device about to be closed...
[19443010D13B291300] [1.7] [1709540136.585] [host] [debug] Shutdown OK
[19443010D13B291300] [1.7] [1709540136.585] [host] [debug] Timesync thread exception caught: Couldn't read data from stream: 'timesync' (X_LINK_ERROR)
[19443010D13B291300] [1.7] [1709540136.585] [host] [debug] Log thread exception caught: Couldn't read data from stream: 'log' (X_LINK_ERROR)
[2024-03-04 09:15:36.586] [depthai] [debug] XLinkResetRemote of linkId: (4)
[2024-03-04 09:15:36.586] [depthai] [debug] DataOutputQueue (video) closed
[19443010D13B291300] [1.7] [1709540136.723] [host] [debug] Device closed, 366
done

jakaskerl

Hi @PepperoniPingu
Was able to reproduce this on RPI.
Will look into it more.

Thanks,
Jaka

PepperoniPingu

Thanks, I look forward to see what you find!

diddy-cam

have seen the same issue. would be nice to get a patch as we're in need of a robust setup that can run for weeks without hiccups