Record and replay stereo videos

tarocal

Hello,

I am trying to record a video using my OAK-D. This vide need to have the setero information as well. So I modified the encode example and used it to record the RGB, mono left and mono right camera feeds. The depthmap recording would not be as useful, as I might have other ways of doing the stereo calculations later on.

I have the three vides ready. But I when replay the three videso at the same time (using the OAD-D), it is very slow. This is understandable, this was not the processors main use, so it is slower. But I am now trying to see if I can use the mobilenet on the video feed (similar to the object tracker example - an example that crashes a lot BTW), The '.get()' function does not seem to work.

The code just hangs there, but the camera heats up. Am I doing something wrong to the processor? Is there a simple way to do this? (I remembr an example from a while ago that did it on images). Is this not a funstionslity that is possible? Should I post my code here?

Details
Host: KUbuntu on VMWare professional 15+
Language: Python 3.6+
DepthAI: 2.4.0.0 (also tried with 2.3.0.0)

Any help would be appreciated.

Update:

I have changed out 'get' for 'tryget' and made the code get a frame only when it is available. But the problem still presists with the Stereo side of things. The error being:

with dai.Device(pipeline) as device: RuntimeError: , LRT: 'StereoDepth(4) - StereoDepth | Invalid resolution: Width 0, Height 0'

I'm assuming there is still no frame ot be obtained, so it fails?

erik

Hello tarocal ,
first off, I apologize for not making this functionality more visible; we do already have record/replay prototype script available here. This functionality will get merged into depthai_demo.py script so people will be able to easily record/replay their recordings with all the extensibility of the depthai_demo script.
About the implementation: the record.py script saves rgb/rectified mono frames, you can optionally use -e to encode them (so frames will be saved as .jpeg). replay.py script will send both mono frames to the StereoDepth node on the device where the depth map will get calculated. It will also send rgb frames to the device for NN inference. Note: this functionality was tested with depthai 2.3.0.0.
Thanks, Erik

mjvdg1203

Hello, This is realy a nice function to have. unfortunately it won't run on a windows machine.(x64).
iam getting the following error :-(
when i try to record is there a quick sollution for this, except using linux.

python record.py -t 10 -m 60 -p "data" --dirty -depth -mono
Connecting to MX 14442C1041D162D700
['left', 'right', 'depth']
Traceback (most recent call last):
  File "C:\data\camera\depthai-experiments\gen2-replay\record.py", line 255, in <module>
    recording_devices.append(Record(dest=dest, device=device, mxId=device_info.getMxId()))
  File "C:\data\camera\depthai-experiments\gen2-replay\record.py", line 203, in __init__
    self.store_p.start()
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'depthai.Device' object

C:\data\camera\depthai-experiments\gen2-replay>Connecting to MX 14442C1041D162D700
['left', 'right', 'depth']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 268, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\data\camera\depthai-experiments\gen2-replay\record.py", line 255, in <module>
    recording_devices.append(Record(dest=dest, device=device, mxId=device_info.getMxId()))
  File "C:\data\camera\depthai-experiments\gen2-replay\record.py", line 203, in __init__
    self.store_p.start()
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Marc\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError(
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

erik

Hello mjvdg1203 ,
could you try running the record.py from the branch replay_memory_fix (github)? There is no support for multiple devices, but from checking the error it seems that could be the issue.
Thanks, Erik

mjvdg1203

erik Thanks erik, but it still is not running. and gives the same error. i will use a linux machine instead.

erik

Thanks for testing mjvdg1203, I actually plan to refactor the record/replay soon and I will make sure to test it on windows as well.
Thanks again, Erik