Trouble replaying video file

jack-t

I have an h265 file that was recorded on the oak-d. I am currently attempting to replay that file using the oak-d in order to run a custom model on each frame. I want to run the model after the original recording has finished which is why I am attempting to use the replay feature.

I am attempting to roughly follow the example here, but I'm having a lot of difficulty pulling the frames from the video. For my current test, I am simply trying to read the first few frames from the h265 file and save them to a new video file.

Here is my current code:

from depthai_sdk import OakCamera, RecordType
import depthai as dai
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('-p', '--path', default="data", type=str, help="Path to the stored replay")
args = parser.parse_args()

with open('video2.h265', 'wb') as videoFile:
    with OakCamera(replay=args.path) as oak:
        color = oak.create_camera('color', encode='H265')
        nn = oak.create_nn('mobilenet-ssd', color)
        # oak.record([color.out.encoded], './', RecordType.VIDEO)
        oak.visualize([color.out.encoded], fps=True)
        
        pipeline = oak.build()

        oak.start()

        print(oak.device.getInputQueueNames())
        print(oak.device.getOutputQueueNames())

        q = oak.device.getOutputQueue('1_bitstream') # Create output queue after calling start()

        try:
            frame_count = 0
            while oak.running() and frame_count < 450:
                if q.has():
                    h265Packet = q.get()  # Blocking call, will wait until a new data has arrived
                    print(h265Packet)
                    h265Packet.getData().tofile(videoFile)  # Appends the packet data to the opened file
                    frame_count +=1 
                    print(frame_count)
                oak.poll()
        except KeyboardInterrupt:
            # Keyboard interrupt (Ctrl + C) detected
            pass


        # xout = pipeline.create(dai.node.XLinkOut)
        # xout.setStreamName('h265')
        # # color.node.video.link(xout.input)

        # videoEnc = pipeline.create(dai.node.VideoEncoder)
        # videoEnc.setDefaultProfilePreset(30, dai.VideoEncoderProperties.Profile.H265_MAIN)

        # oak.video.link(videoEnc.input)
        # videoEnc.bitstream.link(xout.input)

Here is the error I receive when running this code:

pi@oak2:~/jack $ python replay_example.py -p /home/pi/storage/2023-07-12/color.h265
[2023-07-19 15:24:19] INFO [root.__init__:111] Available streams from recording: ['color']
[18443010D1805B1200] [1.1] [2.671] [DetectionNetwork(3)] [warning] Network compiled for 6 shaves, maximum available 15, compiling for 7 shaves likely will yield in better performance
['color_in']
['1_bitstream']
[18443010D1805B1200] [1.1] [4.083] [VideoEncoder(1)] [critical] Out of memory while creating pool for 'bitstream' frames. Number of frames: 4 each with size: 12604928B

After printing this out, the code just hangs, with q.has() always being False.

*Note - working through the example it looks like I would be able to pull the model output without trouble, however I will need to rewrite some of the video file as the model I plan to use will act as a filter for which frames to save and which to not save for future use. I would like to use the oak encoder to write the frames we want to save, as I am running the oak on a rasberry pi, and using cv2 to write those frames via the rasberry pi is too slow to be viable.

jack-t

Update:

Following this example, I was able to get the replay to work and record to a new file, however it is not using the video encoder. When I drilled down, AVWriter uses the av package, and the other video writers seem to use opencv, which is the exact thing I was trying to avoid. This is confirmed by the runtime being roughly 2 fps, to copy a video that was originally recorded at 30 fps on the same oak-d device.

Is what I'm trying to do even possible - can you read from a file using an oak pipeline, then write using the oak videoencoder?

jakaskerl

Hi jack-t
If you wish to force the stream to go through encoder, you can call pipeline = oak.build() and operate with the pipeline the same way you would in the api.

Thoughts?
Jaka

jack-t

I'm not sure what you mean. Is that different than what I was doing in the above code? Can you provide some example code to demonstrate the method you are recommending?

I believe in the above code by setting the encode variable in create_camera I am forcing it through the encoder. I do seem to get a bitstream output as I think one would expect from the encoder, however I am hitting that error about running out of memory, which I am not sure how to rectify. I have found I get the same error whether I use h265 or MJPEG encoding.

Note, that the videos are recorded only on the color camera, and are at 4k @ 30fps. That shouldn't be too much for the encoder in theory right? So I'm not sure why it's hitting a memory error.

jack-t

Update:

Upon some further digging, I was able to get a version of this code to run:

from depthai_sdk import OakCamera, RecordType
import depthai as dai
import argparse
import time

parser = argparse.ArgumentParser()
parser.add_argument('-p', '--path', default="...", type=str, help="Path to the stored replay")
args = parser.parse_args()

with open('video7.mjpeg', 'wb') as videoFile:
    with OakCamera(replay=args.path) as oak:
        color = oak.create_camera('color', encode='MJPEG', fps=30)
        # nn = oak.create_nn('mobilenet-ssd', color)
        # oak.record([color.out.encoded], './', RecordType.VIDEO)
        oak.visualize([color.out.encoded], fps=30)
        
        pipeline = oak.build()

        oak.start()

        print(oak.device.getInputQueueNames())
        print(oak.device.getOutputQueueNames())

        q = oak.device.getOutputQueue('1_bitstream') # Create output queue after calling start()

        try:
            frame_count = 0
            start = time.time()
            while oak.running() and frame_count < 30:
                # if q.has():
                oak.poll()
                h265Packet = q.get()  # Blocking call, will wait until a new data has arrived
                print(h265Packet)
                h265Packet.getData().tofile(videoFile)  # Appends the packet data to the opened file
                frame_count +=1 
                print(frame_count)
            end = time.time()
            print('')
            print(end - start)
        except KeyboardInterrupt:
            # Keyboard interrupt (Ctrl + C) detected
            pass

However, it seems to be only running at ~2 fps. I was expecting that since it is using the video encoder/decoder it would run a good bit faster than this. Is there anything else I can try to speed up the replay process leveraging the oak hardware?

*Note, if I change the encoding to H265 instead of MJPEG I get the same memory error I was getting previously.

jakaskerl

Hi jack-t
It's the initial read of the video that's causing the bad fps. Removing the encoder does speed it up a bit, but it's never full speed. If you wish to achieve higher FPS, only thing you can do currently is to lower the video resolution. Raspberry pi's processing capabilities play a role in this as well; for example, running the same script on my old laptop gives about 15fps.

Thanks,
Jaka

jack-t

If I remove the encoder, how do I get access to the video stream? Do you have an example I can look at?

jakaskerl

Hi jack-t

from depthai_sdk import OakCamera

with OakCamera(replay="path/to/video") as oak:
    color = oak.create_camera('color', fps=30)

    oak.visualize([color], fps=True)  # Display encoded output
    oak.start(blocking=True)

Thanks,
Jaka