Get frames from Oak 1 Max

JaimilDalwadi

Just looking for some documentation or examples on how to save frames coming in from an oak at 60 fps (max) into a folder?

I tried doing this with cv2.imwrite but this only leads to 6-7 frames being saved per second. Any way to save as many frames as possible (either by delaying saving to after the camera connection is closed or converting using ffmpeg)?

I tried using ffmpeg but the cap for converting h265 files to mp4 is 30fps which isn't optimal for my situation.

jakaskerl

Hi JaimilDalwadi
You can try the SDK in case it yields better results.

from depthai_sdk import OakCamera, RecordType
import time

duration = 5
with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080P', fps=60, encode='H265')

    # Sync & save all (encoded) streams
    oak.record([color.out.encoded], './record')
    oak.start()
    start_time = time.monotonic()
    while oak.running():
        if time.monotonic() - start_time > duration:
            break
        oak.poll()

AFAIK the conversion for ffmpeg should't be capped to 30fps. Can you verify?

Thanks,
Jaka

JaimilDalwadi

jakaskerl

JaimilDalwadi

Thank you for the quick reply. Although the code you sent works great, I was hoping to save the frames as separate JPEG files instead of an mp4. So if it ran for 3 seconds, I would hope to get ~90 (if 30 fps) or 180 (if 60 fps) JPEGS. Do you know where I can get documentation on saving separate JPEGS efficiently?

As for the video conversion, it was actually with Video Encoder, my apologies:
VideoEncoder configuration exceeds the total (all the encoder streams combined) 4K@30 (248.8 MPix/s) for H.26x only, or about 450 MPix/s for JPEG only - maximum that the hardware is capable of

[18443010C1B66C0E00] [1.4] [0.915] [VideoEncoder(1)] [critical] VideoEncoder out of resources

jakaskerl

Hi JaimilDalwadi
Any chance you can use multithreading/multiprocessing to save images then? It could work even with OpenCV. Since the images don't need to be synced up, there is no need to worry about running everything in the same thread/process.

Thoughts?
Jaka

JaimilDalwadi

jakaskerl

I can try, but one thing I did was save the frames I got to a dict instead and then tried saving to jpegs, but that still only returned the same # of frames. Is there any reason why there might be an fps drop in the following code?

        controlQueue = device.getInputQueue(controlIn.getStreamName(),maxSize=30, blocking=False)
        qPreview = device.getOutputQueue("preview", maxSize=30, blocking=False)

        # save_video = cv2.VideoWriter('result_video.avi', -1, 60.0, (1920, 1080))
        

        # For Testing: To record time from when the frame is read from the queue until the frame is successfully written to the file
        # endTime = time.time() + 3
        temp = dict()
        fc = 0
        while True:

            #dai.Clock.now()
            img = qPreview.get()
            #time = img.getTimestamp()
            # print(qPreview.getFps())
            # frame_count = 0
            fc += 1
            frame_name = f"frame_{time.time()}.jpg"
            output_path = f"{output_folder}/{frame_name}"
            temp[frame_name] = img
            

            

            if cv2.waitKey(1) == ord('q'):
                break
    print(fc)
    for filename, f in temp.items():
        frame = f.getCvFrame()
        output_path = f"{output_folder}/{filename}"`

jakaskerl

Hi JaimilDalwadi
You can run your code withDEPTHAI_LEVEL=INFO python3 <script_name> if you wish to view the trace for every image taken.

Also what FPS are you getting with the above method?

Thanks,
Jaka

JaimilDalwadi

jakaskerl
Somehow only getting 6-7 frames per second.

jakaskerl

Hi JaimilDalwadi
Can you create a MRE so I can test it locally?

Thanks,
Jaka

JaimilDalwadi

jakaskerl
Hey, I can send one later.

In the meantime, can you refer me on how to decode/save JPEGS from a VideoEncoder bitstream in H265 or MJPEG format?

jakaskerl

Hi JaimilDalwadi
Assuming you have .h265 or .mjpeg files in some folder, you can use ffmpeg library to decode and save the file as the desired file type. (docs)

Thanks,
Jaka

JaimilDalwadi

jakaskerl
Is there no way to do it in realtime as the frames are getting written?

` with open(rawFile, 'wb') as videoFile:
print("Press Ctrl+C to stop encoding...")
try:
while True:
h265Packet = q.get() # Blocking call, will wait until a new data has arrived
h265Packet.getData().tofile(videoFile) # Appends the packet data to the opened file

do something here with the packet data to save as jpegs?

JaimilDalwadi

jakaskerl
Here's an MRE of just saving images directly:


import cv2
import numpy as np
import depthai as dai
import time

#TODO: Figure out how to ensure the fps for the OAK camera has been changed to 60
def save_frames(output_folder):

    pipeline = dai.Pipeline()

    oakCam = pipeline.create(dai.node.ColorCamera)
    print(str(oakCam.getFps()))
    oakCam.setFps(60)
    manip = pipeline.create(dai.node.ImageManip)
    controlIn = pipeline.create(dai.node.XLinkIn)
    manipOut = pipeline.create(dai.node.XLinkOut)
    controlIn.setStreamName('control')
    manipOut.setStreamName("preview")


    topLeft = dai.Point2f(0.2, 0.2)
    bottomRight = dai.Point2f(0.8, 0.8)

    # to save the video using opencv
    # cap = cv2.VideoCapture(0)

    oakCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    manip.setMaxOutputFrameSize(12441600)

    oakCam.video.link(manip.inputImage)
    manip.out.link(manipOut.input)
    controlIn.out.link(oakCam.inputControl)

    with dai.Device(pipeline) as device:
        controlQueue = device.getInputQueue(controlIn.getStreamName(),maxSize=30, blocking=False)
        qPreview = device.getOutputQueue("preview", maxSize=30, blocking=False)

        temp = dict()
        fc = 0
        while True:

            img = qPreview.get()
            fc += 1
            frame_name = f"frame_{time.time()}.jpg"
            output_path = f"{output_folder}/{frame_name}"
            temp[frame_name] = img
            
            if cv2.waitKey(1) == ord('q'):
                break
        
    print(fc)
    for filename, f in temp.items():
        frame = f.getCvFrame()
        output_path = f"{output_folder}/{filename}"
        cv2.imwrite(output_path,frame)
    cv2.destroyAllWindows()

# Example usage
output_folder = f"./testPhotos" # change as needed

save_frames(output_folder)

Here's another MRE of a script to save videos and convert using ffmpeg and trying to save the frames as JPEGS:



##
#!/usr/bin/env python3

# Script to save videos in .h265 format from oak camera

import depthai as dai
import time
import os
import cv2
# Create pipeline
pipeline = dai.Pipeline()

# Define sources and output
camRgb = pipeline.create(dai.node.ColorCamera)
ctrl = dai.CameraControl()
videoEnc = pipeline.create(dai.node.VideoEncoder)
xout = pipeline.create(dai.node.XLinkOut)
stillEnc = pipeline.create(dai.node.VideoEncoder)

xout.setStreamName('h265')

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
fps = 30
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
videoEnc.setDefaultProfilePreset(fps, dai.VideoEncoderProperties.Profile.H265_MAIN)
camRgb.setSensorCrop(0.2, 0.1) # By default 0.25, 0.25 for center crop




# Linking
camRgb.video.link(videoEnc.input)
videoEnc.bitstream.link(xout.input)

with dai.Device(pipeline) as device:
 
    # Output queue will be used to get the encoded data from the output defined above
    q = device.getOutputQueue(name="h265", maxSize=30, blocking=True)

    # The .h265 file is a raw stream file (not playable yet)
    curr = int(round(time.time()))
    rawFile = rf'.\rawStreams\{curr}.h265'
    convertedFile = rf'.\convertedStreams\{curr}.mp4'
    temp = dict()
    fc = 0
    with open(rawFile, 'wb') as videoFile:
        print("Press Ctrl+C to stop encoding...")
        try:
            while True:
                h265Packet = q.get()  # Blocking call, will wait until a new data has arrived
                h265Packet.getData().tofile(videoFile)  # Appends the packet data to the opened file
                fc += 1
                frame_name = f"frame_{int(time.time()*1000)}.jpeg"
                temp[frame_name] = h265Packet


        except KeyboardInterrupt:
            # Keyboard interrupt (Ctrl + C) detected
            pass
    print(fc)
    output_folder = rf".\testPhotos"
    for filename, f in temp.items():
        with open(filename, "wb") as fi:
            fi.write(f.getData())
            print('Image saved to', filename)

    print("To view the encoded data, convert the stream file (.h265) into a video file (.mp4) using a command below:")
    # print("ffmpeg -framerate 30 -i video.h265 -c copy video.mp4")
    command = f"ffmpeg -framerate {fps} -i {rawFile} -c copy {convertedFile}"
    os.system(command)

JaimilDalwadi

If anyone can help,

I managed to save frames at 30 FPS but I think it's capping there because of this warning:

[184430108104DB0E00] [1.1] [0.801] [ColorCamera(0)] [warning] LCM48 supported resolutions: 4_K / 4000X3000 / 5312X6000. Defaulting to 4_K

I have it set to:

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

camRgb.setVideoSize(1920, 1080)

But I'm guessing it defaults to 4k. Can anyone help? This is on the Oak1 Max

jakaskerl

Hi JaimilDalwadi
The lowest sensor resolution for Max is indeed 4K. You can try downscaling it using ISP.
This capping you are referring to, does is come from camera being unable to process images fast enough or the ffmpeg?

Thanks,
Jaka

JaimilDalwadi

jakaskerl

This time I'm not using ffmpeg, I'm simply saving them to a dictionary and then saving them to files at the end like in the below script, so I'm not sure if it's being capped anywhere:

#!/usr/bin/env python3

# Script to save videos in .h265 format from oak camera

import depthai as dai
import time
import os
import cv2
# Create pipeline
pipeline = dai.Pipeline()

# Define sources and output
camRgb = pipeline.create(dai.node.ColorCamera)
ctrl = dai.CameraControl()
xout = pipeline.create(dai.node.XLinkOut)
xoutStill = pipeline.create(dai.node.XLinkOut)


xout.setStreamName('h265')
xoutStill.setStreamName('preview')

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.RGB)
fps = 60
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
camRgb.setPreviewSize(360,360)
# camRgb.setIspScale(1,2)

# camRgb.setVideoSize(1920, 1080)
stillEnc.setDefaultProfilePreset(fps, dai.VideoEncoderProperties.Profile.H265_MAIN)




# Linking

camRgb.preview.link(xoutStill.input)

with dai.Device(pipeline) as device:
    device.setLogLevel(dai.LogLevel.DEBUG)
    device.setLogOutputLevel(dai.LogLevel.DEBUG)
    # Output queue will be used to get the encoded data from the output defined above
    qPreview = device.getOutputQueue(name="preview", maxSize=60,blocking=False)
    curr = int(round(time.time()))
    rawFile = rf'.\rawStreams\{curr}.h265'
    convertedFile = rf'.\convertedStreams\{curr}.mp4'
    temp = dict()
    fc = 0
    starttime = time.time()
    totaltime = 0
    with open(rawFile, 'wb') as videoFile:
        print("Press Ctrl+C to stop encoding...")
        
        try:
            while True:
                fc += 1
                frame = qPreview.get()
                frame_name = f"frame_{int(time.time()*1000)}.jpeg"
                temp[frame_name] = frame
                # cv2.imshow("rgb",frame)


        except KeyboardInterrupt:
            # Keyboard interrupt (Ctrl + C) detected
            totaltime = time.time() - starttime
            pass
    print(f"Total frame count: {fc}")
    print(f"Total time run: {totaltime}")
    print(f"FPS: {fc/totaltime}")
    output_folder = rf".\testPhotos"
    for filename, f in temp.items():
        pathout = f"../testPhotos/{filename}"
        f = f.getCvFrame()
        cv2.imwrite(pathout,f)

jakaskerl

Hi JaimilDalwadi
And the number of frames you are getting?
Since you have set the queue size to 60, when ending the stream with ctrl+c, I think there might still be some frames left in that queue that haven't yet been read, so you get lower number of frames than you should be getting.

Thoughts?
Jaka

JaimilDalwadi

jakaskerl

I was getting around 29 fps. I'm not sure if its the queue size, I tested with the ISP downscaling to (1,2) but it still only gives about the same. Do you think it could be a camera issue? Or something with the configuration such that it keeps using 4k resolution or limits it to 30 fps instead for some reason?

jakaskerl

Hi JaimilDalwadi
I don't know for sure. You can try running your script with DEPTHAI_LEVEL=TRACE python3 <file> to see how much time each frame acquisition took.

Thanks,
Jaka

JaimilDalwadi

jakaskerl

Thank you for all the help! I was able to save frames for x number of seconds on command.

I am facing issues though as it starts to not get any frames. I basically wait on the keyboard press for "c" and set some flags and run the following code:
`

if getImgs and time.time()-starttime <= 1:

frame = qPreview.tryGet()

if frame is not None:

fc += 1

frame_name = f"frame_{int(time.time()*1000)}.jpeg"

temp[frame_name] = frame

`
When I press c the first time, it works and gets around 40 fps, but when I press c the second time after everything has saved, it only gets around 10, and then anytime I press c again, it just doesn't get anything. Could it be anything to do with the video pipeline? Is it possible the queue gets empty for some reason or the camera doesn't get any more frames for some reason?

These are the pipelines:

xoutPrev = pipeline.create(dai.node.XLinkOut)

xoutPrev.setStreamName('preview')

camRgb.video.link(xoutPrev.input)
qPreview = device.getOutputQueue(name="preview", maxSize=10,blocking=False)

jakaskerl

Hi JaimilDalwadi
Can you add the code as well so I can test locally? And make it as minimal as you can.

Thanks,
Jaka