Neural Networks with multiple sequential frames as inputs

KhanCh

Hi all,

i just purchased an OAK D pro and I really like it. Object detections and neural networks with one RGB frame inputs works as a charm.

Now, I ran over this problem: I want to implement neural network to estimante the heart rate of a person. For this, the networks usually need the last 180-128 frames of a webcam as input. Can you give me some advices how I can buffer or save 180 consecutive frames before piping it to the neural network node? The closest things, I found on the forum was the 3 image concat example.

Best,

Khan

vital

KhanCh This example might help:

https://docs.luxonis.com/projects/api/en/latest/samples/MobileNet/video_mobilenet/#video-mobilenetssd

and I just did a little test (I thought it was an interesting thing to practice since I am learning python). Maybe this will be useful? (Probably not, but who knows):

from depthai_sdk import OakCamera
from depthai_sdk.classes import FramePacket
import threading

frameStore = []
framesWanted = 180
frameIter = 0
with OakCamera() as oak:
    color = oak.create_camera('color')
    color.set_num_frames_pool(180)
    
    def checkFrames():
        global frameIter
        while True:
            if frameIter > 180:
                print(frameStore)
                frameIter = 0     
    
    def doFrameStore(packet: FramePacket):
        global frameStore
        global frameIter
        frame = packet.frame
        frameStore.append(frame)
        frameIter +=1
        print(frameIter)
        
    t = threading.Thread(target = checkFrames)
    t.daemon = True
    t.start()
    oak.callback(color.out.main, callback=doFrameStore)
    oak.start(blocking=True)

KhanCh

Hi vital,

thanks a lot for the advice. As far as I can see the 180 img frames are stored locally on the host machine in the frameStore var and not on the OakCamera or am I missing something? So technically, I always put the lastest image frame and store it locally but I never address the frames_pool?

And is it possible to use the frame_pool as input for a nn on the oak directly without going to the host machine? basically:

On the oakd: buffer 180 in the frames_pool => use it as an input for a custom blob model

Best

vital

KhanCh It was just an experiment I did; I don't think it is particularly useful for your case but I wanted to share in case it happened to be. I am not at all a depthai expert -- sorry if there was a misunderstanding.

vital

KhanCh If you want to do it on device maybe look at scripting -- I am not sure if it will be able to hold 180 frames in memory since it only has 512MB total, including what is needed for operations. The two stage detection script creates a queue on device and goes through it. It may give you some ideas:

https://github.com/luxonis/depthai/blob/main/depthai_sdk/src/depthai_sdk/components/template_multi_stage_script.py

KhanCh

Hi vital ,

no worries! For brunching and transporting the data to the host system that good. I also think, it should be possible to buffer all 180 frames on the host machine and then use it as an input for the oak camera again as shown in https://docs.luxonis.com/projects/api/en/latest/samples/MobileNet/video_mobilenet/#video-mobilenetssd . So schematically you do:

Oak-D => frames => buffer on host machine => processing on host machine => oak-d => nn.input => AI-processing.

Basically, to use the host machine as external storage.

I just wondered, that I couldn't find something in the document for accessing img frames from the frames_pool? Does anyone know how to do that?

vital

KhanCh How about this:

https://github.com/luxonis/depthai/blob/main/depthai_sdk/examples/mixed/switch_between_models.py

Take frames from host, send them to script, script takes frames and pushes them to out which is input to nn.

erik

vital / KhanCh , For the heartrate estimation, perhaps you could reuse some of Cortic loigic? Demo here (at 4:14):

Relevant code from their repo:

Quickly glancing over the code, they run face deteciton on OAK, then send frames to the host where they run heartrate estimation.