Sending OAK Camera frames from Python DepthAI API to C++ or C#

yishu_corpex

I'm successfully running my OAK camera Python scripts and I'm able to deploy my custom trained Yolo model on the camera.

However, now I want to access the frames that I get from the OAK camera (within the python script) in C++ or C# without shifting the entire script to C++ using the DepthAI core library. The reason for having this functionality is that it would allow me to directly feed those frames into another Machine Vision based IDE tool (which only supports C# but can get a workaround if can access frames in C++) that we use for our client side environment and user interface.

Currently as a simple solution we save the frames (in the python scripts) on disk and then read them back in that Machine Vision IDE tool. It does work but it adds a lot of overhead to the entire process. The reason for getting access to these frames from the python script in C++ or C# is to reduce the above-mentioned overhead and get direct access to the frames within the Machine Vision IDE tool.

Your support and any help in this regards would be highly appreciated as I am stuck on this part since a long time.

Thanks & Regards

Yishu

jakaskerl

Hi @yishu_corpex
I'd suggest cross-sharing memory between Python and C# (shared memory block):

Python side:

import mmap
import numpy as np
import os

# Example buffer size for a 1920x1080 image with 3 channels (RGB)
buffer_size = 1920 * 1080 * 3

# Create a new memory-mapped file
with mmap.mmap(-1, buffer_size, "Local\\MySharedMemory") as mm:
    frame = np.random.randint(0, 255, (1080, 1920, 3), dtype=np.uint8)  # example frame
    mm.write(frame.tobytes())  # Write the numpy array to shared memory

using System;
using System.IO.MemoryMappedFiles;
using System.Runtime.InteropServices;

public class Program
{
    public static void Main()
    {
        int width = 1920;
        int height = 1080;
        int channels = 3;
        
        using (var mmf = MemoryMappedFile.OpenExisting("MySharedMemory"))
        {
            using (var accessor = mmf.CreateViewAccessor())
            {
                byte[] buffer = new byte[width * height * channels];
                accessor.ReadArray(0, buffer, 0, buffer.Length);

                // Optionally convert to a managed array, handle as needed
            }
        }
    }
}

Thoughts?

yishu_corpex

@jakaskerl That does look like a nice approach. In fact I did give this a try, however after running the python script that you shared above, C# isn't able to find the Memory Mapped File. It errors out and gives the following message.

Exception has occurred: CLR/System.IO.FileNotFoundException

An unhandled exception of type 'System.IO.FileNotFoundException' occurred in System.IO.MemoryMappedFiles.dll: 'Unable to find the specified file.'

I am pretty new to C# so trying to understand what might be the issue here? I look forward to your response.

jakaskerl

Hi @yishu_corpex
Not sure how you were running the script, but you have to make sure the python file stays open when accessing the memory block.

import mmap
import numpy as np
import os
import time  # Importing time for demonstration purposes

# Buffer size for a 1920x1080 image with 3 channels (RGB)
buffer_size = 1920 * 1080 * 3

# Open a memory-mapped file that stays open beyond the 'with' block
mm = mmap.mmap(-1, buffer_size, "Local\\MySharedMemory")

try:
    frame = np.random.randint(0, 255, (1080, 1920, 3), dtype=np.uint8)
    mm.write(frame.tobytes())
    # Keep the Python script running for demonstration
    while True:
        time.sleep(1)  # Sleep to prevent busy waiting
finally:
    mm.close()

Also try running the script as admin if you can.

Thanks,
Jaka

yishu_corpex

Yes, you are right. The python file was closing after the execution. Thanks for your help.

Thanks & Regards

Yishu