Pre-process raw frames from OAK-D Pro W PoE/output predictions

ZhifeiShen

Hi,

I currently have a custom yolov8n model trained and deployed on the OAK camera but I'd like to pre-process the incoming frames before putting them through the neural network. My model is 640x640 and the original training data used to construct the model was 1440x640, barrel distortion corrected, resized and padded to fit the 640x640 dimensions. Is there a way to do the same to the raw frames obtained by the camera?

Second, I'd like to start testing the accuracy of inferences with pre-existing images saved in the computer. Is there also a way to "bypass" the camera feed and continually perform inferences with the onboard neural net on the saved image instead?

Lastly, how would I send the output bounding box coordinates, classes and confidences to be used with another Python script?

For reference I've attached the sample code I'm using to drive the camera/neural net:

`from depthai_sdk import OakCamera, ArgsParser
import argparse

parse arguments

parser = argparse.ArgumentParser()
parser.add_argument("-conf", "--config", help="Trained YOLO json config path", default='model/yolo.json', type=str)
args = ArgsParser.parseArgs(parser)
with OakCamera(args=args) as oak:
color = oak.create_camera('color')
nn = oak.create_nn(args['config'], color, nn_type='yolo', spatial=True)
oak.visualize(nn, fps=True, scale=2/3, visualizer='opencv')
oak.visualize(nn.out.passthrough, fps=True, visualizer='opencv')
oak.start(blocking=True)`

Thanks in advance!

jakaskerl

ZhifeiShen My model is 640x640 and the original training data used to construct the model was 1440x640, barrel distortion corrected, resized and padded to fit the 640x640 dimensions. Is there a way to do the same to the raw frames obtained by the camera?

nn.config_nn("LETTERBOX") i think should work the same as setResizeThumbnail on API to resize and pad the image.

ZhifeiShen Is there also a way to "bypass" the camera feed and continually perform inferences with the onboard neural net on the saved image instead?

You can feed the images manually with XlinkIn nodes, or using SDK: https://docs.luxonis.com/projects/sdk/en/latest/samples/replay/SDK_counter/

ZhifeiShen Lastly, how would I send the output bounding box coordinates, classes and confidences to be used with another Python script?

I guess you could use a socket to transfer the data.

Thanks,
Jaka

ZhifeiShen

jakaskerl

Thanks Jaka, I've tried to implement the code found in the linked pages but I'm running into issues - namely, I want to run a custom YOLOV8 model on a custom image. My current code is here but it crashes when I try to set the saved image "SAVED_IMAGE.png" as packet.frame. How do I make it work?

`import cv2
from depthai_sdk import OakCamera
from depthai_sdk.classes import DetectionPacket

model_name = "CUSTOM_MODEL.json"
curr_img = cv2.imread('./SAVED_IMAGE.png')

def callback(packet: DetectionPacket):
packet.frame = curr_img #### THIS IS THE OFFENDING LINE
visualizer = packet.visualizer
num = len(packet.img_detections.detections)
print('New msgs! Number of items detected:', num)
visualizer.add_text(f"Number of dets: {num}", position=TextPosition.TOP_MID)
visualizer.draw(packet.frame)
cv2.imshow(f'frame {packet.name}', packet.frame)

with OakCamera() as oak:
color = oak.create_camera('color')
nn = oak.create_nn(model_name, color, nn_type='yolo', spatial=True)
nn.config_nn(resize_mode="LETTERBOX")
oak.visualize(nn, callback=callback)
oak.start(blocking=True)`

ERROR MESSAGE:
File "inference_saved_img.py", line 40, in callback packet.frame = curr_img AttributeError: can't set attribute

jakaskerl

Hi ZhifeiShen
Inference has already been done on a detection packet. when inside callback so there is no point setting the frame to yours.
You have to instantiate the cameras with your own images (adding them at the beginning of the pipeline, not at the end).

https://docs.luxonis.com/projects/sdk/en/latest/samples/replay/SDK_counter/

Thanks,
Jaka

ZhifeiShen

jakaskerl

Thanks Jaka, I was able to get the pre-prepare images to work.

However, can you point me to the documentation that does the auto-undistort for the camera? Something like this:

Thanks!

jakaskerl

Hi ZhifeiShen
If you don't intend to use NN, you can use the camera node to automatically undistort the streams. But this is early development and has some issues.

Then there's manual undistort.

Thanks,
Jaka