Unable to get the coordinates and send the value to another Raspberry Pi.

uk0310

Hi,
I'm using OAK-D camera.
I'd like to get the coordinates and send the value to another Raspberry Pi.
How can that be achieved?

erik

Hello uk0310 ,
You can get the spatial coordinates of objects with eg. RGB & MobilenetSSD with spatial data. After streaming them back to the host (I assume in your case RPi 1), you could stream them to second RPi via web, by eg. websockets. Tons of possibilities on this end, but it really depends on your application.
Thanks, Erik

uk0310

erik
Thank you for your reply.
I'd like to perform an object-detection and want to share the coordinates of the bounding-box (plus depth-information).
I could perform an object-detection and show the coordinates of the bounding-box (plus depth-information) in display by using depthai_demo.py.
but I could not retrieve the values.
If i retrieve the values, I'll send the values to another raspberrypi by using UDP communications.
Can I retrieve the values by using RGB & MobilenetSSD with spatial data?

My UDP communication program is mentioned below.
I could send a sample value by using this program.
So, I send the coordinates of the bounding-box (plus depth-information) by replacing "d" with the values.

erik

Hello uk0310 ,
You can also retrieve these values (bounding box, spatial coordintes) from the depthai demo using callbacks.py file, see docs here - from the onNn() function. Same with the example I linked above, you will retrieve SpatialImgDetections messages that have bounding box and XYZ coordiantes inside them.
Thanks, Erik

uk0310

erik
Thank you for your reply.
I could retrieve he coordinates of the bounding-box (plus depth-information) by using SpatialImgDetections.
Thank you so much!

NotGrant

I am new to using CV and have been playing around with it for a school project. We are currently using https://github.com/luxonis/depthai-experiments/tree/master/gen2-yolo/device-decoding with a custom trained Yolov5 model. We can get it show our objects locations, but we cannot figure out how to export the coordinates of the objects. I'm confused as to where to place:
def onNn(nn_packet, decoded_data):
pass
Any help would be greatly appreciated.

erik

Hi NotGrant ,
I would suggest checking the docs here. If you have any questions please let us know.
Thanks, Erik

NotGrant

We are still struggling to access the depth data. We have tried going through the documentation and exploring many examples on Github. We keeping hitting walls. The goal is to have the depth data of objects exported to the terminal with a print command.

import cv2
import numpy as np
import depthai as dai
from depthai import NNData
from depthai_sdk import OakCamera, ArgsParser
from depthai_sdk.managers import PipelineManager
from depthai_sdk.classes import DetectionPacket, Detections
from depthai_sdk.visualize.configs import TextPosition
import argparse


parser = argparse.ArgumentParser()
parser.add_argument("-conf", "--config", help="Trained YOLO json config path", default='json/best.json', type=str)
args = ArgsParser.parseArgs(parser)

def decode(nn_data: NNData) -> Detections:
    print(nn_data)
    layer = nn_data.getFirstLayerFp16()
    print(layer)
    results = np.array(layer).reshape((1, 1, -1, 7))
    dets = Detections(nn_data)
    print(dets)
    for result in results[0][0]:
        if result[2] > 0.5:
            dets.add(result[1], result[2], result[3:])

    return dets


def callback(packet: DetectionPacket):
    visualizer = packet.visualizer

    num = len(packet.img_detections.detections)
    #print(packet.img_detections.detections)
    #print('New msgs! Number of Balls detected:', num)
    #print(packet.imgFrame)
    visualizer.add_text(f"Number of Objects: {num}", position=TextPosition.TOP_MID)
    visualizer.draw(packet.frame)
    cv2.imshow(f'frame {packet.name}', packet.frame)


with OakCamera(args=args) as oak:
    color = oak.create_camera('color')
    nn = oak.create_nn(args['config'], color,spatial=True, decode_fn=decode) #nn_type='yolo' decode_fn=decode
    oak.visualize(nn, callback=callback)
    oak.visualize(nn.out.passthrough,fps=True)
    oak.start(blocking=True)

DaniilPastukhov

Hi @NotGrant,

Could you please try running the following script?

import argparse
import cv2
from depthai_sdk import OakCamera, ArgsParser
from depthai_sdk.classes import DetectionPacket
from depthai_sdk.visualize.configs import TextPosition

parser = argparse.ArgumentParser()
parser.add_argument("-conf", "--config", help="Trained YOLO json config path", default='json/best.json', type=str)
args = ArgsParser.parseArgs(parser)


def callback(packet: DetectionPacket):
    visualizer = packet.visualizer

    num = len(packet.img_detections.detections)
    visualizer.add_text(f"Number of Objects: {num}", position=TextPosition.TOP_MID)
    visualizer.draw(packet.frame)
    cv2.imshow(f'frame {packet.name}', packet.frame)


with OakCamera(args=args) as oak:
    color = oak.create_camera('color')
    nn = oak.create_nn(args['config'], color, nn_type='yolo', spatial=True)  # nn_type='yolo' decode_fn=decode
    oak.visualize(nn, callback=callback)
    oak.visualize(nn.out.passthrough, fps=True)
    oak.start(blocking=True)

Also please specify the path to the blob in your json config. The following configuration worked in my case:

{
    "model": {
        "blob": "../models/yolov5n_coco_416x416_openvino_2021.4_6shave.blob"
    },
    "nn_config":
    {
        "output_format" : "detection",
        "NN_family" : "YOLO",
        "input_size": "416x416",
        "NN_specific_metadata" :
        {
            "classes" : 80,
            "coordinates" : 4,
            "anchors" : [10,13, 16,30, 33,23, 30,61, 62,45, 59,119, 116,90, 156,198, 373,326],
            "anchor_masks" :
            {
                "side52" : [0,1,2],
                "side26" : [3,4,5],
                "side13" : [6,7,8]
            },
            "iou_threshold" : 0.5,
            "confidence_threshold" : 0.5
        }
    },
    "mappings":
    {
        "labels":
        [
            "person",
            "bicycle",
            "car",
            "motorbike",
            "aeroplane",
            "bus",
            "train",
            "truck",
            "boat",
            "traffic light",
            "fire hydrant",
            "stop sign",
            "parking meter",
            "bench",
            "bird",
            "cat",
            "dog",
            "horse",
            "sheep",
            "cow",
            "elephant",
            "bear",
            "zebra",
            "giraffe",
            "backpack",
            "umbrella",
            "handbag",
            "tie",
            "suitcase",
            "frisbee",
            "skis",
            "snowboard",
            "sports ball",
            "kite",
            "baseball bat",
            "baseball glove",
            "skateboard",
            "surfboard",
            "tennis racket",
            "bottle",
            "wine glass",
            "cup",
            "fork",
            "knife",
            "spoon",
            "bowl",
            "banana",
            "apple",
            "sandwich",
            "orange",
            "broccoli",
            "carrot",
            "hot dog",
            "pizza",
            "donut",
            "cake",
            "chair",
            "sofa",
            "pottedplant",
            "bed",
            "diningtable",
            "toilet",
            "tvmonitor",
            "laptop",
            "mouse",
            "remote",
            "keyboard",
            "cell phone",
            "microwave",
            "oven",
            "toaster",
            "sink",
            "refrigerator",
            "book",
            "clock",
            "vase",
            "scissors",
            "teddy bear",
            "hair drier",
            "toothbrush"
        ]
    }
}

Thanks,
Daniil