Output xyz, identification name to console

Selective

First, thank you for the great Oak-D camera!

So, my question.
Lets say I want to run the depthai-demo.py.
How can I output (print) the depth info x y z and identification name directly on the console instead of having the previewout box running the stream?

The goal here is to connect the information to a robot so it can take actions based on what it sees.

Thanks!

Brandon

Hi Selective ,

Thanks for the kind words!

In terms of getting this directly to the console, Szabi recently wrote an example patch on top of depthai_demo.py that actually writes a given class out to CSV, so it is very similar:

So in this case you could print these instead.

What robot are you hoping to control? The reason I ask is that Brandon Gushlaw (of the University of Colorado) made a strawberry-picking robot with OAK-D/depthai:

And I think he'd be willing to share the code used to do this. And he's definitely getting the class information and 3D position to pass this to the robotic arm to control the movement.

And another option, if you're wanting to really get into it, is that we have robot operating system (ROS) drivers, so you can pull the values from there and then use ROS to control the robot.

Thoughts?

Thanks,
Brandon

Selective

Hi Brandon,

Thank you for your quick reply! I will look into it this weekend.

My story is a long one 😉 so I will make a short one.

a few years ago I built a robotic lawn mower, which does not use a boundary wire. Instead I used real-time kinematics, which gives me a position around 2 cm and 100% localization.
This opens a lot of doors...
First, as you may or may not know a robotic lawn mower cuts your lawn in a random pattern, meaning it never knows when your lawn is finished, so it cuts and cuts and cuts, eventually everything is cut and all looks fine.

However, with 100% localization, you can now cut your lawn in a systematic pattern. This means for my 500 sqm lawn, its completely done in 2 hours. And I can look at my phone how much is left etc... In an event of power failure, unwanted weather condition, it will save the position and head home. When everything is good again it will go back to the position and continue the program.

So, last summer I made a small video on my phone, put it on linkedin, then it exploded. I got around 600K views, and companies started to approach me. In the end I created a company, signed a deal with a robotic lawn mower company.

So, why do I need a camera? Well, in random pattern you bump into something and go backwards, turn around and go in a another direction. Simple and it works.
In a systematic program, you can´t do that. You have a pattern to follow, and if you bump into something you need to find a way around the obstacle, this is where the camera comes in, to identify the object, how big is it? what side can I go around it etc....
So right now we have two possibilities, use a camera or use a radar. Radar is easier, costs less etc.. but a camera can do so much more, so I need to explore this possibility.

Drop me an email at: calle at lanstep dot com if you want to see it in action.

Thanks, and I´m sure I will get back to you with more questions 😉
//Carl

Edit: oh, no ROS is getting into my robots, I write everything myself for total control.

Selective

ok, I´m getting close!
I have this minimum code, it will give me what I want, BUT I´m missing the depth information:

`from pathlib import Path

import cv2
import depthai
import time

class DepthAI:
def init(self):
self.device = depthai.Device('', False)
self.p = self.device.create_pipeline(config={
"streams": ["metaout", "depth"],
"ai": {
#blob compiled for maximum 12 shaves
#blob can be generated using: python3 depthai_demo.py -cnn mobilenet-ssd -sh 12
#it will be written to <path_to_depthai>/resources/nn/mobilenet-ssd/mobilenet-ssd.blob.sh12cmx12NCE1
"blob_file": "/home/pi/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob.sh12cmx12NCE1",
"blob_file_config": "/home/pi/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.json",
"shaves" : 12,
"cmx_slices" : 12,
"NN_engines" : 1,
},
})

  self.detections = []

def run(self):
    while True:
        nnet_packets, data_packets = self.p.get_available_nnet_and_data_packets()
        for nnet_packet in nnet_packets:
            self.detections = list(nnet_packet.getDetectedObjects())
            for detection in self.detections:
             print (detection.get_dict())

    del self.p
    del self.device

DepthAI().run()
`
The output:

What am I missing?

Thanks,
Carl

Brandon

Sorry about the delay. Let me pull in Erik and/or Steven to help on this.

Brandon

Just got the answer internally:

I think the config is missing this:
https://github.com/luxonis/depthai/blob/5461419/depthai_helpers/config_manager.py#L256
https://discord.com/channels/790680891252932659/791018949143429121/802360109275676682

So this is the flag that tells DepthAI to calculate the depth for the bounding box.

Thanks,
Brandon

Selective

That did the trick!
I actually circled around that line "calc_dist_to_bb", but I thought it was calling a function which I never found.
But it was a simple True o False statement.. damn it 😉

Thank you Brandon!

Some thoughts:

I guess a lot of people will use these cameras on embedded devices/robots etc.. which really doesnt care about the actually video stream (like me). How about producing examples which outputs the information to the console? Or create a pipeline which is easy to hook up to to get information. Like Im doing now, I want the camera to alert me on things it sees, with the information about depth, and where the object can be found in the camera lens etc.. So my robot can take actions.

Another thing I want to ask you about.
We have this depth view where the image changes color based on the distance to the camera, how can I get that info as text/values to the console? One idea I have is to use the camera as a "radar", to alert me when something is too close to the robot, I dont care what the object is, just that something is really close -> STOP.

Thanks again!
//Carl

erik

Hello Carl,
the video stream is really helpful when we are prototyping/developing a solution and we want to quickly see for example "what on the video got classified as an [object]" to conclude whether our NN is working as expected.
Many examples actually provide the NN result as well (for example age/gender, classification, ocr, people counter, etc. The result is usually just not printed to the console but rather shown on the actual video stream (for easier development).

WRT your question, the video stream of the depth is essentially just an array of distances from the depthAI, so you could easily access that and check if (an area of) distances in front are too close, stop.
Thanks, Erik

Selective

Hi Erik,

Yes, I totally agree with you. The video stream is very useful for debugging /prototyping. I was just asking for some extra examples on how to tap the information as text/values for easy integration. I actually have a working camera now on a robot and avoiding obstacles.
However I find the depth information to be very inaccurate at most times. Is there anything that can be done to raise the quality? The robot is using the code above in this thread.

Also, how can get that depth information from the depth map as arrays? Is there an example somewhere or if you can point in me in the right direction.

Your help is much appreciated! Keep up the good work!

Thanks,
Carl

erik

Hello Carl,
you could check for example this line, if you add print(age) or print(gender_str), it will print the age or gender to the console.
For the depth noise, you could check out filtering, here's a great wsl filter example. Hopefully, this will help in your situation.
In above example (wls-filter), try adding print(type(filtered_disp)) to main.py after line 120 and you will see that filtered_disp is of type numpy.ndarray.
Thanks, Erik

Selective

Thank you Erik,

I will take a look at this as soon as I can, compiling "opencv-contrib-python" as we speak. Takes forever on RPi 😉

Selective

First real outdoor test with my test robot:

OneDrive link, about 3 min long:

https://1drv.ms/v/s!Apv2S-u3rGa7jcYNdeS0GjTQXSq8kQ

This a custom pcb board I designed which runs on a ARM mcu, and one RPi that snaps on to the board.
In this video Im using the disparity stream which detects objects in three ROI, left, front and right.
(Sorry about the Swedish words in the video, I presented this to a group of R&D people.)

Brandon

That's awesome @Selective ! Thanks for sharing! Do you mind if I put that on LinkedIN/Twitter?

Thanks again,
Brandon

Selective

Thanks!

Go for it 😉

Brandon

Selective Thanks! :-)

HughN

Apologies for resurrecting this thread but: a) I only started working with Python a few days ago and I haven't found a solution myself; b) searching this forum and via Google only turned up this thread.

I have been trying to write detection data to a CSV file, first by modifying the demo code, then using the code by @Selective, above. My version is here.

The error shown is:

Traceback (most recent call last):
File "ToCSV.py", line 37, in <module>
DepthAI().run()
File "ToCSV.py", line 29, in run
nnet_packets, data_packets = self.p.get_available_nnet_and_data_packets()
AttributeError: 'DepthAI' object has no attribute 'p'

This means nothing to me (it does appear to be DEFined). Please would it be possible give a newby a prod in the right direction?

Thank you,

Hugh

erik

Hello @HughN ,
your code is actually gen1 of our library. We now have gen2 (docs here), which is much more flexible and powerful. Sorry about the change - but your script won't work with latest depthai library.
I would suggest starting from eg. this example code, and add CSV saving by yourself (after line 139, where you get spatial detections🙂).
Thanks, Erik

HughN

Thanks, @erik - I'll give it a go!

Hugh

HughN

That worked treat. Thank you, @erik.

Hugh

erik

Great, thanks for the feedback🙂