Hi DarshitDesai
As I have mentioned above, instead of stock visualizer, make your own callback function that will run each time there is a frame ready. Tracker and spatials are both available outputs of the NN component https://docs.luxonis.com/projects/sdk/en/latest/components/nn_component/#nncomponent).

Inside that same callback you can either print a list of all xyz values, or maybe make a publish to a ros topic. This is up to you since ROS is not integrated into SDK as of now.

Thanks,
Jaka

      jakaskerl I am still not able to figure out those values, can you tell me the exact api call in the python sdk that I need to type up for getting the x,y,z values?

        Here's my code for your reference

        from depthai_sdk import OakCamera

        import depthai as dai

        from depthai_sdk.classes import DetectionPacket

        def cb(packet: DetectionPacket):

            print(packet.img_detections)

        with OakCamera() as oak:

        color = oak.create_camera('color')

# List of models that are supported out-of-the-box by the SDK:

# https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models

nn = oak.create_nn('yolov8n_coco_640x352', color, tracker=True, spatial=True)

nn.config_nn(resize_mode='stretch')

nn.config_tracker(

    tracker_type=dai.TrackerType.ZERO_TERM_COLOR_HISTOGRAM,

    track_labels=[0], # Track only 1st object from the object map. If unspecified, track all object types

    # track_labels=['person'] # Track only people (for coco datasets, person is 1st object in the map)

    assignment_policy=dai.TrackerIdAssignmentPolicy.SMALLEST_ID,

    max_obj=1, # Max objects to track, which can improve performance

    threshold=0.1 # Tracker threshold

)

nn.config_spatial(

    bb_scale_factor=0.3, # Scaling bounding box before averaging the depth in that ROI

    lower_threshold=500, # Discard depth points below 30cm

    upper_threshold=8000, # Discard depth pints above 10m

    # Average depth points before calculating X and Y spatial coordinates:

    calc_algo=dai.SpatialLocationCalculatorAlgorithm.AVERAGE

)

oak.visualize([nn.out.tracker], fps=True)

# oak.callback(nn.out.tracker, callback=cb)

oak.visualize([nn.out.image_manip], fps=True)

oak.visualize([nn.out.spatials], fps=True)

oak.visualize(nn.out.passthrough)



# oak.start(blocking=True)

oak.start(blocking=True)

          Hi DarshitDesai
          I think you should be using trackerpacket if you are sending trackers as your callback arguments.

          Tracklets should give you a list of all tracked objects and their positions.

          Thanks,
          Jaka

              jakaskerl there are two detection api packets, SpatialMappingBbpacket and TrackerPacket, which of the x,y,z are more accurate or have optimal estimates from the kalman filter?

                  Hi DarshitDesai
                  Tracker packet is the SDK equivalent for the the tracker message in API, so you should use that. Filter can be applied with tracker_config when tracker is enabled.

                  Thanks,
                  Jaka

                      jakaskerl In my code I did use Spatial tracking feature, Wouldn't the spatialmapping packet have good results?

                          Hi DarshitDesai
                          Should work as well yes, since it includes the info for spatials. However there is no tracking here to my knowledge. It's basically just a depth frame with bb mappings.

                          Thanks,
                          Jaka

                            20 days later

                            The depth sdk everytime tries to connect to the internet and download the yolov8n blob model, Is there a way to store it on device and use that? I am using a raspberry pi and it is tedious to connect it to the internet again and again

                              Hi DarshitDesai
                              I think the downloaded models should be stored in cache and run locally once downloaded. Alternatively you can download the json, bin and xml file you get when converting a model and specify it when creating a new camera.

                              with OakCamera(args="model.json") as oak: #ignore this

                              Thanks,
                              Jaka

                                  jakaskerl Also there is one more issue, If I try using two callback functions one for viewing the tracker output using TrackerPacket and one for the spatial locations x,y,z using SpatialMappingBbPacket, it gives me an error. So to debug this I removed the SpatialMappingBbPacket Callback function and tried getting the x,y,z from tracker and viewing the frame output using opencv but it still gives me the same error. I checked the forum, I didn't find similar issues, Can you help debugging it?

                                  Code:

                                  Photo of error

                                    jakaskerl Can you tell me how to do the download step? Do I need to download it from yolov8n documentation and place it in the directory where the code is? Or I know that sdk converts model files into blob files so if there is a way to download the blob from somewhere?

                                        Hi DarshitDesai
                                        Sorry for the confusion, please ignore my last reply.
                                        You need to specify the model when creating a NN component. This can be either a string (already downloaded models) or a path to custom json or blob file.

                                        From component definition:
                                        def create_nn(self, model: Union[str, Dict, Path], ...)

                                        yolov8n_coco_640x352 should be dowloaded inside depthai/depthai_sdk/src/depthai_sdk/nn_models/yolov8n..... It should automatically get picked up when creating a model with it (exactly as you have done in above code).

                                        You don't need to use both visualizer (with callback) and a callback. Use one or the other. I suggest removing both standalone callbacks and just leave the two visualizers.

                                        Thanks,
                                        Jaka

                                            jakaskerl Actually it doesn't get picked up, I don't know why I will definitely check out the path which you wrote above in the RPI File system.

                                            About the callback, wouldn't I need callbacks to extrack x,y,z coordinates. Also I had remove the oak.visualize() functions earlier and it didn't give me the error but it doesn't print the x,y,z outputs too

                                            I commented those visualizer lines

                                            Edit: Ok it worked ignore the things above, I removed the visualizer lines and the spatialmappingbb callback and I am just using the tracker callback. I still have a question, if i still want to visualise the depth data how can I do that? Also what is the coordinate system orientation of the depth frame and rgb frame. (like z axis coming out of the lens, x on the left and y pointing up)

                                                Hi DarshitDesai
                                                Make sure you have the depthai repository installed.

                                                oak.visualize()'s callback is the same as the oak.callback(). You need one of them to extract the xyz coordinates. If you want to visualize as well, use a visualizer without a callback. It should automatically parse all the results and display them.

                                                Thanks,
                                                Jaka

                                                    jakaskerl Thank you can you also please help me with the coordinate system for the depth and rgb frames. Usually it z pointing out of the lens and x and y are in some suitable orientation

                                                        Hi DarshitDesai
                                                        Could you be a bit more specific please? As you have said z is pointing out and y, x are pixel indices, meaning y points down, x points right.

                                                        Thanks,
                                                        Jaka