High latency during Neural Network inference

I converted my custom yolov8s.pt model directly to Myriad .blob format using luxonis tools from depthai-ml training notebook. Upon runnning Spatial Object Tracker in my Oak-D Pro, I am getting less than 1 fps. My project asks for getting precise location of detected objects(x,y,z) in real time. So, is there any way to get higher fps on-device or I was planning of sending the isp color frame from Oak-D to my laptop, perform yolov8 inference on my own laptop and again send the Region Of Interest of bounding box to OakD to get the spatial location. And the problem in my approach is that, there will be delay in this two way communication, and how to sync the exact frames for this whole time-consuming process.