Summary:
I would like to see a snapshot of the visualizer output (e.g. tracked objects) in a web browser. This can be an image capture triggered by an event (i.e. does not need to be streamed with high FPS). In production (standalone), there will be no visualizer on the host, so the snapshot is needed to position and rotate the camera.
In development, I need the usual visualizer output on the host, in addition to the output snapshot in the web browser.
Preferred solution:
I am happy to send an event (i.e. browse to /img) to tell the device to take the output of the visualizer and send it to the webserver script node. Actually, this would be the most efficient approach and avoids streaming.
Less preferred, but acceptable if above preferred solution is impossible:
I'm also happy if I can have the webserver stream at a much lower FPS.
Details:
I have tried several things, so this is not a beginner question. I tried the following:
1 . HTTP Server
https://docs.luxonis.com/projects/api/en/latest/samples/Script/script_http_server/
This works to capture a still image from the camera. When an event happens (in this case browsing /img), we tell the camera to take a still picture:
ctrl = CameraControl()
ctrl.setCaptureStill(True)
and given that the still output is fed to the jpeg encoder:
cam.still.link(jpeg.input)
the encoder sends its output to the webserver sript:
jpeg.bitstream.link(script.inputs['jpeg'])
However, I need to capture the output of the visualizer, or at least be able to draw lines and circles on the frame before it is sent to the webserver node. The problem here is that there is no visualizer.still and no VisualizerControl(), so I'm unable to continue with this approach. Also, when I add a visualizer to the above code, it freezes until I refresh the /img page, so I end up almost single-stepping the Visualizer by refreshing.
Is there a way to run all the pipeline nodes continuously, even when a single node does not empty its queue?
2 . MJPEG Server
https://docs.luxonis.com/projects/api/en/latest/samples/Script/script_mjpeg_server/
This is nice but suffers from a similar problem: if I add the visualizer to see things on the host, the visualizer freezes until I browse /img however, in this case the Visualizer is no longer in single-step, so both videos are playing… until I close the browser, then the Visualizer freezes. I also sometimes got these random errors when I was not doing anything:
(.venv) ~/code > python main.py
[1844301021D93E1300] [192.168.2.241] [7.256] [Script(8)] [warning] Serving at 192.168.2.241:8081
[1844301021D93E1300] [192.168.2.241] [50.711] [Script(8)] [warning] [Errno 32] B
[1844301021D93E1300] [192.168.2.241] [85.700] [Script(8)] [warning] [Errno 32] B
[1844301021D93E1300] [192.168.2.241] [168.946] [Script(8)] [warning] [Errno 32] B
[1844301021D93E1300] [192.168.2.241] [1693772051.843] [host] [warning] Monitor thread (device: 1844301021D93E1300 [192.168.2.241]) - ping was missed, closing the device connection
INFO:root:Closing OAK camera
F: [global] [ 53847] [] tcpipPlatformWrite:300 Cannot find file descriptor by key: 56
(.venv) ~/code > python main.py
[1844301021D93E1300] [192.168.2.241] [7.787] [Script(8)] [warning] Serving at 192.168.2.241:8081
[1844301021D93E1300] [192.168.2.241] [1693772104.544] [host] [warning] Monitor thread (device: 1844301021D93E1300 [192.168.2.241]) - ping was missed, closing the device connection
INFO:root:Closing OAK camera
F: [global] [ 106549] [] tcpipPlatformWrite:300 Cannot find file descriptor by key: 56
(.venv) ~/code >
Moreover, just to test if eventually I'll be able to feed the output of the tracker, I tried to feed any other output that I know, which is the NN passthrough. So, if I try to modify this line:
cam.node.video.link(jpeg2.input)
to:
nn.node.passthrough.link(jpeg2.input)
I get this error:
[1844301021D93E1300] [192.168.2.241] [75.953] [VideoEncoder(7)] [warning] Arrived frame type (8) is not either NV12 or YUV400p (8-bit Gray)
Summary
I have a feeling this has to do with blocking vs non-blocking behavior, or perhaps that all nodes in the pipeline need to run at the same FPS. Also, the fact that the Visualizer output freezes until the web browser refreshes /img tells me that the whole pipeline, including detections and tracking is also frozen.
Any help or code snippets would be greatly appreciated
