Gaze estimation example on multiple detections

VojkoGlaser

Hi.

I would like to implement gaze estimation on multiple detected persons. I've started from gen2-gaze-estimation example, but the example is for only one person. I thought that, if I remove the "break" from for loop and use node.io.tryGetAll in scrypt.py, I would receive estimations for all detected persons, but it doesn't seem to be the case.

What would be the correct approach? Should I node.io.tryGet inside for loop? Can it be the same for loop as node.io.send(img) or should I create another?

Are tryGet and tryGetAll blocking functions?

Is there any good example that could help me with this multiple estimation?

Best regards,

Vojko

erik

Hi VojkoGlaser ,

I would suggest checking how multi-detection multi-model pipeline is implemented in SDK;

Script node (manager): https://github.com/luxonis/depthai/blob/main/depthai_sdk/src/depthai_sdk/components/template_multi_stage_script.py
Class that populates Script node: https://github.com/luxonis/depthai/blob/main/depthai_sdk/src/depthai_sdk/components/multi_stage_nn.py#L69
Host side syncing frames, detections and (multiple) recognitions: https://github.com/luxonis/depthai/blob/main/depthai_sdk/src/depthai_sdk/oak_outputs/xout/xout_nn.py#L259-L410 - Note that because you'd have multiple recognition types (with multiple NN results each), you'd need to update this to have as many recognition types as you'd want.

tryGet and tryGetAll are not blocking - get and getAll are (see docs here)

I hope this helps!
Thanks, Erik

VojkoGlaser

Hi.

I've followed suggested implementations and made some progress. But unfortunately it works only on first frame.

My script is basically the same as in script node (manager) and it seems to work well on the first frame -> for every detection, I get node.warn recognition result received and counter is decreased to 0. But on the next frame I only see the outputs for "Frame sent to recognition", but no recognitions are returned and counter is keeps on rising. Around counter 30 the script stops working completely although "while true" loop and without any warning or error.

Any suggestions and explanations what could be wrong?

Best regards,

Vojko

VojkoGlaser

Here is also a part of debug output if it helps. As mentioned before the script stops working around Counter 26.

Processing frame number: 1

Added frame seq 1

[18443010A101660F00] [6.2] [1.846] [Script(4)] [warning] New msg frame, seq 1

[18443010A101660F00] [6.2] [1.876] [Script(4)] [warning] New msg detections, seq 1

[18443010A101660F00] [6.2] [1.877] [Script(4)] [warning] synced 1. Removed older sync values. len 0

[18443010A101660F00] [6.2] [1.878] [Script(4)] [warning] Frame sent to recognition. Counter 1

[18443010A101660F00] [6.2] [1.879] [Script(4)] [warning] Frame sent to recognition. Counter 2

[18443010A101660F00] [6.2] [1.881] [Script(4)] [warning] Frame sent to recognition. Counter 3

[18443010A101660F00] [6.2] [1.881] [Script(4)] [warning] Frame sent to recognition. Counter 4

[18443010A101660F00] [6.2] [1.892] [Script(4)] [warning] Recognition result received. Counter: 3

[18443010A101660F00] [6.2] [1.897] [Script(4)] [warning] Recognition result received. Counter: 2

[18443010A101660F00] [6.2] [1.902] [Script(4)] [warning] Recognition result received. Counter: 1

[18443010A101660F00] [6.2] [1.907] [Script(4)] [warning] Recognition result received. Counter: 0

Added detection seq 1

Processing frame number: 2

Added frame seq 2

[18443010A101660F00] [6.2] [2.918] [Script(4)] [warning] New msg frame, seq 2

[18443010A101660F00] [6.2] [2.936] [Script(4)] [warning] New msg detections, seq 2

[18443010A101660F00] [6.2] [2.937] [Script(4)] [warning] synced 2. Removed older sync values. len 0

[18443010A101660F00] [6.2] [2.939] [Script(4)] [warning] Frame sent to recognition. Counter 1

[18443010A101660F00] [6.2] [2.939] [Script(4)] [warning] Frame sent to recognition. Counter 2

[18443010A101660F00] [6.2] [2.941] [Script(4)] [warning] Frame sent to recognition. Counter 3

[18443010A101660F00] [6.2] [2.942] [Script(4)] [warning] Frame sent to recognition. Counter 4

[18443010A101660F00] [6.2] [2.944] [Script(4)] [warning] Frame sent to recognition. Counter 5

Added detection seq 2

Processing frame number: 3

Added frame seq 3

[18443010A101660F00] [6.2] [3.926] [Script(4)] [warning] New msg frame, seq 3

[18443010A101660F00] [6.2] [3.946] [Script(4)] [warning] New msg detections, seq 3

[18443010A101660F00] [6.2] [3.947] [Script(4)] [warning] synced 3. Removed older sync values. len 0

[18443010A101660F00] [6.2] [3.949] [Script(4)] [warning] Frame sent to recognition. Counter 6

[18443010A101660F00] [6.2] [3.949] [Script(4)] [warning] Frame sent to recognition. Counter 7

[18443010A101660F00] [6.2] [3.951] [Script(4)] [warning] Frame sent to recognition. Counter 8

[18443010A101660F00] [6.2] [3.953] [Script(4)] [warning] Frame sent to recognition. Counter 9

[18443010A101660F00] [6.2] [3.955] [Script(4)] [warning] Frame sent to recognition. Counter 10

…

VojkoGlaser

I have found my mistake.

I had passthrorugh link set in pipeline from older implementation.

Now I get the recognitions for other frames also.

Best regards,

Vojko

VojkoGlaser

Hi.

I have again stumbled on similar problem as before but on another node. This time on left and right eye imageManip. It works as expected on first frame. The image and the config are sent to the nodes and all results for the first frame are returned. But for the detections on the next frame nothing gets returned from these two nodes although new img and configs are sent to the nodes. As if the nodes are waiting for something to be release and to be able to process new data or the pipeline would be broken...

Any suggestions how to pinpoint where the data flow gets stuck?

Thanks and best regards,

Vojko

erik

Hi VojkoGlaser ,
I'd suggest either using trace mode , or perhaps checking via depthai pipeline graph the FPS you get to each node. Are you reading/disposing ImageManip results (eye frames) fast enough?

VojkoGlaser

Hi.

I hope trace mode would show something (I hoped for something like error: crop out of bounds), but unfortunately it doesn't seem so. At least I didn't see anything. No error at all. How could I check if I'm reading fast enough? I would suspect that in this case the behavior would be more randomly distributed.

Is there a way to upload log files on this forum? Currently I get notice: Uploading files of this type is not allowed.

Here is also image of pipeline from pipeline_graph:

VojkoGlaser

Would you suggest to have multiple scripts instead of one?

Best regards,

Vojko

erik

Hi VojkoGlaser ,
When correctly enabling trace mode, you should get a bunch of logs inside your terminal. You can upload files on google cloud and share the link to it. From the screenshot, it looks like the whole pipeline is stopped (all FPS=0). Splitting a single python script into multiple ones wouldn't make a difference (except if you'd find a bug in the code sooner).
Thanks, Erik

VojkoGlaser

Hi.

I've found and remove one bug. The messages in msgs list weren't removed properly. Now the second frame starts to process, but from 5 detections on the second frame only 4 get processed entirely. The fifth gets stuck in the same place. Again the detections of left and right eye stop to return the results.

Here is the trace log:

https://files.solvesall.com/d/s/vGpw4J5pkQ2x70iKnrxbxNAN4VigGOaI/dGqiI0IDsI-8kVzv65vpMpNDu3LProic-7rPATJtiwgo

Let me know, if you will have problems accessing the file.

Best regards,

Vojko

VojkoGlaser

Hi.

Perhaps I have failed to mention that previous version was processing a video. In case of live stream from camera it successfully processed 26 sequences of one person.

Here are links to logs:

debug: https://files.solvesall.com/d/s/vHKQk4pqVEdbHcDkFEugKjysdzTQaR0P/nS8zuE3klAL6Pv5zT7H2xbU4NxY9GT8A-MbrgsR7Ewgo

trace: https://files.solvesall.com/d/s/vHKQl5zawgMciQWbaDErnCm4SEnyH7RX/ZVecxa4NxLejCXCb-9p-0PYYu_GNO2wy-yLqgwIbEwgo

It's interesting why does it always stop after 26 processed sequences and I have noticed that in later stages sequences can get skipped. I guess this is due to heavy load and the camera starts to skip frames.

Best regards,

Vojko

VojkoGlaser

If my assumption about skipping sequences is correct, I am a little bit worried about the performance on up to 10 people. The is no need for 30 processed frames per second, but at least 4 or 5 would be desirable.

Best regards,

Vojko

erik

Hi VojkoGlaser ,
Depending on queue size and blocking/nonblocking behaviour (docs here). Why would there be a problem if there are skipping sequences?