• DepthAI
  • OAK-FFC-3P-OG output stopped after 15 hours

erik
Sorry I did not notice that you send me PR to depthai python repo, but I am using c++ only depthai.

  • erik replied to this.

    Hi, I tried 2.15 depthai, but same issue happens.
    I am retrieving output of queue by following code:

    bool hasTimeout = false;
          gst_print("calling get\n");
          auto h264packet = queue_out->get<dai::ImgFrame>(std::chrono::duration_cast<std::chrono::seconds>(1s), hasTimeout);
          gst_print("calling get done\n");
          if(hasTimeout){
             gst_print("Retrieve timeout\n");
             return false;
          }

    I also turned on logs by setting environment variable DEPTHAI_LEVEL=trace

    Here is output:

    [2022-02-24 07:28:17.181] [trace] Log vector decoded, size: 3
    [14442C1001DBAACE00] [18187.511] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
    [14442C1001DBAACE00] [18187.511] [system] [info] Temperatures - Average: 47.24 °C, CSS: 48.03 °C, MSS 46.90 °C, UPA: 47.13 °C, DSS: 46.90 °C
    [14442C1001DBAACE00] [18187.511] [system] [info] Cpu Usage - LeonOS 0.30%, LeonRT: 0.17%
    [2022-02-24 07:28:17.707] [trace] Received message from device (h264) - parsing time: 88µs, data size: 14797, object type: 1 object data:
    0000: b9 06 b9 08 18 81 cd 39 81 d0 39 00 01 00 82 00 e1 e8 2f 82 00 e1 e8 2f 00 00 86 58 53 08 00 b9
    0020: 02 86 ec db 03 00 86 b7 b0 9a 05 b9 02 85 0c 47 86 c5 a9 47 00
    calling get done
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    [2022-02-24 07:28:22.182] [trace] Log vector decoded, size: 3
    [14442C1001DBAACE00] [18192.512] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
    [14442C1001DBAACE00] [18192.512] [system] [info] Temperatures - Average: 47.35 °C, CSS: 48.47 °C, MSS 46.23 °C, UPA: 47.58 °C, DSS: 47.13 °C
    [14442C1001DBAACE00] [18192.512] [system] [info] Cpu Usage - LeonOS 0.44%, LeonRT: 0.18%
    [2022-02-24 07:28:22.739] [trace] Received message from device (h264) - parsing time: 89µs, data size: 15647, object type: 1 object data:
    0000: b9 06 b9 08 18 81 1f 3d 81 20 3d 00 01 00 82 00 81 b8 2f 82 00 81 b8 2f 00 00 86 ef 53 08 00 b9
    0020: 02 86 f1 db 03 00 86 b3 fa 8c 07 b9 02 85 11 47 86 c1 f3 39 02
    calling get done
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    calling get
    [2022-02-24 07:28:27.183] [trace] Log vector decoded, size: 3
    [14442C1001DBAACE00] [18197.513] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
    [14442C1001DBAACE00] [18197.513] [system] [info] Temperatures - Average: 47.41 °C, CSS: 47.80 °C, MSS 46.45 °C, UPA: 47.80 °C, DSS: 47.58 °C
    [14442C1001DBAACE00] [18197.513] [system] [info] Cpu Usage - LeonOS 0.23%, LeonRT: 0.17%
    [2022-02-24 07:28:27.772] [trace] Received message from device (h264) - parsing time: 92µs, data size: 15677, object type: 1 object data:
    0000: b9 06 b9 08 18 81 3d 3d 81 40 3d 00 01 00 82 00 21 88 2f 82 00 21 88 2f 00 00 86 86 54 08 00 b9
    0020: 02 86 f6 db 03 00 86 02 bd 7d 09 b9 02 85 16 47 86 7c 38 2c 04
    calling get done
    calling get
    calling get done
    Retrieve timeout
    calling get
    calling get done
    Retrieve timeout
    • erik replied to this.

      Thanks for reporting oto313 , I have forwarded this to firmware engineers and we will take a look at it.

      6 days later

      Any progress? Or horizon when you can investigate this?

      Thanks

      • erik replied to this.

        Hello oto313 , I believe due to the nature of this sporadic issue, debugging and fixing/testing takes quite a long time, but I will recheck with FW engineers on it.
        Thanks, Erik

        Ok thanks for reply. If any help is needed i am happy to do so.

        20 days later

        Now I got also some other error message. And I forgot to mention that i am using luxonis module with raspberry pi hq camera

        [14442C1001DBAACE00] [118.886] [system] [info] Temperatures - Average: 53.06 °C, CSS: 53.99 °C, MSS 53.33 °C, UPA: 51.80 °C, DSS: 53.12 °C
        [14442C1001DBAACE00] [118.886] [system] [info] Cpu Usage - LeonOS 13.14%, LeonRT: 3.92%
        [14442C1001DBAACE00] [123.887] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
        [14442C1001DBAACE00] [123.887] [system] [info] Temperatures - Average: 53.39 °C, CSS: 54.42 °C, MSS 52.90 °C, UPA: 52.68 °C, DSS: 53.55 °C
        [14442C1001DBAACE00] [123.887] [system] [info] Cpu Usage - LeonOS 13.02%, LeonRT: 3.94%
        [14442C1001DBAACE00] [128.888] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
        [14442C1001DBAACE00] [128.888] [system] [info] Temperatures - Average: 53.00 °C, CSS: 54.42 °C, MSS 51.58 °C, UPA: 52.68 °C, DSS: 53.33 °C
        [14442C1001DBAACE00] [128.888] [system] [info] Cpu Usage - LeonOS 13.13%, LeonRT: 4.22%
        [14442C1001DBAACE00] [133.889] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
        [14442C1001DBAACE00] [133.889] [system] [info] Temperatures - Average: 53.33 °C, CSS: 54.21 °C, MSS 53.33 °C, UPA: 52.90 °C, DSS: 52.90 °C
        [14442C1001DBAACE00] [133.889] [system] [info] Cpu Usage - LeonOS 13.11%, LeonRT: 3.95%
        [14442C1001DBAACE00] [138.890] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
        [14442C1001DBAACE00] [138.890] [system] [info] Temperatures - Average: 53.39 °C, CSS: 54.42 °C, MSS 52.68 °C, UPA: 53.12 °C, DSS: 53.33 °C
        [14442C1001DBAACE00] [138.890] [system] [info] Cpu Usage - LeonOS 12.99%, LeonRT: 3.94%
        [14442C1001DBAACE00] [143.891] [system] [info] Memory Usage - DDR: 94.51 / 339.99 MiB, CMX: 2.18 / 2.50 MiB, LeonOS Heap: 21.38 / 78.29 MiB, LeonRT Heap: 3.76 / 41.54 MiB
        [14442C1001DBAACE00] [143.891] [system] [info] Temperatures - Average: 53.22 °C, CSS: 54.42 °C, MSS 52.90 °C, UPA: 52.46 °C, DSS: 53.12 °C
        [14442C1001DBAACE00] [143.891] [system] [info] Cpu Usage - LeonOS 13.18%, LeonRT: 4.01%
        Retrieve timeout
        [14442C1001DBAACE00] [149.513] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'
        Retrieve timeout
        Retrieve timeout
        • erik replied to this.

          oto313 Oh this seems interesting, looks like a firmware crash. Could you provide the full minimal reproducible code, so we can debug it locally?
          Thanks, Erik

          5 days later

          I will try to reproduce it. Normally it does not log any critical message.