R
rexn8r

  • Feb 4, 2023
  • Joined Jan 11, 2022
  • 0 best answers
  • hello

    is it possible to integrate gaze estimation with depthai sdk 1.9.2?

    I did try following code;

    from depthai_sdk import OakCamera
    import blobconverter
    
    with OakCamera() as oak:
        color = oak.create_camera('color')
        path = blobconverter.from_zoo("gaze-estimation-adas-0002", shaves=4,
            compile_params=['-iop head_pose_angles:FP16,right_eye_image:U8,left_eye_image:U8'],
        )
        nn = oak.create_nn(path, color)
        oak.visualize([nn], fps=True)
        oak.start(blocking=True)
    
    **following error thrown;**
    
    Traceback (most recent call last):
      File "test.py", line 11, in <module>
        oak.start(blocking=True)
      File "C:\ProgramData\Anaconda3\lib\site-packages\depthai_sdk\oak_camera.py", line 268, in start
        self.build()  # Build the pipeline
      File "C:\ProgramData\Anaconda3\lib\site-packages\depthai_sdk\oak_camera.py", line 354, in build
        component._update_device_info(self._pipeline, self._oak.device, self._pipeline.getOpenVINOVersion())
      File "C:\ProgramData\Anaconda3\lib\site-packages\depthai_sdk\components\nn_component.py", line 136, in _update_device_info
        raise NotImplementedError()
    NotImplementedError

    I would like to use depthai sdk to detect person and use gaze estimation for heat map creation functionality.

    is it possible?

    thanks

    • erik replied to this.
    • hello

      I downloaded the ONNX model vgg_ilsvrc_16_age_imdb_wiki.onnx from following link;

      https://github.com/onnx/models/tree/main/vision/body_analysis/age_gender/models

      The size of the ONNX model is 513 mb

      i am trying to convert ONNX to blob using blobconverter package using following code;

      import blobconverter

      blob_path = blobconverter.from_onnx(
      model="C:/Users/ABC/Documents/models-main/vision/body_analysis/age_gender/models/vgg_ilsvrc_16_age_imdb_wiki.onnx",
      data_type="FP16",
      shaves=6,
      optimizer_params=[
      "--input_shape=[1,3,224,224]",
      "--mean_values=[127,127,127]",
      "--scale_values=[255,255,255]"
      ],
      )

      I am getting following Bad Request error;

      blob_path = blobconverter.from_onnx(
      File "C:\ProgramData\Anaconda3\lib\site-packages\blobconverter__init.py", line 428, in from_onnx
      return compile_blob(blob_name=Path(model_name).stem, req_data={"name": Path(model_name).stem}, req_files=files, data_type=data_type, **kwargs)
      File "C:\ProgramData\Anaconda3\lib\site-packages\blobconverter__init
      .py", line 322, in compile_blob
      response.raise_for_status()
      File "C:\ProgramData\Anaconda3\lib\site-packages\requests\models.py", line 1021, in raise_for_status
      raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: https://blobconverter.luxonis.com/compile?version=2021.4&no_cache=False

      How do i fix it?

      thanks

      • erik replied to this.
      • hi @KlemenSkrlj

        thanks for your help. much appreciate.

        I did download the model.blob and modified the gen2-age-gender demo to check the result. The code link is here;

        unfortunately, the result is way off the mark.

        Can you provide any pointer from the above code to improve?

        thanks

        • erik replied to this.
        • hi

          thanks for your response.

          Below is the link to download age model in H5 format;

          here's the netron result of the h5 model;

          Below is the link to download age model in onnx format;

          here's the netron result of the onnx model;

          thanks

        • hello,

          i was wondering if someone can help.

          I downloaded UTKFace dataset and trained an Age model using the same.

          The output was tensorflow.keras model in h5 format.

          I did a an inferencing test and it was working with web camera.

          Now I converted the h5 model to onnx format using onnxmltools using onnxmltools.convert_keras(keras_model, target_opset=12)

          I then converted the ONNX model to blob file from https://blobconverter.luxonis.com/ website using following paramters;

          --data_type=FP16 --mean_values=[127.5,127.5,127.5] --scale_values=[255,255,255] -- input_shape=[-1,200,200,3]

          The blob conversion was successful.

          I then downloaded gen2-age-gender detection demo and replaced the "age-gender-recognition-retail-0013" model with my blob file.

          recognition_nn.setBlobPath(str(Path("model.blob").resolve().absolute()))

          when i run the main.py, i get following error;

          Creating Color Camera...
          OAK-D detected, app will display spatial coordiantes
          Creating recognition Neural Network...
          Traceback (most recent call last):
          File "main.py", line 181, in <module>
          device.startPipeline(create_pipeline(stereo))
          File "main.py", line 170, in create_pipeline
          recognition_nn.setBlobPath(nnPath)
          IndexError: invalid map<K, T> key

          can anyone help me how to resolve this error.

          thanks

          • erik replied to this.
          • hi @erik

            thanks for you response.

            I downloaded your code with MRE and ran on my pc having Ubuntu 22.04. I uninstalled depthai and reinstalled 2.17.0.0

            I am getting following error after half a minute;

            I then downloaded gen2 age and gender demo files from github and ran main.py and it just got stuck on the initialization;

            Let me elaborate on the 2nd question in my previous post regarding -90 degree rotation;

            The question is about wrong gender inference when script node is used with -90 degree rotation.

            After the script node is initialized i.e. "image_manip_script.setScript" , following line of code is executed;

            cam.preview.link(image_manip_script.inputs['preview']) <--- (this line present in your gen2 age-gender demo)

            The gender inference was coming out wrong so i wanted to check the preview input frame used by the script node so i added following line in the script node

            node.io['manip_frame'].send(frame)

            Then i added following xlinkout node to check the frame used by the script node;

            frame_xout = pipeline.create(dai.node.XLinkOut)
            frame_xout.setStreamName("frame_xout")
            image_manip_script.outputs['manip_frame'].link(frame_xout.input)


            When i did imshow on the frame_xout what i found was that its was not rotated -90degree and the inference was off mark i.e. wrong gender result

            if you check my first post on the pastebin, these lines are present but commented;

            then what i did was replaced cam.preview.link(image_manip_script.inputs['preview']) with following line;

            manipRgb.out.link(image_manip_script.inputs['preview'])

            and that showed rotated image and also the gender inference was correct.

            so my question is, Is it correct to use manipRgb.out.link instead of cam.preview.link to the script node?

            Sorry to be a pain but we are at a stage of DepthAI integration with our digital signage software where time is running out for us and the success of our solution depends upon the accuracy and stability of Depthai integration before we can proceed for POC. so any help would be highly appreciated.

            we are testing 2 versions of depthai implementation;

            1. Option 1 - Use script node which we are discussing in this post
            2. Option 2 - without script node using dai.NNData() on each detection for age/gender and expression inference which seem to be slower then option 1.

            Any guidance or help would be highly appreciated.

            thanks
            rex

          • hi

            as you said its complex solution to achieve 3 stage nn, what I as a user of the SDK was expecting from luxonis expert is to at least vet the code and let me know if I am on the right path or not so that i can try to debug further.

            for example;

            1. in the script node code, am I cropping the image 62X62 and 64X64 and passing them back to respective input. is this correct way??

            2. cam.preview.link(image_manip_script.inputs['preview']) this line pass the camera frame to the script node. when I output the frame in a window its still in landscape mode and not in -90 degree rotated mode (i thought cam.preview.link(face_det_manip.inputImage) should pass the rotated frame to the script node but its not). so that could have been the reason the inference was off mark. but then i replaced cam.preview.link(image_manip_script.inputs['preview']) with manipRgb.out.link(image_manip_script.inputs['preview']) and it seem to have resolved the inference.

            3. Also the Multimsgsync.py. I have added expression bit in it. is there any code problem which could be causing the script to produce errors listed in the post?

            4. "potentially unsupported config error" - if you look at my script node code, the correct_bb function is already used which i suppose take care of out of bounds problem. and as i mentioned in my original post, i am using following PR which was given to me in following post on github by szabi-luxonis

            https://github.com/luxonis/depthai-experiments/issues/326

            so in my first post also I asked for help to check the code and suggest modification/tweaking so i can do further debugging myself and help get such live example up and running that would benefit not only me as potential customer but also luxonis community.

            hope this will clarify my ranting.

            if you guys have time to vet the pastebin script code and suggest modifications to it then i will try to debug further and see if that resolves the issue.

            thanks

            • erik replied to this.
            • i suppose the luxonis team/experts answering other questions posted after mine suggest its time to move on and find alternative of OAK-D for my project.

              bit disappointing when someone not having huge AI experience trying hard to put together a real world solution and not getting enough support from luxonis ecosystem.

              time to move on....

              • erik replied to this.
              • hello depthai experts,

                I am trying to put up a python script to do following tasks using OAK-D Lite (3 stage nn);

                -- Face Detection
                |__ Age / Gender Detection
                |__ Expression Detection

                The camera is rotated @ -90 degrees.

                I have created a python script using gen2 age/gender/expression demo along with modifications to MultiMsgSync.py as per the code described in the link below;

                detection script code;

                MultiMsgSyncV2.py code;

                The problems:

                1. Some time it throws Processing failed, potentially unsupported config error.

                2. if multiple faces are in front of the camera then msgs = sync.get_msgs() returns none and the script doesn't work there on i.e. it gets stuck on the last frame

                3. The python scripts runs on OAK-D but on OAK-D Lite, it freezes after first 4-5 frames

                I have following PR installed which has fix for the edge problem on the -90 degree rotated camera;

                python3 -m pip install -U --prefer-binary --user --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local "depthai==2.15.4.0.dev+7ce594fa6a2b8437be7fef010a26b1eb5eb1fdf8"

                I would highly appreciate if some expert can throw a light on where could be the problem in the code.

                thanks
                rex

              • hello

                i am trying to combine age & gender demo with emotion using OAK-D lite.

                Age and gender image manip requires 62X62 cropped image size where as emotion requires 64X64.

                i have created 2 node scripts for age/gender and emotion. but when i run the python script, the camera frame gets stuck on the first frame and program halts.

                i looked at 2 stage inference demo and when i executed the main.py, it did not run as it has different camera initialization code.

                Please download the demo python file to check my code;

                [https://www.infokliks.com/detect.zip]

                how do i implement this use case?

                thanks
                rex

              • awesome erik. thanks for your help. much appreciate.

                will check out with my camera.

                cheers
                rex

              • Hello
                I have downloaded age and gender sample from following link;

                https://github.com/luxonis/depthai-experiments/tree/master/gen2-age-gender

                along with age and gender, i would like to calculate the distance of the face.
                The depthapi is quite overwhelming and i looked at spatial detection but could not figure out how to integrate it with gen2-age-gender demo.

                can someone guide me how to implement the depth in the same demo?

                thanks
                rex