AnhTuNguyen

  • Mar 28, 2024
  • Joined Feb 21, 2023
  • 0 best answers
  • jakaskerl
    Thanks !
    However, the behavior of the resize is still very strange to me. I just did a simple test:

    • I used Luxonis Pro W, set resolution of Left camera at THE_400_P.
    • I didn't do any cropping, just simply resized to a random size, in this case i chose 480x640. I tested 2 types of resize: normal resize and resize with thumbnail. I received 2 different FoVs. You can see clearly that the FoV given by Normal resize is smaller than thumbnail resize (look at the right size of images, in the thumbnails you can see the image on the wall which doesnt appear in normal resize). That being said that your normal resize doesnt do only the resize but also other operations (crop maybe ?) ?

    With normal resize

    With thumbnail resize

    • jakaskerl
      Thanks !
      I understand the principle behind each resize, but that doesnt explain why the image cropped by the luxonis has a much smaller FoV than one cropped with opencv.

      • jakaskerl
        thanks for your response. What is the difference between setResizeThumbnail and setResize please ? Is it possible to remove the black border when we do setResizeThumbnail please ?

        • Hello !

          I'm working with the StereoDepth module on the Oak-D ProW and I received this information when i launch my code
          [194430107158771300] [1.2] [1.398] [StereoDepth(4)] [info] Calibrated FOV 97.6715 is over 85 degrees, enabling distortion correction.
          [194430107158771300] [1.2] [1.476] [StereoDepth(4)] [info] Rectification mesh loaded, will override camera intrinsics/extrinsics matrices
          [194430107158771300] [1.2] [1.476] [StereoDepth(4)] [info] Depth baseline: '7.5000005' cm, focal length: '559.57263'
          [194430107158771300] [1.2] [1.476] [StereoDepth(4)] [info] Depth horizontal FOV: '97.6715', vertical FOV: '71.11673'

          Is it saying that the FoV is limited at 85degree after the undistortion processing please ?

          Thanks in advance for you support !

          • erik
            Hello !

            Thanks. I used my oak-d lite camera to do some records, and I believe that the calibration of this camera is not good. When did a test by visualizing the rectifiedLeft-rectifedRight (aligned with RGB), I observe a huge vertical shift (1-2 pixels between left-right).
            To be able to correct these records, I want to understand your calib.json file and your alignement process so that I can correct the records that I made.

            This is the calibration file. In the calibration, i found the last part (rectified rotation). Can I use those metrics to revert the rectification please ?
            {
            "batchName": "",
            "batchTime": 0,
            "boardConf": "",
            "boardCustom": "",
            "boardName": "OAK-D-LITE",
            "boardOptions": 0,
            "boardRev": "R1M1E3",
            "cameraData": [
            [
            2,
            {
            "cameraType": 0,
            "distortionCoeff": [
            -12.104032516479492,
            77.64612579345703,
            0.001263035461306572,
            0.0022347013000398874,
            -77.21680450439453,
            -12.094476699829102,
            77.58488464355469,
            -77.23764038085938,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0
            ],
            "extrinsics": {
            "rotationMatrix": [
            [
            0.9999452233314514,
            0.0073130750097334385,
            0.007484255824238062
            ],
            [
            -0.007313414476811886,
            0.9999732375144958,
            1.7986509192269295e-05
            ],
            [
            -0.007483924273401499,
            -7.272099173860624e-05,
            0.9999719858169556
            ]
            ],
            "specTranslation": {
            "x": 3.75,
            "y": 0.0,
            "z": 0.0
            },
            "toCameraSocket": 0,
            "translation": {
            "x": 3.7195262908935547,
            "y": -0.08316565304994583,
            "z": 0.026069268584251404
            }
            },
            "height": 480,
            "intrinsicMatrix": [
            [
            453.2582092285156,
            0.0,
            336.7252197265625
            ],
            [
            0.0,
            453.2582092285156,
            258.0137939453125
            ],
            [
            0.0,
            0.0,
            1.0
            ]
            ],
            "lensPosition": 0,
            "specHfovDeg": 72.9000015258789,
            "width": 640
            }
            ],
            [
            1,
            {
            "cameraType": 0,
            "distortionCoeff": [
            -9.763245582580566,
            37.59429168701172,
            -0.0005994563107378781,
            6.300376116996631e-05,
            -37.508628845214844,
            -9.76201057434082,
            37.57674026489258,
            -37.51167297363281,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0
            ],
            "extrinsics": {
            "rotationMatrix": [
            [
            0.9999633431434631,
            -0.0022503056097775698,
            -0.008261280134320259
            ],
            [
            0.002276295330375433,
            0.9999924898147583,
            0.0031379053834825754
            ],
            [
            0.008254156447947025,
            -0.0031565953977406025,
            0.9999609589576721
            ]
            ],
            "specTranslation": {
            "x": -7.5,
            "y": 0.0,
            "z": 0.0
            },
            "toCameraSocket": 2,
            "translation": {
            "x": -7.500237464904785,
            "y": 0.01437666080892086,
            "z": -0.12574531137943268
            }
            },
            "height": 480,
            "intrinsicMatrix": [
            [
            457.1339111328125,
            0.0,
            323.1533508300781
            ],
            [
            0.0,
            457.1339111328125,
            249.11166381835938
            ],
            [
            0.0,
            0.0,
            1.0
            ]
            ],
            "lensPosition": 0,
            "specHfovDeg": 72.9000015258789,
            "width": 640
            }
            ],
            [
            0,
            {
            "cameraType": 0,
            "distortionCoeff": [
            -4.017131328582764,
            7.194274425506592,
            0.0006911047967150807,
            -0.0003079548478126526,
            -2.7309975624084473,
            -4.1296186447143555,
            7.666990280151367,
            -3.326430320739746,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0,
            0.0
            ],
            "extrinsics": {
            "rotationMatrix": [
            [
            0.0,
            0.0,
            0.0
            ],
            [
            0.0,
            0.0,
            0.0
            ],
            [
            0.0,
            0.0,
            0.0
            ]
            ],
            "specTranslation": {
            "x": -0.0,
            "y": -0.0,
            "z": -0.0
            },
            "toCameraSocket": -1,
            "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
            }
            },
            "height": 2160,
            "intrinsicMatrix": [
            [
            3003.086181640625,
            0.0,
            1863.3856201171875
            ],
            [
            0.0,
            3003.086181640625,
            1119.2449951171875
            ],
            [
            0.0,
            0.0,
            1.0
            ]
            ],
            "lensPosition": 0,
            "specHfovDeg": 68.7938003540039,
            "width": 3840
            }
            ]
            ],
            "hardwareConf": "",
            "imuExtrinsics": {
            "rotationMatrix": [
            [
            0.0,
            0.0,
            0.0
            ],
            [
            0.0,
            0.0,
            0.0
            ],
            [
            0.0,
            0.0,
            0.0
            ]
            ],
            "specTranslation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
            },
            "toCameraSocket": -1,
            "translation": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
            }
            },
            "miscellaneousData": [],
            "productName": "",
            "stereoRectificationData": {
            "leftCameraSocket": 1,
            "rectifiedRotationLeft": [
            [
            0.9992857575416565,
            0.03516098111867905,
            0.013843906112015247
            ],
            [
            -0.0351836197078228,
            0.9993798732757568,
            0.0013948403066024184
            ],
            [
            -0.01378627773374319,
            -0.0018809227040037513,
            0.9999032020568848
            ]
            ],
            "rectifiedRotationRight": [
            [
            0.9990556240081787,
            0.03747883066534996,
            0.021980637684464455
            ],
            [
            -0.03744276240468025,
            0.9992966651916504,
            -0.0020502633415162563
            ],
            [
            -0.02204201929271221,
            0.0012253113090991974,
            0.9997562766075134
            ]
            ],
            "rightCameraSocket": 2
            },
            "version": 6
            }

            • erik replied to this.
            • Hello !

              I'm working with StereoDepth on the Oak-D Lite and I just find a strange behavior of depthai

              • if I set the disparity/depth align to RECTIFIED_LEFT (stereo.setDepthAlign(depthai.RawStereoDepthConfig.AlgorithmControl.DepthAlign.RECTIFIED_LEFT), I got the frame left like image below:

              • if I set the disparity/depth align to RGB (stereo.setDepthAlign(depthai.CameraBoardSocket.RGB)), I got the frame left like image below:

              If you look at the image border, you will see a huge difference. Therefore, i dont understand the behavior of setDepthAlign. Should it align the Depth/Disparity only ? Why is the frame left / right changed too ?

              Thanks for your support !

              • Hello !

                I want to implement a pipeline which has 2 models. The first model has 2 outputs, and the 2nd model use these 2 outputs as input.
                Is it possible to create a pipeline like that on the luxonis please ?

                Thanks !

                • Hello !

                  I'm trying to deploy my model on the OAK-D lite. I'm able to convert my ONNX to OpenVino IR format without problem. I'm trying now to convert the IR format to blob.

                  The problem is that my model recevies multiple inputs: 1 frame from mono camera (left/right) and 1 input from other sources. The frame coming from mono camera must be encoded as U8 and the other input must be encoded as F16 because of its values.

                  • I tested with option -ip F16 when I compile the model from IR to blob and F16 using image manip during the execution on the luxonis manip.initialConfig.setFrameType(depthai.ImgFrame.Type.RGBF16F16F16p)but I have bug [ImageManip(1)] [error] Processing failed, potentially unsupported config .

                  • I tested with option -iop "frame: U8, info:FP16" when i compiled the model but I have bug saying that
                    ```Check '!tensorFound' failed at /home/jenkins/agent/workspace/private-ci/ie/build-linux-ubuntu20/b/repos/openvino/tools/compile_tool/main.cpp:456:

                    Model doesn't have input/output with tensor name: frame```

                  In this case, is there any solution ?

                  Thank you in advance for your support !

                • Hello !

                  I'm looking at the file calib.json created when I recorded frames with stereo cameras. In the calibration, i'm confusing about the baseline of stereo camera. The cameraData in the calibration file is below.

                  I believe that the first 2 cameras are stereo cameras. When I look at the extrinsic of these 2 cameras, I have:
                  translation = 3.6339592933654785 for the 1st camera and translation=-7.459366321563721 for the 2nd camera. It means that the baseline of stereo camera is about 11cm which is not what you stated (7.5cm) in the camera documentation here .

                  "cameraData": [
                          [
                              2,
                              {
                                  "cameraType": 0,
                                  "distortionCoeff": [
                                      -4.026230335235596,
                                      0.1840355396270752,
                                      -0.002873856807127595,
                                      0.0016747524496167898,
                                      45.5887336730957,
                                      -4.025686740875244,
                                      0.17398102581501007,
                                      45.52424240112305,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0
                                  ],
                                  "extrinsics": {
                                      "rotationMatrix": [
                                          [
                                              0.9999908208847046,
                                              0.004288002382963896,
                                              -0.00014115366502664983
                                          ],
                                          [
                                              -0.004288680851459503,
                                              0.999977171421051,
                                              -0.005220828577876091
                                          ],
                                          [
                                              0.00011876351345563307,
                                              0.005221385974436998,
                                              0.9999863505363464
                                          ]
                                      ],
                                      "specTranslation": {
                                          "x": 3.75,
                                          "y": 0.0,
                                          "z": 0.0
                                      },
                                      "toCameraSocket": 0,
                                      "translation": {
                                          "x": 3.6339592933654785,
                                          "y": -0.009591661393642426,
                                          "z": -0.004193586762994528
                                      }
                                  },
                                  "height": 480,
                                  "intrinsicMatrix": [
                                      [
                                          453.42791748046875,
                                          0.0,
                                          305.33612060546875
                                      ],
                                      [
                                          0.0,
                                          453.42791748046875,
                                          242.4577178955078
                                      ],
                                      [
                                          0.0,
                                          0.0,
                                          1.0
                                      ]
                                  ],
                                  "lensPosition": 0,
                                  "specHfovDeg": 72.9000015258789,
                                  "width": 640
                              }
                          ],
                          [
                              1,
                              {
                                  "cameraType": 0,
                                  "distortionCoeff": [
                                      -6.71436882019043,
                                      84.24278259277344,
                                      -0.00036940438440069556,
                                      0.0008059097453951836,
                                      -118.6076889038086,
                                      -6.681336402893066,
                                      84.14616394042969,
                                      -118.51358795166016,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0
                                  ],
                                  "extrinsics": {
                                      "rotationMatrix": [
                                          [
                                              0.9998267292976379,
                                              -0.0017055005300790071,
                                              -0.01853567361831665
                                          ],
                                          [
                                              0.0017498093657195568,
                                              0.9999956488609314,
                                              0.0023745065554976463
                                          ],
                                          [
                                              0.018531542271375656,
                                              -0.0024065289180725813,
                                              0.9998253583908081
                                          ]
                                      ],
                                      "specTranslation": {
                                          "x": -7.5,
                                          "y": 0.0,
                                          "z": 0.0
                                      },
                                      "toCameraSocket": 2,
                                      "translation": {
                                          "x": -7.459366321563721,
                                          "y": -0.04788130894303322,
                                          "z": -0.13062356412410736
                                      }
                                  },
                                  "height": 480,
                                  "intrinsicMatrix": [
                                      [
                                          455.1717224121094,
                                          0.0,
                                          305.75689697265625
                                      ],
                                      [
                                          0.0,
                                          455.1717224121094,
                                          232.0742645263672
                                      ],
                                      [
                                          0.0,
                                          0.0,
                                          1.0
                                      ]
                                  ],
                                  "lensPosition": 0,
                                  "specHfovDeg": 72.9000015258789,
                                  "width": 640
                              }
                          ],
                          [
                              0,
                              {
                                  "cameraType": 0,
                                  "distortionCoeff": [
                                      -0.8858298063278198,
                                      -0.7882803082466125,
                                      -0.0016143274260684848,
                                      -0.0019167952705174685,
                                      30.069868087768555,
                                      -1.0199025869369507,
                                      -0.5080228447914124,
                                      29.434795379638672,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0,
                                      0.0
                                  ],
                                  "extrinsics": {
                                      "rotationMatrix": [
                                          [
                                              0.0,
                                              0.0,
                                              0.0
                                          ],
                                          [
                                              0.0,
                                              0.0,
                                              0.0
                                          ],
                                          [
                                              0.0,
                                              0.0,
                                              0.0
                                          ]
                                      ],
                                      "specTranslation": {
                                          "x": 0.0,
                                          "y": 0.0,
                                          "z": 0.0
                                      },
                                      "toCameraSocket": -1,
                                      "translation": {
                                          "x": 0.0,
                                          "y": 0.0,
                                          "z": 0.0
                                      }
                                  },
                                  "height": 2160,
                                  "intrinsicMatrix": [
                                      [
                                          2968.8076171875,
                                          0.0,
                                          1872.032958984375
                                      ],
                                      [
                                          0.0,
                                          2968.8076171875,
                                          1102.5250244140625
                                      ],
                                      [
                                          0.0,
                                          0.0,
                                          1.0
                                      ]
                                  ],
                                  "lensPosition": 0,
                                  "specHfovDeg": 68.7938003540039,
                                  "width": 3840
                              }
                          ]
                      ],
                  • jakaskerl

                    Hi !
                    I found the problem. Since I use mono camera for my neural network, I must specify -ip U8 when I compile the openvino IR to blob. My model works well now.

                    Thanks a lot for your time !

                    • jakaskerl

                      Hi !
                      That was the figure I sent above: Manip Left - Manip Right - NN input. NN input should be exactly the same as Manip-Left (first image) but somehow it was corrupted into that strange image in my figure.

                      I tried with Color camera and the right most image was correct --> I only had problem with input come from mono camera - manip.

                      I used 160x160 for the manip and input of model.

                    • jakaskerl

                      Hello !

                      I tested it and the values are in good scaled.

                      I think there is a bug in the manip. I explain the way I test:
                      - In the output of my model, I also output the input image. The goal is to really see how the input of my model looks like inside the pipeline of depthai. I found very strange input image.

                      • from left to right: Left camera - Right camera - Input frame that my model received.
                      • Knowing that I have a OAK-D Lite, rvc2. depthai version is 2.21.2.0
                      • jakaskerl Hello !

                        Sorry for the delay.
                        This is my pipeline code

                        def create_mono(pipeline: dai.Pipeline, socket):
                                cam = pipeline.createMonoCamera()
                                cam.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
                                cam.setBoardSocket(socket)
                        
                                # ImageManip for cropping (face detection NN requires input image of 300x300) and to change frame type
                                manip = pipeline.create(dai.node.ImageManip)
                                manip.initialConfig.setResize(args.HW[1], args.HW[0])
                                manip.initialConfig.setFrameType(dai.ImgFrame.Type.BGR888p)
                                cam.out.link(manip.inputImage)
                                return cam, manip
                        
                        # setup input cameras
                        cam_left, processed_left = create_mono(pipeline, dai.CameraBoardSocket.LEFT)
                        cam_right, processed_right = create_mono(pipeline, dai.CameraBoardSocket.RIGHT)
                        # create left output node
                        xout_left = pipeline.createXLinkOut()
                        xout_left.setStreamName("left")
                        cam_left.out.link(xout_left.input)
                        # create right output node
                        xout_right = pipeline.createXLinkOut()
                        xout_right.setStreamName("right")
                        cam_right.out.link(xout_right.input)
                        
                        # setup neural network node
                        model = pipeline.create(dai.node.NeuralNetwork)
                        model.setBlob(args.blob_path)
                        model.setNumInferenceThreads(2)
                        processed_left.out.link(model.inputs["frame1"])
                        processed_right.out.link(model.inputs["frame2"])
                        • Hello Luxonis team !

                          I'm currently working with the OAK-D lite and i'm trying to use a neural network on that camera. With this model, I want to use the mono camera (left or right) as input of my neural network.

                          The input of my model has 3 channels (even with mono camera i want to have 3 channels because my model was trained on both grayscale and rgb images) and has values between -1 and 1, I apply the normalization during the conversion between ONNX to IR format by using --meanvalues [127.5,127.5,127.5] and --scalevalues [127.5,127.5,127.5] as mention in the documentation here .

                          When I did the test, with mono camera as input (Left/Right), the model just output random values. I tested the same blob on RGB camera and it worked normally. In my case, how can i normalize correctly the image from mono camera ?

                          Thanks in advance for your support !

                          • AntoniaSchroff Sorry for the delay i was quite busy last week.
                            For the first question, can you plug your camera in slot A and D and try this command python3 cam_test.py --cameras rgb,c camd,c with this script.

                            I dont know if you gonna have this problem or not, but I got another problem on my FFC-6P. I will describe the problem and also the solution here. Hope that can help you if you get the same:

                            1. Slot D on the FFC 6P doesnt correspond to socket depthai.CameraBoardSocket.CAM_D but depthai.CameraBoardSocket.CAM_F. And Slot D corresponds to depthai.CameraBoardSocket.CAM_D.
                            2. Slot RGB cannot run at the same time as other RGB slots. When I tested to plut cameras to Slot RGB and D at the same time, depthai only recognized RGB via device.getConnectedCameraFeatures():. Same thing happened when I used Slot RGB and Slot F. When I tested Slot D and Slot F, it worked.

                            For the 2nd question, I'm running FFC-6P from my host computer.

                            • Hello !

                              I'm testing my new OAK-FFC-6P with 2 cameras IMX378-FF by running the script depthai-python/examples/ColorCamera/rgb_preview.py and I get some problems. (I'm using rvc3_support branch)

                              1. Nothing shows on screen when i lanch the script. When I try to check the queue by print(qRgb.has()), I always get False. To make sure that the camera is still working, I test RTSP streaming as mentioned in the documentation, and I can stream what the camera sees).

                              2. When I print device.getCameraSensorNames(), I got {<CameraBoardSocket.RGB: 0>: 'imx412'} knowing that my camera is IMX378. Is that normal ?

                              Thanks for your help !

                              Anh Tu Nguyen