Servet

  • Apr 9, 2024
  • Joined Jan 5, 2024
  • 0 best answers
  • In OAK-D-SR-PoE in Isaac Sim

    jakaskerl good to know we are OK on the size. There is activity on the NVIDIA post, so I will make sure to update here once we get closer to a working solution.

    Thanks for the help so far!

    • In OAK-D-SR-PoE in Isaac Sim

      Hi jakaskerl.

      I should have been more clear and copy-pasted the rgb_preview.py with our slight modifications. We did notice the frame size was different in the default script so we changed it, can you let us know if we did this correctly, I have pasted the code below.

      That is fair -- it is an NVIDIA specific issue after all.

      #!/usr/bin/env python3

import cv2
import depthai as dai


def print_info(_device):
    # Print out usb speed
    print('Usb speed:', _device.getUsbSpeed().name)
    # Bootloader version
    if _device.getBootloaderVersion() is not None:
        print('Bootloader version:', _device.getBootloaderVersion())
    # Device name
    print('Device name:', _device.getDeviceName(), ' Product name:', _device.getProductName())


# Create pipeline
pipeline = dai.Pipeline()

# Define source and output

camRgb = pipeline.create(dai.node.Camera)
xoutRgb = pipeline.create(dai.node.XLinkOut)

xoutRgb.setStreamName("rgb")

# Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_C)
camRgb.setSize(1280, 800) # set size to overwrite default
camRgb.setPreviewSize(1280, 800) # changed from default 1280, 720


# Linking
camRgb.preview.link(xoutRgb.input)

device_info = dai.DeviceInfo(ENV_OAK_D_IP)

# Connect to device and start pipeline
with dai.Device(pipeline, device_info) as device:
    qRgb = device.getOutputQueue(name="rgb", maxSize=1, blocking=False)

    # print_info(device)

    while True:
        inRgb = qRgb.get()  # blocking call, will wait until a new data has arrived

        # Retrieve 'bgr' (opencv format) frame
        cv2.imshow("rgb", inRgb.getCvFrame())

        if cv2.waitKey(1) == ord('q'):
            break

      Appreciate all the help @jakaskerl 🙂

      • In OAK-D-SR-PoE in Isaac Sim

        jakaskerl Hi,

        I am using the rgb_preview.py found in ColorCamera in the depthai python sdk: https://docs.luxonis.com/projects/api/en/latest/samples/ColorCamera/rgb_preview/

        The f_stop is multiplied following the Isaac Sim camera documentation here: https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/isaac_sim_sensors_camera.html#calibrated-camera-sensors

        We are using the fisheye polynomial following the guide in the link above, I am not sure fisheye polynomial is the best approach, but it seems to be what they are suggesting if I am not wrong? What type of "Camera Model" would the OAK-D-SR-PoE fall under?

        Thanks for coming back to us,
        Servet

          • In OAK-D-SR-PoE in Isaac Sim

            Hello 👋 hope some of you can help out! Anything helps 😁

          • In OAK-D-SR-PoE in Isaac Sim

            Hi Luxonis people, this is a crossever episode as it regards both the OAK-D-SR-PoE camera and NVIDIAs Isaac Sim. We are trying to match the physical and simulated camera and are having some issues in achieving this. We have made a quick and dirty setup for initial testing and are seeing some problems. Copy-pasted question on Isaac Sim forums here, hope you can help us! https://forums.developer.nvidia.com/t/importing-oak-d-sr-poe-camera/277960 (also let me know if I should just delete the text below)

            We want to import the OAK-D-SR-PoE camera into Isaac sim, but we are not managing to make the physical and simulated outputs match. We have used a simple setup to try and validate the outputs. More specifically we can see that the amount of space taken up by the box used in our experiments physically and in simulation do not line up, this is after setting up the simulated camera using the intrinsic values and distortion coefficients.

            Setup for initial validation
            Images are taken of a box at set distances. The images are then analyzed and finally the % of the image taken by the box is noted. The experiment is repeated in simulation using the same set distances and finally the % of the image taken up by the physical image is compared to that of the simulated. The idea is to validate that the camera parameters in the simulation roughly match those of the real camera, so we are not looking for millimeter precision.

            • The distances tested are 30, 40, 50, 60, 70, 80 and 100cm and these are measured from the front of the camera to the box surface.

            • The camera is also placed on a box lifting the middle of the camera lens 10.5cm.

            • The box is kept roughly in the middle of the frame to our best ability.

            • The camera intrinsic parameters from the physical camera is used in simulation.

            • The camera distortion coefficients from the physical camera is used in simulation.

            • The box is initially offset by box_thickness/2 in simulation to make sure the box and camera start exactly on top of eachother.

            • We are aware that the actual focal point could be inside the camera housing and that this can mean an error of 1-2 cm since we are measuring from the front of the camera housing (we added this distance in simulation afterwards to check and it makes such a small difference in the % space the box takes up that we are confident it is not the source of the discrepancy between real and simulated images).

            Camera specifications
            These are from the Luxonis page. The bold specs are the ones used in the script to create the simulated camera together with the camera intrinsic values and distortion coefficients.

            Camera Specs
            Stereo pair / ColorToF
            SensorOV9782 (color, PY074) 33D ToFDFOV
            / HFOV / VFOV 89.5° / 80° / 55° 82.3° / 70° / 54.7°

            Resolution1MP (1280x800) VGA (640x480)
            Range / FocusFF: 20cm - ∞ 20cm - 5m
            Max Framerate 120 FPS (800P) 30 FPS (VGA)
            Pixel size 3µm x 3µm 7µm x 7µm
            Lens size 1/4 inch 1/3.2 inch
            F-number 2.0 ±5% 1.45 ± 5%
            Effective Focal Length 2.35mm N/A

            Camera matrix

            [[805.8621215820312, 0.0, 651.9990844726562], [0.0, 805.4034423828125, 412.4564208984375], [0.0, 0.0, 1.0]]

            Distortion coefficients

            k1: -9.155631065368652
            k2: 91.02870178222656
            p1: -0.002392938593402505
            p2: 0.00018919834110420197
            k3: -58.8187255859375
            k4: -9.20724105834961
            k5: 90.6237564086914
            k6: -57.472503662109375
            s1: 0.0
            s2: 0.0
            s3: 0.0
            s4: 0.0
            τx: 0.0
            τy: 0.0

            FOV
            RGB FOV 71.86000061035156, Mono FOV 71.86000061035156. These were reported by the Luxonis calibration read script and we do not know what they are for, we simply used the documented FOVs in the camera specifications.

            Script for defining camera specification
            The script is a slight modification to the one in the Isaac Sim official camera documentation. We changed the division factor from 10 to 100 for the camera.set_focal_length as the documentation said it should be 1/10 of scene units.

            from omni.isaac.sensor import Camera
from omni.isaac.core import World

my_world = World(stage_units_in_meters=1.0)

camera = Camera(prim_path="/World/camera")

camera.initialize()# Camera examplewidth, height = 1280, 800
camera_matrix = [[805.8621215820312, 0.0, 651.9990844726562], [0.0, 805.4034423828125, 412.4564208984375], [0.0, 0.0, 1.0]]
pixel_size = 3 * 1e-3
f_stop = 2.0
focus_distance = 0.2

((fx,_,cx),(_,fy,cy),(_,_,_)) = camera_matrix
horizontal_aperture =  pixel_size * width
vertical_aperture =  pixel_size * height
focal_length_x  = fx * pixel_size
focal_length_y  = fy * pixel_size
focal_length = (focal_length_x + focal_length_y) / 2

camera.set_focal_length(focal_length / 100.0)
#camera.set_focus_distance(focus_distance)
camera.set_focus_distance(0.2)
camera.set_lens_aperture(f_stop * 100.0)
camera.set_horizontal_aperture(horizontal_aperture / 100.0)
camera.set_vertical_aperture(vertical_aperture / 100.0)

camera.set_clipping_range(0.05, 1.0e5)

diagonal_fov = 89.5# from documentation of cameracamera.set_projection_type("fisheyePolynomial")

distortion_coefficients =  [ -9.155631065368652, 91.02870178222656,  -0.002392938593402505, 0.00018919834110420197,  -58.8187255859375,  -9.20724105834961, 90.6237564086914, -57.472503662109375]

camera.set_rational_polynomial_properties(width, height, cx, cy, diagonal_fov, distortion_coefficients)

            Experiment results
            Not including 60cm, 70cm and 80cm to keep post shorter. As you can see the amount of image that is taken up by the carton box do noes not match between the real and simulated images, we have checked and double checked that the camera parameters match but have not been able to identify what is going wrong.

            30cm

            30cm-camera-edited30cm-camera-edited1280×800 75.5 KB

            30cm-simulated30cm-simulated1280×800 40.2 KB

            40cm

            40cm-camera-edited40cm-camera-edited1280×800 82.8 KB

            40cm-simulated40cm-simulated1280×800 52 KB

            50cm

            50cm-camere-edit50cm-camere-edit1280×800 86.8 KB

            50cm-simulated50cm-simulated1280×800 57 KB

            100cm

            100cm-camera-edit100cm-camera-edit1280×800 83.8 KB

            100cm-simulated100cm-simulated1280×800 64.2 KB

            Camera property inside Isaac Sim
            Here are the raw values for the camera in simulation after setting everything up, I have not been successful in identifying which params are going wrong here, hopefully you can help clear it up for us. We used the camera documentation to create the camera using the polynomial f-theta model.

            imageimage395×878 30.6 KB

            Can you help us match our simulated camera to the real-world one?

            Thanks!