How to get precise external timing on OAK-D S2 POE

LLuxonis-Alex · Feb 15, 2024

Hi @fredrik,
For the scope capture above, how were the camera nodes being configured in the pipeline? And the upper signal (ch 1) is applied on FSYNC?
For the continuous streaming with external sync mode, camera FPS should be set to match precisely the input signal rate (e.g. .setFps(20)) and this applied as well: .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
That weird pulse might happen if there's a mismatch between camera sensor internal timings (configured by setFps) and the external signal. Can also check how that signal looks like if the FSYNC signal is not provided - static level. A particularity of OV9282 is that if the signal is missing, the sensors continue to stream at the configured FPS. They will align to the external signal if provided, but its rate needs to stay very close to setFps rate.

Ffredrik · Feb 15, 2024

@erik @Luxonis-Alex

I found on this page: https://docs.luxonis.com/projects/hardware/en/latest/pages/NG9097s2/#ng9097s2 that there should be 5V between pins 5 and 8 of the M8 connector. That isn't the case with my OAK-D S2. What could be the issue?

LLuxonis-Alex · Feb 15, 2024

Pin 5 (VBUS) by default is an input for receiving power (it could also be an alternative to PoE, but only if the application doesn't have high power requirements, depending on what it uses - neural networks, video encoder etc), so no voltage present is expected.

It can be configured as an output if desired by your application (could power some external circuitry), and I believe this should do it, for example from a Script node:

    VBUS_OUT_GPIO_CTRL = 9
    GPIO.setup(VBUS_OUT_GPIO_CTRL, GPIO.OUT, GPIO.PULL_NONE)
    GPIO.write(VBUS_OUT_GPIO_CTRL, 1)  # Enable 5V output

Ffredrik · Feb 15, 2024

Luxonis-Alex

I'm sorry. I tested it again and now it seems to work correctly. I'm not sure if it was hard or software:

Now all cameras are controlled by .setFps.

Ffredrik · Feb 15, 2024

Luxonis-Alex

Perfect, this works!

Ffredrik · Feb 15, 2024

@alex @Luxonis-Alex

I'm able to get warned if the state on the aux GPIO changes.

But I don't understand how to use the interrupt function within this script block. I found the Script in the Nodes section, but an example with an interrupt implemented would help a lot.

The next step is to receive the timer that drives the cameras in the script section. Any ideas, examples or documentation on this?

My idea is as follows:

The three cameras are running on the highest FPS
Start a counter if there is a pulse on AUX GPIO
Count until the scene is captured and send this time to the host

Do you think the script area will be the place to let this happen?

Thanks for your help!

Ffredrik · Feb 16, 2024

@Luxonis-Alex @erik

I made this Script:
script.setScript("""
import time
import GPIO

VBUS_OUT_GPIO_CTRL = 9
GPIO.setup(VBUS_OUT_GPIO_CTRL, GPIO.OUT, GPIO.PULL_NONE)
GPIO.write(VBUS_OUT_GPIO_CTRL, 1)  # Enable 5V output

AUX_GPIO = 52    
GPIO.setup(AUX_GPIO, GPIO.IN, GPIO.PULL_DOWN)

GPIO.setInterrupt(AUX_GPIO, GPIO.RISING, 1)
    
ctrl = CameraControl()
ctrl.setCaptureStill(True)
while True:
    node.warn('waiting for interrupt')
    GPIO.waitInterruptEvent()
    node.io['out'].send(ctrl)

""")

When I'm putting a 10 Hz signal on the AUX GPIO the system works for around 1 minute, then I get this message:
[1944301061CA801300] [192.168.88.253] [83.028] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

[1944301061CA801300] [192.168.88.253] [1708082326.696] [host] [warning] Monitor thread (device: 1944301061CA801300 [192.168.88.253]) - ping was missed, closing the device connection

Any ideas on this?

LLuxonis-Alex · Feb 16, 2024

@fredrik Your code looks good. I can reproduce the same type of failure, with the 10Hz 3.3V signal applied on pin 1 (AUX_GPIO_3V3). I just replaced the script content from this example luxonis/depthai-pythonblob/main/examples/Script/script_camera_control.py
with yours from above. Sometimes it runs for a few minutes, but eventually fails the same. Will debug this firmware problem.

An unrelated note is that AUX_GPIO_3V3 also has another GPIO to control the direction (of a 3.3V<->1.8V level shifter), but by default the 3.3V side is an input, so this code is optional:

    AUX_GPIO_DIR_CTRL = 6   
    GPIO.setup(AUX_GPIO_DIR_CTRL, GPIO.OUT, GPIO.PULL_NONE)
    GPIO.write(AUX_GPIO_DIR_CTRL, 0)  # AUX_GPIO_3V3 (52) direction control: 0=input, 1=output

Mentioning just in case at some point is desired to set it as output. (We should also see about abstracting these in FW.)

However about the usage of .setCaptureStill, internally this doesn't trigger a new frame capture, but just sets a flag for the new frame from the camera to be processed/sent on the ColorCamera .still output. So by default the camera would stream at the 30fps rate, unsynced. I'm not exactly sure if this suits your needs, or you would rather want to change the cameras to fsync-input mode, for example editing in:
luxonis/depthai-pythonblob/main/utilities/cam_test.py
to uncomment cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
and applying an e.g 12V 10Hz signal on FSYNC (M8 pin 2, with GND_ISO on pin 7), then run:
python3 utilities/cam_test.py -rs -cams rgb,c left,m right,m -fps 10

It's still possible to add a script to get events on FSYNC GPIO (which goes both to sensors and to an SoC pin), but currently it might suffer from the same issue that it might crash the app after some time:

script = pipeline.create(dai.node.Script)
script.setScript("""
    import GPIO
    import time

    # Note: level gets inverted due to isolation circuitry
    FSYNC_GPIO = 41
    GPIO.setup(FSYNC_GPIO, GPIO.IN, GPIO.PULL_DOWN)
    GPIO.setInterrupt(FSYNC_GPIO, GPIO.RISING, 128)
    node.warn('Waiting for interrupts...')

    while True:
        ret = GPIO.waitInterruptEvent(FSYNC_GPIO)
        node.warn(f'GPIO interrupt: {ret}')
""")

It seems to run longer (couldn't reproduce a crash yet) with interrupt priority set to 128 (range 0..255) instead of 1, but needs more debugging to understand the issue.

Instead, it should be possible to check the timestamps of the received frames, that are aligned to the host monotonic time (dai.Clock.now())
Adding in cam_test.py e.g before frame = pkt.getCvFrame():
print(c, pkt.getTimestamp().total_seconds(), pkt.getTimestampDevice().total_seconds())
The timestamps by default are aligned to end of exposure window: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames/?highlight=fsin#frame-capture-graphs

Ffredrik · Feb 16, 2024

Luxonis-Alex

@Luxonis-Alex, thank you for your detailed response!

I hope you resolve the firmware issue quickly.

Thank you for explaining the level shifter; I was unaware that I could change the orientation of this circuit. Which is strange because the M8 connector documentation states that it is a GPIO pin.

Your solution to my timing issue does not fit the project. The camera is directed at a conveyor belt with objects going along. The PLC, that controls this conveyor belt, will emit pulses according to the belt's travelled distance. To ensure precise timing, the camera must validate the capture moment with the PLC. I intended to use the FSYNC and STROBE signals on the M8 connector for this. I forgot to note the issue with the rolling shutter sensors:-(.

Now I've ordered the OAK-D PRO POE with global shutter. I hope the resolution is sufficient to identify the objects on the belt.

@erik In my opinion, the product pages should include a large, obvious notice for cameras with rolling shutter and M8 ports. FSYNC appears to work just with the two monochrome cameras.

Thank you both for your support!

LLuxonis-Alex · Feb 16, 2024

For the issue with the GPIO interrupt, looks like setting the priority of 128 (instead of 1, the last param) can be a workaround for now, I had it running for over 3 hours and still good.

About Fsync, that should work as well with the IMX378 rolling shutter, my test above was done with such a device (OAK-D-S2-PoE). But only continuous streaming mode is supported with IMX378, and not arbitrary trigger.
If you need the cameras to run at a higher FPS, all synced, and then just need to decide which frames to send out, that may be possible to implement.
If you have a given camera configuration and fsync signal that fails to get a stream from IMX378, I can have a look.

Ffredrik · Feb 25, 2024

Luxonis-Alex

@Luxonis-Alex I did replace the camera to an OAK-D PRO POE. So all of the cameras have global shutters now.

I am currently confronting two strange situations:

One of the two frames is really dark.

The strobe output peaks a little while the signal is low.

What I don't understand is why one of the two frames is so black. I've set manual exposure and white balance. Anti-banding is turned off. This only occurs when the external trigger is used.

I configured the external trigger as follows:

camRgb.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
monoLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
monoRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

camRgb.initialControl.setExternalTrigger(1, 0)
monoLeft.initialControl.setExternalTrigger(1, 0)
monoRight.initialControl.setExternalTrigger(1, 0)

I guess that the numFramesBurst and numFramesDiscard is already built in for situations like this, but I would like to have some context about this.

Secondly the STROBE output has, in my opinion strange behaviour. The blue signal is the strobe pin, the yellow signal is the FSYNC pin.

This one is with numFramesBurst=1, numFramesDiscard=0

This one is with numFramesBurst=2, numFramesDiscard=1

Why is the peak/short signal within the other pulse?

The strobe pin is connected to a 12V source by a 9.2k resistor.

Thanks for your help.

LLuxonis-Alex · Feb 25, 2024

@fredrik Could you share the full script used for test, we'll check if something might not be configured correctly.
We have some docs here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames
but should probably improve the API description as well, and have it report errors on misconfigurations.

.setFrameSyncMode and .setExternalTrigger should not be set both, but only one, depending on the mode of operation desired:

setFrameSyncMode for continuous streaming mode, where the externally provided signal frequency should match precisely the value preconfigured by .setFps(...) (default 30.0). At sensor side (after the optocoupler logic that inverts the signal), the rising edge at FSIN sensor pin (Frame Sync INput) starts a new frame interval (1/FPS), where the exposure window is aligned to the end of the frame interval - we have frame capture graphs at the above link.
setExternalTrigger for arbitrary time capture. In this mode it's best for .setFps to be set to max (e.g 60 or 120) to avoid missed triggers. The exposure starts immediately after trigger and can't overlap with previous frame's MIPI readout. The first parameter configures how many frames to be captured on a single trigger. On OV9282/OV9782 the first frame(s) after trigger may not have the optimal image quality (sensor specific) and then the second parameter specifies how many to be auto-discarded. E.g. with (4, 3), 4 would be captured, with first 3 discarded, and the remaining of one actually streamed out. The interval between 2 triggers should not be too short, have to take into account the number of frames in a burst, the configured exposure time, and setFps() at which they will be captured.

Ffredrik · Feb 25, 2024

@Luxonis-Alex Ok, I didn't understand how to use the syncmode and ext trigger. But with your explanation, I do. Thanks.

I will use the ext trigger so I don't have to worry about timing from the device that gives the pulse.

I've set the fps to 120, but then there is something weird. Sometimes the stream starts for a few seconds and then runs into this error:
[19443010112BAC2E00] [192.168.88.251] [15.180] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

Somes runs it for a while but when I close it, I got this error dump.

When I used 60 fps, it seemed to work okay. But all the cameras on the Pro should be able to run at 120 fps, right?

With 60 fps I tested the strobe signal again:

The signals are the same as the previous scope screenshots.

Here the most important parts from the code:

def get_camera_settings(camera_settings_path):

config = configparser.ConfigParser()

config.read(camera_settings_path)

ctrl = dai.CameraControl()

exposure = config.getint('CONFIG', 'exposure')

iso = config.getint('CONFIG', 'iso')

whitebalancecolor = config.getint('CONFIG', 'whitebalancecolor')

ctrl.setManualExposure(exposure, iso) # exposure

ctrl.setManualWhiteBalance(whitebalancecolor) # colortemp metaphase illumination is 6000K

ctrl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.OFF) # autowhitebalance mode off

ctrl.setAntiBandingMode(dai.CameraControl.AntiBandingMode.OFF)

saturation = config.getint('CONFIG', 'saturation')

contrast = config.getint('CONFIG', 'contrast')

brightness = config.getint('CONFIG', 'brightness')

sharpness = config.getint('CONFIG', 'sharpness')

luma_denoise = config.getint('CONFIG', 'luma_denoise')

chroma_denoise = config.getint('CONFIG', 'chroma_denoise')

ctrl.setBrightness(brightness)

ctrl.setContrast(contrast)

ctrl.setSaturation(saturation)

ctrl.setSharpness(sharpness)

ctrl.setLumaDenoise(luma_denoise)

ctrl.setChromaDenoise(chroma_denoise)

return ctrl

nnPath, config_path = find_model_and_config(args.directory)

with config_path.open() as f:

config = json.load(f)

nnConfig = config.get("nn_config", {})

# parse input shape

if "input_size" in nnConfig:

W, H = tuple(map(int, nnConfig.get("input_size").split('x')))

# extract metadata

metadata = nnConfig.get("NN_specific_metadata", {})

classes = metadata.get("classes", {})

coordinates = metadata.get("coordinates", {})

anchors = metadata.get("anchors", {})

anchorMasks = metadata.get("anchor_masks", {})

iouThreshold = metadata.get("iou_threshold", {})

confidenceThreshold = metadata.get("confidence_threshold", {})

print(metadata)

# parse labels

nnMappings = config.get("mappings", {})

labels = nnMappings.get("labels", {})

# sync outputs

syncNN = True

# Create pipeline

pipeline = dai.Pipeline()

pipeline.setOpenVINOVersion(version=dai.OpenVINO.VERSION_2021_4)

# Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

detectionNetwork = pipeline.create(dai.node.YoloDetectionNetwork)

xoutRgb = pipeline.create(dai.node.XLinkOut)

nnOut = pipeline.create(dai.node.XLinkOut)

xoutRgb.setStreamName("rgb")

nnOut.setStreamName("nn")

if not args.external_trigger:

fps = 10

# Properties

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_800_P)

camRgb.setInterleaved(False)

camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

camRgb.setPreviewSize(W, H)

# manip = pipeline.create(dai.node.ImageManip)

# manip.setResize(W, H)

# # manip.setKeepAspectRatio(True)

# manip.setFrameType(dai.RawImgFrame.Type.BGR888p)

# camRgb.video.link(manip.inputImage)

# manip.out.link(detectionNetwork.input)

camControlIn = pipeline.create(dai.node.XLinkIn)

camControlIn.setStreamName('camControl')

camControlIn.out.link(camRgb.inputControl)

if args.external_trigger:

# camRgb.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

# monoLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

# monoRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

numFramesBurst = 1

numFramesDiscard = 0

camRgb.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

monoLeft.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

monoRight.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

camRgb.setFps(60)

monoLeft.setFps(60)

monoRight.setFps(60)

else:

camRgb.setFps(fps)

monoLeft.setFps(fps)

monoRight.setFps(fps)

# Network specific settings

detectionNetwork.setConfidenceThreshold(confidenceThreshold)

detectionNetwork.setNumClasses(classes)

detectionNetwork.setCoordinateSize(coordinates)

detectionNetwork.setAnchors(anchors)

detectionNetwork.setAnchorMasks(anchorMasks)

detectionNetwork.setIouThreshold(iouThreshold)

detectionNetwork.setBlobPath(nnPath)

detectionNetwork.setNumInferenceThreads(2)

detectionNetwork.input.setBlocking(False)

# Linking

camRgb.preview.link(detectionNetwork.input)

detectionNetwork.passthrough.link(xoutRgb.input)

detectionNetwork.out.link(nnOut.input)

if args.save_frames:

os.makedirs(args.savelocation, exist_ok=True)

session_dirs = os.listdir(args.savelocation)

output_folder = os.path.join(args.savelocation, 'session_{:05d}'.format(len(session_dirs)))

os.makedirs(output_folder, exist_ok=True)

# Connect to device and start pipeline

with dai.Device(pipeline) as device:

qControl = device.getInputQueue(name="camControl")

qControl.send(get_camera_settings(args.camera_settings))

# Output queues will be used to get the rgb frames and nn data from the outputs defined above

qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)

qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False)

Thanks for your help!

LLuxonis-Alex · Feb 28, 2024

Thanks! I can reproduce this issue with the STROBE as well, there is one more period, vs. the expected number of frames in burst.
However, it appears that the frame being streamed on MIPI properly follows the expected exposure/strobe interval. We'll check more on why OV9282 has this behavior, and if there's a way to disable generating that extra strobe interval.

I took some scope captures with the below configurations. Signals measured at sensor side (1.8V). Fsync signal was generated for test from a Script node (similar to luxonis/depthai-pythonmain → alex-luxonis:depthai-python:fsync_gpio_ffc_4p )

cam[c].setFps(100)  # just to have a nice 10ms strobe period on scope

burst = 1, exposure = 4ms

cam[c].initialControl.setManualExposure(4000, 100)
cam[c].initialControl.setExternalTrigger(1, 0)

^ it can also be noticed that when the unexpected second strobe period starts, there's some extra blanking (idle time) on MIPI, which is a bit weird, but shouldn't cause issues

exposure = 1ms

cam[c].initialControl.setManualExposure(1000, 100)
cam[c].initialControl.setExternalTrigger(1, 0)

burst = 2 (with ignore = 1), exposure = 1ms

cam[c].initialControl.setManualExposure(1000, 100)
cam[c].initialControl.setExternalTrigger(2, 1)

About .setFps(120), that should work as well, and even up to 129.6 (auto-capped if higher) at THE_800_P. In the quick tests I did, it seemed good, I'll test more and see if possible to replicate the crash you have observed. I have tested on the depthai-python develop branch, on top of the v2.24.0.0 release.

And unrelated to the latest discussions, for the crash observed after some time with GPIO.setInterrupt(AUX_GPIO, GPIO.RISING, 1), it's caused by the priority - last parameter of 1. Currently that issue doesn't happen if that number is set 16 or higher (range 0..255). We'll fix this is firmware, to be translated to a safe range.

Ffredrik · Feb 28, 2024

Luxonis-Alex

Nice that you were able to reproduce the error. What strobe pulse should I look for when the scene was captured? Is it the first rising edge?

It's good to know that there will be a firmware upgrade for the interrupt. If I had the time, I would surely test it with a different priority value. For the time being, I'm working on getting my system to operate with the OAK-D PRO as a cam.

Thank you for your replies.

LLuxonis-Alex · Feb 28, 2024

fredrik
Yes, on the OAK device M8 connector where the signals are inverted due to opto-isolators:

the falling edge of FSYNC is the trigger
if no frames are set to be discarded on trigger, then first pulse (inverted, low-active) marks the exposure window: exposure start on first falling edge, exposure end on next rising edge. The frame is streamed out on MIPI immediately after exposure end.