How to get precise external timing on OAK-D S2 POE

erik · Feb 14, 2024

Hi @fredrik ,
I think using GPIO to read when to "capture" frames would be the best option as you mentioned, and then Script node can be used to determine which frames to send to the host.
Looking at connectors, you could use pin1 on m8 connector for GPIO (which is processors pin 52):
https://docs.luxonis.com/projects/hardware/en/latest/pages/NG9097s2/#connectors
WRT Strobe, you could log when you received a gpio signal (log timestamp), as imgFrame carries information about the capture time and exposure time.
Thoughts?
Thanks, Erik

Ffredrik · Feb 15, 2024

Hello, @erik.

Thank you for your prompt reply.

Using GPIO52 appears to be the most effective solution.

But this pin does not have any protection like the FSYNC, correct?
Can the FSYNC and STROBE GPIOs be used in a different way than they are currently?
Is this not achievable because of firmware or hardware issues?

I will keep you updated about the solution.

Best wishes! Fredrik

erik · Feb 15, 2024

Hi @fredrik ,
There is a level shifter (3v3 <-> 1.8). FSYNC signal is connected to the Myriad X chip directly, but depthai FW uses it unless the cameras generate the FSYNC signal themselves. STROBE is not connected to Myriad X, only to the cam sensor.

Ffredrik · Feb 15, 2024

erik

Hi @erik ,

thanks for the explanation. The FSYNC is probably not needed to read as an IO since time logging is already supported in the firmware as far as I know.

To which cam is the STROBE connected?

Is there a description about the timing of the STROBE signal? In this case it's the blue signal. I didn't get yet why it's not one single pulse and what the edges mean.

Hope to hear from you.

erik · Feb 15, 2024

Hi @Luxonis-Alex, do you perhaps know why the strobe has the strange short pulse (rising back up)?

erik · Feb 15, 2024

@fredrik STROBE signal on OAK-D-S2-POE is connected to the left stereo camera (OV9282). It pulls (or should pull) the signal low during the entire exposure of the camera.

Ffredrik · Feb 15, 2024

erik

@Luxonis-Alex @erik I connected the STROBE signal through an 1.2K Ohm resistor to an 12VDC source.

LLuxonis-Alex · Feb 15, 2024

Hi @fredrik,
For the scope capture above, how were the camera nodes being configured in the pipeline? And the upper signal (ch 1) is applied on FSYNC?
For the continuous streaming with external sync mode, camera FPS should be set to match precisely the input signal rate (e.g. .setFps(20)) and this applied as well: .initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
That weird pulse might happen if there's a mismatch between camera sensor internal timings (configured by setFps) and the external signal. Can also check how that signal looks like if the FSYNC signal is not provided - static level. A particularity of OV9282 is that if the signal is missing, the sensors continue to stream at the configured FPS. They will align to the external signal if provided, but its rate needs to stay very close to setFps rate.

Ffredrik · Feb 15, 2024

@erik @Luxonis-Alex

I found on this page: https://docs.luxonis.com/projects/hardware/en/latest/pages/NG9097s2/#ng9097s2 that there should be 5V between pins 5 and 8 of the M8 connector. That isn't the case with my OAK-D S2. What could be the issue?

LLuxonis-Alex · Feb 15, 2024

Pin 5 (VBUS) by default is an input for receiving power (it could also be an alternative to PoE, but only if the application doesn't have high power requirements, depending on what it uses - neural networks, video encoder etc), so no voltage present is expected.

It can be configured as an output if desired by your application (could power some external circuitry), and I believe this should do it, for example from a Script node:

    VBUS_OUT_GPIO_CTRL = 9
    GPIO.setup(VBUS_OUT_GPIO_CTRL, GPIO.OUT, GPIO.PULL_NONE)
    GPIO.write(VBUS_OUT_GPIO_CTRL, 1)  # Enable 5V output

Ffredrik · Feb 15, 2024

Luxonis-Alex

I'm sorry. I tested it again and now it seems to work correctly. I'm not sure if it was hard or software:

Now all cameras are controlled by .setFps.

Ffredrik · Feb 15, 2024

Luxonis-Alex

Perfect, this works!

Ffredrik · Feb 15, 2024

@alex @Luxonis-Alex

I'm able to get warned if the state on the aux GPIO changes.

But I don't understand how to use the interrupt function within this script block. I found the Script in the Nodes section, but an example with an interrupt implemented would help a lot.

The next step is to receive the timer that drives the cameras in the script section. Any ideas, examples or documentation on this?

My idea is as follows:

The three cameras are running on the highest FPS
Start a counter if there is a pulse on AUX GPIO
Count until the scene is captured and send this time to the host

Do you think the script area will be the place to let this happen?

Thanks for your help!

Ffredrik · Feb 16, 2024

@Luxonis-Alex @erik

I made this Script:
script.setScript("""
import time
import GPIO

VBUS_OUT_GPIO_CTRL = 9
GPIO.setup(VBUS_OUT_GPIO_CTRL, GPIO.OUT, GPIO.PULL_NONE)
GPIO.write(VBUS_OUT_GPIO_CTRL, 1)  # Enable 5V output

AUX_GPIO = 52    
GPIO.setup(AUX_GPIO, GPIO.IN, GPIO.PULL_DOWN)

GPIO.setInterrupt(AUX_GPIO, GPIO.RISING, 1)
    
ctrl = CameraControl()
ctrl.setCaptureStill(True)
while True:
    node.warn('waiting for interrupt')
    GPIO.waitInterruptEvent()
    node.io['out'].send(ctrl)

""")

When I'm putting a 10 Hz signal on the AUX GPIO the system works for around 1 minute, then I get this message:
[1944301061CA801300] [192.168.88.253] [83.028] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

[1944301061CA801300] [192.168.88.253] [1708082326.696] [host] [warning] Monitor thread (device: 1944301061CA801300 [192.168.88.253]) - ping was missed, closing the device connection

Any ideas on this?

LLuxonis-Alex · Feb 16, 2024

@fredrik Your code looks good. I can reproduce the same type of failure, with the 10Hz 3.3V signal applied on pin 1 (AUX_GPIO_3V3). I just replaced the script content from this example luxonis/depthai-pythonblob/main/examples/Script/script_camera_control.py
with yours from above. Sometimes it runs for a few minutes, but eventually fails the same. Will debug this firmware problem.

An unrelated note is that AUX_GPIO_3V3 also has another GPIO to control the direction (of a 3.3V<->1.8V level shifter), but by default the 3.3V side is an input, so this code is optional:

    AUX_GPIO_DIR_CTRL = 6   
    GPIO.setup(AUX_GPIO_DIR_CTRL, GPIO.OUT, GPIO.PULL_NONE)
    GPIO.write(AUX_GPIO_DIR_CTRL, 0)  # AUX_GPIO_3V3 (52) direction control: 0=input, 1=output

Mentioning just in case at some point is desired to set it as output. (We should also see about abstracting these in FW.)

However about the usage of .setCaptureStill, internally this doesn't trigger a new frame capture, but just sets a flag for the new frame from the camera to be processed/sent on the ColorCamera .still output. So by default the camera would stream at the 30fps rate, unsynced. I'm not exactly sure if this suits your needs, or you would rather want to change the cameras to fsync-input mode, for example editing in:
luxonis/depthai-pythonblob/main/utilities/cam_test.py
to uncomment cam[c].initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
and applying an e.g 12V 10Hz signal on FSYNC (M8 pin 2, with GND_ISO on pin 7), then run:
python3 utilities/cam_test.py -rs -cams rgb,c left,m right,m -fps 10

It's still possible to add a script to get events on FSYNC GPIO (which goes both to sensors and to an SoC pin), but currently it might suffer from the same issue that it might crash the app after some time:

script = pipeline.create(dai.node.Script)
script.setScript("""
    import GPIO
    import time

    # Note: level gets inverted due to isolation circuitry
    FSYNC_GPIO = 41
    GPIO.setup(FSYNC_GPIO, GPIO.IN, GPIO.PULL_DOWN)
    GPIO.setInterrupt(FSYNC_GPIO, GPIO.RISING, 128)
    node.warn('Waiting for interrupts...')

    while True:
        ret = GPIO.waitInterruptEvent(FSYNC_GPIO)
        node.warn(f'GPIO interrupt: {ret}')
""")

It seems to run longer (couldn't reproduce a crash yet) with interrupt priority set to 128 (range 0..255) instead of 1, but needs more debugging to understand the issue.

Instead, it should be possible to check the timestamps of the received frames, that are aligned to the host monotonic time (dai.Clock.now())
Adding in cam_test.py e.g before frame = pkt.getCvFrame():
print(c, pkt.getTimestamp().total_seconds(), pkt.getTimestampDevice().total_seconds())
The timestamps by default are aligned to end of exposure window: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames/?highlight=fsin#frame-capture-graphs

Ffredrik · Feb 16, 2024

Luxonis-Alex

@Luxonis-Alex, thank you for your detailed response!

I hope you resolve the firmware issue quickly.

Thank you for explaining the level shifter; I was unaware that I could change the orientation of this circuit. Which is strange because the M8 connector documentation states that it is a GPIO pin.

Your solution to my timing issue does not fit the project. The camera is directed at a conveyor belt with objects going along. The PLC, that controls this conveyor belt, will emit pulses according to the belt's travelled distance. To ensure precise timing, the camera must validate the capture moment with the PLC. I intended to use the FSYNC and STROBE signals on the M8 connector for this. I forgot to note the issue with the rolling shutter sensors:-(.

Now I've ordered the OAK-D PRO POE with global shutter. I hope the resolution is sufficient to identify the objects on the belt.

@erik In my opinion, the product pages should include a large, obvious notice for cameras with rolling shutter and M8 ports. FSYNC appears to work just with the two monochrome cameras.

Thank you both for your support!

LLuxonis-Alex · Feb 16, 2024

For the issue with the GPIO interrupt, looks like setting the priority of 128 (instead of 1, the last param) can be a workaround for now, I had it running for over 3 hours and still good.

About Fsync, that should work as well with the IMX378 rolling shutter, my test above was done with such a device (OAK-D-S2-PoE). But only continuous streaming mode is supported with IMX378, and not arbitrary trigger.
If you need the cameras to run at a higher FPS, all synced, and then just need to decide which frames to send out, that may be possible to implement.
If you have a given camera configuration and fsync signal that fails to get a stream from IMX378, I can have a look.

Ffredrik · Feb 25, 2024

Luxonis-Alex

@Luxonis-Alex I did replace the camera to an OAK-D PRO POE. So all of the cameras have global shutters now.

I am currently confronting two strange situations:

One of the two frames is really dark.

The strobe output peaks a little while the signal is low.

What I don't understand is why one of the two frames is so black. I've set manual exposure and white balance. Anti-banding is turned off. This only occurs when the external trigger is used.

I configured the external trigger as follows:

camRgb.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
monoLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
monoRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

camRgb.initialControl.setExternalTrigger(1, 0)
monoLeft.initialControl.setExternalTrigger(1, 0)
monoRight.initialControl.setExternalTrigger(1, 0)

I guess that the numFramesBurst and numFramesDiscard is already built in for situations like this, but I would like to have some context about this.

Secondly the STROBE output has, in my opinion strange behaviour. The blue signal is the strobe pin, the yellow signal is the FSYNC pin.

This one is with numFramesBurst=1, numFramesDiscard=0

This one is with numFramesBurst=2, numFramesDiscard=1

Why is the peak/short signal within the other pulse?

The strobe pin is connected to a 12V source by a 9.2k resistor.

Thanks for your help.

LLuxonis-Alex · Feb 25, 2024

@fredrik Could you share the full script used for test, we'll check if something might not be configured correctly.
We have some docs here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/sync_frames
but should probably improve the API description as well, and have it report errors on misconfigurations.

.setFrameSyncMode and .setExternalTrigger should not be set both, but only one, depending on the mode of operation desired:

setFrameSyncMode for continuous streaming mode, where the externally provided signal frequency should match precisely the value preconfigured by .setFps(...) (default 30.0). At sensor side (after the optocoupler logic that inverts the signal), the rising edge at FSIN sensor pin (Frame Sync INput) starts a new frame interval (1/FPS), where the exposure window is aligned to the end of the frame interval - we have frame capture graphs at the above link.
setExternalTrigger for arbitrary time capture. In this mode it's best for .setFps to be set to max (e.g 60 or 120) to avoid missed triggers. The exposure starts immediately after trigger and can't overlap with previous frame's MIPI readout. The first parameter configures how many frames to be captured on a single trigger. On OV9282/OV9782 the first frame(s) after trigger may not have the optimal image quality (sensor specific) and then the second parameter specifies how many to be auto-discarded. E.g. with (4, 3), 4 would be captured, with first 3 discarded, and the remaining of one actually streamed out. The interval between 2 triggers should not be too short, have to take into account the number of frames in a burst, the configured exposure time, and setFps() at which they will be captured.

Ffredrik · Feb 25, 2024

@Luxonis-Alex Ok, I didn't understand how to use the syncmode and ext trigger. But with your explanation, I do. Thanks.

I will use the ext trigger so I don't have to worry about timing from the device that gives the pulse.

I've set the fps to 120, but then there is something weird. Sometimes the stream starts for a few seconds and then runs into this error:
[19443010112BAC2E00] [192.168.88.251] [15.180] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 00, address: 00000000' '0'

Somes runs it for a while but when I close it, I got this error dump.

When I used 60 fps, it seemed to work okay. But all the cameras on the Pro should be able to run at 120 fps, right?

With 60 fps I tested the strobe signal again:

The signals are the same as the previous scope screenshots.

Here the most important parts from the code:

def get_camera_settings(camera_settings_path):

config = configparser.ConfigParser()

config.read(camera_settings_path)

ctrl = dai.CameraControl()

exposure = config.getint('CONFIG', 'exposure')

iso = config.getint('CONFIG', 'iso')

whitebalancecolor = config.getint('CONFIG', 'whitebalancecolor')

ctrl.setManualExposure(exposure, iso) # exposure

ctrl.setManualWhiteBalance(whitebalancecolor) # colortemp metaphase illumination is 6000K

ctrl.setAutoWhiteBalanceMode(dai.CameraControl.AutoWhiteBalanceMode.OFF) # autowhitebalance mode off

ctrl.setAntiBandingMode(dai.CameraControl.AntiBandingMode.OFF)

saturation = config.getint('CONFIG', 'saturation')

contrast = config.getint('CONFIG', 'contrast')

brightness = config.getint('CONFIG', 'brightness')

sharpness = config.getint('CONFIG', 'sharpness')

luma_denoise = config.getint('CONFIG', 'luma_denoise')

chroma_denoise = config.getint('CONFIG', 'chroma_denoise')

ctrl.setBrightness(brightness)

ctrl.setContrast(contrast)

ctrl.setSaturation(saturation)

ctrl.setSharpness(sharpness)

ctrl.setLumaDenoise(luma_denoise)

ctrl.setChromaDenoise(chroma_denoise)

return ctrl

nnPath, config_path = find_model_and_config(args.directory)

with config_path.open() as f:

config = json.load(f)

nnConfig = config.get("nn_config", {})

# parse input shape

if "input_size" in nnConfig:

W, H = tuple(map(int, nnConfig.get("input_size").split('x')))

# extract metadata

metadata = nnConfig.get("NN_specific_metadata", {})

classes = metadata.get("classes", {})

coordinates = metadata.get("coordinates", {})

anchors = metadata.get("anchors", {})

anchorMasks = metadata.get("anchor_masks", {})

iouThreshold = metadata.get("iou_threshold", {})

confidenceThreshold = metadata.get("confidence_threshold", {})

print(metadata)

# parse labels

nnMappings = config.get("mappings", {})

labels = nnMappings.get("labels", {})

# sync outputs

syncNN = True

# Create pipeline

pipeline = dai.Pipeline()

pipeline.setOpenVINOVersion(version=dai.OpenVINO.VERSION_2021_4)

# Define sources and outputs

camRgb = pipeline.create(dai.node.ColorCamera)

monoLeft = pipeline.create(dai.node.MonoCamera)

monoRight = pipeline.create(dai.node.MonoCamera)

detectionNetwork = pipeline.create(dai.node.YoloDetectionNetwork)

xoutRgb = pipeline.create(dai.node.XLinkOut)

nnOut = pipeline.create(dai.node.XLinkOut)

xoutRgb.setStreamName("rgb")

nnOut.setStreamName("nn")

if not args.external_trigger:

fps = 10

# Properties

camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_800_P)

camRgb.setInterleaved(False)

camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)

camRgb.setPreviewSize(W, H)

# manip = pipeline.create(dai.node.ImageManip)

# manip.setResize(W, H)

# # manip.setKeepAspectRatio(True)

# manip.setFrameType(dai.RawImgFrame.Type.BGR888p)

# camRgb.video.link(manip.inputImage)

# manip.out.link(detectionNetwork.input)

camControlIn = pipeline.create(dai.node.XLinkIn)

camControlIn.setStreamName('camControl')

camControlIn.out.link(camRgb.inputControl)

if args.external_trigger:

# camRgb.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

# monoLeft.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

# monoRight.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)

numFramesBurst = 1

numFramesDiscard = 0

camRgb.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

monoLeft.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

monoRight.initialControl.setExternalTrigger(numFramesBurst, numFramesDiscard)

camRgb.setFps(60)

monoLeft.setFps(60)

monoRight.setFps(60)

else:

camRgb.setFps(fps)

monoLeft.setFps(fps)

monoRight.setFps(fps)

# Network specific settings

detectionNetwork.setConfidenceThreshold(confidenceThreshold)

detectionNetwork.setNumClasses(classes)

detectionNetwork.setCoordinateSize(coordinates)

detectionNetwork.setAnchors(anchors)

detectionNetwork.setAnchorMasks(anchorMasks)

detectionNetwork.setIouThreshold(iouThreshold)

detectionNetwork.setBlobPath(nnPath)

detectionNetwork.setNumInferenceThreads(2)

detectionNetwork.input.setBlocking(False)

# Linking

camRgb.preview.link(detectionNetwork.input)

detectionNetwork.passthrough.link(xoutRgb.input)

detectionNetwork.out.link(nnOut.input)

if args.save_frames:

os.makedirs(args.savelocation, exist_ok=True)

session_dirs = os.listdir(args.savelocation)

output_folder = os.path.join(args.savelocation, 'session_{:05d}'.format(len(session_dirs)))

os.makedirs(output_folder, exist_ok=True)

# Connect to device and start pipeline

with dai.Device(pipeline) as device:

qControl = device.getInputQueue(name="camControl")

qControl.send(get_camera_settings(args.camera_settings))

# Output queues will be used to get the rgb frames and nn data from the outputs defined above

qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=False)

qDet = device.getOutputQueue(name="nn", maxSize=4, blocking=False)

Thanks for your help!