4P PoE - Can't Ping - No available devices

BenjaminKiefer

Hi!

Bought a 4P PoE and the PoE injector you recommended. Connected the board to the power port of the injector via Cat6 ethernet cable and the other (communication) port to my PC directly. My PC got assigned an IP in the range of

169.254.X.X

Tried running the cam_test.py on the latest depthai version but got

No available devices

Then, tried pinging the device with the static IP after setting my PC to the same range as indicated here, butDestination host unreachable.I looked for any potentially relevant traffic through Wireshark, but haven't found anything:

https://cloud.cs.uni-tuebingen.de/index.php/s/Y8XmkqbFzSx8THt

I also tried connecting it to my router, but it did not get recognized. I looked at the logs of the DHCP server, but nothing.

I connected the device via USB-C to my PC as well while it being powered through ethernet, but the cam_test.py did not work still.

I spoke to some system administrator former colleague and he concluded that it must be that it is set to an IP that we don't know and there is no way for us to find out. Would that coincide with your judgement? What can I do?

Thanks!

jakaskerl

BenjaminKiefer Destination host unreachable

This sounds like your PC is routing the traffic through the wrong network interface. Check the routes.
Also, how are you connecting the 4P POE? It has 2 ETH ports and can be confusing sometimes.

Thanks,
Jaka

BenjaminKiefer

Please see the setup in the video. I tried both ports individually doing the following:

Turned on the board by connecting it to power port of the injector
Connect the PC to the other port of the injector
Got IP 169.254.90.235
Started cam_test.py, did not work. not sure what to ping
Set IP to 169.254.1.10 and subnet mask to 255.255.0.0
Started cam_test.py, did not work, pinging 169.254.1.222 did not work.

I had the wifi interface disabled the entire time (only ethernet active).

When I start the cam_test.py script, the indicator lights for 0 0 start to blink.

Do I need to connect it differently?

jakaskerl

Hi @BenjaminKiefer
But you are connecting it to ETH out. IIRC it should be ETH IN port (the other one).

Thanks,
Jaka

BenjaminKiefer

I tried that as well (I tried both ports individually and following the procedure as outlined above).

Am I missing something?

jakaskerl

@BenjaminKiefer
When I connect the POE to the device, five lights turn on.

The device is accessible through 169.254.1.222. Check the network adapter routing. Or maybe just turn off all other wired/wireless interfaces (except the device) to make sure ping goes through the right one.

Thanks,
Jaka

BenjaminKiefer

@jakaskerl Thx for the reply!

For me, only ever one light (of the yellow ones) is turned on:

I did turn off/disable all network interfaces except for ethernet.

Do I have a faulty device? Or can I press any button on the board?

jakaskerl

Hi @BenjaminKiefer
Likely a HW fault. Any chance the realtek chip is missing in the background?

Thanks,
Jaka

BenjaminKiefer

I think it's there?

BenjaminKiefer

Should I initiate getting a replacement unit? If so, who should I contact? Thx!

Luxonis-Alex

Hi @BenjaminKiefer ,
There's a possibility it could be the IMU not being flashed properly in the factory, and it interferes in the bootup process, as it's sharing the SPI bus with the NOR Flash where the depthai bootloader resides.

I see you tried already connecting the device over USB-C with no success. It's good that Ethernet for power was still connected, it's needed on this device model.

While having both Ethernet/PoE and USB connected, you can try putting the device in USB recovery mode:

press and hold the BOOT button (1 in the picture below)
press and then release the RESET button (2), a short press is enough
release the BOOT button

At this point the device should become available, and any depthai apps should be able to run. A note that after one or two runs, the device may revert to default NOR Flash boot, and the above button press procedure would have to be repeated.

Run from depthai-python:
python3 examples/IMU/imu_version.py
^ if this prints version 0.0.0, it confirms the IMU flashing issue. Can reflash with:
python3 examples/IMU/imu_firmware_update.py

Then can disconnect both cables (for power cycle) and connect Ethernet back.

If the issue is still not resolved, you can try also reflashing/updating the bootloader:
python3 examples/bootloader/flash_bootloader.py network

BenjaminKiefer

Thanks for your reply!

I ran

python3 examples/IMU/imu_firmware_update.py

which yielded

[14442C10E1B336D200] [2.6] [6.482] [system] [warning] Calibration Data on device is empty

IMU type: BNO086, firmware version: 3.9.9

So, I did not reflash. Instead, in USB recovery mode, I ran:

(lowlight) PS C:\Users\ben93\PycharmProjects\depthai-python> python .\examples\bootloader\flash_bootloader.py network

[0] 14442C10E1B336D200 [X_LINK_USB_VSC]

Which DepthAI device to flash bootloader for [0..0]: 0

Booting latest bootloader first, will take a tad longer...

Bootloader version to flash: 0.0.26

Flashing NETWORK bootloader...

…
Flashing successful. Took 11.65600000001723 seconds

Unplugging everything and plugging in everything again does not work: no pinging possible, no depthai-script finds the device.

Again, in USB recovery mode, I opened the device manager and saw there is no flashed bootloader.

I tried updating it and got

However, setting the IP through the device manager is not possible (greyed out). Setting it via the poe set ip.py says it was successful:

Found device with name: 2.6

-------------------------------------

"1" to set a static IPv4 address

"2" to set a dynamic IPv4 address

"3" to clear the config

Enter the number: 1

-------------------------------------

Enter IPv4: 169.254.1.222

Enter IPv4 Mask: 255.255.0.0

Enter IPv4 Gateway: 169.254.0.0

Flashing static IPv4 169.254.1.222, mask 255.255.0.0, gateway 169.254.0.0 to the POE device. Enter 'y' to confirm. y

Flashing successful.

I have my IP/mask/gateway set to the following:

But still can't ping. Even if I unplug everything (including PoE).

The next time I'm opening the device manager it shows the first img again (unbooted). Also, the config in the device manager says this:

Is that expected behaviour? If so, what else could I try? Otherwise, could I get a replacement unit?

BenjaminKiefer

Sorry again:

If you think the configuration device manager output looks right, I can try to give another go at the DHCP server? It did not work earlier, but maybe I missed something?

We'd want to get a couple of units but successful testing holds us back a bit.

Thanks!

jakaskerl

@Luxonis-Alex for ping

Luxonis-Alex

@BenjaminKiefer Thanks for the update and sorry for the late reply...

Trying to set the IP address when booted over USB/recovery mode (using USB bootloader, not network bootloader) isn't working, for this it's disabled in device_manager.py. We need to fix poe_set_ip.py to not allow it, it would write to a wrong area in the NOR flash memory.

This log you got I think is the key for the device not booting up normally (it should have shown up over USB as well, without the recovery button press):
[14442C10E1B336D200] [2.6] [6.482] [system] [warning] Calibration Data on device is empty
Even if there are no camera modules to be calibrated attached to the device, its EEPROM should have been factory programmed with some bootup configuration data at least. I'm not sure how this could have been skipped in the factory...

Anyway you can program that manually by putting it again in USB recovery mode and running this script:

python3 oem_eeprom_flasher_OAK_FFC_4P_POE_R2M0E2.py

Note on Windows the environment variable set by the script might not work (the script would complain about permissions when flashing), depending on the terminal. You may need to set it explicitly:

CMD (Command Prompt): set DEPTHAI_ALLOW_FACTORY_FLASHING=868632271
Windows Terminal/PowerShell: $env:DEPTHAI_ALLOW_FACTORY_FLASHING='868632271'

And if you can share with us the logs from running it, would help.

And just in case, can flash the bootloader again if needed:
python3 examples/bootloader/flash_bootloader.py network

Thanks and sorry about this issue

BenjaminKiefer

Hi @Luxonis-Alex

thanks for the reply. Here's what I did:

(lowlight) PS C:\Users\ben93\PycharmProjects\depthai-python\examples\bootloader> python oem_eeprom_flasher_OAK_FFC_4P_POE_R2M0E2.py

[14442C10E1B336D200] [2.6] [6.642] [system] [warning] Calibration Data on device is empty

Name/conn/mxid: / 2.6 / 14442C10E1B336D200

Is EEPROM available: True

=== Backup created: backup__14442C10E1B336D200_20240509_173815_factory_raw.json

No factory calibration: No or invalid EEPROM configuration flashed, error: EEPROM_INVALID_DATA

=== Backup created: backup__14442C10E1B336D200_20240509_173815_user_raw.json

No user calibration: No or invalid EEPROM configuration flashed, error: EEPROM_INVALID_DATA

Double-check the correct device is addressed, this script could cause a soft-brick!

Sure to overwrite and remove all calib data?

>>> Type 'y' and press Enter to proceed, otherwise exits: y

EEPROM flashing...

EEPROM flashing OK, please power-cycle

Then I ran

C:\Users\ben93\anaconda3\envs\lowlight\python.exe C:\Users\ben93\PycharmProjects\depthai-python\examples\IMU\imu_firmware_update.py

IMU type: BNO086, firmware version: 3.9.9, embedded firmware version: 3.9.9

Warning! Flashing IMU firmware can potentially soft brick your device and should be done with caution.

Do not unplug your device while the IMU firmware is flashing.

Type 'y' and press enter to proceed, otherwise exits:

n

Prompt declined, exiting...

pressing "n" because it had a IMU calibration file.
Then I ran

(lowlight) PS C:\Users\ben93\PycharmProjects\depthai-python\examples\bootloader> python .\flash_bootloader.py network

[0] 14442C10E1B336D200 [X_LINK_USB_VSC] current bootloader: 0.0.26

Which DepthAI device to flash bootloader for [0..0]: 0

Warning! Flashing bootloader can potentially soft brick your device and should be done with caution.

Do not unplug your device while the bootloader is flashing.

Type 'y' and press enter to proceed, otherwise exits:

y

Booting latest bootloader first, will take a tad longer...

Bootloader version to flash: 0.0.26

Are you sure you want to flash 'NETWORK' bootloader over current 'USB' bootloader?

Type 'y' and press enter to proceed, otherwise exits:

y

Flashing NETWORK bootloader...

Flashing progress: 2.4%

Flashing progress: 6.0%

Flashing progress: 8.4%

Flashing progress: 12.0%

Flashing progress: 15.6%

Flashing progress: 19.2%

Flashing progress: 22.8%

Flashing progress: 26.4%

Flashing progress: 28.8%

Flashing progress: 32.4%

Flashing progress: 36.0%

Flashing progress: 39.6%

Flashing progress: 42.0%

Flashing progress: 45.6%

Flashing progress: 49.2%

Flashing progress: 52.8%

Flashing progress: 55.2%

Flashing progress: 58.8%

Flashing progress: 62.4%

Flashing progress: 66.0%

Flashing progress: 68.4%

Flashing progress: 80.8%

Flashing progress: 100.0%

Flashing successful. Took 11.687000000005355 seconds

Now, I could ping and can run all scripts over ethernet 👏 🙏 Now, not only the 0 0 light is flashing but also another. Maybe, that's a good/necessary/sufficient indicator for debugging for others as well?

Two follow-up questions if I may (let me know if I should open a new thread):

Does the IMU have the correct calibration file now or do I have to do anything about that?
I plugged in three cams (IMX477->CAMB-2L, IMX477 ->CAMA-4L, IMX462 -> CAMD-4L). I run them on branch camera_controls_misc (depthai==2.24.0.0.dev0+74f0144831d643da9b103a3717ab0479f3dfabc9) with firmware overwrite file "Overriding firmware: imx462_2lane_30fps_depthai-python-74f01448_separateExpLR.fw". I asked you earlier in some mails regarding the color shift for the different cams on some of the ports. I'm getting a significant color shift here again (with both the new firmware and without). Is there any way, I can correct for or avoid this color shift? (the issue is related to another post I created: https://discuss.luxonis.com/d/3880-color-very-blueish-color-gradeexposure-settings)

Luxonis-Alex

That's great that it worked! Now your device should be in the same state regarding calibration/flashing as when a working one would leave factory. We will check what happened in this specific case...

For the IMU, AFAIK no calibration is performed currently.

For the color shift, that's weird. If you swap for example the cameras on ports A and D, so to have respectively IMX462 and IMX477, is it still present?
About the custom FW, it shouldn't be needed anymore if only 3 camera modules would be used, as one of the ports where the I2C bus is shared (B and C) can be left unused. But the 2-lane mod is needed if IMX462 is put on B/C. There's also a quirk on camera detection currently, if no camera is connected on port C, the one on port B will be listed as detected. That's incorrect, and can be ignored. Or it can be avoided by moving the camera module from B to C.

Can you check if changing the white balance helps with the color shift?
In cam_test.py, press 3 to select AWB mode, then cycle around with -_ and += keys.
Printing camera settings per frame (including color temperature) can be toggled with /

And fixed color temperature setting is also possible, we don't have a key control added for that, but can uncomment:
# cam[c].initialControl.setManualWhiteBalance(4000) # light temperature in K, 1000..12000

BenjaminKiefer

Ok, thanks, I'm not using the firmware for the PoE board when 462 on CAMA anymore. Changing ports for 462 and 477 does not solve the problem. Please see the following videos.

I can change the AWB mode and also set manual white balance, but it seems the cams behave differently (same mode/manual white balance yields different images). E.g., switching to different AWB modes keeps the differences of the color:

I think for now, I'd try to tune each cam with an individual existing AWB mode or manual white balance mode (I haven't tried doing that for the cams individually while they are all streaming at the same time, but I assume this should work straigh-forward?).

On another (hopefully last 🙂 ) note:

Sometimes when starting up, it gets stuck in

C:\Users\ben93\anaconda3\envs\lowlight\python.exe C:\Users\ben93\PycharmProjects\depthai-python\utilities\cam_test.py -cams cama,c camb,c camc,c -rs -misc high-conversion-gain=1

Enabled cameras:

cama : color

camb : color

camc : color

DepthAI version: 2.24.0.0.dev0+74f0144831d643da9b103a3717ab0479f3dfabc9

DepthAI path: C:\Users\ben93\anaconda3\envs\lowlight\lib\site-packages\depthai.cp38-win_amd64.pyd"

or I'm getting the known error

Enabled cameras:

cama : color

camb : color

camd : color

DepthAI version: 2.24.0.0.dev0+74f0144831d643da9b103a3717ab0479f3dfabc9

DepthAI path: C:\Users\ben93\anaconda3\envs\lowlight\lib\site-packages\depthai.cp38-win_amd64.pyd

Traceback (most recent call last):

File "C:\Users\ben93\PycharmProjects\depthai-python\utilities\cam_test.py", line 242, in <module>
with dai.Device(\*dai_device_args) as device:
RuntimeError: Failed to connect to device, error message: X_LINK_DEVICE_NOT_FOUND

Could you give me any advice on how I would catch that or what to do about it? Essentially, I want a robust mechanism for starting up the cameras and potentially restarting without hanging or killing of processes. Could you point me to best practices to using your camera in a production setting?

Thanks!

erik

Hi @BenjaminKiefer ,
I don't think diff sensors will have the same image if they have same white balance values. Are AWB values the same across all sensors, or does it sensor have it's own values (I can't remember if we have multiple I2C lines on 4P-PoE, or if multiple sensors share I2C bus). So for now I'd suggest perhaps "alter" the AWB value for one camera based on the actual AWB. So eg. if CAM_B has AWB val of X, then perhaps you could manually set WB for CAM_C to X+y, or something like that.

WRT device, you can first check whether you can detect the device (dai.Device.getAllAvailableDevices()), or do try/except when creating the device. In depthaiv3 we also have a simple auto-reconnection logic that will reconnect to the device if it crashes/disconnects

BenjaminKiefer

Hi @erik

we set all cams to "auto" AWB, but the issue is still there.
We can set the AWB to warm fluorescent, but then this looks odd in other ways/scenarios (yellow-ish).

Could you check whether you can reproduce this issue with the IMX462+IMX477/577?

Thanks!