Thank you @jakaskerl , I soldered a home-made USB connector with the required connections ref https://docs.luxonis.com/projects/hardware/en/latest/pages/NG9097pro/#connectors. I left the AUX 3V3 open but still it worked out well. I then flashed the NETWORK bootloaded, and - no problem connecting over ethernet.
I guess the lessons learned are: Have a USB connector ready when trying to factory reset the device via PoE :-)