Hi Luxonis team, I am working on **PatchCore anomaly detection** and would like to know whether it is possible to run the **complete PatchCore pipeline directly on an OAK4 / RVC4 camera** without relying on a PC for inference. My current PatchCore pipeline is: 1. Input image from the OAK camera 2. Preprocessing / resize 3. Pretrained **WideResNet101** backbone 4. Extract **Layer 2 and Layer 3** feature maps 5. Patchify the feature maps 6. Align Layer 3 with Layer 2 7. Generate the final feature embeddings 8. Store representative normal embeddings in a **FAISS memory bank** 9. During inference, generate embeddings for the test image 10. Perform nearest-neighbor search against the FAISS memory bank 11. Calculate the anomaly score and generate the anomaly result I have already been able to export the **WideResNet101 backbone / PatchCore-related model to ONNX**. My questions are: * Can the **WideResNet101 + PatchCore feature-extraction pipeline** run on the OAK4/RVC4 device? * Can the **FAISS nearest-neighbor search / memory bank** also run directly on the OAK4/RVC4? * If FAISS is not supported on-device, what would be the recommended way to implement the nearest-neighbor search on RVC4? * Can the complete pipeline run in **standalone mode**, with only the camera providing input and the OAK4 producing the anomaly score/result? * Are there any limitations regarding custom ONNX models, dynamic operations, memory usage, or unsupported operators that I should consider? * Would you recommend splitting the pipeline between the RVC4 neural accelerator and the onboard CPU? The main goal is to have: **OAK4 camera → PatchCore inference → anomaly score/result**

Hi @"OmkarDevadas"#3931, Overall, what you are describing should be feasible as a standalone RVC4 application. The complete pipeline would run on the device, combining the Qualcomm HTP/DSP neural accelerator with the onboard CPU to produce the final anomaly result. For an initial implementation, I would suggest splitting the pipeline as follows: * **Preprocessing and resizing:** Handle this as part of the DepthAI pipeline, for example using an `ImageManip` node. * **WideResNet101 backbone:** Convert the model for RVC4 and execute it on the HTP/DSP. * **Feature-map patchification and alignment:** This depends on the implementation. If these steps can be expressed using static-shape operations supported by the RVC4 conversion toolchain, they could be integrated into the WideResNet101 model (or exported as a separate NN model) and executed on the HTP/DSP. Otherwise, they should run on the CPU. * **FAISS nearest-neighbor search:** Run this on the CPU. FAISS is a CPU-side library rather than an operation directly supported by the neural accelerator, and nearest-neighbor search is generally a better fit for the CPU. Some general considerations: * Prefer fixed input and output shapes for accelerator-executed models. * Static operations such as reshaping, interpolation, pooling and concatenation may run on the HTP/DSP if supported by the conversion toolchain. Dynamic or unsupported tensor manipulation should remain on the CPU. * The normal-feature memory bank should generally be created off-device. The resulting bank can then be copied to the OAK4 and loaded during application startup. * WideResNet101 is relatively large, so both its inference latency and the cost of transferring its Layer 2 and Layer 3 feature maps to the CPU should be benchmarked. A smaller backbone may provide a better accuracy–performance trade-off. * After validating the initial implementation, you could try moving patchification, feature alignment and embedding generation into the accelerator model. Ideally, the HTP/DSP would output the final patch embeddings, leaving only the memory-bank search and anomaly scoring to the CPU. Klemen

@"KlemenSkrlj"#p34592 Hi Klemen, following up on my earlier question. I've implemented the split architecture you recommended (WideResNet101 backbone on RVC4, patchify/pooling/FAISS on CPU) and have it running end-to-end. However, I'm seeing a large, consistent discrepancy in the backbone's raw output between the compiled RVC4 model and the same PyTorch model run on my PC. **Setup:** * Model: WideResNet101, exporting layer2 and layer3 raw feature maps (no head) * Converted via Luxonis Hub's Quick Conversions tool, target RVC4, Standard FP16 precision * Preprocessing config: `mean=[153.255,153.255,153.255]`, `scale=[86.7,86.7,86.7]`, `reverse_channels=true`, input `640x640 NCHW` * These mean/scale values follow the documented convention: `input = (input/255.0 - mean) / std` → `mean=255*0.601`, `scale=255*0.340` **The discrepancy:** For the identical image (captured from the camera, transferred and run through the same weights on PC): * Real PyTorch backbone: `layer2 max ≈ 2.03`, `layer3 max ≈ 2.37` * Compiled RVC4 model (same image, same weights): `layer2 max ≈ 13-31` at rest, climbing to `≈770-780` once the camera's auto-exposure settles That's roughly 6-15x too large even at the low end, and 300-400x too large once the image stabilizes. I tested inverting the scale convention (`scale = 1/86.7`) in case the runtime multiplies rather than divides, but as expected that's mathematically identical to the original value and gave the same order-of-magnitude result. **My question:** Is there a different actual preprocessing formula applied at runtime on RVC4/SNPE than the documented `(pixel/255 - mean)/scale`? Or is there a known issue with how Hub's Quick Conversions tool (as opposed to Detailed Conversion / Model Registry) embeds these preprocessing values into the compiled archive? I confirmed via extracting the archive's `config.json` that Quick Conversions initially wrote `mean`/`scale` as `null` and had to patch them into the archive manually afterward using `luxonis_ml.ArchiveGenerator` — wondering if that manual patch is itself not being applied correctly at the SNPE/HTP level despite showing correctly in `config.json`.

@"OmkarDevadas"#p34598 Hi, When converting a model using our tools, including Hub, you typically start with an ONNX model whose inputs require the same preprocessing used during training. The UI therefore prompts you to specify the corresponding mean and scale values. During conversion, we try to bake this preprocessing into the exported model. If this succeeds then the resulting model accepts images in the `[0, 255]` range, with mean subtraction and scale division incorporated as its first operations. This is why the resulting RVC4 model has `null` mean and scale values: no additional preprocessing should be applied when using it in a DepthAI pipeline, since `ImgFrame` data is already in the `[0, 255]` range. Please try removing any preprocessing currently performed in your pipeline and feed the images directly to the `NeuralNetwork` node. Then check whether the outputs match those of the ONNX model. You should also configure the pipeline to provide RGB or BGR input according to the value selected for `Encoding.to` in the UI. Let me know if the issue persists after making these changes. Klemen

**Suggested reply to post:** > Hi Klemen, thanks for the explanation — that makes sense, and confirms `mean=null`/`scale=null` was correct all along, not a bug on our end. > > I removed the manual patch entirely and tested the archive exactly as Hub produced it (no external preprocessing in the DepthAI pipeline, feeding images directly to the NeuralNetwork node). **The discrepancy persists identically** — `layer2`/`layer3` raw output on RVC4 still lands around 300-400x larger than the same PyTorch model on PC, even with the archive completely untouched. > > I also retrained the memory bank on fresh, camera-native images (ruling out any training/domain mismatch) — same result, no change. > > A couple of follow-up questions: > > 1. **I can't find an "Encoding.to" setting anywhere in Quick Conversions** — only Height/Width, Layout, Mean/Scale, Target Precision, SNPE version, and the two "Disable" toggles. Is this setting specific to a different conversion flow (Detailed Conversion / Model Registry), or have I missed it somewhere in Quick Conversions? > > 2. **Separately, I found what looks like a bug in Quick Conversions itself**: setting either "Disable ONNX Simplification" or "Disable ONNX Optimization" to "Yes" causes the conversion to fail immediately with `ValueError: Invalid number of overrides. See --help for more information.` — happening in your tool's internal CLI argument parsing (`modelconverter/cli/utils.py:92`), before the model is even touched. Wanted to flag this separately in case it's useful. > > Given the discrepancy remains even with your exact recommended setup, is there a way to get visibility into what preprocessing/graph the compiled RVC4 model actually contains — e.g., could I inspect the exported model's first few operations to confirm mean/scale really did get baked in as expected?

@"KlemenSkrlj"#p34599 **Suggested reply to post:** > Hi Klemen, thanks for the explanation — that makes sense, and confirms `mean=null`/`scale=null` was correct all along, not a bug on our end. > > I removed the manual patch entirely and tested the archive exactly as Hub produced it (no external preprocessing in the DepthAI pipeline, feeding images directly to the NeuralNetwork node). **The discrepancy persists identically** — `layer2`/`layer3` raw output on RVC4 still lands around 300-400x larger than the same PyTorch model on PC, even with the archive completely untouched. > > I also retrained the memory bank on fresh, camera-native images (ruling out any training/domain mismatch) — same result, no change. > > A couple of follow-up questions: > > 1. **I can't find an "Encoding.to" setting anywhere in Quick Conversions** — only Height/Width, Layout, Mean/Scale, Target Precision, SNPE version, and the two "Disable" toggles. Is this setting specific to a different conversion flow (Detailed Conversion / Model Registry), or have I missed it somewhere in Quick Conversions? > > 2. **Separately, I found what looks like a bug in Quick Conversions itself**: setting either "Disable ONNX Simplification" or "Disable ONNX Optimization" to "Yes" causes the conversion to fail immediately with `ValueError: Invalid number of overrides. See --help for more information.` — happening in your tool's internal CLI argument parsing (`modelconverter/cli/utils.py:92`), before the model is even touched. Wanted to flag this separately in case it's useful. > > Given the discrepancy remains even with your exact recommended setup, is there a way to get visibility into what preprocessing/graph the compiled RVC4 model actually contains — e.g., could I inspect the exported model's first few operations to confirm mean/scale really did get baked in as expected? > >

@"OmkarDevadas"#p34623 1/ Indeed the "Encoding.to" is only present in the Detailed conversion. So if you want to experiment with that feature you should try that out: [upl-image-preview uuid=2387f4b2-73a0-4806-a7ba-313252af4152 url=https://discuss.luxonis.com/assets/files/2026-09-18/1789726892-377224-image.png alt={TEXT?}] 2/ Thanks for reporting this, we'll get that fixed. For next steps I think the best way for you to have full control over the export process is that you try to do it locally with [modelconverter](https://github.com/luxonis/modelconverter). This is the package that is actually ran in the background for all conversion tasks. And doing it this way you can also edit anything that you require since it is an open-source library that we have. If you find some general bug I would also encourage you to make a PR for the fix if you want and we can get it mainlined. Best, Klemen

@"KlemenSkrlj"#p34624 Hi Klemen, following up with results from testing locally with `modelconverter`, as you suggested. **Setup**: Same model, same settings (`mean=153.255`, `scale=86.7`, RVC4, `FP16_STANDARD` quantization mode explicitly set). **What I tested, systematically, at each stage of the pipeline:** 1. Real PyTorch backbone (FP32): `layer2 max ≈ 2.03`, `layer3 max ≈ 2.37` — ground truth 2. Same backbone in FP16 (on PC, isolated from RVC4 entirely): `layer2 max ≈ 2.03`, `layer3 max ≈ 2.37` — confirms FP16 precision itself isn't the issue 3. The modified ONNX file produced by `modelconverter` (with preprocessing baked in), run directly via `onnxruntime`, fed raw 0-255 pixel values: `layer2 max ≈ 2.45`, `layer3 max ≈ 3.00` — also correct, confirming the preprocessing-baking step works correctly 4. The actual OAK4 camera, running this same model: `layer2 max ≈ 700-780`, `layer3 max ≈ 1200-1240` — still wrong, \~300-400x inflated, matching every previous test on real hardware Everything checks out correct right up through the ONNX file. **Is there something specific in the ONNX-to-RVC4 conversion (the `.dlc` compilation or quantization step) that could introduce this kind of distortion**, even when FP16 mode is explicitly set? Or does this pattern suggest the issue is more likely happening at runtime, on the actual RVC4 chip itself?

@"OmkarDevadas"#p34633 What thing you can try is setting `onnx_optimizations` to `False` either through overrides or on the config level of modelconverter. This will make it so no optimizations are tried which might be what is corrupting your model. Otherwise the only thing that should happen between ONNX to RVC4 is this baking in of the preprocessing step. The quantization should be skipped with FP16 mode so that shouldn't affect it. If you can't figure it out I can look into it more hands on as well if you send me a MRE (can be through email if you don't want to share model publicly on klemen.skrlj@luxonis.com). In the MRE please include some input sample, expected output and ONNX model with inference script so I can have it as the base which we want to match. If there is something else relevant include that as well. Best, Klemen

@"KlemenSkrlj"#p34634 Hi Hi Klemen, tried `onnx_optimizations False` as suggested — same exact failure, same exact point: $$ error_code=1200; Unable to load backend library libHtpPrepare.so $$ I dug into this directly — the file exists inside the `2.41.0` image, but it's **0 bytes**: $$ -rwxrwxrwx 1 root root 0 Dec 8 2025 /opt/snpe/lib/x86_64-linux-clang/libHtpPrepare.so $$ This is a corrupted/empty file in the published Docker image itself, not an ONNX optimization issue. I tried other SNPE tool versions (`2.40.0`, `2.32.6`) but both require Qualcomm's auto-download, which 404s without authentication. Could you check/replace this specific file in the `2.41.0` image? Happy to send the full MRE (ONNX model + input sample + expected output) via email if that helps you reproduce it directly.

Can we do Patchcore Anomaly Detection in the OAk camera

OmkarDevadas

KlemenSkrlj
Hi Klemen,

Thanks, that clarifies the execution differences.

One final clarification: when the same model is correctly converted and executed through QNN/HTP on RVC4, should we normally expect the intermediate backbone outputs (layer2/layer3) to be numerically different from the PyTorch outputs on PC?

So, should we consider this type of numerical difference between PyTorch and RVC4/QNN to be expected due to the different backend execution, or should we investigate further if the difference is this large?

KlemenSkrlj

OmkarDevadas
Yes, some numerical differences between PyTorch and RVC4/QNN are expected, including in intermediate backbone outputs. A correctly converted model does not necessarily produce bit-identical tensors across backends. Different kernels, operation fusion and reduction order can affect the results, even when both paths use FP16.

Based on your usecase where this kind of magnitude of change can affect final result I would suggest going further and trying to verify that the modified ONNX (produced by modelconverter internall after all optimizations and simplifications), which gets converted to the RVC4 device format, still matches original PyTorch. And if this matches then go into testing if difference comes when going from ONNX -> RVC4 by running qnn-net-run command directly on device. And the last step is running model in your DepthAI pipeline. This investigation should let you know in which stage of the process you start to get this difference and then have a sense what to investigate deeper.

Best,
Klemen

OmkarDevadas

KlemenSkrlj
Hi Klemen,

One more clarification regarding the Layer2/Layer3 outputs.

We are already seeing numerical differences between the PyTorch Layer2/Layer3 outputs on PC and the Layer2/Layer3 outputs from the OAK4.

When the model is running on the camera, are the Layer2/Layer3 outputs we receive the direct RVC4/QNN inference outputs, or is there any additional processing/transformation being applied to these outputs before they are returned to our application?

Also, could you clarify how the outputs are processed on the camera after the backbone inference to eventually calculate the PatchCore anomaly score?

We want to reproduce the same processing/calculations on the PC so that we can make the PC and OAK4 results directly comparable.

Thanks.

KlemenSkrlj

We are already seeing numerical differences between the PyTorch Layer2/Layer3 outputs on PC and the Layer2/Layer3 outputs from the OAK4.

Yes, that is why I'm suggesting you look at the intermediate steps as well. Because the conversion goes from:
PyTorch model -> ONNX model -> Optimized/Modified ONNX model -> DLC ran on host with emulator -> DLC ran on device with emulator -> DLC ran with DepthAI

Right now you are comparing PyTorch to the final DLC ran with DepthAI. But there are several steps in between that you could investigate to figure out on which stage does the difference happen excetly and that could help you pin down what is the problem.

OmkarDevadas

KlemenSkrlj
Hi Klemen,

Thanks. Here is where I am in the chain (same frame, layer2/layer3 vs PyTorch FP32):

1. PyTorch -> ONNX: match (cosine 1.0)

2. ONNX -> modified ONNX: match (cosine 1.0, raw BGR input; flip and mean/scale verified)

3. DLC on host with emulator: not tested yet

4. DLC on device: not tested yet (SSH password login disabled, OS 1.32.1)

5. DLC with DepthAI: large difference (cosine 0.72 / 0.77; score 3.29 PC vs 2.11 OAK4). Output layout checked, not a transpose issue.

So the difference starts after the modified ONNX.

Questions:

1. For step 3, is snpe-net-run on x86 (from the ModelConverter SNPE 2.41 image) the right way, ?

2. For step 4, is oakctl adb shell the right way to get a shell on the OAK4?

Best,

Omkar

OmkarDevadas

KlemenSkrlj

Hi Klemen,

Thanks, I went through the chain stage by stage. I used the same frame for every test (the exact passthrough frame from DepthAI) and compared the layer2/layer3 values with PyTorch FP32:

PyTorch -> ONNX: same values as PyTorch
Optimized/modified ONNX (from ModelConverter intermediate_outputs): same values as PyTorch
DLC on host:
- CPU, FP32 (snpe-net-run): same values as PyTorch
- HTP emulator, FP16 (qnn-net-run + libQnnHtp.so): almost the same values, with only small FP16 rounding differences
- HTP emulator with the final DLC from the .rvc4.tar.xz archive: almost the same values, identical to the result above
DLC with DepthAI on OAK4: different values, with differences much larger than FP16 rounding

I also checked that the output layout (HWC) and input format (BGR888i, interleaved vs. planar) are not the cause.

So, the exact DLC that runs on the camera gives the same values as PyTorch on the host emulator, but the values are significantly different when the same model is executed on the OAK4 through DepthAI.

Why would the device execution produce different layer2/layer3 values if the same DLC produces almost identical values on the HTP emulator? Could this be related to the device-specific HTP graph preparation, runtime, or some other difference in how DepthAI executes the model on the OAK4?

Best,
Omkar

KlemenSkrlj

For step 3, is snpe-net-run on x86 (from the ModelConverter SNPE 2.41 image) the right way, ?

I'd suggest qnn-net-run as mentioned in one of the earlier answers. Though likely snpe-net-run might also work.

For step 4, is oakctl adb shell the right way to get a shell on the OAK4?

Yes, or if you know the IP of the device you can simply just ssh into it (ssh root@<device ip>).

KlemenSkrlj

In that case I would need to see a MRE (minimal reproducible example) with the ONNX, DLC, ONNX inference script, DepthAI inference script, inputs and expected outputs for this whole procedure so I can investigate it further.
This is not expected and most likely there is either a small inconsistency inside the DepthAI inference script or a real DepthAI core bug. But I'd have to look more deep into it to see what is the real cause.
So if you'd like you can send me the MRE on email (klemen.skrlj@luxonis.com) and we can go from there.

Best,
Klemen

« Previous Page