CUSTOM OCR TRAINING

dhunjoshi

I WANT TO TRAIN CUSTOM OCR . CAN I GET HELP REGARDING THIS. I PREFER TO USE LOCAL SYSTEM FOR TRAINING.

Brandon

So there are two parts to our OCR system:

Text detection. We used EAST, as it allows text to be oriented at odd angles/etc. I am pinging our engineer on this on if he did retraining. Anyway, here is one example of retraining it in TensorFlow, which then should be compatible with our platform, as OpenVINO supports TensorFlow.
OCR. This takes the region found from the text detection, and runs the actual OCR on it. For this network, we use the OCR model from Intel: https://docs.openvinotoolkit.org/2019_R1/_text_recognition_0012_description_text_recognition_0012.html

I do not at this time know if Intel gives a reference on how to retrain that. For some of their networks, they do give retraining. Looking quickly. Not immediately seeing it.

From the notes on it, "VGG16-like backbone and bidirectional LSTM encoder-decoder". So likely any network that is similar to this could be used instead, as long as it is on a similar backbone (or one of these), or uses neural operations supported by OpenVINO for the VPU (OAK-D is the VPU in this context), see here.

demoacct01

i tried EAST and result as shown. EAST was suggested by Brandon

By the way Brandon is there any OCR examples ?

dhunjoshi

Can OCR will work in attached image. i have to retrain with new images?

Brandon

Thanks @demoacct01 . To your question demoacct01 , we do have an in-progress example but it is crashing on some of the warp/re-shape when taking the EAST bounding boxes and feeding them through WARP/deWARP on the Myriad X.

We just fixed the crash (got the message that it was fixed just now, actually). So the last step is the OCR output decoding. But the whole pipeline is actually now running.

Here's the Github issue:
https://github.com/luxonis/depthai/issues/124

And here's the WIP PR for the whole demo flow:
https://github.com/luxonis/depthai-experiments/pull/26

We will likely update the PR shortly with the whole flow working, excepting the host-side decoding of the OCR. We're discussing internally the timing on that.

Thoughts?

Thanks again,
Brandon

demoacct01

Hi Brandon, git checkout the branch gen2_ocr and gave it a test run. It runs slow ... :-(

I tried to move the paper (with text on it) I was holding in front of the camera. Was like it never went past the first few frame or something. The output screen just freeze with the bounding box on some texts on the paper I was holding.

When I did pip install -r requirements.txt, I got this error.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
depthai-gui 1.0.7 requires depthai==0.0.2.1+d436ec6b629c09b92c58d869e80aac52367a3aa9, but you have depthai 0.0.2.1+9430403bc960388da512c6c8936c27f8d1fa8b2d which is incompatible.

Could this cause the "freeze" ?

Brandon

Hi demoacct01 ,

Sorry about the trouble. The OCR is likely the slow part. The detection itself is a lot faster:

This example is still a work in progress. Luxonis-Alex is finishing it up now. We actually lost the engineer who was working on this (another one to Amazon!) so it got delayed as a result.

This PR is what will allow for the flow to be on DepthAI/OAK directly:
https://github.com/luxonis/depthai-shared/pull/16

It's what took a lot of time/debugging here. So now that this is out of the way, hopefully we will have the whole flow working well soon.

Thanks and sorry about the delay.
-Brandon

demoacct01

No problem Brandon ... I am working on some OCR project at the moment. So was eager to see a working sample from your end.

I hope I didn't misunderstood your response. Are you trying to say that if I "git checkout gen2_ocr" and run main.py I should see a speedup improvement? I tested it and it was slightly faster but not as fast as what was shown in the youtube. I realised the youtube was a last year video. So I am a bit confused now whether the fix is completed or not.

Luxonis-Alex

Hi @demoacct01 ,
The demo in the Youtube video is only running the text detection (bounding boxes) network, whereas the Gen2 one https://github.com/luxonis/depthai-experiments/pull/26 is running both text detection and text recognition, passing each cropped bounding box to the recognition model. So depending on the number of detections, it may be slower.

I'm working now on this Gen2 example, moving the rotated cropping from host (OpenCV warp) to run on device (with the recent rotated cropping/rescale added in ImageManip node), and will add a decoder for the recognition model outputs as well: https://docs.openvinotoolkit.org/2020.1/_models_intel_text_recognition_0012_description_text_recognition_0012.html