Choosing the right model - docTR documentation (original) (raw)

The full Optical Character Recognition task can be seen as two consecutive tasks: text detection and text recognition. Either performed at once or separately, to each task corresponds a type of deep learning architecture.

For a given task, docTR provides a Predictor, which is composed of 2 components:

• PreProcessor: a module in charge of making inputs directly usable by the deep learning model.
• Model: a deep learning model, implemented with all supported deep learning backends (PyTorch) along with its specific post-processor to make outputs structured and reusable.

Text Detection¶

The task consists of localizing textual elements in a given image. While those text elements can represent many things, in docTR, we will consider uninterrupted character sequences (words). Additionally, the localization can take several forms: from straight bounding boxes (delimited by the 2D coordinates of the top-left and bottom-right corner), to polygons, or binary segmentation (flagging which pixels belong to this element, and which don’t). Our latest detection models works with rotated and skewed documents!

Available architectures¶

The following architectures are currently supported:

• linknet_resnet18
• linknet_resnet34
• linknet_resnet50
• db_resnet50
• db_mobilenet_v3_large
• fast_tiny
• fast_small
• fast_base

For a comprehensive comparison, we have compiled a detailed benchmark on publicly available datasets:

All text detection models above have been evaluated using both the training and evaluation sets of FUNSD and CORD (cf. doctr.datasets). Explanations about the metrics being used are available in Task evaluation.

Disclaimer: both FUNSD subsets combined have 199 pages which might not be representative enough of the model capabilities

Seconds per iteration (with a batch size of 1) is computed after a warmup phase of 100 tensors, by measuring the average number of processed tensors per second over 1000 samples. Those results were obtained on a 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz.

Detection predictors¶

detection_predictor wraps your detection model to make it easily useable with your favorite deep learning framework seamlessly.

import numpy as np from doctr.models import detection_predictor model = detection_predictor('db_resnet50') dummy_img = (255 * np.random.rand(800, 600, 3)).astype(np.uint8) out = model([dummy_img])

You can pass specific boolean arguments to the predictor: * pretrained: if you want to use a model that has been pretrained on a specific dataset, setting pretrained=True this will load the corresponding weights. If pretrained=False, which is the default, would otherwise lead to a random initialization and would lead to no/useless results. * assume_straight_pages: if you work with straight documents only, it will fit straight bounding boxes to the text areas. * preserve_aspect_ratio: if you want to preserve the aspect ratio of your documents while resizing before sending them to the model. * symmetric_pad: if you choose to preserve the aspect ratio, it will pad the image symmetrically and not from the bottom-right.

For instance, this snippet will instantiates a detection predictor able to detect text on rotated documents while preserving the aspect ratio:

from doctr.models import detection_predictor predictor = detection_predictor('db_resnet50', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)

Text Recognition¶

The task consists of transcribing the character sequence in a given image.

Available architectures¶

The following architectures are currently supported:

• crnn_vgg16_bn
• crnn_mobilenet_v3_small
• crnn_mobilenet_v3_large
• sar_resnet31
• master
• vitstr_small
• vitstr_base
• parseq
• viptr_tiny

For a comprehensive comparison, we have compiled a detailed benchmark on publicly available datasets:

All text recognition models above have been evaluated using both the training and evaluation sets of FUNSD and CORD (cf. doctr.datasets). Explanations about the metric being used (exact match) are available in Task evaluation.

While most of our recognition models were trained on our french vocab (cf. Supported Vocabs), you can easily access the vocab of any model as follows:

from doctr.models import recognition_predictor predictor = recognition_predictor('crnn_vgg16_bn') print(predictor.model.cfg['vocab'])

Disclaimer: both FUNSD subsets combine have 30595 word-level crops which might not be representative enough of the model capabilities

Seconds per iteration (with a batch size of 64) is computed after a warmup phase of 100 tensors, by measuring the average number of processed tensors per second over 1000 samples. Those results were obtained on a 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz.

Recognition predictors¶

recognition_predictor wraps your recognition model to make it easily useable with your favorite deep learning framework seamlessly.

import numpy as np from doctr.models import recognition_predictor model = recognition_predictor('crnn_vgg16_bn') dummy_img = (255 * np.random.rand(50, 150, 3)).astype(np.uint8) out = model([dummy_img])

End-to-End OCR¶

The task consists of both localizing and transcribing textual elements in a given image.

Available architectures¶

You can use any combination of detection and recognition models supported by docTR.

For a comprehensive comparison, we have compiled a detailed benchmark on publicly available datasets:

All OCR models above have been evaluated using both the training and evaluation sets of FUNSD and CORD (cf. doctr.datasets). Explanations about the metrics being used are available in Task evaluation.

Disclaimer: both FUNSD subsets combine have 199 pages which might not be representative enough of the model capabilities

Two-stage approaches¶

Those architectures involve one stage of text detection, and one stage of text recognition. The text detection will be used to produces cropped images that will be passed into the text recognition block. Everything is wrapped up with ocr_predictor.

import numpy as np from doctr.models import ocr_predictor model = ocr_predictor('db_resnet50', 'crnn_vgg16_bn', pretrained=True) input_page = (255 * np.random.rand(800, 600, 3)).astype(np.uint8) out = model([input_page])

You can pass specific boolean arguments to the predictor:

• assume_straight_pages: if you work with straight documents only, it will fit straight bounding boxes to the text areas.
• preserve_aspect_ratio: if you want to preserve the aspect ratio of your documents while resizing before sending them to the model.
• symmetric_pad: if you choose to preserve the aspect ratio, it will pad the image symmetrically and not from the bottom-right.

Those 3 are going straight to the detection predictor, as mentioned above (in the detection part).

Additional arguments which can be passed to the ocr_predictor are:

• export_as_straight_boxes: If you work with rotated and skewed documents but you still want to export straight bounding boxes and not polygons, set it to True.
• straighten_pages: If you want to straighten the pages before sending them to the detection model, set it to True.

For instance, this snippet instantiates an end-to-end ocr_predictor working with rotated documents, which preserves the aspect ratio of the documents, and returns polygons:

from doctr.models import ocr_predictor model = ocr_predictor('linknet_resnet18', pretrained=True, assume_straight_pages=False, preserve_aspect_ratio=True)

Additionally, you can change the batch size of the underlying detection and recognition predictors to optimize the performance depending on your hardware:

• det_bs: batch size for the detection model (default: 2)
• reco_bs: batch size for the recognition model (default: 128)

from doctr.models import ocr_predictor model = ocr_predictor(pretrained=True, det_bs=4, reco_bs=1024)

To modify the output structure you can pass the following arguments to the predictor which will be handled by the underlying DocumentBuilder:

• resolve_lines: whether words should be automatically grouped into lines (default: True)
• resolve_blocks: whether lines should be automatically grouped into blocks (default: False)
• paragraph_break: relative length of the minimum space separating paragraphs (default: 0.035)

For example to disable the automatic grouping of lines into blocks:

from doctr.models import ocr_predictor model = ocr_predictor(pretrained=True, resolve_blocks=False)

Running the predictors on GPU¶

You can run the predictors on GPU by specifying the appropriate device.

Here’s how to do it for both NVIDIA and Apple Silicon (MPS) GPUs:

import torch from doctr.models import ocr_predictor

For NVIDIA GPU

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') predictor = ocr_predictor(pretrained=True).to(device)

Alternatively: predictor = ocr_predictor(pretrained=True).cuda()

For Apple Silicon (MPS)

device = torch.device('mps' if torch.backends.mps.is_available() else 'cpu') predictor = ocr_predictor(pretrained=True).to(device)

The same approach applies to all standalone predictors:

• recognition_predictor
• detection_predictor
• crop_orientation_predictor
• page_orientation_predictor

Just create the predictor instance and move it to the appropriate device. To enable half-precision inference, you can append .half() after moving the predictor to the device.

What should I do with the output?¶

The ocr_predictor returns a Document object with a nested structure (with Page, Block, Line, Word, Artefact). To get a better understanding of our document model, check our Document structure section

Here is a typical Document layout:

Document( (pages): [Page( dimensions=(340, 600) (blocks): [Block( (lines): [Line( (words): [ Word(value='No.', confidence=0.91), Word(value='RECEIPT', confidence=0.99), Word(value='DATE', confidence=0.96), ] )] (artefacts): [] )] )] )

To get only the text content of the Document, you can use the render method:

text_output = result.render()

For reference, here is the output for the Document above:

You can also export them as a nested dict, more appropriate for JSON format:

json_output = result.export()

For reference, here is the export for the same Document as above:

{ 'pages': [ { 'page_idx': 0, 'dimensions': (340, 600), 'orientation': {'value': None, 'confidence': None}, 'language': {'value': None, 'confidence': None}, 'blocks': [ { 'geometry': ((0.1357421875, 0.0361328125), (0.8564453125, 0.8603515625)), 'lines': [ { 'geometry': ((0.1357421875, 0.0361328125), (0.8564453125, 0.8603515625)), 'words': [ { 'value': 'No.', 'confidence': 0.914085328578949, 'geometry': ((0.5478515625, 0.06640625), (0.5810546875, 0.0966796875)), 'objectness_score': 0.96, 'crop_orientation': {'value': 0, 'confidence': None}, }, { 'value': 'RECEIPT', 'confidence': 0.9949972033500671, 'geometry': ((0.1357421875, 0.0361328125), (0.51171875, 0.1630859375)), 'objectness_score': 0.99, 'crop_orientation': {'value': 0, 'confidence': None}, }, { 'value': 'DATE', 'confidence': 0.9578408598899841, 'geometry': ((0.1396484375, 0.3232421875), (0.185546875, 0.3515625)), 'objectness_score': 0.99, 'crop_orientation': {'value': 0, 'confidence': None}, } ] } ], 'artefacts': [] } ] } ] }

To export the output as XML (hocr-format) you can use the export_as_xml method:

xml_output = result.export_as_xml() for output in xml_output: xml_bytes_string = output[0] xml_element = output[1]

For reference, here is a sample XML byte string output:

Advanced options¶

We provide a few advanced options to customize the behavior of the predictor to your needs:

• Modify the binarization threshold for the detection model.
• Modify the box threshold for the detection model.

This is useful to detect (possible less) text regions more accurately with a higher threshold, or to detect more text regions with a lower threshold.

import numpy as np from doctr.models import ocr_predictor predictor = ocr_predictor('db_resnet50', 'crnn_vgg16_bn', pretrained=True)

Modify the binarization threshold and the box threshold

predictor.det_predictor.model.postprocessor.bin_thresh = 0.5 predictor.det_predictor.model.postprocessor.box_thresh = 0.2

input_page = (255 * np.random.rand(800, 600, 3)).astype(np.uint8) out = predictor([input_page])

• Disable page orientation classification

If you deal with documents which contains only small rotations (~ -45 to 45 degrees), you can disable the page orientation classification to speed up the inference.

This will only have an effect with assume_straight_pages=False and/or straighten_pages=True and/or detect_orientation=True.

from doctr.models import ocr_predictor model = ocr_predictor(pretrained=True, assume_straight_pages=False, disable_page_orientation=True)

• Disable crop orientation classification

If you deal with documents which contains only horizontal text, you can disable the crop orientation classification to speed up the inference.

This will only have an effect with assume_straight_pages=False and/or straighten_pages=True.

from doctr.models import ocr_predictor model = ocr_predictor(pretrained=True, assume_straight_pages=False, disable_crop_orientation=True)

• Add a hook to the ocr_predictor to manipulate the location predictions before the crops are passed to the recognition model.

from doctr.models import ocr_predictor

class CustomHook: def call(self, loc_preds): # Manipulate the location predictions here # 1. The output structure needs to be the same as the input location predictions # 2. Be aware that the coordinates are relative and needs to be between 0 and 1 return loc_preds

my_hook = CustomHook()

predictor = ocr_predictor(pretrained=True)

Add a hook in the middle of the pipeline

predictor.add_hook(my_hook)

You can also add multiple hooks which will be executed sequentially

for hook in [my_hook, my_hook, my_hook]: predictor.add_hook(hook)