GitHub - FireRedTeam/FireRed-OCR (original) (raw)

🤗 HuggingFace | 🤖 ModelScope | 🖥️ Demo | 📄 Technical Report | 🐈 GitHub

Figure 1: Performance comparison on the OmniDocBench v1.5 benchmark. FireRed-OCR achieves state-of-the-art performance among end-to-end solutions, ranking first with a score above 92%.

FireRed-OCR is a systematic framework designed to specialize general Large Vision-Language Models (LVLMs) into high-performance, pixel-precise structural document parsing experts.

General VLMs frequently suffer from "Structural Hallucination" (e.g., disordered rows, invented formulas) when processing complex documents. FireRed-OCR addresses this by shifting the paradigm from "impressionist" text generation to "structural engineering," achieving State-of-the-Art (SOTA) results on authoritative benchmarks like OmniDocBench v1.5.

✨ Key Features

• SOTA Performance: Achieves 92.94% overall score on OmniDocBench v1.5, significantly outperforming DeepSeek-OCR 2, OCRVerse, and massive general VLMs (e.g., Gemini-3.0 Pro，Qwen3-VL-235B).
• Structural Integrity: Utilizing Format-Constrained GRPO (Group Relative Policy Optimization), the model enforces strict syntactic validity, eliminating common errors like unclosed tables or invalid LaTeX formulas.
• "Geometry + Semantics" Data Factory: A novel data engine that uses geometric feature clustering and multi-dimensional tagging to synthesize balanced datasets, effectively handling long-tail layouts.
• Progressive Training Pipeline:
  1. Multi-task Pre-alignment: Establishes spatial grounding.
  2. Specialized SFT: Standardizes full-image Markdown output.
  3. Format-Constrained GRPO: Self-correction via Reinforcement Learning.
• In-the-Wild Robustness: Demonstrates superior resilience on complex, non-standard layouts (FireRedBench) compared to traditional pipeline systems like PaddleOCR.

📰 News

• 2026.02.28: We released FireRed-OCR-2B weights. Check more details in the Model Zoo section.

🗂️ Model Zoo

🏗️ Model Architecture

The FireRed-OCR framework transforms a general VLM into a structural expert through a three-stage progressive training strategy:

1. Stage 1: Multi-task Pre-alignment: Trains the model on detection, region recognition, and layout-to-markdown tasks to ground visual perception.
2. Stage 2: Specialized SFT: Fine-tunes on a high-quality, standardized Markdown dataset to ensure logical consistency and hierarchical expression.
3. Stage 3: Format-Constrained GRPO: Applies Reinforcement Learning with specific rewards for Formula Syntax, Table Integrity, Hierarchical Closure, and Text Accuracy.

⚡️ Quick Start

FireRed-OCR is based on the Qwen3-VL architecture. You can use the following code snippets to generate structured Markdown from document images.

1. Install Dependencies

pip install transformers pip install qwen-vl-utils git clone https://github.com/FireRedTeam/FireRed-OCR.git cd FireRed-OCR

2. Inference

from transformers import Qwen3VLForConditionalGeneration, AutoProcessor from conv_for_infer import generate_conv

Load the model

model = Qwen3VLForConditionalGeneration.from_pretrained( "FireRedTeam/FireRed-OCR", torch_dtype=torch.bfloat16, device_map="auto", )

We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.

model = Qwen3VLForConditionalGeneration.from_pretrained(

"Qwen/FireRed-OCR,

dtype=torch.bfloat16,

attn_implementation="flash_attention_2",

device_map="auto",

)

processor = AutoProcessor.from_pretrained("FireRedTeam/FireRed-OCR")

Prepare Input

image_path = "./examples/complex_table.png" messages = generate_conv(image_path)

Preparation for inference

inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt" ) inputs = inputs.to(model.device)

Inference: Generation of the output

generated_ids = model.generate(**inputs, max_new_tokens=8192) generated_ids_trimmed = [ out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids) ] output_text = processor.batch_decode( generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False ) print(output_text)

📊 Benchmark

We evaluate FireRed-OCR on OmniDocBench v1.5 and FireRedBench.

OmniDocBench v1.5

FireRedBench

Additional Benchmarks

For PaddleOCR-VL-1.5 and GLM-OCR on OCRBench, scores are reported as API / pure VLM.

📜 License Agreement

The code and the weights of FireRed-OCR are licensed under Apache 2.0.

🖊️ Citation

We kindly encourage citation of our work if you find it useful.

@article{fireredocr, title={FireRed-OCR Technical Report}, author={Super Intelligence Team, Xiaohongshu Inc.}, year={2026}, eprint={2603.01840}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2603.01840} }

⚠️ Ethics Statement

FireRed-OCR is a technical tool designed for document digitization and structural parsing.

• Prohibited Use: This project must not be used to generate or process content that is illegal, defamatory, pornographic, harmful, or that violates the privacy, rights, or interests of individuals or organizations.
• User Responsibility: Users are solely responsible for any content generated using this project. The authors and contributors assume no responsibility or liability for any misuse of the codebase or for any consequences resulting from its use.

🤝 Acknowledgements

We would like to thank the developers of the amazing open-source projects, including Qwen-VL, PaddleOCR, olmOCR and the broader OCR community.

⭐ Star History