Training a 3B State-Space Vision Model for Indic Languages
AI Engineergo watch the original →
the gist
Sarvam built a 3B parameter vision-language model using a state-space architecture to perform block-level OCR, achieving state-of-the-art results on Indic languages by prioritizing a four-stage curriculum and verifiable reinforcement learning.
Architecture and Training Strategy
The model utilizes a state-space model (SSM) backbone rather than a transformer to handle long documents, which can reach 10,000 visual tokens per page. While transformers incur quadratic compute and memory costs as sequence length increases, the SSM maintains constant memory and linear compute growth. The team adopted a contrarian block-level OCR approach, wrapping the model in layout and reading-order harnesses to improve document intelligence.
Training follows a four-stage curriculum:
- Stage 1: Text-only pre-training on 13 trillion tokens, including English, 22 Indian languages, math, and code, to establish a strong language prior.
- Stage 2: Continual pre-training on 300 million image-text pairs to develop general vision capabilities.
- Stage 3: Supervised fine-tuning on 100 million OCR samples covering diverse document components like tables and handwritten text.
- Stage 4: Reinforcement learning (RL) using machine-verifiable rewards based on character error rates, table structure, and grammar.
Data and Production Impact
Because Indic languages are largely underrepresented in common crawl data, the team built a proprietary data engine to generate synthetic and real-world document data. This pipeline enables the model to resolve ambiguous or smudged text by leveraging its language prior. The model is currently deployed in an agentic workbench called 'Aure', which facilitates human-in-the-loop proofreading and digitization. It is currently processing 35 million pages for banking, insurance, and government sectors.