Scaling Agreement Data Extraction with Purpose-Built VLMs
AI Engineergo watch the original →
the gist
Docusign and NVIDIA replaced generic layout models with a 900M parameter purpose-built VLM to extract structured table data from agreements 20x faster than general-purpose alternatives.
The Breakthrough
Docusign and NVIDIA developed a purpose-built 900 million parameter vision language model (VLM) specifically for document table extraction, which outperforms general-purpose models by preserving semantic structure and layout in complex, nested, or merged-cell tables.
What Actually Worked
- Model Specialization: The team shifted from generic VLM architectures to a compact 900M parameter model optimized solely for extraction tasks rather than generation, significantly reducing latency and compute costs.
- Single-Pass Processing: The architecture replaces a multi-stage stack of separate layout and table-detection models with a single-shot inference pass that simultaneously identifies reading order, semantic formatting, and table structure.
- Quantization and Optimization: The model currently runs on FP16 precision, with active development paths toward FP8 and FP4 quantization to further optimize throughput on NVIDIA hardware.
- Hybrid Pipeline: Docusign utilizes a dual-route pipeline where the purpose-built VLM handles complex table extraction, while standard OCR processes the remaining text and metadata fields to ensure high-fidelity data ingestion.
Context
Enterprise agreements are frequently trapped in unstructured formats like PDFs and PNGs, rendering critical pricing tiers, SKUs, and service-level agreements (SLAs) unqueryable. Docusign processes approximately one million agreements daily, and traditional line-by-line text extraction consistently fails to parse the hierarchical and tabular nature of these documents. By leveraging a smaller, specialized model, the team achieved a 20x increase in table extraction speed compared to general open-source alternatives, enabling the conversion of legacy contract data into structured, actionable insights for downstream procurement and finance systems.
Notable Quotes
- "A purpose-built small model ran table extraction around twenty times faster than the general alternatives they tested."
- "It's a tiny small model that you can use; it's not a generator, it's more of an extractor at the end of the day."
Content References
{"type": "tool", "title": "NVIDIA Neotron", "url": "https://github.com/NVIDIA/Neotron", "context": "mentioned"}, {"type": "tool", "title": "NVIDIA NIM", "url": "https://www.nvidia.com/en-us/ai-data-center/nim/", "context": "mentioned"}, {"type": "dataset", "title": "RD TableBench", "context": "cited"}