ChartNet: Scaling Vision-Language Models via Synthetic Chart Data

The Breakthrough

ChartNet demonstrates that smaller vision-language models (VLMs) can match or exceed the performance of massive frontier models on chart-understanding tasks by training on a large-scale, synthetically generated dataset that pairs visual chart renderings with their underlying programmatic code and structured metadata.

Synthetic Data Generation Pipeline

The pipeline leverages the fact that most charts are created via code, allowing for automated, high-volume data generation through a two-stage process:

• Seed Translation: The system begins with a seed set of real-world chart images, using a VLM to reverse-engineer the approximate plotting code that generated them.
• Iterative Augmentation: An LLM performs iterative modifications within the code space to create variations of the original charts, which are then executed to render new image-code pairs.
• Metadata Grounding: Because the charts are generated from code, the system automatically extracts accurate supporting metadata, including the underlying data tables, natural language summaries, and multi-step reasoning traces for question-answering.
• Quality Filtering: A dedicated filtering stage ensures that the synthetic outputs meet specific quality thresholds before being included in the final training mixture.

Performance and Scaling

Training on ChartNet consistently improves performance across various model families, ranging from sub-one-billion parameter models to seven-billion parameter models. Notably, a two-billion parameter model trained on this dataset outperformed GPT-4o on specific chart and table extraction benchmarks. The dataset has been integrated into the core training mixtures for next-generation VLM development at IBM Research, proving that carefully structured synthetic data can effectively substitute for the massive scale typically required for frontier-level reasoning capabilities.