Improving Small Model Tool Use via RL Instead of Scaling

AI Engineer2026-06-10go watch the original →

the gist

Fine-tuning a 4B parameter model with RL on tool-use behaviors outperforms a 235B model on financial analysis tasks by teaching it to inspect schemas and self-correct errors.

The Breakthrough

By shifting focus from model size to tool-use discipline, a 4B parameter model was fine-tuned with Reinforcement Learning (RL) to outperform a 235B parameter model on complex financial reasoning tasks. The breakthrough relies on teaching the model to inspect environments and self-correct, rather than relying on inherent reasoning capabilities.

What Actually Worked

Environment-Aware Tooling: Instead of guessing queries, the model was trained to first call get_table_name to discover available tables and get_table_info to inspect schemas before executing SQL.
Error-Driven Self-Correction: The model was trained to observe SQL execution errors (such as missing columns) and perform a corrective action to identify the correct schema, rather than hallucinating an answer when the initial query failed.
GRPO Training: The team utilized Group Relative Policy Optimization (GRPO) to fine-tune the 4B model, achieving significant performance gains in a 21-hour training job costing under $500.
Curriculum Simplification: Contrary to expectations, training exclusively on single-table questions yielded the greatest performance uplift, which generalized effectively to harder multi-table reasoning tasks.
Rubric-Based Evals: The team decomposed model responses into rubrics to identify specific behavioral failures (e.g., failure to inspect schema) before generating targeted training data.

Before / After

FinQA Reasoning Benchmark: Accuracy improved from 13.9% to 26.6% on multi-table reasoning tasks after the 4B model was fine-tuned on single-table data.
Pass@1 Performance: The 4B model achieved a doubling in success rate compared to its pre-fine-tuned state, successfully navigating tool-use sequences where the 235B model hallucinated.

Context

Large models often fail at tool-use tasks because they attempt to reason through problems without first verifying the environment state. By treating tool-use as a behavioral problem rather than a knowledge-retrieval problem, the research team demonstrated that smaller, on-premise models can be made production-ready for enterprise financial applications without the inference costs or data privacy risks associated with massive models.