Needle: A 26M Parameter Function-Calling Model

The Breakthrough

Cactus Compute developed Needle, a 26M parameter model distilled from Gemini 3.1 that performs single-shot function calling at 6,000 tokens per second (prefill) and 1,200 tokens per second (decode) by removing the feed-forward network (FFN) entirely from the transformer architecture.

Architecture and Training

The team identified that function calling relies on retrieval and assembly, tasks handled by attention mechanisms, rather than the per-position feature rewriting provided by FFNs. By removing the FFN, which typically accounts for two-thirds of a transformer's parameters, they achieved significant size reduction while maintaining performance. To compensate for the loss of the FFN, the model uses the following techniques:

• Gated residuals: Each layer learns a scalar between 0 and 1 to determine the retention of attention output.
• Zero-centered RMS norm: Normalization is initialized as an identity function to stabilize the network.
• Dual-optimizer strategy: The model uses Muon on QKVO projections to maintain weight orthogonality while using AdamW for the remaining parameters.
• Quantization-aware training: The model is trained with fake INT4 quantization every 100 steps to ensure robustness for edge deployment.
• Weighted loss: Argument values receive four times the gradient signal compared to structural tokens to prioritize accuracy in tool parameter extraction.

Deployment and Limitations

Needle is designed for edge hardware, such as smartwatches and glasses, where cloud round-trips are impractical. The repository includes a local web UI, a Python interface, and a full data generation pipeline for synthetic function call data. The model is currently restricted to single-step function calling and lacks multi-step reasoning or conversational memory, making it unsuitable for ambiguous prompts where the tool selection is not immediately obvious.