Needle 3: Lightweight On-Device Function Calling

Architecture and Performance

Needle 3 is a 35MB automation-focused model that replaces standard transformer feed-forward networks with a multi-layer perceptron and an n-gram embedding layer. This architecture allows the model to output structured JSON schemas directly from unstructured text without requiring a separate parser or GPU acceleration. The model is sliceable, meaning developers can use as few as two layers (13MB) for extreme resource constraints or the full 20 layers for higher complexity, achieving inference speeds of approximately 66 milliseconds on a standard CPU.

Implementation and Gotchas

To use Needle 3, developers define standard Python functions with docstrings and a specific decorator to map inputs to the schema. Because the model is highly specialized, it lacks the general-purpose reasoning of larger LLMs and will return empty responses if no relevant function definition exists.

Key operational constraints include:

• State Management: The model retains history, which can lead to incorrect function calls if the state is not explicitly reset between queries.
• Default Values: Failing to define default values in function signatures often results in hallucinated or missing parameters.
• Input Quality: The model struggles with unstructured, conversational, or messy text that deviates from the expected schema structure.
• Classification Limits: While capable of semantic similarity tasks via embeddings, the model is less effective at intent classification than dedicated System 1 models.