Inkling: A 975B MoE Model Designed for Fine-Tuning
Better Stackgo watch the original →
the gist
Inkling is a 975B parameter mixture-of-experts model that accepts raw audio and image patches without encoders, specifically optimized for fine-tuning on narrow, high-volume tasks rather than general-purpose reasoning.
Architecture and Multimodal Processing
Inkling is a 975B parameter mixture-of-experts (MoE) model where each layer contains 256 experts, with only 41B parameters active per token. Unlike standard multimodal models that utilize separate encoders for audio or vision—which can introduce data loss through compression—Inkling processes raw input directly. Audio is fed as spectrograms, and images are processed as 40x40 pixel patches. These inputs are tokenized into the same latent space as text, allowing the model to process raw sensory data without intermediate transcription or description layers.
Fine-Tuning Strategy
The model is designed for LoRA (Low-Rank Adaptation) rather than full parameter retraining. This approach keeps the base weights frozen while injecting smaller, lightweight matrices to capture task-specific patterns. This method reduces computational overhead and allows for rapid iteration on narrow, high-volume tasks. The platform Tinker facilitates this process by automating dataset generation and training, as demonstrated by a model trained to avoid the letter 'e' while maintaining coherent output.
Performance and Use Cases
Inkling is not intended as a general-purpose reasoning model; it underperforms on agentic benchmarks like Terminal Bench (63.8 vs 82.7 for competitors). It is best suited for:
- Audio-specific tasks: It is currently the only model on the Tinker platform capable of processing raw audio, with available recipes for speech recognition and speaking style classification.
- Domain-specific classification: As seen in legal citation extraction, fine-tuning can significantly improve F1 scores (e.g., from 0.56 to 0.68) while maintaining lower latency than larger generalist models.
- Granular control: The model provides a 'thinking effort' slider (0 to 1), allowing users to tune the compute-per-token dynamically.
Before / After
- Legal Citation Extraction (Harvey): Fine-tuning improved the F1 score from 0.56 to 0.68 compared to the base model performance.
- Instruction Following: Inkling scored 79.8 on instruction following benchmarks, outperforming GLM 5.2 which scored 73.3.
- Agentic Reasoning (Terminal Bench): Inkling scored 63.8, while competing models reached 82.7.