Building a Guitar That Speaks and Sings Using AI
AI Engineergo watch the original →
the gist
Todd Fisher built a JUCE plugin that routes speech-to-text and LLM responses through a guitar, using energy-based segmentation and pitch-shifted vocal samples to simulate speech and singing.
Speech Synthesis and Segmentation
The project uses a custom JUCE plugin to map text-to-speech output onto guitar notes. To handle word segmentation, the author initially attempted energy gap segmentation, which cuts audio at near-zero decibel levels. Because natural speech often lacks silence between words, he implemented a sonority peak syllabifier to identify vowel-heavy regions. Combining these two methods provided a baseline, though manual adjustment of segment boundaries remains necessary for high accuracy.
Pitch Detection and Singing Synthesis
To enable the guitar to sing, the system uses the YIN algorithm to extract the fundamental frequency from live guitar input. This frequency drives a synthesized tone shaped by an ADSR (Attack, Decay, Sustain, Release) envelope. The output is processed through a vocoder to shape the synthesized tone using pre-baked vocal samples. Because real-time pitch shifting is computationally expensive, the author uses the World vocoder library to pre-process samples from the VocalSet dataset, mapping specific vowel samples to individual frets and notes.
Conversational Integration
The live demonstration uses a local LLM to generate responses from spoken input. The pipeline captures audio via a microphone, converts it to text using Whisper, processes the text through a local LLM, and feeds the resulting string into the JUCE plugin for playback through the guitar strings. This setup allows the guitar to answer questions in real-time, albeit with some latency and speech-synthesis artifacts.