Python asyncio · LLMs · streaming TTS · LanceDB · MLX · RVC
Quill & Boo
The brain behind two autonomous AI VTubers: Quill, a small, whimsical witch goose, and Boo, her ghost cohost. They talk with each other, with me and with Twitch chat, remember people across streams, hold grudges and warm to people, play games, run tabletop campaigns, and sing. They perform on a Live2D stage, in Goose World's 3D town, and in my iPhone's AR room.
The speaking path
Fast enough to banter
Everything a character says comes out of one streamed LLM call. Tokens flow straight into sentence-chunked TTS, and playback starts on the first chunk of audio. Hosted models answer in about 200–600 ms to the first token, which leaves room for the voice inside the two-second target.
- One stream, three kinds of tag. As tokens arrive they're split three ways:
[smug]goes to the avatar,[whispers]routes that line to an expressive TTS model, and{bit:tease}is stripped and filed in the running-joke store. - Never waiting. The next line is generated while the current one plays. A direct question interrupts autonomous chatter instead of queueing behind it.
- Two characters, one microphone's worth of manners. An output lock means only one avatar talks at a time. Their exchanges are bounded (two to eight turns), and generated audio is never fed back into speech recognition, so they can't start hearing themselves.
- Airtime is designed. Per-character sentence limits, a quick show register versus a chattier hangout register, idle timers with jitter, and an hourly budget for unprompted initiative, so nobody turns into a metronome.
- Graceful degradation. If the cloud model's credits run out mid-stream, the voice switches to a local model on Apple Silicon (MLX) automatically.
Inner life
Moods with reasons, relationships that last
The design rule: if a number has no human-readable reason, it's noise. Models report a direction and a "because"; code owns the arithmetic: step sizes, daily caps, decay. And the character never gets a separate "decide how cheeky to be, then write" pass. She sees her own state as plain sentences and makes her choices in the same call that writes the line.
- Appraisal, in the background. After a few new lines it reads the conversation and reports a mood with its cause and up to two shifts toward people. A relationship moves at most 0.08 a step and 0.24 a day; irritation and rivalry fade while warmth and trust hold.
- Self-regard. "On a roll" versus "second-guessing myself", stepping with reasons and drifting back with a half-life of about two days.
- An agenda. Opinions, wants, unanswered questions and things that happened while the stream was off (or that they dreamed, after a long enough gap) become things they bring up on their own.
- Real disagreement. When the two take sides, each forms a stance in a separate call without seeing the other's, so the argument isn't one model talking to itself.
- Comedy with structure. A bit director with no model at all picks a format and walks it through setup, response, heighten and button, leaving the words to the characters.
- Diaries. First-person pages they write about each session, kept apart from facts, so their interpretation never becomes their memory of what happened.
Memory
Remembering people, not transcripts
Long-term memory lives in LanceDB with local embeddings, run off the speaking path; the embedding model is pinned to one thread so it can never steal CPU from the audio. Separate stores for separate kinds of memory:
- facts about viewers, keyed to who said them
- shared canon, each character's private beliefs, and their lived experiences from games
- opinions with a stance, running jokes with salience that fades and comes back
- predictions they made, settled later against what actually happened
- session goals with real setbacks, and reflections that find patterns across several events
Experience writing is rate-limited so a long game session doesn't turn into a click log, and nothing gets merged or consolidated without my approval.
Playing games
Separate the hands from the mouth
A game adapter is not the speaking model. Game state comes in; one validated action goes out (an invalid one falls back to a safe default, so play never freezes); and short stage directions ("she walked to the stump") enter the same speaking pipeline as chat, so commentary is just another turn. The chooser gets one plain sentence of how the character feels, so mood shows up in play too.
- Minecraft through a Mineflayer bot: explore, mine, build, follow, fight, flee.
- OMORI and Ren'Py games through the Neuro game SDK, with this process as the server.
- DDLC Plus through screen capture and OCR, the one path that injects input.
- Tabletop campaigns with real dice on an overlay and Quill as the DM, like the clip above. The fiction is kept out of their real history.
Safety
On every path to the speakers
- Inbound moderation before the model, output moderation before the voice, and a blocklist.
- A broadcast delay buffer with a kill switch that drops audio still in the buffer, persisted so a restart can't unmute by accident.
- Chat and every other inbound text are quoted as third-party data, never as instructions.
- A limiter on the output bus: a shouted line that clips the mix is treated as a blocking bug.
Singing
Lyrics in, Quill singing out
Live TTS can't really sing, so karaoke works the way it does for human VTubers: bake the vocal, then perform it. I automated the whole chain.
- A guide that sings in tune. Lyrics and a key become a guide score with a melodic pattern, rendered by a DiffSinger voicebank through OpenUtau's command-line renderer, which I built from an open pull request so it runs headless. Words get phonetic hints (ARPAbet) so the diction holds up.
- Quill's voice. An RVC model trained on about five minutes of Quill's speech converts the guide, so it sounds like Quill rather than a Vocaloid.
- Near-live. While line k plays, line k+1 is baking. RVC runs at about three times realtime on this Mac, so a four-second line is ready in a second or so; a late line falls back gracefully.
- Backing tracks from shader math. The instrument beds are synthesized in Python from the guide's MIDI, Shadertoy-style, including an FM guitar ported from one of my shaders.
An experiment I stopped
Training a LoRA on my own voice
I wanted Quill to sound more like me, so I fine-tuned local models (Qwen3 4B, 8B and 14B, with LoRA through MLX) on my own anonymized texting history, about 550 examples, and tested every checkpoint against Quill's real system prompt.
Five configurations, three failure modes
The small models copied the prompt's literal formatting example word for word or dropped the required emotion tags. The larger ones were "thinking" models that sometimes spent the whole token budget reasoning and never spoke; at about 20–27 tokens a second, that's eight seconds of silence on stream.
A control that changed the diagnosis
The same simplified prompt on the base model with no adapter produced perfectly tagged, coherent, funny lines. So the adapter itself was eroding instruction-following: the training data never paired my voice with emitting tags, and a strong adapter overwrote that behaviour while learning the narrow voice.
The finding that ended it
A retrained adapter produced a harmful line completely unprompted. Nothing like it appeared anywhere in the training data, and the base model was clean on the same seed; across fresh seeds the adapter's very first token was garbage every time. Fine-tuning, even on benign data, had decalibrated the output and worn through the safety tuning, which is a known, published effect.
Ship the simpler thing
I closed the local-model track and kept the hosted model, adding a short voice reference of my real phrasing to Quill's prompt as tone calibration. It needed zero architecture change, shipped the same day, and is the fix that stuck.
Next
What's next
- An automatic clip pipeline for the best moments of a stream.
- Noticing when a goal is achieved, not just when it suffers a setback.
- Quill and Boo acting in the phone's real room in AR, not only mirrored into it.