Unity 6 URP · iOS ARKit · Python · LLM agents

GooseWorld

A 3D VTuber app for the Mac, an iPhone app that puts your character in your real room, and a little town for my two AI characters to live in. Two codebases, one show: the world renders and simulates, and a separate brain decides what Quill and Boo say and do.

Goose Steimann2026 · ongoingMac · iPhone 15 Pro

A day in Goose World, recorded in the app with its own clip recorder; the sound is the world's own (unmute for it). Golden hour on Main Street, dusk coming on as the clock runs, the river at night, Quill and Boo at the arcade's rhythm game, claw machine, Pond Pong and photo booth, then a soak at the bath house.

2apps: Mac (Unity 6 URP) and iPhone (ARKit)

< 2 starget from a line of chat to the cast's first spoken word

1,098automated EditMode tests in the world

322 → 39 MBtexture memory after the performance pass

Why

Warudo doesn't run on a Mac, and my characters needed somewhere to live

Warudo is the 3D VTuber app most streamers use, and it's Windows-only. I wanted the same thing natively on Apple Silicon: load an avatar, drive it with your face, walk it around, and send it to OBS with a transparent background. I track it against Warudo feature by feature: 62 of 95 rows are done, including all but one of the features needed to stream.

The second reason was Quill and Boo, the AI characters I'd been building. They were a flat Live2D overlay. I wanted them in a world where they could walk to the café, play the arcade and mess up the kitchen, with me in it too.

The iPhone app

Your character, in your real room, wearing your face

In AR World the back camera tracks the room while the front TrueDepth camera tracks your face, so the character you walk around your living room makes your expressions in real time. Walk with the left thumb, tap to jump. On an iPhone 15 Pro it uses the LiDAR room mesh for placement, occlusion and shadows.

Two cameras, one session. ARKit can run face tracking on the front camera during world tracking on the back, but not every feature at once: people occlusion and front-camera face tracking compete. The app asks for everything, then logs exactly which features ARKit dropped for the current configuration, instead of failing silently.
Lit like the room. The character's lighting blends ARKit's environment probes with spherical-harmonics ambient and a camera grade; ARKit's light estimate only nudges the key light.
The cast comes too. Quill and Boo stay on the Mac (one brain, one memory). The phone mirrors their poses at 15 fps, plays their voices from where they stand, and builds a NavMesh from the LiDAR scan so they can wander your real room.
Clips without UI. Hold the shutter to record. Clips are the AR camera rendered again, so none of the controls end up in the video.

On device so far. Verified on an iPhone 15 Pro: the room mesh, occlusion, shadows and placement, walking, paint that stays on the room, and the clip recorder (the clip below). The cast wandering your room and the lighting blend are built and going through on-device checks now.

AR World on an iPhone 15 Pro, recorded with the app's own clip button (so no controls in the picture). A pancake character walks around my living room, and the paintball blaster's paint stays where it lands: on the carpet, the cushions, the table.

The world

A town with opening hours

The world is a small town by a river: a house for Quill and Boo, a laundromat, a public bath, a café, a hawker centre, a shrine and an arcade, all walkable. Shops keep real hours. Shut, the shutters come down, the signs go dark and the machines refuse you. Most things can be used: the café's kopi machine makes a cup you can drink, the washing machines run a cycle, and the arcade has three playable games.

A sky that follows the clock. A skybox per part of the day, crossfaded in a custom shader as the sun moves; distance fog and river mist change by the hour, and reflections are only redrawn when the blend has moved enough to matter.
A cutaway camera. Indoors, walls between the camera and whoever it's framing fade out with a dithered dissolve and ceilings lift off, so the camera never has to swing to find you.
Perception through walls. The brain asks the world what each character can see and hear. Walls and shut doors block sight; you hear your own room and the ones next door.
Captions as part of the render. The cast's lines type in with their voices, Animal Crossing style, on an overlay camera stacked after post, so they show up in OBS and in screenshots.

Following my character down Main Street as Quill's line types in: Race you to Pixel Pond! — The follow camera on Main Street, with Quill's line typing in with her voice.

Quill and Boo at the Pixel Pond rhythm game, the scores readable on its screen — Quill vs Boo at the rhythm game. Their timing comes from a skill value that grows a little every song.

The river bend at night: a willow, a stone lantern, lit windows and fireflies — The river bend at 21:30: lit windows across the water, the stone lantern, fireflies.

Quill and Boo soaking in the bath house tub in the steam, Boo's caption reading ...best. day. ever. — End of the day at Hana no Yu, the bath house.

Quill and Boo's corner shophouse at dusk, with flower boxes and lanterns — Quill and Boo's house, a corner shophouse with a café downstairs, at 18:00.

The arcade interior with neon, claw machines and cabinets — Pixel Pond, the arcade, once its neon finally glowed (see below).

The brain: neuro-goose

Two AI characters who live in it

Quill is a small, whimsical witch goose; Boo is her ghost cohost. They're autonomous VTubers run by neuro-goose, a single Python asyncio process I built alongside the world. Goose World is one of their stages: the brain sends them the same commands I use, and the world tells the brain what each of them can actually see and hear.

chat · voice · world events→priority queue→context + LLM (streamed)→safety→streaming TTS→voice, face and actions in the world

They have moods with reasons, relationships that last, memory of people across streams, game agency and a singing pipeline. The brain has its own case study →

Rendering and performance

Bugs worth telling

The neon that never glowed

No Blender model's emission ever drew in the built app: walk-in signs, lanterns, lit windows, the arcade's neon. Changing the glow strength produced identical frames. The build had no URP Lit _EMISSION shader variant, the one an imported GLB's emissive material needs, so it was stripped. The fix preloads the emissive variants, plus a test that loads every bundled building and checks each glowing material's variant exists.

Fog that only existed in the editor

Distance fog looked right in the editor and was missing in builds. Fog is switched on at runtime, so automatic fog stripping saw no scene using it and dropped every fog variant.

"Shadow acne" that was z-fighting

Awnings, umbrellas and cupboards flickered with a hatch pattern. It wasn't shadows: the model kits built thin surfaces as a front and a back face in the same plane, and exported every material double-sided, so both drew at the same depth. Imported models now drop the covered back face on load.

The stutter that wasn't mine

A crane shot stuttered in 20–30 second windows of ~20 ms frames. The cause was a local LLM on the same Mac streaming an 18 GB model through the same unified memory every ~70 seconds. I proved it by reproducing the stutter on demand with 20 seconds of synthetic GPU memory traffic (13 → 25.5 ms, for exactly that long).

A performance pass with numbers

Texture memory went from 597 textures / 322 MB to 154 / 39 MB by sharing identical images and compressing opaque ones as BC1, with night shots identical in a same-build A/B. Preloaded shader variants went from 114 keyword sets to the 40 actually used, cutting a cold build from about 45 to 20 minutes. The UI's state poll went from 76 ms and 23 MB of garbage on the main thread to a cached snapshot built on a worker.

Fix the bug class, not the bug

"What's the floor under this point?" was one lookup that took the highest thing from 20 m up. It caused three different bugs: café seats that saw the rooftop's furniture, a plushie placed on top of a claw machine, and a toy stowed on an easel. I replaced it with two precise questions (the walkable floor of this storey; the surface in reach to put something on), moved all 32 callers over, and added a test that scans the source so the old lookup can't come back.

The far river bank at dusk, its buildings completely dark — Before The far bank at dusk: completely dark.

The far river bank at dusk with windows and lamps glowing — After Each window gets its own pane that lights on its own schedule, plus porch lamps and signs, for six extra draws per backdrop.

Finish the on-device checks for the newer AR features, and record the AR clip for this page.
Hand-made rigs for Quill and Boo. The automatic rig is fine for walking and posing, but their big sleeves and long hair follow the nearest bone.
Give the cast the phone's room to act in, not just to be mirrored into.