I Built My Family a Voice Assistant That Never Touches the Cloud

What this is: the implementation guide for a LAN-only family voice assistant. A wall tablet runs a thin browser client. A small Python server on a Windows box does voice activity detection, streaming speech to text, and GPU text to speech. The brain is a 9-billion-parameter local model served by llama.cpp with a deliberately small tool belt. No cloud speech service, no cloud brain, no monthly fee.

Environment used here: Windows 11 box with RTX 5060 Ti 16 GB cards, Python 3.11, an old x86 tablet running a Chromium-based OS as the kiosk. The entire voice chain runs on one card (GPU 0); the second card sits idle by design. Versions are pinned in section 2. Substitute your own hardware; the pipeline steps do not change.

Ground truth: every command, config, and number below was executed or measured on this setup during 2026-09-06, with the single-GPU consolidation re-measured on 2026-09-07. Nothing is from a spec sheet. Where a number is an estimate it is labeled as one.

Follow it top to bottom, or hand it to an AI agent as the spec (section 11).

0. Goal and acceptance

You end up with a kiosk a child can walk up to, press one button, ask a question of, and hear answered with live captions, where no voice audio ever leaves your LAN.

Acceptance criteria used for this build (all met, all measured):

Time: one working day for the full pipeline (the voice path alone was a morning). Cost: zero new hardware; all parts were already owned.

1. Architecture

tablet kiosk page (mic, speaker, ASK/STOP, captions)
   |  WebSocket, raw PCM16 audio, both directions
   v
server.py  (Python 3.11, asyncio, port 8443)
   Silero VAD          speech start/end on 32 ms frames
   faster-whisper      streaming partials, then a final transcript
   utterance manager   merge pauses, roll never-ending speech, session memory
   routers first       clock, weather, math, unit conversion: no LLM hop
   brain               local 9B via llama.cpp, 5-tool allowlist
   Kokoro-82M TTS      GPU, streamed chunks back to the tablet
   barge-in            new speech cuts the audio instantly

Component versions as built:

GPU map after the single-card consolidation: the whole voice chain rides GPU 0. The brain takes 11.2 GB of 16 GB at 16K context and loads in 48 s; whisper small.en and Kokoro share the same card in bursts. GPU 1 sits at 0 MiB while the kiosk runs, by design, so the second card stays free for other work. The earlier layout put the brain on GPU 1 at 131K context (13.3 GB) and kept speech to text on the CPU. Dropping the brain to 16K, still several times what the kiosk needs with its rolling session memory, shrank the KV cache about 8 times and made the whole chain fit one card.

2. Prerequisites

Hardware

Software install order (Windows):

# 1. Python 3.11, then the venv, then torch from the CUDA index FIRST.
#    Do not let later packages pull a CPU torch. This bit the build twice.
py -3.11 -m venv C:\oracle\venv
C:\oracle\venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cu128

# 2. Everything else (faster-whisper, silero-vad, kokoro, numpy, pint,
#    tzdata, websockets). A plain requirements.txt works once torch exists.
pip install faster-whisper silero-vad kokoro numpy pint tzdata websockets

Note (dependency order): torch first, pinned to your GPU’s CUDA index. Kokoro and silero-vad declare torch as a dependency, and letting pip resolve it after the fact installs a CPU wheel on a GPU machine. Check with python -c "import torch; print(torch.cuda.is_available())" before going on.

Models to download

Note (brain choice): the A/B results that picked the 9B are in section 3. Any strong 8 to 14B instruct model with working tool calling will fit this pipeline; the wiring is identical.

3. The brain: serve the model first

The brain is a text model behind an OpenAI-compatible API. Get it serving before you write any voice code; the voice server only ever speaks HTTP to it. The model is served by llama.cpp on GPU 0, alias aux, 16K context (11.2 GB, loads in 48 s):

@echo off
title 9B Brain (8081)
llama-server.exe ^
  -m "E:\models\qwythos-9b\qwythos-9B-...Q8_0.gguf" ^
  --alias aux ^
  --host 0.0.0.0 --port 8081 ^
  -c 16384 ^
  -np 1 ^
  -dev CUDA0
pause

Note (the single-card layout): 16K context covers the kiosk with room to spare, and any other client sharing this brain also runs at 16K. If you ever need a long-context session, a backup launcher serves the old 131K config on GPU 1 on the same port, so stop the main brain first: the two cannot run together. The stop and restore launchers call the same bat names, so tearing the chain down and bringing it back takes no extra steps. Measured: a full STOP-RESTORE cycle brings the whole chain up in about 42 s.

Note (thinking must be off for voice): Qwen-family servers configured with reasoning enabled burn the whole token budget into thinking text, and you get empty answers or thinking-preamble answers. Voice latency cannot afford that. Disable thinking per request with template arguments:

{ "chat_template_kwargs": { "enable_thinking": false } }

Verify the running model answers without a thinking preamble before wiring anything else. Check model identity with the model list endpoint, GPU memory, and the process command line together; a launcher bat on the desktop can describe a different model than the one actually loaded. Trust the process.

Why this model won the A/B (measured, same controlled two-pass loop):

Note (model honesty): model cards will not tell you which model quietly forgets to call its tools. Run the A/B yourself with the exact tool loop you will ship.

4. The deterministic routers: facts that never touch the LLM

Date, time, weather, arithmetic, and unit conversion are answered by code, not by the model. The routers run first, and the brain only sees questions they do not cover.

Note (small models and facts): a 9B is a wonderful explainer and an unreliable fact machine. Asked to ground on real web results, it invented a Super Bowl winner that contradicted the results in front of it. Anything factual and mechanical belongs in code. Routers are free, instant, and cannot hallucinate.

5. The scoped brain: a five-tool allowlist

When the routers do not cover the question, the brain gets exactly five tools, defined as OpenAI function schemas and enforced by construction:

The loop is one brain call. If the reply contains tool calls, run them, speak a backchannel (“Let me check on that for you”) so the user is not staring at silence, then make the second call with results attached. There is no shell, no file access, and no way for a prompt to reach host-side powers on the kiosk path.

The tempting thing I did and then disabled. A full agent framework was wired in as the brain for an afternoon: sessions, skills, memory, the whole toolset. It worked, and then it was turned off. A kid-facing kiosk must never sit behind an agent with host powers. Even a friendly 9B behind a friendly persona is one clever prompt away from trouble when the toolset includes the machine itself. If you want an agent brain later, run it as a separate service with its own least-privilege identity, never embedded in the voice path. The escalation path that remains: a one-line configuration change points the brain at a much larger model on another box for questions a 9B cannot answer well.

The persona. The assistant identity lives in a plain text prompt file: warm, precise, honest, concise; answers of 2 to 4 short sentences; no markdown, lists, or symbols, because everything is read aloud; always call a tool for anything time sensitive; base answers strictly on tool results when they exist; say “I’m not sure” when that is the truth. The model in use is uncensored, so the persona and the answer policy are the only ceiling. Test the adversarial kid prompts yourself before letting children near it.

6. The voice server: ears, then mouth

Single-file asyncio server, port 8443. The tunable constants, as shipped:

Speech to text. faster-whisper small.en on CUDA, 16 kHz. Live partials stream to the tablet at a 0.3 to 0.5 s cadence while the person is still talking. Final transcripts land around 200 ms after speech ends (measured 202 ms). The earlier CPU build (base.en, int8) ran 343 to 470 ms and sometimes split one sentence into two finals; the larger model on the GPU is also steadier on kid voices.

Voice activity detection. Silero neural VAD, threshold 0.5 with a 350 ms end point. An energy-based VAD was tried first and failed in a living room; the neural VAD was roughly a tenfold improvement. The client has a mic gain slider so input level is fixed before it ever reaches the detector.

Utterance management. Three rules make the state machine survive a room that is never silent:

1. The 0.6 s grace period merges pauses mid-sentence.
2. If speech rolls past about six seconds with no pause, finalize what you have and keep a two second tail, so a question never waits forever for a silence that is not coming.
3. Session memory is six Q/A turns, rolling, per connection. A page reload or socket drop is a fresh conversation; there is no background context accumulation anywhere in the kiosk path.

Text to speech, and why it must be on the GPU. Kokoro-82M, KPipeline on CUDA, warm female voice at rate 1.15. Two numbers decided this:

The CPU floor is phonemization plus the first forward pass, per answer; warm text cannot hide it. Text to speech realtime factor on GPU measured roughly 30 to 100x (7.8 s of audio streams in about 250 ms).

Note (chunking): Kokoro will not chunk long text on its own. Without a split pattern forcing roughly two second chunks, a long answer becomes one giant segment and produces 6 to 12 s of dead air before anything streams. Force the chunks. Everything that reaches the speech engine is stripped of markdown, symbols, and links first; the same clean text feeds the caption line.

Controls on the wire. The client sends raw PCM16 over the WebSocket and the server sends audio plus JSON state back. Control messages are typed: {type:"stop"}, {type:"reset"}. STOP has three meanings depending on state: speaking means cut instantly; mid-question means finalize and answer now; thinking or idle means suppress any in-flight answer and stay disarmed for 60 s. Reset cuts speech, wipes history, disarms, and speaks an acknowledgment. A reset also works by voice: an anchored table of spoken phrases (start over, never mind, forget it, reset, and variants) matches the whole utterance, so “how do you reset a tablet” still reaches the brain.

7. The client page

The tablet page is deliberately dumb: capture mic, stream audio, play audio, render state. The pieces that matter:

• One ASK button. First tap grants mic permission and connects; the question is queued until the WebSocket is open. Without the queue, the first question of a session is silently eaten by the connection race.
• Mic monitor stays connected. The WebAudio graph only pulls audio while the monitor node is connected to the destination. Removing it kills the stream silently. This bug looks like a dead mic.
• State pills. connected, listening, thinking, checking, speaking, say more, plus a connection dot. A voice UI that cannot show what it is doing feels broken.
• STOP and reset buttons wired to the typed control messages above.
• No caching. The server sends Cache-Control: no-store. The tablet once served a stale cached client, and the symptom looked like the server dying. No-store fixed it for good.
• HTTPS or a secure-origin exception. Browsers require a secure context for the mic. On the LAN kiosk this was solved with the Chromium flag that treats an insecure origin as secure for development; production should use a real certificate from a local CA.

8. Verification battery

Run these against the live server with real audio over the real socket. Unit tests are not enough; the byte bug in section 9 was invisible to them and caught by the first end-to-end test.

9. Avoid My Mistakes

1. PCM16 over a WebSocket arrives as raw bytes. Unpack to signed integers (struct.unpack) before any audio math or speech to text. Feeding whisper raw byte values pegs every sample at 255 and returns silent empty results. The quietest, most expensive bug of the day.
2. Keep all decibel math in one unit. Normalized float dBFS everywhere. Mixing raw integer samples into float-designed thresholds means VAD never detects the end of speech in real room tone.
3. Energy VAD is useless in a noisy living room. Neural VAD or nothing, and never-silent rooms still need rolling finalization.
4. Verify with real audio through the real socket, not unit tests.
5. CPU text to speech has a per-answer floor. Warm text does not fix it. If your latency budget matters, the GPU move is not optional.
6. Unchunked text to speech creates dead air. Force sentence-sized chunks so streaming actually streams.
7. Small models cannot be trusted with facts or tool discipline. Deterministic routers for anything mechanical; strict allowlist; persona that demands tool use; measure the A/B yourself.
8. Qwen-family forced reasoning eats the budget into thinking text. Disable it per request when latency matters.
9. Thread to asyncio handoffs need the thread-safe call. A bare queue put from a worker thread is not safe. Route through the event loop and use an event for cut signals.
10. The UI state must mirror the server state. A pill that says connected while the server is armed will make you think the assistant is broken. Resend the authoritative state on every transition. A real follow-up bug is still open in this area: the second follow-up after a long answer went idle. Prime suspects are the 8 s window being eaten by a 47-word answer and the client pill resetting while the server is armed. Fix batch: resend armed state on every roll, widen the window.
11. Trust the running process, not the launcher files. Triangulate with the model list endpoint, GPU memory, and the process command line.
12. Windows path discipline. Native executables want C:/ style paths; flag-heavy commands need path conversion disabled. Small traps, repeated losses. Never kill python by image name; kill by PID.

10. Day-2 operations

@echo off
title Voice Kiosk Server (8443)
cd /d "%~dp0"
"C:\oracle\venv\Scripts\python.exe" server.py
pause

• Start: launch the bat, always a visible window. Expect the boot markers in order: stt model loaded, silero ready, ears listening, tts warmed. The brain takes about 48 s to load cold; a STOP-RESTORE cycle brings the full chain up in about 42 s.
• Stop: find the PID on port 8443 and kill it by PID. Never taskkill /IM python.exe.
• Brain health: the model endpoint answers, GPU 0 memory is near 11.2 GB at 16K context, GPU 1 stays at 0 MiB, and the model name in /v1/models matches the launcher.
• GPU 1 is free while the kiosk runs: point other work at the second card without stopping the voice chain. If a title misbehaves on it, the full-teardown stop still exists as the fallback.
• Kiosk restart: plain page reload. The server sends no-store, so there is no cache to fight.

11. Hand it to an agent

This guide is structured so an AI agent can execute it. Point an agent with shell access at this page, have it work the phases in order and report after each one: serve the brain, stand up the routers, build the audio server, wire the client, then run the verification battery from section 8. Keep yourself in the approval path for anything that touches the network or the model serving config. If privacy is the point, keep the agent’s brain on hardware you control too; a model served from your own network means the build session never leaves your LAN.

12. What I left for later

• Wake word (“Hey Cortana”) with a false-accept budget of two per day before it gets rejected. Tap-to-talk ships first.
• Kid-voice speech to text: a word-error-rate harness on two kids’ voices. Whisper already moved to the GPU as small.en; the harness decides whether a larger model earns the VRAM.
• A far-field USB mic puck; the tablet’s built-in mic is near-field and the living room is not.
• A photoreal avatar. The client ships a stylized hologram presence and a full-scene background view instead.
• The only outbound path today is the allowlisted web search tool, and it is a text call from the server, never voice audio.

Recap: serve a local 9B with thinking off, route facts around it in code, scope its tools by construction, detect voice with Silero, transcribe with faster-whisper on CPU, speak with Kokoro on GPU in two-second chunks, and let a state machine, not a model, own the conversation. One working day, zero dollars a month, and no voice audio leaves the house.