I Built My Family a Voice Assistant That Never Touches the Cloud

What this is: the implementation guide for a LAN-only family voice assistant. The client is a thin browser page: at home it runs on a wall tablet, and the same page works on any phone or laptop. The server is one Python file: a hand-rolled WebSocket endpoint that does voice activity detection, streaming speech to text, deterministic routers for facts, a scoped local brain for the rest, and text to speech through a three-layer voice chain that ends in an OmniVoice model fine-tuned on 213 clean voice lines of Reach Cortana from my own legally-owned MCC install. No cloud speech service, no cloud brain, no monthly fee.

Environment used here: Windows 11 box with two RTX 5060 Ti 16 GB cards (32 GB total), Python 3.11 venvs. The wall kiosk is a Chuwi Ubook X Pro tablet on FydOS; any device with a browser is a valid client. The current server source, the auth module, the test battery, and the browser client are downloadable with this guide (section 1 file manifest). The trained voice models are external assets built in the companion guide: How I Gave My Kiosk a Voice Clone Trained on 213 Lines of Reach Cortana.

Verified here: every command and config below ran on this setup during 2026-09-06, and the figures behind them come from it, with the voice clone arc and final two-card layout completed on 2026-09-07. Nothing is from a spec sheet. The published source copies are sanitized: the dormant full-agent bridge is removed, LAN IPs and the internal search-service and WAN hosts are replaced with placeholders, the two client pages ship without the local scene image and avatar art, and typographic em dashes are normalized to hyphens. Secret files (the password hash, the cookie secret, certs, logs) are never included in anything published. Apart from those sanitizing edits, everything else matches the running server line for line.

Fan project notice: this is an unofficial fan project. It is not affiliated with, sponsored by, or endorsed by Microsoft, 343 Industries, or the Halo franchise. No game assets or voice clips are redistributed on this site.

Follow it top to bottom, or give it to an AI agent to build from (section 13). Replication needs three things this page does not ship: the brain model file, the 66-book KJV database, and the trained voice sidecars (section 2 names all four gaps).

0. Goal and acceptance

You end up with a kiosk a child can walk up to, press one button, ask a question of, and hear answered with live captions, in a voice your family recognizes, where no voice audio ever leaves your LAN.

Acceptance criteria used for this build (all met):

Time: one working day for the full pipeline (2026-09-06), plus one day for the voice clone arc (2026-09-07, covered in its own guide). Cost: zero new hardware; all parts were already owned.

1. Architecture and source manifest

browser client on any device -> login card -> (mic, speaker, ASK, captions)
   |  WebSocket, raw PCM16 audio, both directions, cookie-gated
   v
server.py  (Python 3.11, asyncio, 8443 plain + 8444 TLS, both password-gated)
   kiosk_auth.py       PBKDF2-HMAC-SHA256 gate: POST /auth -> HMAC cookie
                       (HttpOnly, 90 d) required for pages AND WS upgrade
   Silero VAD          200 ms speech end; echo-safe, AUTO_BARGE_IN off
   faster-whisper      streaming partials, then a final transcript (CUDA)
   utterance manager   grace 0.3 s, echo utterances dropped, session memory
   routers first       clock, bible, weather, math, unit conversion: no LLM hop
   brain               Qwen3-14B Q4_K_M via llama.cpp, alias aux, 5-tool allowlist
   voice chain         per piece, automatic fallback:
                         OmniVoice Cortana clone  (sidecar :18083, GPU0)
                         -> Kokoro + RVC timbre   (sidecar :18082, GPU1)
                         -> pure Kokoro
   interaction         BUTTON-ONLY: fresh ASK tap per question; tap while
                       she speaks = interrupt (no auto follow-up window)

Everything that runs on the box is hosted on this site under one folder, https://tech.mytrapp.com/files/cortana-kiosk/, and each file name below is a direct download link (there is no separate download page):

Earlier test files for superseded behaviors (follow-up window, energy VAD, speech-length, Kokoro smoke) are not published: they test modes the current build removed. The voice-lab log and the finetune data stay on the box (companion guide, section 8).

The server is deliberately stdlib asyncio plus three model libraries. There is no web framework, no websocket library on the server side (the client tests use the websockets package, the server implements RFC 6455 by hand, section 6), and no audio framework: PCM16 in, PCM16 out.

Component versions as built:

The GPU map, and how it is pinned. Two facts in the code decide it, and both are easy to get wrong. First, server.py calls os.environ.setdefault("CUDA_VISIBLE_DEVICES", "1") before any torch or whisper import, so the whole server process (whisper, Kokoro, brain client) sees physical GPU1. Second, the OmniVoice sidecar is spawned by the server with CUDA_VISIBLE_DEVICES=0 in its environment, so the clone owns GPU0. The RVC sidecar defaults to GPU1. Net result, verified with nvidia-smi: GPU0 5.4 of 16 GB (OmniVoice), GPU1 12.3 of 16 GB (brain 11.1 plus whisper, Kokoro, and RVC about 1.2). The setdefault is deliberate: an explicit launcher env still wins, so the bat file can pin the server without editing code.

Note (stale comments lie): server.py’s own header comment still says GPU0 stays free for a game and the whole chain runs on GPU1. That comment predates the OmniVoice sidecar taking GPU0. The comments in this file do not all describe the shipped state; the environment variables and nvidia-smi do.

2. Prerequisites and install

Hardware

Software install order (Windows):

py -3.11 -m venv C:\oracle\venv
C:\oracle\venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install faster-whisper silero-vad kokoro numpy pint tzdata websockets

requirements.txt as published carries the warning in its own comments: install torch from the CUDA index FIRST, because kokoro and silero-vad declare torch as a dependency and pip resolving it later installs a CPU wheel on a GPU machine. Check with python -c "import torch; print(torch.cuda.is_available())" before going on.

The model-load block, verbatim (why the box still boots if the GPU path breaks):

_MODEL_LOAD_START = time.time()
os.environ.setdefault("CUDA_VISIBLE_DEVICES", "1")

try:
    import numpy as np
    from faster_whisper import WhisperModel
    try:
        MODEL = WhisperModel("small.en", device="cuda", compute_type="int8_float16")
        _STT_DEV = "cuda"
    except Exception as e:
        print(f"[stt] CUDA init failed ({e}) - falling back to CPU small.en", flush=True)
        MODEL = WhisperModel("small.en", device="cpu", compute_type="int8", cpu_threads=8)
        _STT_DEV = "cpu"
    print(f"[stt] small.en ({_STT_DEV}) loaded in {time.time()-_MODEL_LOAD_START:.1f}s", flush=True)
except Exception as e:
    MODEL = None
    print(f"[stt] MODEL UNAVAILABLE: {e}", flush=True)

try:
    from silero_vad import load_silero_vad, VADIterator
    _SILERO = load_silero_vad()
    print("[vad] silero ready", flush=True)
except Exception as e:
    _SILERO = None
    print(f"[vad] silero UNAVAILABLE: {e}", flush=True)

Two deliberate design points: speech to text is small.en in int8_float16 on CUDA, and the whole import block is wrapped so a model failure degrades the server instead of killing it. MODEL None means transcription returns empty and the server still runs; Silero None means a crude energy fallback takes over VAD (section 6). A kiosk that answers wrong is better than a kiosk that is down, and the log line tells you which degradation you are in.

Models to have on disk before first boot:

The whisper and silero models download on first use. Kokoro downloads its voice pack on first use. The brain is served by a separate llama-server process (alias aux, port 8081); the launcher bat on the box carries the --alias aux flag as a standing rule, and the model path is the only thing you substitute. The RVC and OmniVoice sidecar models come from the companion voice-clone guide; the server health-checks them at boot and runs without them (section 7).

Not shipped with this guide. Four things the build needs that this page names but does not hand you, so you know where the work is:

• The brain server binary. The kiosk talks to llama-server over its OpenAI-compatible endpoint. Get llama.cpp from github.com/ggml-org/llama.cpp and build it for your CUDA version (docs/build.md in that repo). The launcher starts it with --port 8081 --alias aux --ctx-size 16384 --cache-type-k q8_0, and the model path is the only thing you substitute.
• The brain launcher bat. The restore bat is published in the manifest, but the Desktop launcher it calls is a local file with local paths. Section 11 lists what it has to do; it is about twenty lines around one llama-server call.
• The KJV database builder. Section 4 shows the schema (FTS5, one row per verse, 66 books, 24,995 verses) and the exact query the router runs. The loader that reads a KJV text file into it is on the box.
• The search backend. web_search points at an internal search service that this site replaces with a placeholder host. Any endpoint that takes a query and returns JSON titles and snippets will do; keep the truncation on the way back.

And two setup steps the published code cannot do for you: minting the self-signed EC pair under certs/ (any local-CA tool; the listener skips itself when the files are missing), and registering the boot task. The box creates it as schtasks /Create /TN Cortana_Kiosk /TR "<your restore bat>" /SC ONSTART /RU SYSTEM. The exact task name matters, because the STOP bat escalates to schtasks /End /TN Cortana_Kiosk when a plain kill fails on a SYSTEM-owned process.

3. The brain: one HTTP call, scoped

The voice server never talks to a model library. It posts to the llama.cpp OpenAI-compatible endpoint on 127.0.0.1:8081, exactly like a script would:

BRAIN_URL = "http://127.0.0.1:8081/v1/chat/completions"
BRAIN_MODEL = "aux"

def llm_respond(messages, tools=None, timeout=BRAIN_TIMEOUT):
    payload = {"model": BRAIN_MODEL, "messages": messages,
               "max_tokens": 300, "temperature": 0.5, "stream": False,
               "chat_template_kwargs": {"enable_thinking": False}}
    if tools:
        payload["tools"] = tools
    req = urllib.request.Request(BRAIN_URL, data=json.dumps(payload).encode(),
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        d = json.loads(r.read().decode("utf-8", "replace"))
    msg = (d.get("choices") or [{}])[0].get("message", {})
    if isinstance(msg.get("content"), str):
        msg["content"] = msg["content"].strip()
    return msg

Three things in that payload matter for voice:

1. enable_thinking: False. The Qwen family, served with reasoning on, burns the token budget into thinking text and the answer arrives late or as a preamble. Voice latency cannot afford it, so thinking is disabled per request.
2. max_tokens: 300. The persona caps answers at 2 to 4 short sentences, so a real answer is far under this. The cap is the guard against a runaway ramble, and it bounds worst-case time to first audio.
3. stream: False. Streaming would add bookkeeping for no gain at these sizes; the TTS pipeline streams instead (section 6).

Why a scoped brain, not an agent. A full agent framework was wired in for an afternoon: sessions, skills, memory, a whole toolset. It worked, and it was turned off. A kid-facing kiosk must never sit behind an agent with host powers. Even a friendly model behind a friendly persona is one clever prompt away from trouble when the toolset includes the machine itself. What ships is a text brain with exactly five tools (section 5), no shell, no file access, and no way for a prompt to reach host-side powers on the kiosk path. The dormant agent bridge is removed from the published source. If you want an agent brain later, run it as a separate service with its own least-privilege identity, never embedded in the voice path.

Why a 14B and how it was picked. Earlier A/Bs on this box ran candidates through the exact two-pass tool loop the kiosk ships, with real audio, and kept score on tool-call discipline, not speed: a 9B Q8_0 handled 4 of 5 calls at ~75 t/s standalone (five prompts per model, one run each, on this box), a Nemotron Q4 promised a tool call and skipped it (3 of 5), and a 35B-A3B Q6 did not fit the two cards. When the voice arc gave the brain its own card, the kiosk moved to Qwen3-14B Q4_K_M, the largest model that fits GPU1 beside STT and the fallback voice layers, at 16K context with q8_0 KV cache (~11.1 GB). Model cards will not tell you which model quietly forgets to call its tools; run the A/B with the exact loop you will ship.

4. Deterministic routers: facts never reach the LLM

Anything a regex can own outright is answered in code: correct and instant. The chain, in order, from deterministic_answer():

def deterministic_answer(question):
    try:
        a = clock_answer(question)
        if a:
            return a
        a = _router_bible(question)      # exact KJV quote, never paraphrased
        if a:
            return a
        a = _router_weather(question)    # first: "is it hot in Phoenix?" style
        if a:
            return a
        a = _router_math(question)
        if a:
            return a
        a = _router_convert(question)
        if a:
            return a
    except Exception as e:
        print(f"[router] ERROR {type(e).__name__}: {e} - falling through to brain", flush=True)
    return None

Every router returns a speech-ready string or None. None falls through to the brain, so a router that is not sure never blocks the answer. The whole chain is wrapped: a crash in one router logs and falls through instead of killing the question.

Clock. Regexes match date and time phrasings (“what time is it”, “what’s today”, “current date”, and variants). One guard matters: if the question contains “in [Capitalized]” but not “in the”, the router returns None so “what time is it in London” reaches the brain’s timezone tool instead of answering local time wrongly.

Math. One regex owns the pattern, an AST walk evaluates it, and there is no eval() anywhere:

MATH_Q_RE = re.compile(
    r"\b(?:what(?:'s| is)?|how much is)\s+(-?\d+(?:\.\d+)?)\s*"
    r"(\+|x|×|\*|\/|-|times|multiplied by|plus|minus|divided by|over)\s*"
    r"(-?\d+(?:\.\d+)?)\s*\??$", re.I)

The evaluator walks ast.parse(expr, mode="eval") and allows only binary and unary arithmetic nodes, so the expression can never call anything. Division by zero returns None. Results format as ints when whole (is 42.) and rounded to 4 places otherwise.

Unit conversion. A spoken-unit table maps “km”, “mi”, “lb”, “degrees”, and the rest into pint quantities, plus a reverse pattern for “how many X are in Y”:

CONVERT_UNITS = {
    "fahrenheit": "degF", "°f": "degF", "celsius": "degC", "°c": "degC",
    "km": "kilometer", "kilometers": "kilometer", "mi": "mile", "miles": "mile",
    "lb": "pound", "lbs": "pound", "kg": "kilogram", "kilos": "kilogram",
    ...
}

Temperature is special-cased before pint because pint chokes on offset units: 98.6 degF to degC is converted by hand through a Celsius pivot. _spoken_unit() turns “degC” back into “degrees Celsius” so the answer reads naturally aloud.

Weather. Two open APIs, no key: the Open-Meteo geocoder turns a place name into coordinates, then the forecast endpoint returns current conditions plus today’s high and low. A WMO code map turns numeric weather codes into spoken words (“overcast”, “light drizzle”, “thunderstorm with hail”). The router handles two speech traps: “San Diego CA” becomes “San Diego, CA” (the geocoder wants the comma form, and if the first call looks empty the router retries once with the state split out), and trailing time words (“right now”, “today”, “tonight”, “this weekend”) are stripped before geocoding so “what’s the weather in Oceanside today” asks about Oceanside, not a place literally named “Oceanside today”.

Bible, the exact-text router. This router exists because of a question from my kids. They asked the kiosk “On which day did God make the sun?” and the 14B brain answered that it did not have that information. That is a coverage gap no prompt fixes, so the whole King James text went onto the box: the complete Bible in markdown, indexed into a read-only SQLite database (FTS5, 66 books, 24,995 verses). Any verse is now a lookup instead of a memory test, and the quoted text stays in the session history, so a follow-up like “what does that mean?” reaches the brain with the verse already in front of it. An exact quote is also a fact no small model can be trusted with:

_BIBLE_CANON = """Genesis Exodus Leviticus Numbers Deuteronomy ... Revelation""".split()
assert len(_BIBLE_CANON) == 66, f"bible book list must have 66 entries, has {len(_BIBLE_CANON)}"

The book list has to be exact, and the assert is the guard: one earlier version had 65 names, Gospel John was missing, and John 3:16 died silently. Database labels are underscore canonical (1_Corinthians, Song_of_Solomon); the router maps any spoken or written form to the label, including singular “Psalm” to “Psalms”. Spoken numbers resolve through word tables: “psalm twenty three verse one” becomes chapter 23, verse 1, with full teens and tens-plus-unit composition (chapter and verse counts never exceed 176, so the table covers everything). The lookup is one read-only parameterized query:

select text from kjv where book=? and chapter=? and verse=?

A miss at any step returns None, so an unparseable reference goes to the brain or web search rather than a paraphrased scripture. A hit is spoken in the form “Book chapter c, verse v says, [exact text].” Before every server restart, run import server plus the 31-case bible battery: one earlier edit left an unbalanced parenthesis in the speech-reference regex and the server could not import at all. The running process predated the edit and masked it, so a restart then would have killed the kiosk outright.

5. The persona and the five tools

The identity is a plain string in server.py. The operative rules, verbatim from the source:

You are Cortana, a wise and friendly AI presence living in the Trapp family home.
Kids of all ages ask you science and history questions. Answer concisely in 2-4 short
sentences that sound natural when spoken aloud. Be warm and precise. Never use markdown,
lists or symbols.
Never begin an answer with "I don't know", "I don't have that information", or "I can't
help with that" ... you have live tools for web search, weather, math, unit conversion,
and the current date and time. ALWAYS search or call a tool first for anything uncertain,
time-sensitive or current ... you do not know today's date and must use get_datetime.
After a successful search you may say "I checked". Only if a tool actually fails or
returns nothing useful may you say you could not find it.
When tool results are present in the conversation, base your answer STRICTLY on those
results - never answer from memory about facts the tools just provided.

(The published source keeps the full text; the ellipses above skip one parenthetical clause.) The sharp edge is the second paragraph: the persona is forbidden from answering “I don’t know”. It must search or call a tool FIRST, may say “I checked” after a success, and only admits failure after a real failed search. The tools and the persona are the only ceiling, which is why the model choice in section 3 matters: an uncensored model with a strict persona needs the persona to be the enforcement point.

The tool allowlist, exactly as sent to the brain:

TOOLS = [
    {"type": "function", "function": {"name": "web_search", "description":
        "Search the web for current information: news, weather reports, recent events, facts, prices, sports. "
        "Use whenever the user asks anything time-sensitive or outside your own knowledge.",
        "parameters": {"type": "object", "properties": {"query": {"type": "string",
        "description": "Concise search query, plain words"}}, "required": ["query"]}}},
    {"type": "function", "function": {"name": "get_weather", "description":
        "Get current conditions and today's forecast for a named place (no location needed from user).",
        "parameters": {"type": "object", "properties": {"place": {"type": "string",
        "description": "City or town, optionally with state/region (e.g. 'Oceanside, California')"}},
        "required": ["place"]}}},
    {"type": "function", "function": {"name": "calculate", "description":
        "Evaluate a math expression exactly (numbers with + - * / ** and parentheses).",
        "parameters": {"type": "object", "properties": {"expr": {"type": "string",
        "description": "Expression with + - * / ** ( ) and numbers"}}, "required": ["expr"]}}},
    {"type": "function", "function": {"name": "convert", "description":
        "Convert between units of measurement. Examples: '42 km to mi', '5 lb to kg', "
        "'98.6 degF to degC', '2 hours to seconds', '1 gallon to liters'.",
        "parameters": {"type": "object", "properties": {"text": {"type": "string",
        "description": "The conversion, e.g. '42 km to mi' or '98.6 degF to degC'"}},
        "required": ["text"]}}},
    {"type": "function", "function": {"name": "get_datetime", "description":
        "Get the current date, time, weekday and time zone. ALWAYS use for questions about "
        "today's date, the time, or what day it is. Optional zone parameter for other cities "
        "(e.g. 'Europe/London'); omit for local time.",
        "parameters": {"type": "object", "properties": {"zone": {"type": "string",
        "description": "Optional IANA time zone, e.g. America/New_York or Europe/London"}},
        "required": []}}},
]

Five tools by construction. The answer loop runs at most two passes: one brain call, and if it returns tool calls, they execute and a second call goes out with results attached. While the tools run, the server speaks a canned backchannel (“Let me check on that for you.”) so the user is not staring at silence; the canned line is synthesized once at first use and cached. Implementation notes that keep the tools honest:

• web_search returns the top five results as “title. snippet (source: url)” lines, truncated to 2,200 characters, so a model cannot drown in raw HTML. It goes to the local search service; a prompt can never reach it with anything but a query string.
• tool_weather is shared by the deterministic weather router (section 4) and the brain’s tool path, so both answer from the same data.
• run_tool wraps every implementation and returns a plain string on any exception: Tool X failed: TypeName: message. The model sees a string, never a traceback.

6. The voice pipeline, in code

The tunables:

SR = 16000
PARTIAL_GAP = 0.7     # s between live partial transcriptions
PARTIAL_WIN = 4.0     # s of trailing audio shown as partial
MIN_PARTIAL = 0.45    # s of speech before first partial
GRACE = 0.3           # s pause allowed mid-utterance without splitting (0.6 -> 0.3 for snappier answers)
POST_TTS_LOCKOUT_S = 0.8  # ignore speech starting this soon after her stream ends (speaker echo tail)
AUTO_BARGE_IN = False     # VAD never cuts her mid-answer; tap/STOP are the interrupts
HISTORY_TURNS = 24    # prior Q/A turns kept per WS session
FOLLOWUP_WINDOW = 0.0 # BUTTON-ONLY: she disarms after every answer; a fresh ASK tap starts each question
IDLE_RESET_S = 30 * 60  # quiet auto-reset: wipe session memory after this long with no question
RESET_RE = re.compile(r"^((start over|let'?s start over|new conversation|never ?mind|forget (it|that|everything)|(please )?reset( the conversation| this| it| now| please)?) ?(please)?)$", re.I)

HISTORY_TURNS is 24, not 6, and that was probe-validated: a cold prefill with 24 turns costs about 380 ms more than 6, and appended turns hit KV prefix reuse (~0.95 s) once the context is warm. A family kiosk with a rolling conversation notices better pronoun grounding, not lag. One caveat on the timing figures in the config above: GRACE, the silence floor, and the TTS lead were all retuned across the same two days, so a delta credited to a single knob (0.45 s from the silence window, for instance) is directional rather than isolated. The history list stores Q/A pairs and is sliced to HISTORY_TURNS * 2 entries after every exchange. Each WebSocket connection owns its Session, so a page reload is a fresh conversation, and IDLE_RESET_S wipes the history after 30 quiet minutes: nothing accumulates anywhere.

The WebSocket protocol, hand-rolled. The server implements RFC 6455 directly: SHA-1 accept handshake, masked client frames, ping/pong, close, and a 64 MB frame cap. Client to server messages:

Server to client: state (idle/speech/processing/thinking/tool/error), partial and final (transcripts with a revision counter and latency in ms), answer_state (asked/thinking/tool/done/error), ask_state (armed, with fu: true during the follow-up window), tts_start, tts_end, tts_stop, and binary frames of PCM16 at 24 kHz for playback. Every HTTP response carries Cache-Control: no-store, because the tablet once served a stale cached client and the symptom looked like the server dying.

Note (unpack before you do math): raw PCM16 arrives over the WebSocket as a byte buffer. Convert it to signed integers with struct.unpack before any level or VAD math. Summing or thresholding the raw bytes compiles, runs, and quietly moves every threshold in the audio path, which is exactly the bug in section 10 that the unit tests could not see: their fixtures were already float arrays.

Note (Windows path discipline): native executables want C:/ style paths, and flag-heavy command lines need path conversion disabled. Kill python by PID, never by image name. On a box running three python processes, taskkill /IM python.exe takes down the one you did not mean to stop.

Resampling and VAD. Audio arrives at whatever rate the client declares (48 kHz from browsers). A streaming linear resampler drops it to 16 kHz in 512-sample windows with a fractional position counter, so no frame boundary artifacts and no per-utterance buffering. Each window is scored two ways: an RMS noise floor in dBFS that adapts only while not in speech (clamped to [-60, -25]), and Silero VAD with threshold 0.5 and a 200 ms minimum silence (shortened from 350 ms so an answer finalizes about 0.45 s sooner). The noise floor is a gate for transcription, not the speech decision; Silero owns speech. If Silero is unavailable, the crude energy fallback takes over (floor plus 10 dB above it is speech). This is the “one unit everywhere” rule in practice: all level math is normalized float dBFS against 32768, so the noise floor, the gate, and the energy fallback all speak the same scale.

Partials, rolling finals, and the grace merge. While speech is in progress, the server transcribes the trailing PARTIAL_WIN window at most once per PARTIAL_GAP, so captions flash at a 0.7 s cadence. Two guards keep noise from becoming captions: a too-quiet window (floor plus 6 dB) is skipped, and a partial only ships when the state is not idle. When speech has rolled past about 6 seconds with no pause, the server emits a ROLLING-FINAL of everything so far and keeps the last 2 seconds as a tail, so a never-silent room never waits forever for a silence that is not coming. On speech end, a grace task sleeps GRACE seconds; if speech resumes it merges, otherwise it finalizes. finalize() waits for any in-flight transcription, discards noise-only utterances, transcribes once, and hands the text to the answer gate.

The answer gate. maybe_answer() is the whole conversation policy in three lines: a six-word-or-less utterance matching RESET_RE resets the conversation by voice; otherwise the utterance must arrive while armed and be at least two words. An ASK tap arms; in button-only mode she disarms after every spoken answer, so each question is a fresh tap; STOP disarms for 60 seconds. There is no voice-activated follow-up window anymore: a long answer used to re-arm the window at send-complete, while the client speaker was still seconds behind the stream, and the window caught her own trailing audio as a new question. Removing the window removed the self-answer (kiosk changelog, 09-07 night).

Echo safety and interrupt. Automatic barge-in is OFF (AUTO_BARGE_IN = False). The history behind it is the best debugging story in this project: her TTS plays through the speakers, a device mic with weak or absent echo cancellation picks it up, and Silero VAD fired a speech start at her own sentence pauses (quiet past the min-silence threshold reads as a fresh start), which cut her off mid-sentence and then transcribed and answered her own echo. The fix has three parts: VAD never cuts her mid-answer; speech that starts while she is talking, or within 0.8 s (POST_TTS_LOCKOUT_S) after her stream ends, is marked drop_utt and discarded before transcription, with a log line naming the dropped echo utterance; and the only interrupts left are intentional ones. Tap ASK or STOP while she is speaking and she cuts instantly, and the tap clears the echo lockout so your next words count as a real question. STOP is context-aware: if she is speaking it cuts TTS; if you are mid-question it finalizes the buffered utterance NOW (answer-now, no grace wait); if she is thinking or idle it suppresses any in-flight answer’s speech and disarms for 60 seconds. Suppression, not cancellation: the brain may still be working, and its answer is computed, logged, and silenced, so a later STOP cannot resurrect it.

TTS streaming. speak() runs the synthesis worker in a thread and shuttles completed segments through an asyncio queue using the thread-safe handoff (loop.call_soon_threadsafe), because a bare put from a worker thread is not safe. It holds at most 4 segments to the socket, sending each piece in TTS_FRAME-sized chunks as soon as it is ready. First audio lands in a few hundred ms because the first segment ships the instant it exists, not when the whole answer is done. The text is stripped for speech first by clean_for_speech(): markdown links become their text, URLs and bullet symbols are removed, and emoji and symbol ranges are deleted (Kokoro cannot speak them, and cp1252 console logs cannot print them). The same function rewrites scripture references into spoken form: “John 3:16” becomes “John chapter 3, verse 16”. The whole displayed answer is spoken, capped at 1,500 characters as a runaway guard, because the persona already caps answers at 2 to 4 sentences.

7. The voice chain: sidecars and fallback

The clone and the timbre converter do not live in server.py. They are separate processes the server spawns and calls over HTTP, so a crash in a voice model can never take the kiosk down, and each engine gets its own GPU. The chain, per sentence piece, in order:

The sidecar HTTP protocol is small and raw. Health: GET /health returns JSON with a ready boolean. Conversion: POST /convert with raw int16 PCM in the body returns converted int16 PCM (RVC). Synthesis: POST /synth with {"text": ..., "speed": 1.35} returns int16 PCM at 24 kHz (OmniVoice). No JSON wrapping of audio anywhere; the body is the samples.

The server spawns a sidecar if health fails, polls up to 75 s (RVC, cold load about 45 s on this box) or 90 s (OmniVoice, about 25 s here), and logs the exact boot markers:

[rvc] sidecar ready (trained Reach voice online)
[omni] sidecar ready (OmniVoice Cortana live)

A mid-answer failure trips a circuit breaker with a cooldown: RVC falls back to pure Kokoro for 120 s, OmniVoice for 180 s, then re-probes. So a flaky sidecar degrades the voice instead of repeating a failing call 40 times a minute.

The piece-splitting rule that fixed cut-out speech. OmniVoice generation costs about one second fixed per call regardless of piece size. Comma-level fragments (~1 s of audio each) could not keep the gapless player fed, and the symptom was intermittent speech cutting out. The fix is in _omni_pieces(): split on sentence boundaries into pieces of 1.5 to 7.9 s of audio, with a 200-character ceiling that falls back to comma granularity for pathological run-ons, so a never-ending sentence still streams. Kokoro’s own split pattern stays comma-granular because its per-call cost is tiny and fine chunks stream earliest. The two engines want opposite chunking, and both are correct for their cost model.

Boot order. warm_tts() at startup synthesizes a 3-sentence warmup text through Kokoro (tuning oneDNN inference shapes so the first real answer does not pay the shape-benchmark tax), then boots RVC, then OmniVoice. The kiosk is answerable on pure Kokoro while the clone loads, because warm_tts runs as a background task after the listeners are up: [srv] ears listening prints before any voice model is warm.

8. The client page

Two clients, one codebase of shared logic: the default path serves the full-scene client, and /oracle serves the earlier procedural hologram client as a fallback. Both are deliberately dumb: capture mic, stream audio, play audio, render state. The client is a web page, not an app: no install, no store, same URL on every device. The published copies of both pages ship without the local scene image and avatar art; what is published is the client code itself. The pieces that matter, each one bought with a real bug:

• One ASK button, pressed per question. First tap grants mic permission and connects; the question is queued until the WebSocket is open (without the queue, the first question of a session is silently eaten by the connection race). In button-only mode every question is a fresh tap, and tapping while she is speaking is the intentional interrupt: it cuts her and clears the echo lockout.
• Mic monitor stays connected. The WebAudio graph only pulls audio while the monitor node is connected to the destination. Removing it kills the stream silently. This bug looks like a dead mic.
• State pills. connected, listening, thinking, checking, speaking, say more, plus a connection dot. A voice UI that cannot show what it is doing feels broken.
• STOP, reset, and the mic gain slider. STOP and the reset button are wired to the typed control messages in section 6; the slider fixes input level before it ever reaches the VAD.
• Phone support. iOS detection plus an audio context unlock on the first tap. Any phone reaches the kiosk over the TLS mirror on 8444, accepts the self-signed cert once, and works. The tablet still runs on the plain HTTP listener; migrating it to the TLS URL removes the need for the browser insecure-origin flag it used in early builds.
• No caching. The server sends Cache-Control: no-store. A client browser once served a stale cached copy, and the symptom looked like the server dying.

9. Verification battery

The tests talk to the live server over the real socket with real audio; unit tests are not enough, and the byte bug in section 10 was invisible to them. All files are in the manifest; all results are from the box on 2026-09-06/07:

The e2e test is worth reading before you trust any change: it opens the socket, sends hello with the sample rate, arms the gate, streams a WAV in 2,048-sample chunks at 1.6x real time (the tablet streams faster than real time so speech never starves), pads 18 seconds of silence, and waits for tts_end. The WAV is a 16 kHz mono file; the published copy expects cache/jfk.wav, so substitute any 16 kHz speech file (the script boosts quiet samples 16x before sending, which is itself a lesson: the classic test clip is quiet).

10. Avoid My Mistakes

1. PCM16 over a WebSocket arrives as raw bytes. Unpack to signed integers (struct.unpack) before any audio math. Feeding whisper raw byte values pegs every sample at 255 and returns silent empty results. The quietest, most expensive bug of the day.
2. Keep all level math in one unit. Normalized float dBFS everywhere, referenced to 32768. Mixing raw integer samples into float-designed thresholds means VAD never detects the end of speech in real room tone.
3. Energy VAD is useless in a noisy living room. Neural VAD or nothing, and never-silent rooms still need rolling finalization.
4. Verify with real audio through the real socket, not unit tests.
5. Comma-level pieces starve a gapless voice player. A fixed per-call generation cost means fat pieces, not thin ones. Unchunked Kokoro has the opposite failure: force chunks so streaming actually streams.
6. A bad edit can kill the next restart before it happens. The server ran fine on a regex edit with an unbalanced parenthesis, because the running process predated the edit. Run import server plus the test battery after every edit, even when the live kiosk looks healthy.
7. An emoji can silence an answer. A brain reply with an emoji crashed a print() on the cp1252 system console before the text reached the speaker: caption text was there, audio was not. Fix: reconfigure stdout/stderr with errors="replace" at import and strip symbol ranges in clean_for_speech.
8. The bible book list must have 66 entries. Gospel John went missing once and John 3:16 died silently. The assert on _BIBLE_CANON is not decoration.
9. Small models cannot be trusted with facts or tool discipline. Deterministic routers for anything mechanical, the bible text included; strict allowlist; persona that demands tool use; measure the A/B yourself with the exact loop you ship.
10. Qwen-family forced reasoning eats the budget into thinking text. Disable it per request when latency matters.
11. Thread to asyncio handoffs need the thread-safe call. A bare queue put from a worker thread is not safe. Route through the event loop.
12. CUDA_VISIBLE_DEVICES must be set before torch imports, and the server’s setdefault pins GPU1 while the OmniVoice spawn overrides to GPU0. The order of those two lines is the whole GPU map.
13. Stale comments and stale diagrams lie. The header comment in the shipped server still describes the pre-OmniVoice layout, and the project status doc’s diagram still says base.en on CPU while the code loads small.en on CUDA. Trust nvidia-smi, the env vars, and the code, not the prose.
14. The follow-up window self-answered her own trailing audio. The client play queue lags seconds behind the server stream: speak() returns when bytes are SENT, not when the speaker finishes. The old 8 s window armed at send-complete, so her own still-playing trailing sentences were caught as questions and answered, worse over the WAN. The fix was my call: button-only mode, window removed (FOLLOWUP_WINDOW = 0.0). If you ever re-enable the window, resend the armed state on every roll and add gate-fail logging; the UI state must mirror the server state or the pill lies.
15. Trust the running process, not the launcher files. Triangulate with the model list endpoint, GPU memory, and the process command line. A launcher bat can describe a different model than the one actually loaded.
16. Windows path discipline. Native executables want C:/ style paths; flag-heavy commands need path conversion disabled. Never kill python by image name; kill by PID.

11. Day-2 operations

• Bring it back: cortana-restore-14b.bat (published in the manifest). The order is enforced by the file: it starts the brain launcher (the bat it calls lives on the Desktop) and polls http://127.0.0.1:8081/v1/models until HTTP 200, up to about 4 minutes for a cold model load. Only then does it start the kiosk server and poll :8443 until HTTP 200, up to about 2 minutes. It finishes with a status block that probes the TLS mirror too. Expect the boot markers in the kiosk window in order: stt loaded, silero ready, ears listening, tts warmed, then the RVC and OmniVoice sidecar ready lines. Both bat files honor NOPAUSE=1 for unattended runs. One naming trap: the Desktop launcher the RESTORE file calls, and the banner the file prints, still say qwythos 9B. That name is legacy, kept for muscle memory; the brain it starts is the Qwen3-14B from section 3. Verify with the model list endpoint, never with file names (section 10, stale names lie).
• Verify alive: netstat -ano | findstr ":8443 :8444 :8081 :18082 :18083" all LISTEN, then run python e2e_test.py. The kiosk log line that matters is the OmniVoice ready line; without it, answers silently fall to Kokoro.
• Tear it all down: Cortana-STOP.bat (published in the manifest). It kills the brain on :8081 and the kiosk on :8443 by PID from netstat, then polls up to 40 s until both ports release. The kiosk has run as the SYSTEM scheduled task Cortana_Kiosk since 09-06, and a plain taskkill on a SYSTEM-owned process fails silently, so the bat detects the port still listening and escalates to schtasks /End /TN Cortana_Kiosk with a UAC prompt. That is the file you run before gaming: it frees the VRAM both services hold. The voice sidecars are children of the server, which hands them its own PID at spawn, so they exit with it. Never taskkill /IM python.exe.
• Brain health: the model endpoint answers, /v1/models shows aux, and GPU memory matches the map: GPU0 near 5.4 GB, GPU1 near 12.3 GB.
• Auth ops: every listener is gated (LAN and WAN alike). The password hash lives in kiosk-pass.json (PBKDF2-HMAC-SHA256, 240k iterations) and the cookie secret in kiosk-secret.bin, both created on first use via kiosk_auth.setup_password(); neither is ever published or snapshotted. Change the password by calling the same function locally. The rate limiter is per-IP and in-memory: five bad passwords escalate a block from 30 s up to 10 min, and a server restart resets it. Sessions are 90-day HMAC cookies, so a password change does not log anyone out until the secret is regenerated. Local automation (the e2e test) reads a plaintext password file that exists only on the box; the published test documents how to point it at your own.
• Sidecar logs: rvc_sidecar.log and omni_sidecar.log next to the sidecar scripts in the kiosk’s sidecar cache directory.
• Phones: https://<lan-ip>:8444, accept the self-signed cert once. The tablet keeps the plain listener until you migrate its bookmark to the TLS URL.
• Kiosk restart: plain page reload. The server sends no-store, so there is no cache to fight.
• The TLS cert is a self-signed EC pair under certs/; regenerate it with your normal local-CA tooling when it expires. The listener skips itself gracefully if the cert files are missing.

12. Threat model

This kiosk is a microphone and a speaker on your LAN with an HTTP server in front of them: it takes audio and text from any client that can reach the port, spawns child processes, and holds voice models on two cards. That is worth four honest paragraphs before you copy it.

What the guardrails stop. Every listener is password-gated, LAN and WAN alike: a login card, a 90-day HMAC cookie, and a per-IP limiter that walks from a 30 s block up to 10 minutes after five bad passwords. The brain has five tools and no more. Tool inputs are parsed rather than evaluated, so there is no eval on model output and the math tool walks an AST. The search tool truncates what it returns. Anything a regex can own outright is answered in code, so facts never reach the model at all, which also means the model cannot be argued into inventing them. Voice audio never leaves the house; the only outbound path is that one search call.

What they cannot stop. All four, plainly. The brain reads search results from the open web, so a page can put sentences in front of a model that answers your family’s questions; a regex gate on tool output loses to obfuscation eventually, and the durable answer is a brain on hardware you control with a short tool list. The kiosk’s plaintext automation password sits on the box by design, so anyone with a shell there already has the front door. Revoking a session means regenerating the cookie secret, because the cookies are self-contained HMAC and there is no session table to delete from. And if you publish the listener through a proxy, a password and a rate limiter are the whole distance between the internet and a microphone in your living room.

Least privilege, which this build does not do yet. The kiosk runs as the SYSTEM scheduled task Cortana_Kiosk, and that is more privilege than it needs. A dedicated non-admin account owning only the kiosk directory, the venv, and the two GPUs is the right shape. The tradeoff is that the boot task then has to store that account’s credentials. If you keep SYSTEM, know that you are doing it for convenience.

Minimum hardening. Keep the plaintext automation password and kiosk-pass.json out of every snapshot and every backup you put on a share. Point the phone-facing TLS mirror at a certificate your devices trust instead of teaching people to click through a warning. Leave the WAN door closed unless you actually need it, and if you need it, put a real reverse proxy with TLS in front rather than exposing the listener. Never give the voice sidecars a command interface, not even a debug one.

13. Hand it to an agent

This guide is structured so an AI agent can execute it. Have the agent download the published sources from the file manifest above: the current server, the auth module, the authenticated e2e, requirements, the launcher, both bats, and the clients. Build the venv per section 2, then work in order: serve the brain (alias aux on 8081), stand up the routers, run the audio server, and execute the verification battery from section 9 against the live socket. Three assets are external and must exist before the full chain passes: the brain model file (your path, your quant), the bible database (the 66-book KJV index, built by the tool that lives on the box), and the trained voice sidecars (the companion guide’s output). Plus one setup step the published code cannot do for you: create the password gate’s hash and secret with kiosk_auth.setup_password() and point the published e2e at your own password file, because the box’s plaintext copy is never published. Without those assets the server boots and answers in fallback voices, which is itself a useful first milestone: pure Kokoro proves the whole pipeline before the clone is wired in.

Keep yourself in the approval path for anything that touches the network, the model serving config, or the voice sidecars. If privacy is the point, keep the agent’s brain on hardware you control too; a model served from your own network means the build session never leaves your LAN.

14. What I left for later

• Wake word (“Hey Cortana”) with a false-accept budget of two per day before it gets rejected. Tap-to-talk ships first.
• Kid-voice speech to text: a word-error-rate harness on two kids' voices. small.en moved to CUDA in int8_float16; the harness decides whether a larger model earns the VRAM.
• A far-field USB mic puck; the tablet’s built-in mic is near-field and the living room is not.
• A shared family context across devices. Today each device owns its session history over its own WebSocket; shared context needs a deliberate design.
• The only outbound path today is the allowlisted web search tool, and it is a text call from the server, never voice audio.