Memory-informed first words, then an informed transition
Robby engineering study · 27 October 2026 · two shared RTX 3090 GPUs · authored synthetic dialogue only.
Live demo (on the demo host) · Architecture, source and reproduction
What the user hears
Memory-aware local reaction → Qwen-informed transition → full Qwen answer
The first local call sees the current request, recent conversation, approximate speaker identity and compact excerpts of the already retrieved wiki entries. It does not see Qwen's answer yet. It can say a tiny grounded answer such as “Deine Freundin heißt Alice” instead of a thinking/waiting announcement. With no matching evidence it must not invent recognition. Current-session self-introductions work even with persistent memory off.
The second local call sees the actual first one or two Qwen sentences, plus the reaction already chosen. It writes a short contextual transition. For jokes only the setup is supplied, so the transition does not receive the punchline. Exact-output tasks and poems bypass both introductions. Invalid, repeated or inappropriate prefaces are skipped rather than replaced by canned filler.
Why parallel text generation alone did not remove the audio gap
Previously the single shared TTS server finished the first utterance before admitting another. Having Qwen's text ready did not make its audio ready. The new internal WebSocket admits independent utterances to one CUDA owner and one set of weights. Global semantic and local acoustic-token decoding share real tensor batches; each utterance retains separate KV and codec state. Codec decoding itself restores one stream's state at a time. Main-answer audio is prefetched immediately, while the second local model call runs.
Two active streams are selected. Three can improve aggregate work while slowing the stream the listener currently needs. That produced early underruns in the retained three-stream runs. The foreground emits a 160-ms first audio block, then 400-ms blocks; the browser starts with a 220-ms reserve. Main utterances retain final forced alignment and fades. Short prefaces use 10-ms fades without alignment holdback. All conversational trials below use TTS temperature 0.8; the isolated deterministic cache test used zero.
Actual HTTPS browser timing
Times start from typed submission. Playback is the browser's scheduled AudioContext start, not a measured physical sound event. ASR is not part of this test.
Startup is different: the retained first post-restart trial began scheduled playback after 2,473 ms. Its TTS first-PCM service time was about 189 ms; admission/readiness waiting accounted for most of the extra onset. The two subsequent warm staged trials above do not establish a tail-latency guarantee. Network jitter, concurrent users, model switches and compilation can add latency.
Server experiments and retained variants
These earlier server runs used a virtual 140-ms playback reserve; the selected browser uses 220 ms. They are stochastic, not equal-text/equal-length paired quality trials. The main PCM-ready and delivered columns distinguish useful prefetch from the intentionally later playback order. Earlier three-stream runs were retained even when a later case could not start because the shared service was occupied.
Memory grounding: all final local outputs
Two calls per authored case. These examples use fictional test facts, never the visitor’s real wiki. Earlier failed variants are linked below: they were too cautious about known speakers, mishandled English or missed session-only evidence.
Actual memory browser check
Saved authored Bob/Alice facts, reconnected with a zero recent-history budget, then asked: Ich bin Bob. Weißt du noch, wie meine Freundin heißt?
First spoken reaction: Deine Freundin heißt Alice.. It received 2 retrieved entries. Opt-in, disable and reset passed; the isolated test workspace was reset afterward.
Listening examples
Mono 96-kbps MP3 embedded below. Audio is concatenated in spoken order: these files demonstrate wording and voice, not network pauses. Receipts retain the chunk timing for those pauses. Short-preface prosody may still differ from the following sentence.
Warum ist der Himmel blau?
reaction: Das lohnt einen Blick.
informed_bridge: Schauen wir uns die Lichtstreuung an.
answer: Das liegt an der sogenannten Rayleigh-Streuung!
answer: Das Sonnenlicht besteht aus vielen Farben, aber wenn es durch die Luft geht, streuen die kleinen Luftmoleküle das blaue Licht viel stärker als das rote.
answer: Deshalb sehen wir den Himmel blau, während die Sonne selbst eher gelblich wirkt.
Heute war ein anstrengender Tag. Wie kann ich etwas abschalten?
reaction: Puh, das klingt nach viel.
informed_bridge: Ein paar Ideen für dich.
answer: Das tut mir leid, aber ich habe da ein paar Ideen für dich.
Retained failure of substantive answer quality: Qwen announced ideas but did not actually provide them in this trial. Batching does not solve main-model completeness or factual reliability.
Please speak English. Why do cats purr?
reaction: That deserves a closer look.
informed_bridge: The healing aspect matters.
answer: Cats purr to communicate contentment, reduce stress, and even accelerate their own healing.
Reproduction, evidence and limits
Run the scripts from the published robby/ folder using its pinned stack. The internal TTS transport is loopback-only, shares the existing engine admission lock and does not replace the normal Playground API. No private recordings, credentials or real user memories are published.
python run_unit_checks.py
python benchmark_live_batch.py
python benchmark_two_stage.py --output two-stage-audio-new.json
python benchmark_reaction_memory.py --output two-stage-memory-new.json
python browser_two_stage_check.py --url https://YOUR-DEMO
python browser_two_stage_memory.py --url https://YOUR-DEMO
python browser_two_stage_cancel.py # edit URL for another deployment
One repeated memory browser run failed before the memory assertion because the existing exact-word TTS path rejected a SCRIPT/transcript mismatch. Its original model wording was not captured, so the cause is not established. The harness now retains all events and surfaces memory errors promptly. A fresh run passed; this does not prove that formatting failure is fixed.
First reaction text remains generated at temperature 0.35; informed-link text at 0.25. TTS is 0.8. One local model is reused, not a second local LLM download. The truncated temperature-zero batching pilot had sample-identical PCM across its tested schedules; that is not a proof of stochastic or perceptual equivalence at 0.8. Some links remain generic or grammatically thin; no human MOS/listening panel was run. Speaker matches and retrieved facts remain fallible. These typed/browser tests do not newly certify acoustic Stop, noisy-room ASR, speaker recognition or universal sub-900-ms latency.
Full implementation notes and privacy boundaries