Open live demo · Implementation and reproduction · Test protocol

Who is speaking — and who is Robby?

October 27, 2026 · Explicit roles, multiple voice prototypes, optional remembered profiles and cautious attention to background sounds.

Results first

CheckObserved resultMeaning / limits
Offline regression567 passedRoles, prototype updates, weak continuity, consented import, camera binding, background guards; no accuracy estimate.
HTTPS microphone / role replay6 turns passedTwo authored voices; real ASR/ECAPA/Ears/TTS. Reconnect, consent withdrawal and forget also checked.
Current role probes10/10Six deterministic name lookups and four actual Qwen generations; do not call this ten LLM accuracy tests.
Repeated background voicePASSA cautious question produced real PCM; the rejected voice was not enrolled.
Optional camera pathPASSCPU model inference, two faces, motion/held-mouth negatives, gaze and consent; public imagery and fake device.
Background-question prompt5/5Real Qwen calls after introductions; lexical question check and manual reading, not a general semantic score.

These are development regressions with authored synthetic speech and public imagery. They do not certify human speaker accuracy, arbitrary German grammar, or reliability on every physical microphone/camera.

Which model made the mistake?

The exact historical quote “Du bist Christoph, wie du dir selbst genannt hast” was not in retained logs. Its individual source cannot be proven. The configured main model is Qwen 3.8 27B through OpenRouter/Cerebras; local filler reactions remain off by default.

The real browser replay did reproduce a related error: the acoustic endpoint said Christoph, yet Qwen said “Du bist Selma” after a previous Selma turn. Another probe dismissed a user correction with “Ach du Scherzkeks!”. Those failed attempts remain downloadable below.

The system prompt and current-turn header now distinguish the human user's “I” from Robby's “I”. Narrow questions such as “Wie heiße ich?” read the named current speaker directly: “Du heißt Christoph.” Weak continuity asks for confirmation. General questions still use the language model. No extra correction model call is added. The UI labels every actual reply model, including the local identity controller.

Actual spoken browser replies

Authored input clipAcoustic nameReply textActual sourcePCM chunks
christoph-introChristophHallo Christoph! Es freut mich sehr, dich kennenzulernen. Ich bin Robby, dein kleiner Roboter-Kumpel.Main reply: qwen/qwen3.8-27b · Cerebras13
who-calmChristophDu heißt Christoph.Main reply: Session identity · name lookup · local controller3
who-playfulChristophDu heißt Christoph.Main reply: Session identity · name lookup · local controller3
who-calmChristophDu heißt Christoph.Main reply: Session identity · name lookup · local controller3
selma-introSelmaHallo Selma! Schön, dass du da bist.Main reply: qwen/qwen3.8-27b · Cerebras5
who-calmChristophDu heißt Christoph.Main reply: Session identity · name lookup · local controller3

The replay includes calm/playful intonation, restoring a consented profile after a real connection restart, a Selma introduction, and Christoph returning. It also verifies disabling persistence and forgetting profiles. The earlier failed six-turn replay is retained for comparison.

Tested source: 9ca2d88b340e7fd2c568083cb553a054fe5a852a69a46af4e6fd9caf4cac0eb2. Every receipt retains its own source revision; earlier diagnostic runs must not be presented as tests of the latest revision.

Better continuity without accepting every voice

Open matching defaults to cosine 0.32. A recent named last speaker can remain a tentative match down to max(0.24, threshold − 0.08), only within 60 seconds and with at least a 0.08 lead over other voices. Such weak matches never add prototypes. Strong different voices and explicit name corrections retain their separate meaning.

A person can have up to eight ECAPA prototypes. Explicit corrections can add another vocal mode; strong automatic additions require 0.55 cosine and a 0.12 rival margin. Name questions have 90-second per-person and 45-second global cooldowns. These are heuristic thresholds, not identity probabilities or secure authentication.

Profiles, memory and controls

Uncertain vocal age/gender and timbre impressions are ON by default. The existing GPU Humaneness Ears model describes perceived register, brightness, resonance, warmth and related vocal qualities. Up to twelve accepted observations form a tentative profile. These are audible impressions, not someone's actual age, gender identity or personality. They never decide who a person is.

The lower People Robby knows · voice profiles panel shows names, prototype counts and traits. Optional browser persistence requires everyone's permission and is OFF by default. It saves approved named profiles locally; new visitors remain session-only until consent is renewed. Embeddings are never sent to Qwen. Turning it off removes that voice store; Forget all clears face/voice profiles. Separately consented face storage can be managed in Recognition & privacy settings.

Facts, relationships and interests use the existing separate, default-off wiki memory. Vocal guesses are not promoted to facts. Forgetting biometric profiles does not silently delete that wiki; it has its own reset.

Camera: available, but still opt-in

Camera auto-start, webcam Face ID and face-VAD gating remain OFF by default. Enable the camera and webcam Face ID to use optional face/voice association. Following position is independent. A missing connection between visual identity and the newer sentence pipeline is now restored: an unambiguous time-aligned speaking face can bind to its named voice, without requiring camera-gated speech.

Existing MediaPipe landmarks and face-api embeddings run in CPU browser workers. The test detected two public-image faces and produced repeatable 128-D descriptors. It also rejected a held-open mouth, static mouth, jitter and rigid head movement as speaking evidence. This is not a human webcam identification study.

ComponentSamplesMeasured timeScope
MediaPipe · two public faces641.1 msIncludes initial sample; CPU browser worker, not physical webcam accuracy.
Face-api descriptor · 128-D3481.0 msOn demand, throttled, worker CPU; not every camera frame.
Geometric speaking-state · two facessynthetic loop0.089 msGeometry calculation only; excludes landmark inference.
Ears GPU · impressions OFF145.66 msOne authored 4.43-s clip; GPU inference/rules only, not end-to-end.
Ears GPU · impressions ON121.93 msSame clip after warmup. This is not evidence that adding traits speeds inference up.
Validated proactive voice question5451.2 msProvider generation + output check; at most one retry. Excludes ASR, capture and TTS; does not delay normal foreground streaming.

Curiosity that waits its turn

Prominent sounds and unfamiliar audible-voice questions are on by default. Quiet-background filtering remains independent. Two sufficiently audible rejected background segments within 25 seconds can prompt one tentative question; 120-second cooldown, playback guards, user permissions and idle scheduling prevent repeated intrusion. This does not enroll the background voice or transfer conversation ownership.

Ich habe gerade kurz eine zweite Stimme wahrgenommen – ist da noch jemand bei dir oder läuft vielleicht einfach nur der Fernseher?

This test produced 8 real PCM chunks; background enrollment was blocked: True. It used two quiet repeats of an authored synthetic weather voice, not an actual TV. Existing music and sound tests are in the environment report.

A repeated run correctly routed the observation but only said “Hallo Christoph! Schön, dass du da bist!”. That is preserved as a semantic failure, despite successful playback. The current controller request states the background question at the latest message position; the strengthened test now requires a relevant question as well as audio and no enrollment. Five extra real-provider probes are also retained.

Prompting alone still produced one irrelevant greeting in five probes. The optional voice question now waits for a relevant-question check before TTS. A bad draft gets at most one regeneration; if both fail, that optional comment stays silent. Only the first matching question is spoken. This narrow lexical guard is not a general semantic truth checker.

Sound hypotheses explicitly allow a television, a recording, another person, or the same person speaking differently. The GPU captioner and CPU AudioSet candidates continue alongside ASR; no extra model blocks an ordinary foreground turn.

Reproduce, inspect and improve

python run_unit_checks.py
python check_social_dialogue.py
python check_social_curiosity_prompt.py
python generate_social_fixtures.py
python browser_social_profiles.py --url https://YOUR-HOST
python browser_social_curiosity.py --url https://YOUR-HOST
python browser_multimodal_checks.py --url https://YOUR-HOST/desk-robot/
python build_social_report.py

Use the normal service stack and existing adversarial fixture cache; see the linked implementation guide for setup. Run microphone suites sequentially: the live ASR has one session slot. Preserve actual provider/model metadata and deployment hashes. Never publish private audio, voiceprints or credentials.

Next evaluation: consenting held-out people, distance/microphone changes, realistic overlapping television, similar voices, emotional variation and explicit corrections. Measure false identity merges as well as missed matches, duplicate profiles, unnecessary name questions and false proactive questions. More permissive thresholds alone can make these trade-offs worse.

Raw receipts, including failures