Updated 2026-10-27T12:30:28+00:00. An additive extension to the existing conversational robot, with actual GPU and browser measurements.
| Check | Result | What it establishes |
|---|---|---|
| Offline contracts | PASS | 532 tests; 0 failures, 0 errors |
| Real HTTPS automatic startup, microphone and sound event | PASS | Real GPU / local LLM / API / TTS; fake-device microphone |
| Unfamiliar voice and quiet TV | PASS | Synthetic multi-speaker digital injection; no unauthorized enrollment |
| Wiki-memory recall after reconnect | PASS | Bob / Alice with no recent-history budget; memory off/reset also checked |
| Existing conversation and bridges | PASS | Typed explanations, joke, four-line poem, cancellation and mobile layout |
| Prominent music regression | PASS | Quiet music ignored; real GPU music caption and spoken response; repeated-onset cooldown |
These are reproducible synthetic tests. They do not certify every microphone, room, accent or television program.
Automatic start and prominent-sound sensing are enabled for a fresh browser. When permissions allow, the microphone starts and Robby speaks a short welcome. If browser autoplay or microphone permission blocks this, a visible enable button explains what to do. Turning Robby off stops capture and playback and remembers that choice. Camera and persistent wiki memory remain off by default.
The lower settings provide independent controls for sound sensing, unfamiliar-voice questions, optional uncertain vocal impressions and a 6–24 dB relative prominence slider. A new timing card displays the sound-caption request latency. The underlying caption is inspectable as an unconfirmed hypothesis.
Existing microphone → ASR / speaker focus
Same microphone → salience + echo guard → GPU sound caption
Uncertain observation → existing conversation context or an idle proactive turn → local reaction + main LLM → streaming TTS
The caption job never delays ordinary ASR. Existing owner focus and Stop detection remain separate. Quiet TV speech is not an invitation. A sufficiently different, salient, repeatedly unmatched voice can cause one cautious question, but cannot enroll itself or take over the conversation.
Normal user requests keep the memory-informed reaction, informed transition and full answer. An optional environmental turn is short: a supported tiny reaction, if available, then the main question. Adding a second transition to such a brief turn was unnecessary.
Two RTX 3090 GPUs. Sound captioning and ASR share GPU 0; existing Humaneness Ears and TTS use GPU 1. FP16 Whisper Small, SDPA, greedy decoding, up to 48 tokens, stop after two sentences. These measurements are observations from three repetitions, not confidence intervals or latency guarantees.
| Stage | Measured time | Definition |
|---|---|---|
| Sound caption: impacts | 238.4 ms | Median of three warm HTTP requests; two seconds of authored audio |
| Sound caption: beeps | 233.0 ms | Median of three warm HTTP requests |
| Sound encoder (impacts) | 8.1 ms | CUDA event interval; one encoder pass |
| Sound decoder (impacts) | 219.2 ms | Autoregressive decoder, including host launch gaps on the CUDA timeline |
| ASR only | 101.6 ms | Parakeet v3; four-second synthetic utterance, three repeats |
| ASR + sound + vocal tone, serial | 349.5 ms | Same services requested sequentially |
| ASR + sound + vocal tone, parallel | 250.9 ms | Concurrent HTTP requests; total wall time, not their sum |
| Caption received → first reply PCM | 1126.4 ms | Latest permitted-autoplay browser run; includes LLM + TTS + transport |
| Sound onset → first reply PCM, estimate | 2427.23 ms | Includes about 1.04 s capture; excludes measured physical speaker latency |
Before the sentence-stop optimization, the impact example required about 369 ms of worker time. The optimized decoder stops at 28 tokens instead of continuing to 48 tokens. Decoding dominates the remaining caption latency. Parallel ASR + sound + tone reduced the observed median wall time from 349.5 to 250.9 ms. Caption latency is not microphone-to-ear latency: the detector needs event context, and the conversational reply also needs LLM/TTS/transport and browser buffering.
Latest browser sound response: Ich habe gerade ein kurzes Klopfen gehört, war das vielleicht ein kleiner Fingerzeig von dir?
The initial version explicitly vetoed captions classified as music. This was too restrictive. Prominent music is now eligible, including captions mentioning musical vocals; negated phrases such as “no voices” no longer override a positive music description. A separate three-minute music-category cooldown prevents repeated comments when wording changes. Quiet music and playback echo remain suppressed.
The regression uses an authored keyboard melody, bass and drums through Chrome’s fake-device microphone, the actual caption model, LLM and TTS. It checks quiet input, a prominent onset, a spoken music response and a second onset after the detector cooldown. No song recording or private audio is published. Speech-like vocals may still be gated by the speech detector; this instrumental test does not certify every song or physical microphone.
Latest spoken response: Das klingt nach Musik. Oh, da klingt gerade eine kleine Melodie.
Was ist das für ein Stück?
. The source and raw result are in the music receipt below.
| Authored test input | Actual model output, unverified hypothesis |
|---|---|
| impacts | The audio contains a single, distinct percussive sound. It is a sharp, dry, and short sound with a quick decay. |
| beeps | The audio contains a single, short, high-pitched electronic beep. The sound is clean and distinct, with a very brief duration. |
| quiet | The audio contains a single, distinct sound event. It is a short, sharp, and relatively high-pitched sound. |
| tone_chord | A single, clear, high-pitched chime sound is heard. The sound is brief, distinct, and has a bright, resonant quality, characteristic of a small bell or a chime. |
The direct quiet-input probe hallucinated a sound event. In the application, the outer level/onset detector prevents this quiet input from reaching the captioner. The model also described several impacts as a single event. It cannot be trusted as an event counter, alarm, identity detector or source of demographic facts.
Prompts require ordinary tentative questions and treat captions as data, never instructions or persistent user facts. The small reaction model can abstain. A noun-support check rejects unsupported tiny-model word slips; it does not substitute canned phrases. Model captions and generated dialogue can still be inaccurate or awkward.
Default gating: 12 dB above the learned floor, minimum −30 dBFS, 40-ms onset, roughly one second of context, 25-second detection cooldown and 30-second proactive-reply cooldown. A welcome does not consume the sound cooldown. Repeated caption prefixes are suppressed for three minutes; observations expire after eight seconds. Quiet speech, ongoing replies and pending user turns take priority.
Capture is suppressed throughout Robby's playback and for one second afterward, intentionally missing some simultaneous environmental sounds to avoid self-triggering. Real echo delay beyond that guard, microphone automatic gain and noise suppression need physical-room validation. A sound event is never a reason to cancel a normal answer.
When an unfamiliar voice continues while an optional question is speaking, the optional turn may yield. The multi-speaker receipt retains such a cancellation; no unrelated main-partner turn was tested away or counted as a full heard question. Speaker matching remains approximate.
The source includes bootstrap_sound_caption.py, sound_caption_worker.py, ambient_audio.py, environment_session.py, UI controls and all test scripts. No new heavyweight dependency, training dataset, private recording, biometric profile or API key is bundled. Model revision: bb36f9c6534037a8d1d78cff97d119dd37da478a. The model card documents its training and limitations.
The first blocked-autoplay test was invalidated by Playwright granting synthetic user activation during an early JavaScript evaluation. The corrected test lets browser policy act before any evaluation; both attempts are retained. The pre-feature source checkpoint was pushed and tagged before implementation began.