Updated 2026-10-27T13:59:47.152947+00:00. EfficientAT MN04 supplies small, independent AudioSet hints alongside GPU sound captioning. Normal ASR does not wait for this optional branch. This report separates working data flow from actual recognition quality.
| Check | Result | What was verified |
|---|---|---|
| Offline regression | 551 passed | No network, physical microphone or model accuracy claim. |
| Expanded HTTPS suite | 13 cases passed | Actual CPU/GPU models, Parakeet, Qwen, TTS and browser PCM scheduling. Synthetic digital input. |
| Browser fake microphone | PASS | Real capture worklet; quiet music, music response, repeat cooldown, no false user turn. |
| Memory after reconnect | PASS | Local reaction remembers Alice using opt-in wiki with zero recent-history budget; off/reset verified. |
| Reply flow and UI | PASS | Optional reaction → informed main answer; poem, off switch, cancellation and mobile layout. Receipt includes its source revision. |
Useful but fallible: candidate labels helped expose rain, vehicle and cat sounds that the captioner described differently. They can also be wrong: rhythmic impacts received a high Music score. The browser completed its routes and speech playback, but the dog wording and glass-break interpretation still need improvement. No physical room, microphone-array or representative accuracy claim is made.
Runtime fingerprint for the expanded suite: f086cee87229863d2051e6ca20d413cc6cd281f533f2bb8f03543345f408b82a. Each other receipt records its own revision; a prior result is not silently relabeled as a current test.
| Candidate | Scope | Decision |
|---|---|---|
| EfficientAT MN04 | 983,599 parameters; 4,066,513-byte checkpoint; 527 AudioSet labels | Implemented on two CPU threads. Fits existing PyTorch stack and leaves VRAM to speech. |
| EfficientAT MN10 | Upstream larger 4.88-M parameter variant | Possible follow-up accuracy/latency trade-off; not benchmarked here. |
| YAMNet | MobileNet-based AudioSet classifier, 521 classes | Valid lightweight alternative; not deployed or benchmarked. Would introduce a different model runtime/conversion path. |
Sources: EfficientAT official repository and model table; official YAMNet documentation. AudioSet is the label/training resource, not a spatial localizer. Upstream benchmark numbers are not Robby test accuracy.
MN04 revision a425fdce92572e602a1d5634799bd9f1f2efa806; checkpoint SHA256 899a8c6217063f6941793102a2780b3dd788ab11fce468d9cf5f0577c453321c. Bootstrap downloads only eight inference/source files and one small weight file, retaining the upstream MIT license. CPU inference uses FP32; upstream notes that precision can affect results.
The two models receive the same captured audio independently. MN04 does not become a required serial gate before Whisper. Its request has a 180-ms deadline; unavailable, overloaded or late hints are dropped. The CPU worker accepts one job at a time and rejects a stale-work queue. The deadline overlaps the caption request; in the unusual case of a faster caption, joining can extend that branch up to the event deadline. Microphone ingress and normal ASR never wait for it.
Labels do not re-prompt Whisper's decoder. They join its caption as a second evidence source for the LLM. A strong music-vs-speech hint can also correct routing. This avoids an extra serial caption pass. Disagreement is passed explicitly as uncertainty, not resolved by treating an uncalibrated score as truth.
Temporal, not spatial: overlapping 1.28-second windows with 0.64-second stride show where a category was suggested. These are coarse classifier windows, not trained event boundaries. A mono microphone cannot supply a reliable left/right angle; spatial localization requires suitable synchronized microphone hardware and another algorithm.
| Measurement | n | Median | p95 (interpolated) | Definition |
|---|---|---|---|---|
| CPU MN04 worker, full clip + time windows | 14 | 37.94 ms | 43.27 ms | 14 five-second diagnostic clips; two CPU threads. No VRAM. |
| Caption only, HTTP wall time | 6 | 340.81 ms | 408.07 ms | 3.2-second identical authored music; alternating order. |
| Caption + MN04, parallel HTTP wall time | 6 | 273.25 ms | 386.21 ms | Same music; six repetitions. Difference is timing variation, not proof of speedup. |
| Caption + optional tags inside live session | 15 | 298.77 ms | 422.13 ms | Includes local HTTP; excludes capture, conversational pause, LLM and TTS. |
The paired test alternates six caption-only and six parallel requests on identical audio. Lower parallel wall time in this small run is timing variation, not an acceleration claim. Capture/context collection (about 1.6 seconds for transients, 3.2 for sustained sounds), idle scheduling, API generation, TTS and browser buffering remain separate. CPU classification time is not microphone-to-ear latency.
First two fixed metadata rows per class, not selected for successful predictions. Twelve public ESC-50 clips plus two authored controls. Original audio stays in the excluded local cache; exact URLs, hashes and license credits are recorded in the receipts.
| Input label | GPU caption, fallible | CPU top 3, uncalibrated scores | CPU ms |
|---|---|---|---|
| dog-1 | The audio contains a single, distinct dog bark. The bark is relatively short and sharp, with a medium pitch. | Bark 0.758; Dog 0.757; Animal 0.733 | 50.57 |
| dog-2 | A single, sharp, and distinct percussive sound is heard. It has a dry, crisp quality, resembling a quick, forceful impact or a single strike. | Animal 0.735; Dog 0.652; Bark 0.611 | 36.37 |
| cat-1 | The audio features a distinct, high-pitched meow sound. The meow is short and clear, with a slight wavering in pitch. | Cat 0.850; Animal 0.824; Domestic animals, pets 0.777 | 38.14 |
| cat-2 | The audio contains a loud, sustained siren sound. The siren has a rising and falling pitch, characteristic of emergency vehicles. | Cat 0.907; Animal 0.874; Domestic animals, pets 0.846 | 37.84 |
| glass_breaking-1 | The audio contains a single, sharp, percussive sound. It sounds like a single, forceful strike, possibly a drum hit or a similar impact. | Breaking 0.690; Machine gun 0.488; Glass 0.486 | 39.34 |
| glass_breaking-2 | The audio contains a single, sharp, and loud gunshot sound. The sound is brief and impactful, with a clear attack and decay. | Breaking 0.861; Coin (dropping) 0.728; Glass 0.659 | 38.04 |
| car_horn-1 | The audio features a single, distinct, and sharp metallic clang. The sound is brief and has a clear, resonant quality. | Vehicle horn, car horn, honking 0.579; Vehicle 0.566; Silence 0.426 | 39.18 |
| car_horn-2 | The audio features a short, sharp sound, similar to a gunshot or a gunshot. The sound is abrupt and percussive, with a quick decay. | Vehicle horn, car horn, honking 0.845; Music 0.631; Silence 0.430 | 36.76 |
| engine-1 | The audio features a distinct, low-frequency rumble, accompanied by a higher-pitched whine. The sounds are consistent and do not vary significantly in intensity. | Vehicle 0.685; Truck 0.291; Engine 0.240 | 38.11 |
| engine-2 | The audio features a continuous, low-frequency hum, consistent with the sound of a large engine or machinery. The sound is steady and has a deep, resonant quality. | Vehicle 0.769; Truck 0.473; Bus 0.308 | 35.61 |
| rain-1 | The audio contains the sound of liquid being poured. The sound is continuous and consistent, with a moderate volume. | Rain 0.602; Rain on surface 0.481; Raindrop 0.454 | 38.14 |
| rain-2 | The audio features the sound of a vehicle. The sound is a low rumble, possibly from an engine, and a distinct whooshing sound, possibly from the vehicle moving at a moderate speed. | Rain on surface 0.697; Rain 0.626; Raindrop 0.406 | 36.49 |
| authored_music | The audio features a repetitive, rhythmic electronic beat. A synthesized bassline is prominent, along with a simple, repeating melody. | Music 0.910; Musical instrument 0.143; Electric piano 0.126 | 36.89 |
| authored_impacts | The audio contains a single, distinct percussive sound. It is a sharp, dry, and short sound with a quick decay. | Music 0.779; Drum machine 0.657; Silence 0.426 | 35.05 |
The caption-only detector replay routed 42/42 examples across onset, forced-positive VAD and already-playing modes. Routing is not correctness. In particular, music-like impacts, cat/siren confusion, vague rain descriptions and glass/impact confusion prevent any claim of perfect recognition. Raising thresholds alone would also discard genuine quiet sounds.
| Case | Data-flow check | PCM chunks | Generated reply | Meaning / limitation |
|---|---|---|---|---|
| music_onset | PASS | 5 | Oh, da läuft gerade ein ziemlich treibender elektronischer Track. | Music acknowledged/answered; precise instrument descriptions are not verified. |
| music_already_playing | PASS | 4 | Oh, das klingt nach fröhlicher Musik! | Music acknowledged/answered; precise instrument descriptions are not verified. |
| dog | PASS | 4 | Habt ihr gerade ein Hundegefühl gehört? | Wording defect: invented “Hundegefühl”; not a good user-facing description. |
| car_horn | PASS | 9 | Hörst du das? Ein kurzes Hupen – war das vielleicht gerade ein Auto draußen? | Inspect exact reply; routing/completion is the automated assertion. |
| glass_breaking | PASS | 6 | Hi, ich habe gerade kurz ein klares Klanggeräusch gehört, war das etwa ein Drum-Hit? | SEMANTIC ERROR: described as a drum/impact instead of identifying glass. |
| cat | PASS | 11 | Hallo? Ich habe gerade ein leises Meow gehört – ist da etwa eine Katze im Raum? | Tentative cat identification is consistent with the source. |
| after_question_music_bridge_False | PASS | 7 | Das klingt nach einem schnellen elektronischen Track mit einer treibenden Bassline und einer sehr repetitiven Synth-Melodie. | Music acknowledged/answered; precise instrument descriptions are not verified. |
| after_question_music_bridge_True | PASS | 28 | Das klingt nach Musik. Ein schneller elektronischer Track mit einer sehr treibenden Basslinie. Klingt, als würde jemand gerade auf „Party-Modus“ drücken! | Music acknowledged/answered; precise instrument descriptions are not verified. |
| spoken_question_then_music | PASS | 17 | Ich höre gerade etwas Musik im Hintergrund. Es klingt nach einem elektrischen Gitarrenriff mit einem gleichmäßigen Drumschlag und Bass. Ist das ein Song, der dir besonders gefällt? Und wie darf ich dich nennen? | Music acknowledged/answered; precise instrument descriptions are not verified. |
| christoph-intro | PASS | 14 | Mir auch, Christoph! | Inspect exact reply; routing/completion is the automated assertion. |
| selma-followup-clear | PASS | 11 | Wie wär’s mit einer Mini-Geschichten-Challenge: Du gibst mir ein Wort, ich erfinde eine verrückte Kurzgeschichte, und du drehst das Ende um? Und wie darf ich dich nennen? | Inspect exact reply; routing/completion is the automated assertion. |
| selma-intro | PASS | 7 | Hallo Selma! Schön, dass du da bist – jetzt sind wir ein richtiges Trio! | Inspect exact reply; routing/completion is the automated assertion. |
| selma-followup-clear | PASS | 13 | Wie wäre es mit einer kleinen Wort-Challenge? Ihr beide denkt euch je ein verrücktes Wort aus, und ich baue daraus eine minikurze, absurde Geschichte – mit Robotern, Katzen und mindestens einem schiefen Kuchen. | Inspect exact reply; routing/completion is the automated assertion. |
Expanded tests inject deterministic PCM through the actual browser's 20-ms microphone sender, including playback-position tags. The separate music test uses Chrome's fake microphone through the capture worklet. Real model services, hosted/local LLMs and TTS are used. Completion and PCM scheduling do not certify a human listener's physical speakers.
The earlier “text shown but not spoken” bug was premature browser cancellation: raw level/VAD could cancel an optional environmental reply even when the activity was just sound. Environmental replies now ignore that generic browser cancellation source. Server-verified user interruption, Stop and manual controls remain active. The music test injects high level/VAD during a reply and checks that it sends no false avatar:cancel. The UI distinguishes waiting-for-audio, scheduled audio and cancellation.
The conservative heuristic requires all three: relative level below 0.28 of the session reference, no confident known-voice match (cosine below 0.62), and no explicit address or self-introduction. With no reference it accepts the speaker. Loudness and estimated SNR alone are not intent classifiers. The filter is independent and can be disabled.
The speaker browser replay covers Christoph, an unfamiliar foreground guest, “Ich bin Selma”, later Selma and quiet unrelated background speech at gain 0.1. A narrow recent-name-answer rule can attach a lower-confidence vocal variant to the just-created unnamed guest, avoiding a duplicate person. This does not guarantee rejection of loud TV or acceptance of every soft user. The protocol defines a held-out voice/level/noise matrix for further threshold search.
python run_unit_checks.py python bootstrap_sound_events.py # Separate loopback process; existing GPU caption worker is also required: python sound_event_worker.py python check_sound_events.py python check_environment_diversity.py # Sequential browser sessions, with the normal demo stack running: python browser_environment_music.py --url https://YOUR-DEMO python browser_environment_expanded.py --url https://YOUR-DEMO --group all python browser_two_stage_check.py --url https://YOUR-DEMO python browser_two_stage_memory.py --url https://YOUR-DEMO
The main launcher accepts --with-sound-captions for both sidecars. Use matching PyTorch/torchaudio/torchvision wheels; tested 2.6.0+cu124 / 2.6.0+cu124 / 0.21.0+cu124. Optional event settings: ROBO_EVENT_CACHE, ROBO_EVENT_URL, ROBO_EVENT_PORT. Default loopback port: 8804. Runtime audio is held briefly in RAM and not saved.
The test protocol separates required future scenarios from executed evidence and describes licenses, scene generation, physical-room limitations and controlled threshold searches. Retained failed attempts include a busy public ASR slot, a generic after-question reply and a speaker-name split. They are not silently deleted or reported as passes. The older report is archived here.
Recommended next steps: blind semantic rating on held-out sound mixtures; ablate MN04 vs MN10 on identical clips; music-plus-speech and real singing; loud television versus quiet addressed speech; actual room/echo/device tests. Fine-grained event boundaries or direction-of-arrival would need a separately validated model/hardware path.