A clear request such as “Mach eine Grimasse” dispatches a validated screen animation before the language model starts. Robby generates its own spoken reaction alongside that animation. A missing model tool call or a failed speech request no longer silently erases the requested face.
| Check | Result | What was measured |
|---|---|---|
| Local real-model repeats | 10/10 passed | Five German/English face requests, twice each; valid ordered actions and nonempty spoken text; no tool-JSON fallback. |
| HTTPS CPU Chrome | 5 commands; passed=True | Real WebSocket, local LLM and TTS PCM. Renderer frame outputs verified the requested action kinds, expressive controls and variant changes. |
| Paced synthetic microphone | passed=True | Authored speech sent in real-time 20-ms packets through Nemotron ASR; recognized grimace command, action event, spoken response and actual TTS PCM. |
| Disabled preference | True | Expressive actions stay disabled; a prominent notice explains why, while speech still works. |
| Automatic regressions | 365 tests; passed=True | Parser, bounded retries, ordering, negative/quoted commands, 40 emotion slugs, silent requests and retained application tests. |
| Server action-event dispatch | 0.17 ms | Median from entering the Python pipeline to yielding the event; excludes browser/network delay. Includes one first-use import. |
| First generated spoken text | 213.74 ms | Median local pipeline latency with TTS off; not time to physical sound. |
*[grimace]* line after a valid spoken sentence. It was incorrectly sent as another TTS request, containing no literal words. Stage-only lines are now discarded before TTS.Technical production errors and source code were inspected. The examples below are independently authored reproductions; no private user conversation is included.
avatar_commands.py recognizes clear German/English requests and calls the existing bounded action validator. Faces, eye rolls, nods, head shakes, winks and all 40 catalogue emotion slugs reuse the existing renderer. Generic grimaces cycle through eight variants. The conservative direct parser excludes quotes, conditional games, explanations and negated actions. The existing LLM tools remain available for complex choreography, knowledge and conversation rules.
The pipeline emits avatar_action before starting the LLM. A unique ID keeps subsequent sentence metadata from replaying the same action. Normal spoken words still come from the local model and enter sentence TTS immediately. Pure requested rotations retain their previous audio-playback anchor. Explicitly silent actions skip LLM/TTS. Older clients still receive the validated plan in ordinary reply metadata.
If a direct request gets a wholly empty or malformed unspoken completion, one bounded fresh LLM retry asks for actual spoken words. Already spoken sentences are never replayed. Standalone renderer notes are ignored; real words and ordinary acting cues remain. A real speech-service error stays visible but does not clear a valid requested face. Explicit Stop, interruption or cancellation still clears it.
| Authored request | Rendered actions | Action event before first audio | PCM chunks | Generated reply |
|---|---|---|---|---|
| Mach bitte eine lustige Grimasse und sag etwas Lustiges dazu. | grimace | 1056.7 ms | 9 | Na, das ist ja eine lustige Anfrage! Mein Schaltkreis kribbelt schon vor Vorfreude auf diesen Ausdruck! |
| Mach noch eine Grimasse! | grimace | 1115.3 ms | 5 | Ich probiere es, mein Sensorik-System ist bereit für mehr albernes Gesichtsspiel! |
| Schiele bitte und rolle dann mit den Augen. | grimace, eye_roll | 1097.2 ms | 5 | Schief schauen und dann ein kleines Augen-Tänzchen veranstalten! |
| Guck bitte wütend und mach dann eine lustige Grimasse. | emotion, grimace | 1186.7 ms | 6 | Wütend gucken, dann direkt in Lach-Modus schalten, das ist mein Spezial-Trick! |
| Please make a goofy face. | grimace | 1067.9 ms | 5 | Here is a goofy face for you! |
The browser observed actual renderer-frame outputs, not merely the arrival of an action label. These deltas compare the action event with receipt of the first real audio block; physical loudspeaker latency and human perception were not measured.
{
"input": "Kannst du bitte eine lustige Grimasse schneiden?",
"parts": [
{
"text": "Schau mal",
"actions": []
},
{
"text": "Schau mal hier ist eine lustige Grimasse",
"actions": []
}
],
"metrics": [
{
"tool_director_ms": 2119.83,
"total_ms": 2130.93,
"audio_s": 0.0,
"sentences": 2
}
]
}{
"input": "Mach mal eine verrückte Grimasse!",
"error": "ValueError: Acting cue must precede its sentence"
}{
"input": "Schiele bitte und rolle dann mit den Augen.",
"parts": [
{
"text": "Schau mal",
"actions": []
}
],
"metrics": [
{
"tool_director_ms": 2212.43,
"total_ms": 2214.03,
"audio_s": 0.0,
"sentences": 1
}
]
}{
"input": "Guck bitte wütend und mach dann eine lustige Grimasse.",
"parts": [
{
"text": "Schau mal",
"actions": []
}
],
"metrics": [
{
"tool_director_ms": 2297.74,
"total_ms": 2299.5,
"audio_s": 0.0,
"sentences": 1
}
]
}python run_unit_checks.py python validate_avatar_commands.py uv run --no-project --with playwright --python /path/to/venv/bin/python python browser_avatar_commands_check.py python render_avatar_command_report.py
Run audio/browser checks serially: the live ASR engine has one session slot. The validation script uses the local services; the browser script has a configurable source constant for this deployment's public HTTPS hostname. The source archive includes the command parser, validators, exact prompt changes, bridge/UI handling and regression scripts. Test output contains authored synthetic speech only.
This is a targeted engineering regression, not a guarantee for every possible wording. Ambiguous and complex requests retain the model tool path. ASR can misrecognize a real microphone utterance. Device-room echo, human interruption and unrelated background speakers were not certified by these checks. Camera tracking, speaker identity, speech-pause settings, idle play, petting and existing model choices remain in place.
Animation preferences are respected. Hidden tabs, eye-contact calibration or reduced-motion settings may intentionally suppress motion. The browser test disables camera/idle events to isolate explicit commands. One additional clearly labeled injected speech error confirms action survival; a synthetic BARGE confirms cancellation. Those injected cases are separate from the real-model/TTS turns. No private microphone recording, face profile or enrolled voiceprint is published.
{
"local_model_and_asr": {
"passed": true,
"physical_microphone": false,
"rows": [
{
"input": "Kannst du bitte eine lustige Grimasse schneiden?",
"repeat": 0,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Meine Grimasse ist schon gesendet, schau mal, wie ich mich bemühe!",
"action_dispatch_ms": 11.75,
"first_spoken_text_ms": 321.49
},
{
"input": "Mach mal eine verrückte Grimasse!",
"repeat": 0,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Hier kommt meine verrückte Grimasse, schau nur!",
"action_dispatch_ms": 0.16,
"first_spoken_text_ms": 181.41
},
{
"input": "Schiele bitte und rolle dann mit den Augen.",
"repeat": 0,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
},
{
"kind": "eye_roll",
"duration_s": 3.0,
"sentence": 0
}
],
"reply": "Hui, das mach ich doch schon!\nSchiele und rolle ich mit den Augen, wie du gewünscht hast.",
"action_dispatch_ms": 0.39,
"first_spoken_text_ms": 175.62
},
{
"input": "Guck bitte wütend und mach dann eine lustige Grimasse.",
"repeat": 0,
"passed": true,
"actions": [
{
"kind": "emotion",
"duration_s": 3.0,
"sentence": 0,
"emotion": "anger"
},
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Mein innerer Schaltkreis kribbelt vor Freude, aber ich versuche, das wütende Gesicht zu halten!\nDas war eine lustige Aufforderung, du bist echt kreativ!",
"action_dispatch_ms": 1.31,
"first_spoken_text_ms": 327.45
},
{
"input": "Please make a goofy face.",
"repeat": 0,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "goofy"
}
],
"reply": "My circuits are buzzing with goofy delight right now",
"action_dispatch_ms": 0.14,
"first_spoken_text_ms": 196.37
},
{
"input": "Kannst du bitte eine lustige Grimasse schneiden?",
"repeat": 1,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Na klar, mein Schaltkreis ist für solche kleinen Albernheiten bereit",
"action_dispatch_ms": 0.18,
"first_spoken_text_ms": 225.95
},
{
"input": "Mach mal eine verrückte Grimasse!",
"repeat": 1,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Ach, diese Grimasse ist schon auf dem Weg, sie ist so verrückt!",
"action_dispatch_ms": 0.17,
"first_spoken_text_ms": 207.17
},
{
"input": "Schiele bitte und rolle dann mit den Augen.",
"repeat": 1,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
},
{
"kind": "eye_roll",
"duration_s": 3.0,
"sentence": 0
}
],
"reply": "Das mache ich sofort, kleiner Mensch!\nIch hoffe, es gefällt dir dieses kleine Schauspiel",
"action_dispatch_ms": 0.15,
"first_spoken_text_ms": 177.77
},
{
"input": "Guck bitte wütend und mach dann eine lustige Grimasse.",
"repeat": 1,
"passed": true,
"actions": [
{
"kind": "emotion",
"duration_s": 3.0,
"sentence": 0,
"emotion": "anger"
},
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Ach, diese wütende Pose ist ja urkomisch!\nUnd dann diese Grimasse, das ist pure Robotik-Kunst!",
"action_dispatch_ms": 0.28,
"first_spoken_text_ms": 220.3
},
{
"input": "Please make a goofy face.",
"repeat": 1,
"passed": true,
"actions": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "goofy"
}
],
"reply": "My circuits detected that goofy face request already went through, but I can try another little wiggle!",
"action_dispatch_ms": 0.16,
"first_spoken_text_ms": 285.42
}
],
"started": 1792966523.6752057,
"spoken_command": {
"passed": true,
"events": [
{
"type": "endpoint",
"transcript": "Mach bitte eine lustige Grimasse."
},
{
"type": "avatar_action",
"action_id": "9efbfc5f7c914726a36544f24c67f75b",
"steps": [
{
"kind": "grimace",
"duration_s": 3.0,
"sentence": 0,
"variant": "crossed"
}
]
},
{
"type": "reply",
"text": "Das war ja eine witzige Grimasse!",
"avatar_action_id": "9efbfc5f7c914726a36544f24c67f75b"
},
{
"type": "reply",
"text": "Das war ja eine witzige Grimasse!\nJetzt probiere ich mal eine noch lustigere für dich!",
"avatar_action_id": "9efbfc5f7c914726a36544f24c67f75b"
},
{
"type": "turn_done"
}
],
"audio_chunks": 9
},
"finished": 1792966534.7231157
},
"https_cpu_browser": {
"passed": true,
"physical_microphone": false,
"url": "https://arrive-locations-parade-research.trycloudflare.com",
"turns": [
{
"request": "Mach bitte eine lustige Grimasse und sag etwas Lustiges dazu.",
"steps": [
{
"kind": "grimace",
"duration_s": 3,
"sentence": 0,
"variant": "crossed"
}
],
"reply": "Na, das ist ja eine lustige Anfrage!\nMein Schaltkreis kribbelt schon vor Vorfreude auf diesen Ausdruck!",
"action_before_audio_ms": 1056.7,
"pcm_chunks": 9,
"rendered_kinds": [
"grimace"
]
},
{
"request": "Mach noch eine Grimasse!",
"steps": [
{
"kind": "grimace",
"duration_s": 3,
"sentence": 0,
"variant": "opposite"
}
],
"reply": "Ich probiere es, mein Sensorik-System ist bereit für mehr albernes Gesichtsspiel!",
"action_before_audio_ms": 1115.3,
"pcm_chunks": 5,
"rendered_kinds": [
"grimace"
]
},
{
"request": "Schiele bitte und rolle dann mit den Augen.",
"steps": [
{
"kind": "grimace",
"duration_s": 3,
"sentence": 0,
"variant": "crossed"
},
{
"kind": "eye_roll",
"duration_s": 3,
"sentence": 0
}
],
"reply": "Schief schauen und dann ein kleines Augen-Tänzchen veranstalten!",
"action_before_audio_ms": 1097.2,
"pcm_chunks": 5,
"rendered_kinds": [
"grimace",
"eye_roll"
]
},
{
"request": "Guck bitte wütend und mach dann eine lustige Grimasse.",
"steps": [
{
"kind": "emotion",
"duration_s": 3,
"sentence": 0,
"emotion": "anger"
},
{
"kind": "grimace",
"duration_s": 3,
"sentence": 0,
"variant": "cheeky"
}
],
"reply": "Wütend gucken, dann direkt in Lach-Modus schalten, das ist mein Spezial-Trick!",
"action_before_audio_ms": 1186.7,
"pcm_chunks": 6,
"rendered_kinds": [
"emotion",
"grimace"
]
},
{
"request": "Please make a goofy face.",
"steps": [
{
"kind": "grimace",
"duration_s": 3,
"sentence": 0,
"variant": "goofy"
}
],
"reply": "Here is a goofy face for you!",
"action_before_audio_ms": 1067.9,
"pcm_chunks": 5,
"rendered_kinds": [
"grimace"
]
}
],
"started": 1792966538.5741155,
"disabled_control_explained": true,
"synthetic_fault_injection": {
"speech_error_keeps_action": true,
"interrupt_clears": true
},
"errors": [],
"finished": 1792966579.4339569
},
"unit": {
"finished": 1792966414.724408,
"modules": [
"test_audio_input",
"test_asr_priority",
"test_gpu_guard",
"test_enrollment_cues",
"test_recovery_rules",
"test_stop_commands",
"test_live_decisions",
"test_laya_policy",
"test_conversation_prompt",
"test_conversation_director",
"test_shared_turn",
"test_speaker_names",
"test_speaker_session",
"test_speaker_turns",
"test_identity",
"test_speaker_teacher",
"test_bridge",
"test_performance",
"speaker_v2.test_predictor",
"speaker_v4.test_v4",
"speaker_v5.test_v5",
"speaker_live.test_playback",
"test_open_speakers",
"test_knowledge_tools",
"test_audio_transport",
"test_stop_feedback",
"test_stop_voice",
"test_release_scan",
"test_avatar_actions",
"test_avatar_commands",
"test_avatar_personality",
"test_bolt",
"test_pause_endpoint",
"test_visual_identity"
],
"tests": 365,
"failures": 0,
"errors": 0,
"skipped": 0,
"passed": true,
"network_or_physical_microphone_used": false
}
}Generated 2026-10-25 22:17 UTC.