How the final annotations were evaluated
| Criterion | Mean | 95% clip interval | Weight | How to read it |
|---|---|---|---|---|
| Weighted composite | 3.263 / 5 | 3.192–3.332 | 100% | Fixed weighted score, equal weight per recording. |
| Whole-recording caption fidelity | 3.82 / 5 | 3.70–3.92 | 15% | Does GENERAL describe the audible voice, performance and environment accurately, without invented demographic/identity/psychological claims? |
| Whole-recording emotion fit | 3.74 / 5 | 3.60–3.86 | 10% | Are the whole-utterance emotion descriptions supported by audible affect, including intensity and changes? Important omissions reduce usefulness; restrained descriptions are appropriate when affect is ambiguous. |
| Local speech emotion fit | 2.78 / 5 | 2.62–2.92 | 20% | Do emotion cues fit their particular spoken segments and intensity? Consider both false emotion tags and important missing affect. No emotion tag is not a claim of neutrality: do not penalize appropriate restraint on ambiguous segments. |
| Whole-recording talking-style fit | 3.86 / 5 | 3.76–3.94 | 15% | Do GENERAL's speaking-style claims match audible tempo, articulation, breathiness, pitch movement, register, emphasis and performance style? |
| Local speech delivery fit | 3.10 / 5 | 3.02–3.20 | 10% | Do local delivery/voice tags fit the corresponding spoken segment rather than merely repeating a global trait? |
| Burst-description fit | 3.02 / 5 | 2.86–3.16 | 10% | Does each free-form burst caption match the actual nonverbal vocal sound in its selected interval? Reward acoustic accuracy, not verbosity. |
| Burst-timing fit | 2.52 / 5 | 2.36–2.68 | 10% | Do absolute event starts/ends cover the audible events without unnecessary unrelated speech? |
| Event precision rating | 2.88 / 5 | 2.74–3.00 | 5% | Are proposed intervals real nonverbal vocal events rather than spoken syllables or spurious detections? This is a reviewer rating, not measured detection precision. |
| Event coverage rating | 3.46 / 5 | 3.32–3.60 | 5% | Are salient audible nonverbal events covered? This is a reviewer rating, not measured detection recall. |
| Transcript fidelity (diagnostic) | 4.30 / 5 | 4.14–4.46 | Diagnostic | Does the recognized word sequence match what is heard? This is diagnostic, not part of the weighted composite. |
| Speech-timing fit (diagnostic) | 3.42 / 5 | 3.28–3.56 | Diagnostic | Do speech budgets and pause intervals fit the audio? This is diagnostic, separate from mechanical formatting validation. |
| Readability (diagnostic) | 3.86 / 5 | 3.76–3.94 | Diagnostic | Is the annotation understandable, concise and useful? This is diagnostic, not part of the weighted composite. |
We asked an automated reviewer to listen to each original recording and check the final annotation against its sound.
What was evaluated
All 50 reused synthetic acting recordings were included: 45 English and 5 German. The emitted GENERAL/SCRIPT annotations contain 662 timed speech segments, derived from 443 source sentences, and 360 selected event intervals. The reviewer was given the full audio, final text, local cue details and exact segment/event start and end times. Intended TTS scripts, target emotions and reference captions were withheld.
The reviewer was Gemini 3.8 Flash, with standard parallel API calls, structured JSON, temperature 0 and a low thinking setting. The completed responses report gemini-3.8-flash. All 662 local segment IDs were checked for exact review coverage. The default configuration was frozen before these ratings and was not retuned using this evaluation.
The scale and the composite
Every rating is an integer from 0 to 5: 0 unusable/wrong; 1 major errors; 2 partly supported; 3 useful with notable errors; 4 accurate with minor errors; 5 fully correct and useful. The composite has fixed weights:
| Criterion | Weight | What it asks |
|---|---|---|
| Whole-recording caption fidelity | 15% | Does GENERAL match the audible voice and setting without invented demographic/identity claims? |
| Whole-recording emotion fit | 10% | Are the global affect and its intensity/change supported by the sound? |
| Local speech emotion fit | 20% | Do cues fit their own segment? Are salient emotions missed or fine-grained labels overclaimed? |
| Whole-recording talking-style fit | 15% | Do tempo, breathiness, register, articulation and delivery claims match the performance? |
| Local delivery fit | 10% | Do local style cues match that segment rather than repeating a generic trait? |
| Burst-description fit | 10% | Does the prose describe the selected nonverbal sound accurately? |
| Burst-timing fit | 10% | Do event starts and ends cover the sound without unrelated speech? |
| Event precision rating | 5% | Are selected events real vocal events rather than speech syllables or spurious proposals? |
| Event coverage rating | 5% | Are salient audible nonverbal events represented? |
Transcript fidelity, speech-timing fit and readability are separate diagnostics. Event precision/coverage are subjective ordinal ratings, not measured precision or recall rates. If no event is audible and none is proposed, event ratings are 5. A missing emotion cue is not a claim of neutrality: restraint on ambiguous speech is appropriate, while salient omissions reduce usefulness.
Aggregation and uncertainty
Recordings receive equal weight. We compute the weighted score for each recording and then average across the 50 recordings. Segment ratings are diagnostic; recordings with more segments do not dominate the composite. The segment-weighted diagnostic means (3.068 emotion / 3.384 delivery) are consequently different from the clip-level local means.
95% intervals use 10,000 bootstrap resamples of recordings with seed 20261011. They describe clip-sampling uncertainty, not bias or reliability of the single reviewer. Five German examples cannot establish German-language generalization.
Reading the results
The composite is 3.263/5. The stronger components are transcript fidelity (4.30), global talking style (3.86) and global acoustic caption fidelity (3.82). The weaker components are local emotion fit (2.78) and event timing (2.52). Reviewer notes flag implausibly specific emotion tags and redundant or overlapping proposals. Some true events overlap speech, so overlap alone is not an error.
The demo shows every output and the corresponding reviewer notes, including low-rated cases. Mechanical format checks pass for all 50 recordings, but those checks only establish word/event retention and timing arithmetic. They are independent of the audio-quality ratings.
This new rubric has different criteria and weights from the earlier MOSS/procedural comparison. Its composite must not be used as a direct numerical before/after comparison. Independent Humaneness Ears score benchmarks and MOSS validation token losses also measure different tasks.
Reproduce or audit
Download the complete summary, 50 responses and usage metadata, canonical annotations, format checks and rubric. Every review records the exact annotation hash and audio hash. Gemini's proposed audible-event list is reviewer output, not human ground truth.
The repository includes evaluation/review.py and evaluation/summarize.py. A private local manifest supplies each audio path and annotation path; it is deliberately not a public machine-specific manifest. To rerun, create a JSONL manifest with id, language, duration_s, audio_path and annotation_path, extract/download the example audio, and use your own API credential.
export GEMINI_API_KEY=YOUR_OWN_KEY python evaluation/review.py --manifest local_manifest.jsonl --out review_output --workers 4 --max-cost-usd 5 python evaluation/summarize.py --reviews review_output --out summary.json
The review code maintains a persistent budget ledger and reserves conservative costs before requests. Incomplete requests remain reserved for inspection instead of silently retrying billable calls. Responses can vary on rerun even at temperature 0.
Returned usage totals were 334,502 input and 49,577 output tokens, with no reported thinking tokens. At published rates of $0.75/M input and $3.75/M output on 11 October 2026, the estimate is $0.43679. These are list-price estimates, not an invoice; update rate arguments for future pricing.