Directing a speech model

A technical report on the MOSS voice-acting line: how a general-purpose text-to-speech base model was turned, over two full fine-tunes and roughly a thousand adapters, into a model that takes stage directions — and what happened when the project then tried to give it the two things a director actually gives an actor, timing and intensity. One of them is now solved to within 0.08 s. The other has resisted four training objectives, and the reason turns out to be measurable.

LAION · Christoph Schuhmann and collaborators · written 26 August 2026, expanded and re-verified 28 August 2026. The current research round runs on JUPITER (Jülich Supercomputing Centre), NVIDIA GH200, SLURM project reformo; the two full fine-tunes that produced the shipped model ran earlier, on an 8×A100 cluster. Two topics have their own pages: activation forensics and steering (§16) and the trajectory corpus (§17).

1. What the system is

MOSS Voice-Acting is a text-to-speech model that is directed rather than configured. Instead of exposing a handful of knobs — speed, pitch, an “emotion” enum — it takes a written stage direction in natural language, of the kind a director would give a voice actor, and performs the line accordingly. It speaks English and German, outputs 48 kHz audio, can clone a voice from a reference recording, and can produce non-verbal vocal bursts — sighs, gasps, chuckles, groans — inside a spoken line.

The production checkpoint is laion/moss-tts-local-transformer-4.55b-voice-acting-v2, 4.55 billion parameters. It did not appear in one step. §3 reconstructs the road: an off-the-shelf OpenMOSS base model, a full fine-tune on distilled voice-acting data plus a rank-256 adapter merged into it (“v1”), then a second six-dataset full fine-tune selected on generated audio rather than on loss (“v2”).

Around that model sits an ecosystem this report describes end to end:

• a family of LoRA adapters (small trainable weight deltas) for emotions, character archetypes, voice identities, timbre dimensions, vocal bursts and register — roughly a thousand of them across the project;
• a set of learned scoring models used as measurement instruments — they decide which of many generated takes is kept, and they are how every claim in this report is checked;
• a corpus of ~3.1 million annotated utterances, roughly 38 % of them generated by the model itself and filtered, the rest real recorded speech;
• and the current experimental line: a timed-script prompt format plus supervised fine-tuning, preference tuning and reinforcement learning aimed at one specific capability — making the model perform an emotion at a requested intensity.

The short version of the result. Timing control works. Ask for a sentence that takes 3.4 seconds followed by a 0.6-second pause and a 0.3-second sigh, and you get it — the median error on the total clip length is 0.08 s and essentially every clip lands within 0.5 s. Intensity control does not work yet. Ask for an emotion in the 90th-to-98th percentile of how strongly that emotion appears in the corpus, and the model delivers something around the 34th percentile. Four training objectives have been aimed at that gap. Three things have moved it, all of them by less than a hundredth: data — a small adapter trained on the most intense and most genuine 8 % of the corpus (§13.3); overdriving that adapter past its trained strength (§13.4); and a better-constructed contrast in the preference corpus, which as of 28 August produces the best model in the project and is the first preference tuning here that raises intensity without costing intelligibility (§15.4).

2. Terms, defined once

3. The base model and the two full fine-tunes: v1.5 → v1 → v2

The production model was not trained from scratch and did not arrive in one step. It is the end of a four-stage chain, and every stage constrains what the later ones could do. The chain is worth laying out before anything else, because the emotional-intensity ceiling that Part II keeps running into is partly inherited from it.

3.1 The base checkpoint and its representation

MOSS-TTS-Local-Transformer-v1.5 is an autoregressive local-transformer TTS model: a large semantic backbone predicts one hidden state per audio frame, and a small second transformer expands that hidden state into the codec tokens for the frame. It has three parts:

4.13 B of the 4.55 B parameters are trainable; the remainder is the frozen codec-side embedding. The audio tokenizer is MOSS-Audio-Tokenizer-v2: a residual vector-quantised codec with 12 codebooks of 1024 entries at 12.5 frames per second, decoding to native 48 kHz audio. Two consequences run through everything below:

• One frame is exactly 80 ms. A clip of duration d seconds is floor(d × 12.5) frames. This is the unit in which length is requested and in which length error is measured.
• Each generated position carries 13 channels (12 codebooks plus a binary stop head), so the model's negative log-likelihood sits at roughly 54 nats per position. Any loss weight that has to be commensurate with it — a DPO anchor, for instance — must be scaled accordingly; getting that wrong once turned a preference run into a plain supervised pass (§11.1).

The base model's prompt surface, which v1 and v2 both inherit, is two fields: instruction carrying a GENERAL: voice description and a SCRIPT: block with inline parenthesised delivery cues and [pause] markers, and text carrying the same spoken words plain. Recommended sampling with a reference clip: temperature 0.8–0.9, top-p 0.9, top-k 40–50.

3.2 Stage one — voice-acting v1

Two things happened here, and they are often conflated because the public artefact is their merge.

(a) A full-parameter fine-tune of v1.5 on DramaBox-style voice-acting data. “DramaBox-style” means the data is formatted as a structured director's instruction plus the spoken line, and it was filtered for vocal bursts and strong emotion. The checkpoint, with its checkpoint-cont-* continuations, lives in the private repository TTS-AGI/moss-dramabox-ft.

(b) A rank-256 voice-acting LoRA on top of that, α = 256, over all attention and MLP projections, trained on 1 million samples (the adapter repository's root is the 100 k checkpoint; checkpoint-1m/ is the full run), then merged into the weights. Training: 8×GPU DDP, bf16, gradient checkpointing, global batch 128, warmup 0.03 with linear decay.

The public release laion/moss-tts-local-transformer-4.55b-voice-acting is that merged model — “the voice-acting LoRA is already folded into the base fine-tune”, in the card's own words. Its card names four data sources, and this mix is the first place where the balance between synthetic and real material is set:

(c) A vocal-burst LoRA. Rank 256 again, trained on a 100 k inline-burst dataset and resumed from the 1 M voice-acting adapter rather than started fresh. Its training data (TTS-AGI/moss-inline-vocal-bursts, private) writes the bursts into both the instruction and the text, which is what teaches the model that a burst tag corresponds to an event in the audio. The merged result was called merged_inline, and a further all-parameter fine-tune was run from it. A separate library of per-class burst adapters — chuckle, breathy giggle, growl, fearful gasp, contented sigh, and so on — is published as laion/vocal-burst-lora-adapters.

3.3 Stage two — voice-acting v2, the six-dataset full fine-tune

This is the production model. It started on 2026-07-22 and ran roughly 14–15 hours on 8×A100-80GB — not on JUPITER. Its training scripts lived on an ephemeral RAM disk that has since been wiped; the surviving record is the run notes, the published repositories and the live results pages, which is why this section is reconstructed rather than quoted from a script.

Where the six datasets come from, and what they are. The prep_* half is prepared external material: prep_top3_neutral is the top-3 reward-selected takes of a neutral-delivery set, prep_emolia_elise is a cut of the Emolia German emotional-speech corpus, and prep_a1plus is an additional voice-acting collection (published tokenised, without per-clip scores, as TTS-AGI/additional-data-a1-plus, 31,306 rows). The our_* half is the model family's own generations, kept by a best-of-N reward: our_top3 the top-3 takes per prompt, our_emolia the Emolia-conditioned generations, our_adult the adult-register set. The Gemini-generated named-character material (TTS-AGI/gemini-adult-voices, 30 named characters × 1,000 clips) sits in the same family of synthetic sources.

3.4 Determinism and serving

Randomness lives entirely in the autoregressive token sampling (the torch RNG, fixed by a seed); the codec decoder is deterministic. The same seed therefore gives byte-identical audio despite a non-zero sampling temperature — verified at MAE 0.0 across two seeded generations. That is what makes the paired sweeps of §13.4 possible at all.

Serving throughput was characterised with SGLang-Omni on a JUPITER GH200 (aarch64): 27.7× realtime at concurrency 32 and 31.2× at concurrency 128 on a 5-second sweep, against 18.8× for the plain transformers baseline — a 1.47–1.66× speedup, saturating around concurrency 96–128 at roughly 6.4 clips/s. On 30-second clips it peaks at 85.3× realtime at concurrency 64. The lesson in that pair of numbers is to always quote a realtime factor together with the clip length it was measured on; the advertised ~35×-per-A100 figure was not reproduced locally.

3.5 The numerics that cost hours, recorded because they generalise

• v1.5 has no loss in forward(). It is a base model that returns only last_hidden_state (architecture MossTTSLocalModel). The channel-wise loss has to be computed externally, by re-projecting the hidden states through the local transformer and the text_lm_head / audio_lm_heads — the reference implementation is compute_supervised_loss_from_hidden in moss_tts_local_v1.5/finetuning/sft.py. Labels must be packed with the 13-channel MossTTSLocalV15SFTDataset, not the delay-pattern dataset of the sibling architecture.
• bf16 full fine-tuning NaNs unless AdamW's eps is raised to 1e-4. At the default 1e-8 the term underflows in bf16 and the loss is NaN after step 1.
• Gradient checkpointing must use the module-wrapper path — configure_gradient_checkpointing(scope="global", use_reentrant=False, impl="module_wrapper"), not model.gradient_checkpointing_enable(), which yielded NaN gradients.
• With a manual DataLoader the batch must be moved to the device explicitly. The global forward tolerates CPU input; the manual loss uses the labels directly and does not.
• Checkpoints are ~27 GB each (8 GB model + 16.5 GB bf16 optimizer state + 9 GB of save_pretrained for the best copy), so they were written to a 504 GB RAM-backed /tmp, keeping the last three and the best two.

Two of these — bf16 eps and the checkpointing path — are the same class of failure: a numerical default that is correct in fp32 and silently wrong in bf16. The sibling diffusion-model project in the same group hit the identical class from the other side, where bf16's unit in the last place exceeded a typical Adam update and training silently froze; the fix there was fp32 master weights. The transferable rule is that bf16 needs its optimizer numerics checked explicitly, not inherited.

4. The adapter ecosystem built on v1 and v2

Before the current research round, the project's main mechanism for control was adapters. Every one is a PEFT LoRA attached to the same frozen base model, and — this is what makes them stackable — they all declare the same 23 target-module patterns: the global Qwen3 stack's q, k, v, o, gate, up, down_proj; the local GPT-2 decoder's c_attn, c_proj, fc_in, fc_out; and all twelve audio_lm_heads.0…11. They differ only in rank and in what they were trained on.

4.1 Emotion LoRAs: three generations, and what each one taught

v1 — the rescue result (2026-07-26). Evolutionary search over natural-language prompts had failed outright to elicit Anger and Amusement from the base model. Training a per-emotion rank-64 LoRA on emotion-heavy audio rescued them. The ablation used the best checkpoint and the same emotional prompt in both arms, so the LoRA is the only variable (EmoNet z; higher = more of the emotion):

Rank 64 became the default (rank 128 overfits the ~1,000-clip buckets), and the best adapter was consistently an intermediate checkpoint — 20 epochs drives the loss to 0.03–0.4 and overfits. Speaker similarity was preserved at 0.41–0.58.

Overnight scale-up and the mixture study (2026-07-27). 34 emotions were trained on natural EmoLia top-1000 subsets. Two things came out. First, there is no single best rank: Anger and Fear peak at rank 128, Sadness at 64, Amusement only moves at all at rank 32. Second, and more consequential, a seven-way mixture study over three sources — D cross-folder DACVAE, E natural EmoLia, G LAION's Got Talent (high-emotion synthetic voice acting) — found that Got-Talent wins: pure Got-Talent is best for Fear (z = +2.62), Distress, Sadness and Anger; only Amusement preferred a three-way mix. High-emotion synthetic data became the primary emotion-LoRA training source. Clear intensity lifts at this stage: Amusement +1.29, Sadness +0.59, Anger +0.52, Fear +0.47 — with many emotions near zero, because natural speech is subtler.

v3 — 40 of 40, and the synthetic character dialled back. The shipped set, TTS-AGI/moss-emotion-loras-v3, is 40 adapters at rank 32, α 64, single-phase, trained against v2. Its distinguishing change is a cap: Got-Talent material is limited to ≤ 25 % per bucket, so at least ~60 % of every mix is natural or expressive non-Got-Talent material — EmoLia, DACVAE, the Gemini-generated adult-voice set, and the mitermix collection. The card states the reason plainly: this “removes the synthetic Got-Talent character that v2 had”. So the trajectory of this line is a full round trip — synthetic data was found to be the strongest single source, and then had to be capped because it stamped its own character on the output.

4.2 The full 40-emotion sweep: 11 responders and 29 that barely move

One adapter per emotion, best of ranks {16, 32, 64}, ~1,000–1,400 clips per emotion (natural EmoLia topped up with Got-Talent, and about 25 % of rows carrying a same-voice contrasting-emotion reference). The best adapter per emotion was chosen by a gated composite, z + 0.15·blend + 0.5·spk + 0.1·genu — note that the emotion term is gated and the quality terms are already there, which is the ancestor of every reward used later in this report.

Three findings from that sweep have kept their value:

• Rank can collapse an emotion. Emotional Numbness scores −1.68 at rank 32 while ranks 16 and 64 give +0.8 and +0.7. There is no universal rank; low rank (16) wins for narrow or data-scarce emotions.
• Half-merge often beats full merge on total reward — Fatigue, Sexual Lust, Jealousy, Helplessness and Bitterness all peak at λ ≈ 0.5. Only Sadness and Pleasure keep gaining to 100–150 %. Base-native emotions (Interest, Concentration, Impatience) are best prompt-only, with no adapter at all.
• An adapter buys less over a good prompt than it looks. Against the best prompt found by evolutionary search, the mean LoRA lift is only +0.30.

A reference-audio ablation on the same sweep is worth one line because it is so emotion-specific: attaching a same-voice, contrasting-emotion reference to ~50 % of rows helps Amusement enormously (+2.93 z, beating even the best reference-free mixture) and is mixed or small for everything else, with no improvement in speaker similarity. It is a per-emotion option at around 25 % of rows, not a default.

4.3 Character LoRAs, and the geometry of voice space

Voice space is a continuum, not a set of types. The 567 character reference voices, z-scored over their 57 VoiceNet dimensions and clustered at k = 12 and k = 18, give a silhouette score below 0.2 everywhere; k = 18 adds only about 5 % of variance and is less separated than k = 12. Twelve is a workable practical taxonomy; eighteen just subdivides the same families. The poles of the space are bright / high / feminine at one end and dark / low / gravelly / masculine at the other. An honest caveat travels with it: the 120 creature adapters skew masculine and monstrous, so the feminine and clean clusters get loose matches — the “High-Bright-Feminine” centroid maps to a guttural imp, which is a conceptual mismatch. Good matches: Playful-Feminine → elf, Full-Chesty → ogre, Silken → titan, Gravel → skeleton.

A cross-lingual reference does not clone a voice; it invents one. Over 1,280 generations conditioning German and English output on Japanese anime references, mean speaker similarity to the reference was 0.188 against an unrelated-speaker floor of 0.105 and a self-similarity of 1.000. Pitch tracks (r = 0.591) but compresses toward the model's own register (median F0 182 → 157 Hz). English transfers better than German (0.217 vs 0.158), and best-of-8 lifts it to 0.294. The recommendation is to use cross-lingual references to mine new character voices, never to clone — a limit that returns in §20 as a constraint on dubbing.

The corpora behind the character work

4.4 VoiceNet-dimension LoRAs

One rank-32 adapter per pole of each VoiceNet dimension, trained on v2. The published evolution study covers 86 dimensions across high and low poles. Δ₁₀₀ below is the measured shift at full dose:

Averaged over all of them, λ = 0.5 delivers about 36 % of the full-strength shift (mean in-direction shift +0.350 at λ = 1.0 against +0.157 at λ = 0.5) — so a half-strength merge buys a large fraction of the effect at much lower risk to speaker identity. That matches the search agent's independent canary finding that tempered doses of 0.4–0.5 preserve identity where 1.0 and above maximise the target and destroy it.

A practical trick from the manual that has no obvious analogue elsewhere: use VoiceNet adapters as stabilisers. Adding vn_CLRT_high @ 0.4 and vn_STRU_high @ 0.3 (plus vn_S_NARR_high for storytelling) protects intelligibility when a strong emotion or style adapter would otherwise push word error rate up; adding +0.5·ESTH to the reward biases toward “nice to listen to”.

Bucket feasibility, measured on game-voice data. Per-dimension buckets at three poles — high (≥ 5), mid (3–4), low (≤ 1), capped at 1,500 each — give 154 trainable buckets (≥ 200 clips) across 57 dimensions. Sixteen dimensions have no high bucket at all in game voices (S_TECH, VALN, S_NEWS, STRU, SMTH, R_ORAL, R_MASK, R_MIXD, R_HEAD, EXPL, DFLU, ESTH, COGL, CHNK, BRGT, BKGN) and need podcast or empathic-speech fill; EXPL has essentially no variation in that data at all.

4.5 Explicitness LoRAs

Six adapters, published both publicly and in a gated form requiring manual approval, an 18+ confirmation and name/email collection: TTS-AGI/moss-explicitness-loras. The set is aesthetic_mix_r32 (the recommended default), adult_r32, a1_r32, and raw_r16 / raw_r32 / raw_r64, of which raw_r32 gives the largest Sexual-Lust lift. The recommended recipe is the aesthetic adapter at 0.2–0.8 with the emotion adapter held at 0.35–0.75.

4.6 A negative result worth keeping: conditioning on 99 numeric scores

An attempt to bypass natural language entirely. Ninety-nine numbers — the 40 EmoNet emotions, the 57 VoiceNet dimensions, genuineness and vocal-burst blend — were each projected into the model's 2560-dimensional latent space by a tiny per-score projector, and the 99 resulting rows were spliced into the prompt between the backbone's <|vision_start|> and <|vision_end|> tokens, using the otherwise-unused <|image_pad|> token as a placeholder. A rank-128 LoRA and the 99 projectors were trained jointly, warm-started from an existing adapter.

The first thing measured was the ceiling: how much of a conditioning signal survives a round trip through the codec at all. Encode a clip, decode it, re-score it, and correlate:

The outcome of the experiment itself was negative: the projected-score conditioning path came out slightly below plain v2 on the two proxies in a head-to-head evaluation (genuineness −0.12, blend −0.44). Numeric conditioning is promising and did not beat text in this iteration. The correlation ceilings, though, are permanently useful, and the 0.823 for blend explains a recurring difficulty later.

4.7 Merging: what a dose actually does

Merge weight is an intensity dial, and it is roughly linear. Averaged over adapters and multiple seeds, λ = 0.5 recovers about 40 % of the full-strength gain, with no snap-on, no collapse and naturalness held. (With a fixed seed the same curve looks like a staircase rather than a smooth line — an artefact of sampling, not of the adapter.)

And it destroys speaker identity, monotonically. Measured as ECAPA speaker- embedding similarity to the reference clip, on cross-lingual Japanese → German/English generations:

Anchors from the same encoder: a reference against itself scores 1.000, two different speakers score 0.105. Published guidance: keep λ ≤ 0.5 if reference fidelity matters, 0.75–1.0 if only the emotion matters. Overdriving also truncates: in a controlled stacking run, one sentence and one seed produced 3.84 s with the voice adapter alone and 1.68 s — under half — with an emotion adapter at dose 1.9 stacked on top. An overdriven stack does not merely sound wrong; it stops early.

Difference vectors. Taking Effect = mean z at λ = 1.0 minus mean z at λ = 0, and recording for each measured dimension which adapter raises it most and which lowers it most, turns the adapter library into a set of directions rather than a set of labels. Highlights: the Amusement adapter raises Intoxication by +1.19 as well as Amusement by +1.80; the Sadness adapter is a broad negative lever, raising Embarrassment +1.08, Disappointment +0.85 and Fear +0.66; and subtracting Teasing (−1.70 on Concentration) is a “more concentrated” direction. Hope at −0.83 on Contentment behaves the same way.

4.8 Vocal-burst merging: the dose knee and a blind evaluator

An in-sentence merge-dose curve over 1,536 generations, 8 families × 64 classes:

λ = 0.75 gives 91 % of the full-dose burst presence at the peak of genuineness and a better blend; presence is bought against blend roughly one for one. Two exceptions: Sexual Lust peaks at 0.75 and then drops, Fear is best at 1.0. Five classes never fire inside a sentence at all — hiss, kissing, lip-smack, playful whistle, slurping. Rank barely matters (composite 0.416 / 0.424 / 0.427 for ranks 16 / 32 / 64).

The bare-burst prior is the opposite: in isolated-burst sweeps λ = 1.0 won every class. Do not carry that setting into sentence generation.

4.9 Contained emotion — a mechanism that works, and where it reverses

“Contained” means the emotion is present but held in, which is what most adult dramatic performance actually is. The mechanism that works is specific: keep the emotion and drop vulnerability — a light vn_VULN_low at 0.06–0.18 plus a parenthetical restraint cue, over a character-adapter scaffold with the emotion at about 0.4. What does not work: turning the emotion down, adding tension, or applying heavy Emotional-Numbness or vn_TENS_high — all of those armour the delivery into a news-anchor flatness.

A deep paired re-run over 1,878 clips measured ΔVULN as the suppressed cohort's mean minus the free cohort's; negative means the masking worked:

The rule that falls out: masking works when the audible signal is effort, arousal or social surface and vulnerability is incidental; it fails or reverses when vulnerability is the emotion's core — for Sadness, Pride, Distress and Affection, ship the free take. This result overturned an earlier “moderate contains, intense flattens” story that turned out to be an artefact of four anchors and sparse unpaired data.

4.10 “Text carries the condition”

A large ablation — 106 growing to 124 arms, up to 3,968 candidates — found that several apparently “vocal” dimensions are in fact properties of the written line, and no adapter moves them.

• Disfluency is a text property. Adapter-alone levels are indistinguishable (0.27 / 0.44 / 0.71 / 0.71). Writing hesitation into the line — ellipsis, filled pauses, a false start, and a cue naming it — ladders DFLU from 0.44 to 2.07, at a cost of +0.176 WER. Ellipses drive the dimension harder (span 1.54 vs 1.07) while [pause] tags are gentler on intelligibility (0.122 vs 0.238 WER) — so ship both. [pause] and [long pause] were verified not to be spoken aloud.
• Intoxication headline: writing the state into the text scales measured emotion strength by ×8.5 (0.068 → 0.582) and genuineness by +40 %, while doubling WER.
• Explicitness has two conflated targets. The EXPL register dimension is essentially not adapter-drivable (ladder span −0.006), and vulgar text gives only a modest span (0.26–0.43). The explicitness LoRA in the adult-intimacy sense is real: adult text plus inline burst tags — without a burst adapter — lifts blend from 1.72 to 6.77; adding a burst adapter lowered it.
• Chunking is a clean negative. Not text-drivable this way; every recipe scored at or below baseline at about 2.4× the WER.
• Language dependence is real. Heavy broken-speech treatment on the most-broken emotions gives +0.703 emotion and +0.637 genuineness; the moderate version is a negative result in English (−0.224) and a positive one in German (+0.113). Decide per language.

4.11 Notation, ranking and the reward form

Three further ranking decisions were each measured on 28,698 stored candidates:

• Rank on raw scores, not z-scored within the group. Z-scoring normalises blend inside each group, so a group whose takes are all badly blended still hands its best one a good score. Going raw lifted the median winner's blend from 3.65 to 4.67 (+28 %).
• Multiply by (1 − WER); never divide by (1 + WER). The division form is too lenient and produced more runaway generations (18 vs 14). Worse, since most candidates have a negative core score, dividing a negative number by a larger denominator makes it larger — so the division form actively rewards transcription errors. The real fix for runaways is a length-fit ceiling: expected duration = words / 2.8, free up to 1.6×, quadratic falloff after; runaways 14 → 7 at a ~15 % genuineness cost.
• Character-matching groups zero the emotion term entirely: reward = (genu + blend) × (1 − WER) × identity × length-fit.

The production matrix at the end of this line was 169 distinct adapter keys — 13 character, 40 emotion, 114 VoiceNet, plus explicitness and sports.

4.12 Best-of-N economics, and why ASR is the swappable cost

The 16 → 32 step is worth +4.46 % reward, −8.95 % duration error and +0.254 blend; 32 → 64 costs 3.4× for the same increment. Crucially, generation time is flat from batch 16 to 32 (20.34 → 20.54 s on a GH200) because the decode loop is latency-bound, so a batch of 16 wastes half the GPU. N = 32 is the operating point, and peak VRAM there is about 23 GB.

The dominant cost is then the speech recogniser, and it is swappable: Parakeet-v3 runs at 52× realtime and is 12 % of a 32-candidate group's cost; Whisper-turbo 12.7× and 32 %; CrisperWhisper 3.1× and 66 %. Rank on Parakeet and re-transcribe only the top three with CrisperWhisper — on a 40 × 3,000 production run that took the bill from 2,952 to 1,148 GPU-hours.

Two more rules from the same body of work, both of which recur later: rank whole assemblies, not parts — across 108 rounds the top-ranked assembly used the best candidate of every part in only 11 rounds (10.2 %) — and n = 8 per cell is noise; use n ≥ 30 with the base model in the same batch.

4.13 What building all of them taught about rank, dose and stacking

Rank is not where the quality is

The voice-profile pilot trained each of ten voices at ranks 16, 8 and 4 in a single process against one shared frozen base, stepping on the same micro-batches with the same seed — so rank is the only difference between the three adapters. On 1,920 held-out clips per arm:

The base→rank-4 effect is about ninety times the rank-16→rank-4 gap, and the inter-rank gaps are indistinguishable from zero on every axis measured. Nine of the ten voices ship at rank 4 — 8.6 M trainable parameters. (Speaker similarity here is cosine distance between ECAPA embeddings from speechbrain/spkrec-ecapa-voxceleb.)

Dose is not what carries the result

Across the acting corpus, emotion-adapter dose correlates −0.226 with the measured emotion peak and +0.084 with genuineness. In the manual's words: “you cannot buy a turn by raising λ.” 98.6 % of the 492 doses actually used across the corpus sit in the 0.35–0.75 band. What carries the result is the written line (§4.10).

Only 22 of 40 emotion adapters beat a good prompt

The manual's own emotion index reports that only 22 of the 40 emotion LoRAs produce a positive lift over a well-written prompt with no adapter. Biggest wins: Sexual Lust +0.25, Fatigue +0.21, Helplessness +0.18, Jealousy and Envy +0.17. Biggest losses: Distress −0.18, Concentration −0.12, Anger −0.12, Relief −0.11, Disappointment −0.10. Eight emotions — Interest, Concentration, Impatience and Irritability, Amusement, Hope, Doubt, Disgust, Pride — are best served by a neutral prompt and no adapter at all.

Stacking is a real engineering problem

Because every family targets the same 23 modules, a voice adapter and an emotion adapter are always fighting over the same weights, and at equal λ the larger-rank adapter dominates. Voice adapters are rank 4; emotion, burst, VoiceNet and character adapters are rank 32. That asymmetry — not any target-module difference — is why every non-voice λ is below 1 unless that adapter is the condition.

Two library-level traps are documented with their symptoms, and both killed multi-hour runs: PeftModel.active_adapter is a plain attribute that base_model.set_adapter() does not update, so it stays pinned to the first adapter ever loaded and raises KeyError forever once that adapter is evicted; and a snapshot of the trained scalings goes stale the moment another adapter is loaded, so a dose computation must consult both dictionaries and record the trained value on first sight. A third surfaced today and is recorded in §14.

5. Five hundred synthetic voice profiles

A large part of the training material for the current round is not recorded speech but voice profiles: synthetic speakers, each generated by the v2 model with the adapters of §4, each exercised across the whole expressive range — 40 emotions, 57 VoiceNet dimensions, acting edge cases, character clusters, vocal bursts, English and German — and each filtered by the scoring models before anything was kept.

The selection rule that produced them is the direct ancestor of the reward used in Part II:

reward = (genuineness + w_blend·blend + 1.25·target) × (1 − WER)
score  = reward × durationFloor × lengthCeiling × identityRank × floorPenalty

Three of its design choices were each measured on 28,698 stored candidates, and each is a small lesson (the first two are §4.11 seen from the data side):

• Raw scores, not z-scored within a group — median winner's blend 3.65 → 4.74 (+30 %), at a cost of −15 % genuineness.
• Multiply by (1 − WER), never divide by (1 + WER). On a 1,000-clip control set a literal WER × quality filter put 602 of 1,000 candidates at exactly 0 and then selected the half with worse word error rate.
• Identity is ranked at 19 % weight, never gated high. Among listener-confirmed same-speaker pairs the median ECAPA similarity was 0.632 and 55 % fell below a 0.68 gate — gating there would have discarded most genuinely correct takes.

The largest published artefact from this stage is laion/laion-voice-profiles-annotated — 28,212,933 rows, 5.8 TB.

6. The measurement instruments, and how they were trained

Nothing in this project can be evaluated by listening at the scale at which it is generated — a single GRPO run produces 1,400–1,700 fully scored clips per hour. So the project runs on a set of learned scoring models. They do three different jobs, and it is worth keeping those apart: selection (which of 32 or 64 candidate takes to keep), reward (what an RL objective maximises) and evaluation (the numbers in this report). The same models do all three, which is a known circularity and is discussed in §6.8.

6.1 VoiceCLAP — the shared audio embedding

VoiceCLAP is a CLAP-style (Contrastive Language–Audio Pretraining) dual-tower model: an audio encoder and a text encoder trained so that a clip and a sentence describing how it sounds land close together in one shared vector space. The workhorse here is laion/voiceclap-commercial:

A controlled ablation reported on the card shows that removing the non-commercial data costs nothing — the commercial-only model wins 4 of 5 benchmarks against the full-mix model, trailing only on fine-grained intensity ranking of synthetic audio. Two recovery experiments were tried and discarded: adding VoxCeleb1/2 hurt, and upweighting the safe corpora was flat.

A larger sibling exists, laion/voiceclap-large-v2 — a rank-16 LoRA (α 32) on the 7 B LCO-Embedding-Omni-7B (Qwen2.5-Omni thinker), producing a 3584-d embedding and setting the better numbers on the human-annotated VoiceNet benchmark (emotion balanced accuracy 0.7069 against voiceclap-small's 0.6754). It is not what the pipeline runs on, because the 768-d commercial model comes within a hair of it on the two heads that matter at 1/4.7 the embedding width.

The architectural fact that matters most here is that one VoiceCLAP encode feeds three separate instruments. In the production scorer (pp_scores_fast.FastScorer) a clip is encoded exactly twice in total — once by VoiceCLAP-commercial and once by BUD-E-Whisper — and every downstream head reads one of those two vectors.

6.2 Genuineness: does it sound like a person or like a reading?

There are two generations of this predictor, and conflating them is easy, so both are described.

Generation 1 — the Empathic-Insight “Authenticity” head

The original genuineness signal is one of the expert heads in the Empathic-Insight-Voice suite: the repository ships 55 checkpoints, of which 40 are emotions and 15 are demographic/vocal/psychological attributes — and Authenticity and Arousal are two of those 15. It is trained exactly like an emotion expert: a BUD-E-Whisper encoder is fine-tuned once and then frozen, and a single MLP is trained on its flattened sequence embedding. This is the head occupying slot 97 of the project's 99-dimensional scorer (va_rescore.NinetyNineScorer), and it is what filtered the data for the v1/v2-era emotion buckets. Its published private siblings are laion/genuineness and laion/vocal-burst-blend.

One number from that generation is worth carrying forward: inter-rater agreement on authenticity is very low — Cronbach's α ≈ 0.185 on n = 300, far weaker than the emotion labels. The head is modelling a genuinely subjective target. That is precisely why genuineness is down-weighted (≈ 0.1 in the selection score of §4.2) and used as a gate rather than as a primary objective.

Generation 2 — the VoiceCLAP head the current pipeline uses

laion/voiceclap-commercial-genuineness predicts a single continuous score in [0, 6] for how much a clip sounds like a real, lived-in spoken moment rather than a rehearsed or synthetic read. Anchors from its own card: 0–2 reads as rehearsed, flat or synthetic; 3–4 plausibly natural; 5–6 sounds like a genuine, unscripted moment. It explicitly is not about audio fidelity — it is about natural timing, breath, micro-imperfections and emotional grounding.

6.3 Vocal-burst blend: does the sigh belong to the sentence?

laion/voiceclap-commercial-vocalburst-blend predicts, on a 0–10 scale, how naturally a non-verbal burst blends into the speech around it. Anchors from its card: 0 = disconnected — spliced or pasted in, robotic, or the wrong emotion for the context, and also assigned to clips with no genuine burst at all; 5 = fits the context but sounds performed or stagey; 10 = totally organic, indistinguishable from a natural human reaction. Intermediate values interpolate; output is clamped to [0, 10].

Architecturally it is the same shape as the genuineness head — frozen VoiceCLAP-commercial 768-d → L2-normalise → standardise → Linear(768,50) → GELU → Dropout(0.2) → Linear(50,1), about 38 k parameters — which is why one encode yields both scores. Its card publishes a comparison at equal head architecture across three embedders:

These two heads are why the project's data selection is not naive. As §13.1 shows, the most intense recordings in the corpus are on average the least genuine, so selecting on intensity alone selects for overacting. Genuineness and blend are the gate that stops that.

6.4 Empathic-Insight-Voice-Plus: 40 emotion heads and 4 quality heads

laion/Empathic-Insight-Voice-Plus is not one model but a suite of independent single-output MLP regression experts on top of a frozen BUD-E-Whisper encoder (a fine-tuned Whisper-Small producing a 1500 × 768 sequence).

• Emotion / attribute experts (from Empathic-Insight-Voice-Small, 55 checkpoints = 40 emotions + 15 attributes): flatten the whole sequence and project it down — Flatten(1,152,000) → Linear(…,64) → ReLU → Dropout → Linear(64,64) → … → Linear(16,1), about 73.7 M parameters each, dominated by that first projection. Trained on LAION's Got Talent (~5,000 h of multilingual synthetic voice acting) plus an in-the-wild voice-snippet collection of similar size. The taxonomy and benchmark come from the EmoNet-Voice work.
• Quality experts (the 4 that -Plus adds): pooled features instead of the full sequence — mean + min + max + std over the sequence dimension, 3072-d — and therefore only ~200 k parameters each. Three are distilled from Microsoft's DNSMOS (overall quality, speech quality, background quality) and one from Meta's AudioBox content-enjoyment model. Validation on 200 samples each: overall MAE 0.26 / r 0.899; speech 0.30 / 0.517; background 0.35 / 0.865; content enjoyment 0.34 / 0.691.

The production scorer loads 44 heads — the 40 emotions and the 4 quality experts — and leaves the attribute heads unloaded, because the VoiceNet regressor (§6.5) covers those dimensions. Emotion outputs are unipolar, nominally 0 → 4 (0 = the emotion is absent, 4+ = extreme manifestation, card maximum 4.5), but the heads are unbounded regressors and real corpus values overshoot — the project's own normalisation statistics record a maximum of 8.20 on the Anger head.

The 40 emotions, verified against the repository's file listing, are the axis along which everything in this report is requested and measured:

Affection · Amusement · Anger · Astonishment/Surprise · Awe · Bitterness · Concentration · Confusion · Contemplation · Contempt · Contentment · Disappointment · Disgust · Distress · Doubt · Elation · Embarrassment · Emotional Numbness · Fatigue/Exhaustion · Fear · Helplessness · Hope/Enthusiasm/Optimism · Impatience and Irritability · Infatuation · Interest · Intoxication/Altered States of Consciousness · Jealousy & Envy · Longing · Malevolence/Malice · Pain · Pleasure/Ecstasy · Pride · Relief · Sadness · Sexual Lust · Shame · Sourness · Teasing · Thankfulness/Gratitude · Triumph

That earlier defect is worth repeating as its own warning: the emotion list originally had 39 entries — Jealousy_&_Envy was simply missing, so its head was never loaded and that emotion could never be predicted. Because heads are keyed by name and not by index, the other 39 were still correctly labelled. It was a missing emotion, not a misaligned one, and only a count caught it. The comment recording that fix is still in the scorer source, three lines above the string that now causes the lookup to fail.

6.5 VoiceNet: 57 dimensions of how a voice sounds

laion/voicenet-dimension-predictors-commercial is a bank of 114 small MLP heads — 57 dimensions × 2 (a regression head and a classification head) — on the same frozen 768-d VoiceCLAP-commercial embedding. One encode yields all 57 dimensions. Outputs are ordinal levels, 0–6 for most dimensions, 0–4 for background noise and 0–2 for content appropriateness.

How it was trained. Labels are laion/emolia-voicenet-gemini-annotations — Emolia clips each re-scored on a single VoiceNet dimension by Gemini-3.5-flash, combining three annotation rounds (round 3 topped up the most data-starved score levels of the hardest dimensions). Each unique clip is encoded once. For every dimension three class-balancing variants are trained (buckets capped at 3× / 4× / 5× the second-smallest usable bucket), and the shipped default is chosen per dimension on a frozen validation split — lowest MAE for the regression head, highest accuracy for the classification head, chosen independently. Regression uses Huber (δ 1.5), H = 64, dropout 0.33; classification uses cross-entropy, H = 48, dropout 0.33; Adam, 50 epochs, best-val checkpoint.

Best-per-dimension results: mean regression MAE 0.744, mean Pearson 0.79, mean within-one-level accuracy 0.870, mean classification accuracy 0.617. Per-dimension quality varies a great deal — head resonance reaches MAE 0.478 with r = 0.88, while cognitive load sits at 1.052.

Two polarity traps, and a bug they caused. GEND runs feminine at 0 to masculine at 6, and BKGN runs noisy at 0 to clean at 4 — higher means less noise. Both are counter-intuitive, and both were inverted in an earlier version of the procedural caption generator, so that corpus-wide the prose descriptions of gender and background noise ran backwards while the numeric columns were correct throughout. The fix was verified end to end on the shipped strings rather than on the ladder tables: across 40 sampled voices, zero captions contradict the voice's measured gender class.

6.6 The burst locator and the burst detector

Bursts are handled by two models in sequence, because “where is it” and “what is it” are different problems.

laion/vocalburst-locator does frame-level binary segmentation at 20 ms resolution: a Whisper-small encoder adapted with a rank-8 (α 16) LoRA on q_proj/v_proj and then merged, followed by Linear(768→384) → GELU → Conv1d(384, kernel 7) → Linear(384→1) → sigmoid, giving 1500 per-frame probabilities over a 30-second window, post-processed into (start, end, confidence) events. Event F1 at IoU 0.5 on 992 real in-the-wild expressive-speech clips: 0.607 for v2 against 0.152 for v1 — and v1 fails in a specific way, reaching usable precision only at threshold 0.80 where recall collapses to 0.21.

laion/vocal-burst-detector-v2 then classifies each located span: an MLP (Linear(768,256) → BatchNorm → GELU → Dropout(0.3) → Linear(256,83), ~219 k parameters) on the VoiceCLAP-commercial embedding of the cut, over an 83-class taxonomy — 82 vocalisations plus no_burst at index 82. Argmax accuracy on the official 1,940-clip validation set is 58.1 % overall: 70.4 % on clips with a single pure burst, 39.4 % on composites. Five classes that are hand or body impacts rather than vocalisations — blowing a kiss, finger snaps, hand scratching head, hand slaps, slap face — are folded into no_burst and masked at inference, so the recognised label space is 77 burst classes plus no_burst. A burst is additionally vetoed when P(no_burst) ≥ 0.50.

The project's reward pipeline runs the locator at threshold 0.5, merge gap 0.2 s, minimum duration 0.08 s, pads each span by 0.05 s before classification, and drops anything under 0.10 s (LOC_THR, LOC_GAP, LOC_MINDUR = 0.5, 0.2, 0.08; BURST_PAD, MIN_BURST, NOBURST_GATE = 0.05, 0.10, 0.50). The card's own recommendation is slightly tighter — merge gap 0.10 and minimum duration 0.10 — and the difference has not been ablated.

6.7 Turning raw scores into a scale you can ask for

Everything above produces numbers on incomparable scales. Measured on the project corpus, the Interest head has a median of 2.082 and never outputs zero, while the Infatuation head has a median of −0.017 and outputs zero on 87.7 % of clips. A reward built on raw values would have been an Interest detector.

The answer is the corpus ECDF: every raw score is mapped to its percentile within the full 3,147,802-row corpus, per head. Four intensity bands are defined on that percentile scale, and — this is the point — the same four cutoffs are used by the prompt builder, by the reward function and by the evaluation harness, so what the prompt asks for is exactly what gets measured.

The zero-point defect, measured 25 August 2026

A plain ECDF percentile turned out to be still not comparable across the 40 heads, for a reason that is obvious in retrospect. A clip with a raw score of exactly zero — no measurable emotion at all — lands at a percentile that depends entirely on how often that head outputs zero:

Asking for band 0.90–0.98 was therefore a different task for every emotion: on Awe the model starts 0.08 below the target having produced nothing, while on Affection it starts at zero. Averaged over 40 heads, the reported figure was measuring where each head's zero point happens to fall, not how well the emotion was performed.

The fix is a per-head affine rescale of the tail above the zero point, p' = max(0, (p − p₀) / (1 − p₀)) where p₀ is the percentile of a raw score of zero. It was verified to map raw 0 to exactly 0 on all 40 heads while remaining monotone within each head — so every comparison between two clips of the same emotion, which is what DPO pairs and GRPO group rankings consist of, is unchanged. Only absolute targets and cross-head averages move, which is precisely what was broken.

6.8 How far these instruments can be trusted

This is the section that ought to be read before any number below is quoted elsewhere.

• No human listening study has been run. Not on the corpus, not on the adapters, not on any model comparison in this report.
• The project's own estimate of agreement between these learned scorers and human listeners on spoken material is about ρ = +0.21. That is a weak positive correlation. It is enough to rank two arms of the same experiment; it is not enough to certify quality.
• The findings protocol records seven separate occasions on which an automatic metric or a job status reported a success that did not exist, each found only by checking something the metric did not cover. Two examples: an aesthetics head scored an arm at 4.12 while human-facing judges scored the same audio 0 out of 30; a page reporting 89 groups as “improved” was serving audio from a different run entirely, because the audio key was not unique across runs.
• Every head above is trained on labels produced by another model — Gemini-3.1-Pro for genuineness, Gemini-3.5-flash for the VoiceNet dimensions, DNSMOS and AudioBox for the quality experts, forced alignment for burst timing. The genuineness card says it plainly: treat scores as a soft ranking signal, not ground truth.

Two mitigations have actually worked here and are recommended rather than merely stated: always run a control arm that isolates the claimed cause, and verify identity from an independent namespace — never infer “this file is that record” from a shared key.

7. The prompting model and the voice-acting manual

The shipped v2 model is documented in a public manual: MOSS Voice-Acting — Manual & Studies Hub. It is 25 chapters: a hub, an emotion-conditioning manual with one page per emotion, twelve recipe chapters, and an eight-page tree recording an autonomous casting agent's four generations of runs. It is unusually forthright — a large fraction of it is a record of things that did not work, with the measurements attached.

7.1 The prompt surface the model ships with

Two fields, exactly:

instruction =
GENERAL: <the standing description of the voice and the situation>
SCRIPT:
(delivery cue) "the spoken line, (burst) with tags inline, [pause] and pauses in squares"

text =
<exactly the same spoken words, plain — no cues, no brackets>

The words appear in both fields — plain in text, cue-annotated inside SCRIPT: — and that redundancy is deliberate and matches the training data. The manual's notation rules, verbatim:

1. Round brackets carry delivery cues and vocal bursts: (quietly, with lethal control), (screams). This is the only inline notation the model was trained on.
2. Square brackets carry pauses: [pause], [long pause]. Never put one directly after a burst — that is the truncation shape.
3. Never capitalise a tag. (SCREAM) is read out as “ess see arr ee ay em”. Lower-case everything inside brackets.
4. Give the line enough words. At ~2.8 words/s a line under 8 words cannot fill three seconds. Ten words is the working minimum.

A worked example, copied verbatim from the manual's casting-agent prompt archive (adapters Scream @ 0.9, Anger @ 0.4, Distress @ 0.35):

GENERAL: Start wounded and breathless, then rebuild into nearly screaming but controlled
rage. The final words should land cold and absolute, with the voice hardened by pain. Place
the vocal burst inline and continue speaking immediately afterward.
SCRIPT:
(a broken breath turning into a controlled eruption) "Don't you dare touch me." (Scream)
"You don't get to cry now. You don't get to make this about your guilt. Pack your things,
and get out before I say something neither of us can take back."

There are no duration tags in the shipped prompt format. Length is controlled only indirectly, through the tokens field (target frames, ≈ words × 6) and max_new_frames. The manual gives a specific warning for non-verbal parts: tokens defaults to the number of words, and a scream's text has almost none, so the model is told to make ~5 tokens and stops after 0.3 s — for non-verbal material you must set about 12.5 tokens per intended second by hand.

This is exactly the gap the current research round closes. Everything from §9 onward is about training a prompt surface in which duration is stated explicitly, per sentence, per pause and per burst.

7.2 How intensity is requested, and why the manual says the prose does the work

Intensity in the shipped format is not a tag and not a number. It is requested three ways, and the manual is emphatic about their ranking:

1. The written line and the parenthetical cue. The manual's own chapter title is “Genuineness comes from the writing, not from the dose”. The rule for a cue is: make it an action the character performs, not a description of a tone. “The sentence collapses halfway and they have to start it again” is a direction; “sad, resigned” is a label, and a label is an invitation to narrate.
2. The adapter dose λ, which as §4.13 showed correlates negatively with the measured emotion peak.
3. A declared 2 × 2 condition — {moderate, intense} × {free, suppressed} — used in the contained-emotion chapter (§4.9).

One finding here anticipates the whole of §12: a maximise objective cannot produce a moderate emotion. The manual states it directly — “a maximize objective always drifts to intense; a believable moderate/subtle emotion scores low against ‘maximize’ and dies out of the population.” The fix used there is a target-band fitness function, which is precisely the shape the current reward uses.

The manual also independently reports the cross-head incomparability of §6.7, in its own units: “Anger/Sadness/Fear reach ~2.4–3.7 on the detector; Amusement tops out ~1.7 even when strong — do not compare raw emotion scores across emotions; compare each against its own free take.”

7.3 Best-of-N is not optional, and it is nearly free

Because emotion strength, burst placement and prosody are seed-dependent, the recommended way to use this model is to generate many candidates and rank them. The economics are in §4.12; the practical consequences the manual adds are:

• For a mid-sentence burst specifically, a burst appears in roughly 50 % of takes, so five candidates give about a 97 % chance that at least one has it.
• Rank whole assemblies, not parts (10.2 % agreement between the best assembly and the part-wise best).
• At scale, transcribe for ranking with a fast ASR model and re-transcribe only the top three with the accurate one.

8. The v2 agent's reliability limits, stated plainly

The manual documents an agent built on the v2 model: it writes a ~30-second performance as 2–4 parts, generates each best-of-16, assembles the best combination across parts, ranks with the local scorer stack, and has a supervisor listen afterwards and give feedback for the next round. Seven generations of it were run — four complete, three aborted — over nine challenges, roughly 2,000 generations. A separate autonomous LoRA-plus-prompt search agent (LAION-AI/voice-acting-search-agent) ran an overnight programme of 163 agent generations and ~10 k scored takes on a single A100; its notable finding is about the brain, not the speech model — a Gemma-4-31B QAT policy won decisively on single-dimension missions using only 22 tool calls, while a Gemini supervisor was an effective active director (tracking the objective at r = 0.72) and a local audio judge was usable only as a lenient score-only gate, because active steering by it was harmful.

That agent can merge the LoRAs, and the audio it produces is more emotional than what the current SFT/DPO line produces. It is also, on the project's own measurements, not reliable on character consistency. This section states the limits because they are the honest state of the system and the manual documents every one of them.

8.1 The voice drifts

• Without an explicit continuity mechanism, every part is a different actor. Verbatim: “If you omit it, every part is an independent draw of an unspecified speaker. The model has no reason to pick the same voice twice, and it does not. A three-part assembly is then three different actors, and no amount of crossfading hides it.”
• Reference conditioning alone is not enough. Later parts conditioned only on a reference clip reached mean similarity 0.625, with 69 % below 0.75 and 23 % below 0.55. It keeps timbre roughly right but starts a fresh utterance with its own level and attack, heard as a cut.
• Naive chaining collapses identity. Continuing from the full previous part gave a minimum similarity of 0.280 across an assembly; continuing from an anchor plus a 4-second tail gave 0.835. Two causes are named: drift compounds, and a long prefix drags the old emotion forward (peak emotion fell 2.29 → 2.05 under full chaining).
• Ranking does not fix it; only excluding does. “Ranking the best of one batch is not enough: if a whole batch drifts, the least-bad take still ships, which is how a listener ends up hearing a second actor mid-scene.” And: “A penalty is not a floor. Multiplying a sub-floor take's score by 0.25 still let an excellent-sounding take from below the floor win. Exclude them.” The sequence score used is ½ mean + ½ minimum, so one bad part cannot be averaged away.
• Every remaining identity failure in the corpus is a non-verbal clip. Spoken clips: mean similarity 0.816, 0 % below 0.55. Non-verbal clips: 0.610, 38 % below 0.55.
• Cross-lingual reference conditioning does not clone at all (§4.3): mean 0.188 against a 0.105 unrelated-speaker floor.

And the thresholds themselves are wrong when set high. The casting playbook originally used similarity gates of 0.82 / 0.75 / 0.68. The voice-consistency chapter overrides them: regenerate below 0.58, repair with voice conversion below 0.45, reject below 0.40 — and calls that correction “the single most important thing on this page”. The evidence: among 170 human/model-confirmed same-speaker pairs the median ECAPA similarity is 0.632, 55 % fall below 0.68 and 84 % below 0.82, while the highest score reached by a genuinely different speaker was 0.816. Four embedding models were tried (ECAPA AUC 0.847, VoiceNet 0.814, WavLM timbre 0.794, VoiceCLAP 0.770) and none gives a clean threshold. The conclusion: ranking on identity is fine, gating on it is not — and never raise the floor “to be safe”, because the failure that creates (flat, over-clean delivery) is less visible than a voice change but affects every take rather than a few.

8.2 Under strong emotion, output goes strange

• High emotion dose degrades everything else. For Anger at a high dose: “blend craters from 0.446 to 0.129, quality from 0.552 to 0.261, and WER more than doubles (0.202 to 0.570) … it drags in Malevolence and Contempt while stripping Warmth (−0.48) and articulation clarity, so the anger sounds cruel and slurred rather than heated.”
• Pushing one dimension to its extreme garbles words. “Round 1 failed because ‘maximum volume’ made the model garble.” The mitigation is a safety clause in the prompt: “roaring but the words still land”, “whispered but every word clear”.
• The truncation wall. A named failure mode: the model produces the burst and then emits end-of-sequence, regardless of frame budget or prompt. A silence beat straight after a burst triggers it reliably. Six prompting strategies were tried across 1,152 generations to instruct the model to continue; every one scored lower than the plain inline tag. The conclusion: “you cannot prompt your way out of the truncation. Lower the dose and select over candidates.” And burst-presence alone cannot see this failure — a take that produces a perfect gasp and then stops scores presence 1.0.
• Capitalised tags are spelled out. (SCREAM) becomes “ess-see-arr-ee-ay-em”. The corpus's single clip whose entire purpose was to be a scream transcribed as “Age.” / “Atch! Itch!” / “Akish.”
• The narration failure. The commonest continuation failure is subtle: the voice stays correct but the model stops acting and simply reads the remaining words out. It is measured as narration = ½(S_NARR + S_NEWS) − ½(S_DRAM + S_CONV).
• The dials saturate. Asked for tempo levels 1/2/3/4, the model produced 2.97 / 2.05 / 2.79 / 2.82 — the real range is about 2 to 3, and asking for 4 buys nothing over asking for 3.
• One take holds one emotion. A single generation conditioned on one caption does not turn from composure to collapse; the turn has to happen between generations.
• Whole batches sometimes decode to empty audio, and continuing from an empty clip poisons the next part (0.367 similarity downstream). Empty decodes are a normal silent failure of this model, not an exception — production code must assert on the decoded output.

8.3 And the local reward does not predict a listener

“Short answer: on spoken scenes, a little — ρ ≈ 0.21, about 4 % of the variance. On non-verbal scenes, nothing, and the sign is wrong. There is currently no configuration in which the local stack can replace the listener.”

Two of the reward's own components point the wrong way against listener preference — genuineness at −0.143 and blend at −0.148; only emotion peak (+0.179) is useful. An optimally weighted linear combination of everything the local stack measures explains under 3 % of a listener's within-round preference. The manual's conclusion is that re-weighting cannot fix this, because the information is not in the feature set.

Two further findings sharpen it. The casting score in use was measured to correlate +0.91 to +0.98 with its own (1 − WER) multiplier and +0.00 to +0.04 with genuineness — it ranks assemblies by intelligibility and speaker continuity and is close to blind to everything else it claims to measure. And three of its eight terms were dead: the emotion-peak term was saturated at its cap in 92.4 % of assemblies, and the arc term sat at exactly 1.0 in 84 % and took three distinct values ever.

9. The timed script

Listening to the round-1 supervised-plus-preference model, two faults dominated:

1. Uncontrolled vocal bursts — a sigh that should last 0.5 s runs 5 s.
2. Hallucinated content — text appears that was never prompted.

Both are the same missing signal: the prompt never stated how long anything may take. The current round exists to supply it. All of it runs on JUPITER: GH200, 4 GPUs per node, 102 GB each, SLURM account reformo, partition booster.

9.1 The format

The instruction block is unchanged in shape from the shipped format — the same Reference(s) / Instruction / Tokens / Quality / Sound Event / Ambient Sound / Language / Text template — but the SCRIPT: block is now a rendered timed script:

[0.9 seconds pause] [3.4 seconds duration] Erster Satz. [0.6 seconds pause]
(contented sigh, 0.4 seconds) [4.1 seconds duration] Zweiter Satz.

There are exactly four kinds of tag, told apart by their brackets:

The disambiguation rule in one line: square bracket = a number of seconds; round bracket with a number = a vocal burst; round bracket without a number = a delivery direction. That is the only thing separating a burst from a direction, which is why directions are always emitted with no number at all. There is one soft spot: in the untimed 30 % of samples nothing carries a number, so a burst renders as a bare (chuckle) — the same shape as a direction. What still separates them there is length: a burst label is one or two words, a direction is a phrase.

Segmentation rules, all measured against the corpus: split at sentence ends; split again at a vocal burst, at the nearest word gap; split any remaining segment over 12 s at its largest internal gap provided both halves stay above 2 s. A duration is measured from the first word onset to the last word offset of that segment, so two sentences of 12 s and 8 s produce [12.0 seconds duration] and [8.0 seconds duration], never a single 20-second tag. Gaps under 0.20 s are folded into the neighbouring speech rather than printed, so the printed numbers still sum to the clip length. Bursts under 0.05 s are dropped, and a burst that overlaps speech prints only the part that does not overlap.

Verification, done the right way. The emitted string is re-parsed by a separate function with no access to the source data, and the recovered numbers compared against the true clip length. Residual median 0.010 s on 8,614 real-speech rows and 0.020 s on 14,359 voice-profile rows; within rounding tolerance on 99.9 % / 98.7 %; the transcript comes back byte-identical on 100 % of rows. This is the pattern §6.8 recommends: make at least one leg of a check independent of the pipeline that produced the thing being checked.

9.2 The inline delivery direction

The round-2 format dropped the parenthesised delivery direction that round 1 had carried — by accident, and the accident became the round's central negative result (§10.2). Round 3 puts it back, inside the script, before each duration tag. A direction is composed of five pieces:

Only the first segment gets the full direction; later segments get a short reminder ((still clearly amused)), and segments under 1.6 s often get none — repeating a thirty-word note in front of a 0.6-second line buries the line it is meant to shape. When a row has no emotion label at all — a VoiceNet-axis row, for instance — the opening segment gets the corpus's own most common direction instead.

A limitation kept deliberately rather than silently repaired. All 40 emotions resolve to exactly two harvested phrasings each — one expressed, one suppressed — drawn from only 10 distinct family-level texts. Some pairings read oddly: intoxication, pain and shame all inherit the fear family, so an intoxication direction can say “the fear plain in the voice”. This is not a bug in the generator — it is exactly what the round-1 corpus contains (425 of the intoxication rows use that phrasing verbatim), and the model was already trained on those pairings. It is kept for comparability and flagged as the first thing to revisit if intensity control still lags for those emotions.

Two bugs in this generator were found and fixed on 25 August, seventeen minutes before the round-3 run launched — one that would have made every row render the same intensity word, and one that left the untimed 30 % of samples with no direction at all. Both would have made the round meaningless without changing any training-time number. §10.4 gives them in full, with the pre-fix code.

9.3 What varies between training samples

Nothing is fixed; each sample draws from a seeded generator, so the model sees the whole range and never requires any single element to be present. Measured end to end on 62,672 real corpus rows through the training loader:

The Instruction field itself takes one of four shapes: gen_both (a GENERAL: line and a SCRIPT: block) at 45 % for voice profiles and 75 % for real speech; caption_tpl (a natural-language instruction instead of the GENERAL: line) at 30 % for voice profiles only; gen_general at 15 %; gen_script at 10 %.

Realised band mix over those 62,672 rows: 58.8 % extreme, 23.3 % intense, 10.1 % moderate, 7.8 % faint. That skew is not an accident — it comes from training on the “extreme subset” — and §10.11 argues it is a prime suspect for the failure that follows.

9.4 The corpus

The extreme subset is worth one paragraph because of how its definition had to change. The original rule was “the top 10 % of each of the 40 emotions plus the top and bottom 10 % of each VoiceNet dimension”. Measured, that rule selects essentially the whole corpus: the emotion top-10 % union alone is 89.01 %, the VoiceNet top-plus-bottom is 99.37 %, and the combination is 99.84 % of the full corpus. With 154 overlapping axes, a per-axis decile is not a filter. It was re-implemented as a fixed quota per axis; at 6,000 rows per emotion each pick is the top 0.19 % of its axis.

Band composition of the subset — 59.5 % extreme, 23.8 % intense, 9.9 % moderate, 6.8 % faint — against the full corpus's 45.6 / 32.4 / 12.8 / 9.3 %.

9.5 The evaluation harness

Every model comparison in the rest of this report uses the same harness, so the rows are comparable to each other and to nothing else:

• 80 prompts — one per emotion × {English, German} — built by the same function the GRPO trainer samples from, so the evaluation cannot drift away from the trained format.
• 4 completions per prompt = 320 clips per model. One completion would make every per-emotion number a sample of size one, and the within-group spread is itself a quantity GRPO needs to be non-zero, so it is reported.
• Band requested: intense (percentile 0.90–0.98), inline directions always on.
• Per clip: emotion band achievement, quality (blend and genuineness as corpus percentiles), burst realisation, word error rate against the script's own plain text, and duration error against the length the script's own numbers sum to.

The evaluation deliberately uses a stricter yardstick than training. Training needs a gradient, so its band score ramps; evaluation asks a different question — did the clip land in the band that was requested — and answers it with a plateau that pays nothing outside the band. Keeping the two apart is what makes numbers published before and after a reward change comparable.

Burst realisation deserves a note, because it is not an obvious metric. It is an F1 over prompted-versus-detected bursts in which each match is weighted by how well its duration matches — a matched class scores 1.0 and a mismatched one 0.35, times exp(−|Δduration| / 0.5), with a 1.5 s tolerance on position. Recall alone would reward emitting a burst everywhere and precision alone would reward emitting none; the original listening complaint was bursts that are too long, so the duration term is what carries the signal, and a matched pair with the wrong length scores near zero.

10. Supervised rounds: timing solved, emotion lost, format regained

Three supervised rounds have run. Round 1 established the line: all 4.13 B parameters, 3 epochs over the 3,147,802-row corpus, global batch 4,096, 2,232 steps, 64 nodes × 4 GH200 in 1 h 53 min, held-out loss improving monotonically 4.7076 → 4.6314 across all twelve evaluations. Rounds 2 and 3 are much shorter runs on top, whose purpose is the prompt format rather than more data: each is 2 epochs over the 398,282-row extreme subset, 712 steps on 32 nodes.

The sequence is the most instructive thing in this report, because round 2 was a success and a failure in the same run, the failure was caused by a line that was deleted by accident, and round 3 exists only to put it back. The rest of this section is that story in order: what round 2 won, what it lost, how the loss was diagnosed, the two further bugs found seventeen minutes before round 3 launched, and exactly what round 3 changed.

A comparability note that the protocol records rather than hides: round 2 ran at learning rate 5e-6 and round 3 at 7e-6, so their validation losses are not comparable to each other; nor are they comparable across the round-2 → round-3 prompt change, which altered the objective. §10.7 argues that in this regime the validation loss is barely comparable to itself.

10.1 Round 2 — timing is solved, and emotion control disappears

320 generated clips per model, band intense requested:

(These three rows are on the pre-correction percentile scale of §6.7 and are therefore comparable to each other but not to the tables from §11 onward.)

Timing is solved. Median duration error 0.08 s — one audio frame — and every one of 320 clips within half a second of the length its script's own numbers add up to. That is the capability the timed script was built to deliver, it was delivered in a single 29-minute run, and nothing since has degraded it.

Emotion is not. Asked for percentile 0.90–0.98, every round-2 model lands at 0.49–0.51 — the corpus median, which is what a model produces when it is not being told anything at all. Three models with three different amounts of preference tuning all sitting on the median is not a training-strength problem; it is the signature of a missing input.

10.2 The diagnosis: what round 2 deleted

The cause was found by reading the diff between the round-1 and round-2 prompt renderers. Round 2 had dropped the parenthesised delivery direction entirely.

Round 1 had put a direction in front of the spoken text on 55 % of voice-profile samples and 85 % of real-speech samples — (intensely amused: letting it out / not hiding it, warm and open, unguarded) and the like. Round 2 replaced that slot with the timed script, and the direction did not survive the replacement. Nobody removed it deliberately; the new renderer simply did not carry it forward.

What was left was description, not instruction. The emotion still appeared, in the GENERAL: line, as a clause saying what the source recording sounded like — “reads as contentment”. That tells the model what kind of clip this is. It does not tell it what to do. The model was trained for two epochs on a format in which no one ever asked it to perform anything, and it learned exactly that: produce the corpus median.

"wer": 4.545454545454546,
"hyp": " Ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, ah, …"

10.3 What round 3 changed: the direction goes back, inside the script

Round 3's entire purpose is to put the delivery direction back and train on it, in the position the timed script leaves for it: in round brackets, without a number, standing immediately before the duration tag of the segment it applies to. §9.2 gives the five pieces a direction is built from and §9.1 gives the bracket rule that keeps a direction distinguishable from a vocal burst. Nothing else about the format changed.

Two consequences of that decision are worth stating before the results, because they bound what round 3 could possibly have achieved:

• The direction carries the intensity word, and nothing else does. The band — faintly, clearly, intensely, overwhelmingly — appears only in the direction. If the direction is absent, or if the band is computed wrongly, the model has no intensity signal in the prompt at all. Both of those turned out to be live faults; §10.4 is them.
• The direction is emitted from the row's own measured percentile, using the same cutoffs as the reward and the evaluation harness. What the prompt asks for is therefore exactly what gets measured, which is what makes the comparison in §10.8 meaningful.

10.4 Two prompt bugs, found and fixed seventeen minutes before the run started

Both were found by rendering real corpus rows through the training loader and reading the output, which is a cheap check that this project now runs before every prompt-format change. Both would have made the round meaningless in a way that no training-time metric would have revealed. The backup files the fixes left behind are timestamped 09:54:21 and 09:55:19 on 25 August; the SFT-3 job started at 10:12:48 the same morning.

band = "intense" if row.get("in_extreme") else "moderate"

The prompt-format hash moved from 64d956b8bb97f0d9 to f90a7b16b4cea093 — the hash is written into every checkpoint's state file, so a checkpoint can always be matched to the exact prompt renderer that produced its training data. That is the mechanism by which the round-2 accident was findable at all.

10.5 The round-3 training configuration

One correction to a figure that has circulated internally: round 3 was not trained on 247,049 rows. That number is the emotion-LoRA corpus of §13.2, built the same evening by a different selection script, and the coincidence of dates is the likely source of the conflation. Round 3 trained on corpus_x.

10.6 The validation series, in full

10.7 Why that series is nearly useless, and what it is good for anyway

The whole run moves the held-out loss by 0.0032 nats, and it moves the wrong way for the first three evaluations. Against a model whose negative log-likelihood is about 54 nats per position — each supervised position carries 13 channels (§3.1) — that is a relative change of six parts in a hundred thousand.

The reason is structural and it applies to every short fine-tune in this project. The loss is dominated by predicting audio codes, and a change to the prompt format changes almost nothing about how hard the audio is to predict. The whole intervention — putting a parenthesised phrase back in front of each segment — is a few dozen text tokens against several hundred audio frames per example. A metric that averages over both cannot see it.

10.8 What actually moved in the weights

The export step compares every tensor against the base checkpoint, which is a cheap check that a run did what it claimed and touched what it should have:

Two things are worth reading off that. Round 3 moved more of the model than round 2 did, both in count and in magnitude — 378 tensors against 375, and a peak relative change 31 % larger — which is consistent with the higher learning rate and with the fact that round 3 introduced a new token pattern into every sample rather than merely reshuffling an existing one. And roughly 47 tensors were untouched by both runs, which is the expected shape for a fine-tune that leaves embeddings and certain normalisation parameters alone.

This is a sanity check, not a result. A run that reported zero tensors changed, or all 425, would have been a bug; 378 with a peak delta of 2.7 % is an ordinary short fine-tune. The figure is recorded because a later run in this project did report a suspicious tensor count, and it was a bug (§19.9).

10.9 Round 3 — training the directions in fixes the collapse

Both models on the same 320 prompts, all carrying inline directions:

The collapse is gone. Word error rate falls from 0.447 to 0.099 — a factor of 4.5 — and it falls on both languages, from 0.574 to 0.070 on English and from 0.320 to 0.127 on German. Timing improves at the same time (median duration error 0.100 → 0.080 s, and 100 % of clips inside half a second against 92.8 %), and so does burst realisation (0.224 → 0.356) and burst hit rate (0.516 → 0.666). One 31-minute run on 32 nodes bought all of that.

It is worth being precise about what was and was not demonstrated. Round 3 did not teach the model a new capability. It taught the model a format — and the size of the effect is a measure of how completely a transformer will fail on a prompt shape it has not seen, even when every individual element of that shape is familiar. The directions round 3 trains on are the same directions round 1 trained on. Two epochs of not seeing them was enough to make them poison.

Rescored on the zero-point-corrected percentile scale of §6.7, with no regeneration, the picture is worse and more honest:

4.4 % of clips land in the band that was requested. That number, not the composite reward, is the one that states the problem: asked 320 times for a specific intensity, the model delivers it fourteen times.

10.10 A check that had to be done before considering a retrain

If the zero-point defect had also corrupted the training labels — the emo_strength percentile from which each direction's intensity adverb is derived — the whole round would have needed redoing. It did not. If the labels were a floor artefact, heads with a high zero point would win almost always. They do not: the winning head has a real signal (raw > 0.01) on 99.9 % of rows, including 99.9 % of the rows labelled at percentile ≥ 0.90, and the 40 heads are evenly represented at 3–4 % each. Repeating the supervised round on corrected labels would produce the same labels.

10.11 The standing hypothesis for why intensity does not move

The training subset is 59.5 % extreme and 6.8 % faint. The model sees “overwhelmingly” almost always and “faintly” almost never — so there is little contrast from which to learn that the word controls anything. Note that this is the same failure as bug 1 of §10.4, one order of magnitude weaker: bug 1 would have given the model one intensity word on every row; the corpus as shipped gives it four words in a 59/24/10/7 mix. The fix for the bug was obvious. The fix for the skew is the same fix, and it has not been applied.

The full corpus holds roughly 290,000 faint and 400,000 moderate rows, so a band-balanced selection of the same size is feasible. This remains untested and is the most obvious unexplored lever in the project.

11. Preference tuning: seven runs, four corpora, and one metric that lies

Preference tuning is the objective this project has spent the most compute on, and it is the one that produced the shipped model. It has also produced a completely consistent negative result on the capability it was aimed at. This section covers all of it: how the objective is configured, every run that was launched, how each of the four preference corpora was constructed and what each was meant to fix, which checkpoint is best and why, and what the training telemetry does and does not say.

11.1 How DPO is set up here

DPO trains on pairs: a chosen audio sequence and a rejected one for the same prompt, pushing up the log-probability of the chosen relative to the rejected. Two configuration choices carry most of the weight, and both were arrived at by getting them wrong first.

11.2 Every preference run

Seven runs exist on disk. Two of them are not results — one trained on the wrong base and is kept deliberately (§19.9), one was superseded before it was evaluated — and they are listed anyway, because a table of five successes is a different document from a table of seven runs.

None of the three long runs finished an epoch. All three hit a six-hour wall clock somewhere between a third and two thirds of the way through, and in all three cases the best checkpoint was the last one taken — which means nobody knows whether the remaining half helps, and it is an open task rather than a finding.

11.3 Corpus v1, and 88.9 % of it going missing

The preference corpus has five pair families. The rejected side is constructed, not sampled — which is the design decision that makes the whole approach cheap, since no second generation pass is needed:

Version 1 of that corpus came out at only 236,166 pairs across 128 shards (1.7 GB), and the reason is a good example of a cost that gets stated rather than buried: the builder looked word timings up in the supervised corpus, which contains only the rows selected for supervised training. Every voice-profile pair whose chosen clip was not in that selection was dropped — 946,266 pairs, 88.9 % of the emotion arm — and because the length arm is capped to the emotion arm's size, the whole corpus shrank with it. What survived:

A correction to an earlier figure that propagated through the protocol and through an earlier draft of this report: v1 is 236,166 pairs, not 236,656. The parquet shards and the build log both give 236,166, and the family counts force it arithmetically — 118,328 on the emotion arm against 117,838 on the length arm. It appears to be a digit transposition that was then copied forward.

11.4 Corpus v2 — the timings were never missing

They were in the annotation product's tar files all along: <uid>.json members carrying MMS-FA word alignments in exactly the {w, s, e} form the renderer expects. A sidecar keyed by clip id was built in three stages on a single node — plan (3,451,531 references → 2,146,797 distinct ids over 2,000 shards, 85 s), bursts (1 s per task), words (84 s per task).

The rebuild changes exactly one function, so pairs that already worked come out byte-identical and a v1/v2 comparison isolates the effect of data volume alone:

The v2 drop count splits as 2,981 rows with no annotation at all and 803 with an annotation but no usable duration; the protocol labels the 803 as the no-annotation count, which is the wrong one of the two. The 11.9 % pre-sidecar coverage figure quoted elsewhere could not be reproduced — the v1 auxiliary table was overwritten by the rebuild — so it is stated here as a protocol claim rather than a measurement.

The 57 : 43 ratio is a deliberate departure from the original brief's strict equality, kept because the length arm has already done its work — timing is solved — and the emotion arm is what remains unsolved.

What eight times the data bought. Reward 0.4668 → 0.4687, emotion percentile 0.3518 → 0.3401, word error rate 0.1117 → 0.1094, burst realisation 0.3973 → 0.3929. That is: a rounding-level gain in composite reward, a fall in the emotional intensity the corpus was enlarged to improve, and no material change anywhere else. Eight times the preference pairs on the arm that matters moved the target metric in the wrong direction. It is the cleanest evidence in this section that the emotion ceiling is not a data-volume problem, and it is the measurement that motivated changing the construction rather than the size.

11.5 The classifier-free-guidance corpus

Every contrast built so far pits an intense clip against a mild one — which a model can win by being generically expressive, without ever reading the instruction. The classifier-free-guidance (CFG) construction tries to make that impossible:

1. Per voice profile and per dimension (40 emotion heads plus the VoiceNet axes), rank that voice's clips by the measured value — not the prompt, not the intended bucket.
2. Pair a top-1 % clip with a bottom-1 % clip of the same voice, with durations within ~10 %. (A DPO pair shares one prompt and therefore one length budget; a length mismatch would teach length instead of intensity.)
3. Emit each pair twice with the roles flipped. Under a “high” instruction the intense clip is chosen; under a “low” instruction the mild one is. Both clips appear as chosen and as rejected, so the model cannot win by a global preference for intense audio and must read the instruction.
4. Remove the words. The prompt keeps the direction, the duration and pause tags, the burst tags and the frame budget; the spoken text becomes ..., so the preference cannot be decided from the words.

A rendered CFG prompt, verbatim:

<user_inst>
- Reference(s):
<|audio|>
- Instruction:
GENERAL: A voice; timbre is very dark
- Tokens:
101
- Quality:
None
...
- Text:
(very dark, and hold it there; otherwise exactly as this voice normally speaks)
[5.5 seconds duration] ... [0.5 seconds pause] [2.0 seconds duration] ...
</user_inst>

Composition, counted over all 128 shards of the built artefact — 2,327,904 pairs, 17 GB:

The two CFG halves are not merely balanced to within a tenth of a percentage point, as an earlier write-up of this build reported — they are exactly equal, 237,209 each, which is what a correct flipped-role duplication must produce and is therefore a stronger check than the approximate one. The prompt-format hash moved to 073aeb09dc923376.

11.6 The contrastive-families corpus

Two further families were added on top of the CFG corpus, aimed at two measured defects rather than at a general idea. §15 gives their construction, their selection rules and the per-family learning curves. In summary: p2_emox (309,128 rows) pits intense against intense — same speaker, matched length, emotion A against emotion B, with the instruction naming A — so that turning intensity up is worth nothing and only reading the instruction pays; p2_len (59,389 rows) keeps speaker, text and emotion fixed and varies only the duration. Mixed onto the CFG corpus: 2,327,904 + 368,517 = 2,696,421 pairs, the new families 13.7 %.

11.7 Which checkpoint is best, and why

All rows on the same 80 prompts, 4 completions each, inline delivery directions, zero-point-corrected percentile scale, plateau yardstick — so they are comparable to each other and to nothing else:

Step 5022 of the contrastive-families run is the best model this project has produced, and the case for it rests on three things rather than on the composite reward alone, since the top four rows span 0.0059 in reward and this evaluation cannot resolve ±0.01.

1. It is the only preference-tuned model whose emotion percentile beats plain SFT-3. 0.3541 against 0.3494. Every other DPO checkpoint on this table is below the supervised baseline on the metric all of them were built to raise — v2 at 0.3401, CFG at 0.3373. The two new families were built for exactly this and it is the one thing they changed.
2. Its word error rate beats the supervised baseline too. 0.0977 against 0.0987. Before the CFG corpus, every preference run in this project paid for its gains in intelligibility; two corpora in a row have now stopped doing that, and the contrastive-families run does it while also raising emotion.
3. It does not give up the structural gains. Burst realisation 0.4180 and hit rate 0.762, against the supervised 0.3564 and 0.666. It is slightly below the CFG checkpoint on both (0.4271 / 0.772) — that is the one axis on which the previous best is still ahead, and it is the honest cost.

Its neighbour at step 3906 is a statistical tie on reward, is better on word error rate (0.0916, the lowest of any model in the project), and is worse on emotion (0.3478) and on burst realisation. Step 5022 was chosen because emotion is the unsolved capability and 3906's WER advantage of 0.006 is inside noise. Both are published.

What none of them did. The target band is 0.90–0.98. The best model in the project reaches 0.354. Four preference corpora, seven runs and something over a hundred node-hours have moved that number from 0.3494 to 0.3541 — a gain of 0.005 on a scale where the goal is 0.55 away. The corpora are not what is wrong, and §11.9 argues that the objective may be.

11.8 rew_chosen: a metric that is uninformative across runs and anti-informative within one

The DPO trainer logs two reward terms per evaluation: rew_chosen, the implicit reward on the preferred sequence, and rew_rejected, the same on the rejected one. A healthy preference run should raise the first while lowering the second.

Within a run the metric moves the wrong way as the model improves. From dpo_sft3:

Preference accuracy climbs from 0.865 to 0.981 while rew_chosen gets three times worse — and step 864 is the checkpoint that generates best. The policy is suppressing the chosen audio while learning the ranking: it makes the bad audio catastrophically less likely and the good audio slightly less likely. Both go down; one goes down less. That satisfies the objective exactly and is not what anyone wanted.

11.9 Why none of it moved emotion — one explanation refuted, three standing

Refuted: “the high side was only locally intense.” The obvious explanation is that pairing within a voice makes “high” a relative target — a voice profile that never gets angry has a top-1 % anger clip that is barely angry. Measured over 224,750 voice-profile rows in 500 voices, each voice's own top 1 % per emotion head scores a median 0.995 on the global zero-point-corrected scale, and 99.7 % reach the 0.90 band. The chosen side was genuinely extreme, and the training signal pointed at real extremes.

1. Recognition is not production, and the telemetry supports it. DPO raises the chosen sequence's likelihood only relative to the rejected one. At the CFG run's step 4476, rew_chosen is −1.12 against rew_rejected −46.6. The policy is not making intense audio more likely; it is making mild audio catastrophically less likely under a “high” instruction and intense audio slightly less likely. Pushing everything down, one thing less, does not move the sampling mode. Ranking two fixed recordings is a discrimination task and the model solved it (0.5625 → 0.9750). Sampling draws from wherever the probability mass actually sits.
   Test: measure the absolute log-probability of held-out intense audio before and after a preference run. If it fell, the fix is an absolute anchor — a larger NLL weight than 0.013, or a β below 30. This costs one evaluation pass and is the cheapest decisive experiment in the project. It is still not instrumented.
2. The text-free regime never occurs at inference. The CFG and p2_emox rows are the only ones where the spoken words are absent; at evaluation every prompt has words. What generalises out of that regime is structural — the duration, pause and burst tags mean the same thing with or without words — and that is exactly the profile of the gains: burst hit rate 0.709 → 0.772, burst realisation 0.3929 → 0.4271, WER 0.1094 → 0.0950, all structural. The semantic mapping from an intensity word to an acoustic target has no lexical content to attach to when the direction sits next to silence.
   Test: the same pairs with the words present, only the requested level flipped.
3. The symmetry that prevents drift also prevents push. Whatever gradient pushes towards intensity under “high” is mirrored by an equal gradient away from it under “low”. The net movement of the unconditional distribution is zero by construction. That is correct behaviour for a conditioning signal and useless for raising a ceiling. If the goal is a model able to go further than it currently can, symmetric contrast cannot supply it — only data at a higher ceiling, or an objective with an absolute target, can.
   Test: an asymmetric 70/30 mixture.

A fourth possibility has since arrived from a different direction and is recorded in §16: measured inside the model, the direction that means “more of this emotion” and the direction that means “higher measured quality” have a cosine of −0.75 in mid-stack and −0.954 at the last state before audio codes are chosen. If those two are close to opposite instructions, then any objective that rewards emotion and quality together is asking for something the representation makes difficult, and every reward in this project does exactly that.

12. GRPO: five runs, and one structural negative result

GRPO generates G completions for the same prompt with sampling, scores each with the measurement models, normalises the rewards within the group (A = (r − mean) / std), and pushes the policy towards the completions with positive advantage. With one inner update per rollout the PPO ratio is exactly 1, the clipping term is inert, and the loss reduces to −(A · logp).mean(). The whole learning signal is therefore the spread of scores inside a group.

reward = ( 0.40·r_emo + 0.40·r_qual + 0.20·r_burst ) × wer_factor(WER)

r_emo    band_score(percentile reached on the requested emotion head, requested band)
r_qual   0.5·percentile(vocal-burst blend) + 0.5·percentile(genuineness)
r_burst  F1 over prompted vs detected bursts, weighted by duration match
WER      Whisper-large-v3-turbo against the script's own plain text

12.1 Runs 1 and 3 were not a GRPO result

Both fell, and the cause was outside the optimiser. Run 3 was given prompts with inline delivery directions — the right idea — but the model had never been trained on that format, because round 2 had dropped directions by accident (§10.2). It collapsed, and GRPO spent 2.7 hours optimising against a reward dominated by that collapse. The conclusion is not that GRPO needs tuning; it is that a reward computed on out-of-distribution prompts measures the prompt, not the policy.

12.2 Run 4 — the first run that neither deadlocks nor collapses

88 steps, 176 rollout groups, 2.00 h on one 4-GPU node, from SFT-3, KL β 0.02, corrected percentile scale. skipped = 0 for the entire run: the rank-divergence fix (§19.1) held.

Burst realisation rises 0.561 → 0.669 and WER falls 0.128 → 0.098. Emotion is flat within noise. (An earlier reading of single steps had suggested an emotion collapse; the aggregate over 176 groups does not support it, and per-step numbers of two groups each are far too noisy to read as a trend — a small methodological correction that is recorded rather than quietly dropped.)

12.3 The decisive measurement

Splitting run 4 by the band the prompt asked for, where emo is the band score of the percentile achieved against the percentile band requested:

Supporting numbers, run 4: mean within-group reward standard deviation 0.1063 — that is the entire learning signal per step. Across groups, corr(reward, emo) = +0.753, corr(reward, burst) = +0.310, corr(reward, wer) = −0.432, with std(emo) = 0.292, std(burst) = 0.339, std(wer) = 0.122. The reward is not ignoring emotion — the policy cannot move it where it is being asked to.

12.4 Run 5 — four fixes for that, and why they made it worse

Four levers were built and all four default off, so run 4 stays reproducible:

1. A monotone ramp. Replace the flat floor of the band score with a straight ramp from 0 at percentile 0 to 1.0 at the band's lower edge, so the slope is non-zero everywhere below the target. Verified before the run: for band extreme, percentiles 0.20 / 0.35 / 0.50 score 0.000 / 0.000 / 0.000 under the old shape and 0.204 / 0.357 / 0.510 under the new one.
2. A WER-free allowance. WER below 0.15 costs nothing; the old slope applies to the excess only. The old form taxed every gain in emotion that cost a hair of intelligibility, and the ASR hypothesis of an intensely delivered line genuinely is a little worse than that of a flat one.
3. A band curriculum, starting at 5/30/40/25 (extreme/intense/moderate/faint) and interpolating to the briefed 25/25/25/25.
4. Dump the best completion of each group (most emotional with WER ≤ 0.20) with its prompt and codes.

On the strict evaluation yardstick, run 5 is the worst of the four models on emotion (0.3275 against SFT-3's 0.3494 — see §13.3). The training-time gain was the mechanical artefact it had been predicted to be: lever 1 inflates the number by paying for partial progress, lever 3 inflates it by asking easier questions. The realised band mix confirms the second: v4 ran 0.28 moderate / 0.28 intense / 0.22 extreme / 0.21 faint, v5 ran 0.36 / 0.30 / 0.15 / 0.20 — the curriculum moved the run onto the easier half and the run ended before it came back.

Hypothesis, stated as a hypothesis: a ramp that pays for partial progress supplies a gradient but removes the incentive to arrive. A completion that moves slightly toward the target now collects reward without reaching the band, and the strict yardstick only counts arrival. Levers 1 and 3 push in the same direction and it is the wrong one. A ramp that is steeper near the band, or a rank-within-group reward that keeps the target absolute, would test this.

12.5 What GRPO did produce: data

Lever 4 is the one that paid. Run 5 saved the most emotional usable completion of every group: 204 clips at median emotion percentile 0.641 and median WER 0.049, median 148 frames — 73 moderate, 61 intense, 41 faint, 29 extreme. High emotion and clean speech in the same clip is the combination that has been missing from every evaluation in this project. The model can do it; it does not do it on command. Two hours of rollout on one 4-GPU node therefore yields supervised training material at roughly 100 usable clips per GPU-hour — and best-of-N supervised training on that harvest is currently a more promising route than further reward shaping.

Hyperparameters, for reproduction: group size 16 (32 in run 3), 2 groups per optimizer step, micro-batch 2 completions, LoRA rank 32 / α 64 (68.7 M trainable of 4.27 B), learning rate 2e-6 constant after 20 warmup steps, KL β 0.02 with the k3 estimator against the same model with the adapter disabled (so no second copy in memory — measured KL stayed at 2–3e−4 and the constraint was never binding), sampling temperature 1.0 / top-p 0.95 / top-k 50, max 340 new frames (27.2 s), prompt pool 4,000 rows, checkpoint every 25 steps, 2-hour stopping rule. Prompt mix: 50 % carry vocal bursts, 35 % carry two emotions, 50 % carry a reference clip, 85 % carry a delivery direction. Throughput: one 4-GPU node produces 45–52 optimizer steps per hour at G = 16, i.e. 1,440–1,680 generated and fully scored clips per hour. Loading the three measurement models costs about 40 s per rank.

13. Specialist adapters: the only thing that has moved intensity

13.1 Why a quality gate is not optional

Measured on a 360-clip listening sample decoded from the corpus's own audio codes — one clip per emotion per tier:

Genuineness falls as intensity rises, and both quality axes sit below the corpus median of 0.50 throughout. The most intense recordings are on average the least genuine. An adapter trained on intensity alone would be taught to overact.

13.2 The general emotion LoRA: two arms and one gate

• Emotion arm — the top 1 % of any of the 40 heads, on the zero-point-corrected percentile scale.
• VoiceNet arm — several VoiceNet axes imply strong emotion without naming one, and the emotion heads do not reliably fire on them. Two-sided (top and bottom 1 %) for AROU, VALN, ARSH, VALS — high valence is elation, low valence is despair, and both are extremes. Top only for TENS, VOLT, VULN, EXPL, RANG, EMPH, VFLX, and the ranting / ASMR / whisper / cartoon / drama style axes.
• Gate — normalised vocal-burst blend AND genuineness both above the corpus median.

Two things fall out of this. First, the VoiceNet arm added about 40 % more material than the emotion heads alone would have found, and the axes that contributed are the affective ones — high valence (16,726 rows qualifying on that axis alone), explosiveness (14,437), arousal shift (9,890), valence shift (8,164), velocity flux (4,680), low arousal (3,323), low valence (2,383), high arousal (1,972). The stylistic axes — ranting, ASMR, whispering, cartoonish — contributed nothing in the top eight: their top 1 % was either already in the emotion arm or failed the quality gate.

Second, the source mix flipped. The gated selection is 65.4 % real speech, 34.6 % voice profiles — the reverse of the ungated listening sample (273 synthetic clips to 87). The gate removes synthetic extremes disproportionately, which supports the reading that the synthetic “top 1 %” is often overacted.

13.3 Result: rank 16 wins, and it is the first thing to move the number

Three adapters on the same 247,049-row corpus, 5 epochs, cosine 1e-4 → 5e-6, α fixed at 2 × rank so the effective scaling is identical and rank is the only variable, global batch 256 (4 × 2 × 32 on 8 nodes), 4,395 steps, 2 h 35 to 2 h 40 each. Base SFT-3, evaluated alone on it.

Below is every evaluation in the project that is on the corrected percentile scale — eighteen runs, all on the same 80 prompts, the same 4 completions per prompt, the same strict plateau yardstick, ranked by composite reward. This table replaces every partial comparison published earlier. The top two rows were added on 28 August, after the run of §15 finished; the emotion-term column is not recorded for them.

Three readings of that table:

• Two things in the project raise the emotion percentile above the supervised baseline, and they do it in opposite ways. The rank-16 emotion adapter reaches 0.3572 against 0.3494 — the largest lift on the table — but buys it with word error rate, 0.0987 → 0.1239. The contrastive-families preference adapter reaches 0.3541 and lowers word error rate to 0.0977. One is more extreme data; the other is a better-constructed contrast. The target band remains 0.90–0.98 and neither is close.
• The bare SFT-3 base ranks eleventh of eighteen. Seven of the ten arms that beat it are preference-tuned variants. Since the CFG corpus, preference tuning has stopped costing intelligibility: the top three rows all have a lower word error rate than the supervised baseline, which nothing before them managed.
• The ordering r16 > r64 > r32 is not monotone in rank, so this is not yet a capacity story; with one seed per rank the r32/r64 difference is within what seed noise could plausibly produce. What the ranking does support is that more capacity is not the lever here — the largest adapter is not the best one.

Six further evaluations exist and are deliberately excluded from that ranking, because they predate the percentile correction (§6.7): eval_sft2_dir, eval_sft2_dirv3, eval_dpo2_dir, eval_grpo1_dir, eval_grpo3_dir and eval_sft3_dir_UNCORRECTED_ECDF. Their word error rates (0.45–0.51 for the SFT-2-based ones) are comparable and genuinely bad; their reward and emotion columns are not.

13.4 Merge weight: two sweeps, two answers, and why both are right

An obvious next question is whether merging an adapter more strongly pushes emotion further. Two sweeps have now been run, and they came out differently. Reported together, because the difference between them is the finding.

Sweep A — one general adapter, band intense, 80 prompts: no trend

The single pooled emotion LoRA (rank 16, trained on all 247,049 gated rows) evaluated at seven merge weights on the standard harness. Those are rows 5, 8, 10, 11, 12, 15 and the baseline of the table above, gathered here:

Emotion percentile spans 0.3315–0.3572 with no monotone structure and a peak at the trained value. Reward spans 0.015 — inside what these evaluations show between arms that ought to be equivalent. All seven runs are complete (320 clips each). On this axis, with this adapter, at this band, merge weight is not a dial for intensity.

Sweep B — 31 per-emotion adapters, band extreme, with bursts: a clear monotone rise

The second sweep is a different object. It takes the per-emotion bucket adapters of §13.5, each on prompts naming its own emotion at the extreme band with vocal bursts requested, and scales each adapter's contribution over six weights. Design details that make it a strong measurement: the same eight prompts are used at every weight for a given adapter, the torch seed is reset to a fixed value before each weight, and weight 0 is the same loaded model with the LoRA scaling zeroed rather than a separately loaded baseline — so a difference cannot come from anything but the adapter. 31 adapters × 6 weights × 8 clips = 1,488 scored generations.

Emotion percentile rises monotonically from weight 0.5 onward and is +0.084 higher at weight 2.0 than at weight 0 (paired over 248 clips, bootstrap 95 % CI [+0.048, +0.123]); at weight 1.5 the paired gain is +0.064, CI [+0.030, +0.098]. 27 of 31 adapters are higher at the top weight than at zero — the three that are not are Helplessness (−0.185), Bitterness (−0.104) and Contempt (−0.024) — and the fourth, Jealousy and Envy, is the dead metric of §6.4 and reads 0.000 everywhere. At weight 1.5 the split is 25 up, 5 down, 1 tied. Per-adapter argmax weight: 2.0 for 13 adapters, 1.5 for 7, 1.0 for 6, 0.5 for 2, 0.25 for 2 and 0.0 for 1.

Timing is untouched at every weight — mean absolute duration error stays inside 0.115–0.133 s across the whole sweep, so overdriving an adapter does not cost the capability that was actually solved.

Three caveats belong with sweep B, none of which changes the direction of the result:

• Genuineness and blend are ceiling-bound. Their medians are 1.0 at every weight; blend percentile hits 1.0 in 225–239 of 248 clips and genuineness in 190–208. The rise from 0.817 to 0.880 and 0.925 to 0.969 is real but is happening in the top few percent of a saturated scale, and should not be read as a large quality gain.
• Burst realisation does not rise — it peaks at weight 0.5 (0.3035) and falls to 0.2227 at 2.0, and is exactly zero on 104–129 of 248 clips at every weight. Overdriving helps emotion and does not help bursts.
• The whole word-error story is an English-side story. Split by language: German WER is 0.059 at weight 0 and 0.059 at weight 2.0 — flat — while English runs 0.270 to 0.304. The German half of the sweep shows the emotion gain with no intelligibility cost at all.

Why the two sweeps disagree, and which to believe for what. They are not the same experiment. Sweep A scales a single general adapter trained on all 154 axes at once, on the intense band, against a prompt set that mostly does not request bursts; sweep B scales per-emotion specialists on the extreme band, on prompts that name the adapter's own emotion and request a burst. The reconciling reading is that a general adapter has no single direction to overdrive — scaling it amplifies an average of 154 directions, which is close to noise — while a specialist adapter has exactly one, and scaling that one works. Both results stand; sweep A is the reason not to expect band-dependent scaling of a general adapter to work, and sweep B is the reason the per-emotion adapters ship with a recommended weight above 1.

Coverage note: sweep B was launched when 31 of the 40 per-emotion adapters existed. Amusement, Embarrassment, Intoxication, Relief, Sadness, Sexual Lust, Shame, Thankfulness and Triumph are absent from it. Re-running it now would cover all 40.

13.5 One adapter per emotion — and the selectivity problem

The natural next step is 40 adapters, one per emotion, rather than one general one. All 40 have now been trained (rank 16, α 32, 5 epochs, cosine 1e-4 → 5e-6, 34.4 M trainable parameters, each on the gated top 1 % of its own emotion, all finishing with zero non-finite batches). Training set sizes run from Sourness at 2,663 rows to Intoxication at 15,725; step counts from 3,330 to 19,660; wall time from 42 to 224 minutes per adapter.

Seventeen of them have been measured against the best general checkpoint — SFT-3 + CFG-DPO — on identical prompts, identical sampling and identical seeds. The seventeen are Affection, Anger, Astonishment/Surprise, Awe, Bitterness, Concentration, Contempt, Contentment, Disgust, Elation, Emotional Numbness, Jealousy and Envy, Longing, Malevolence/Malice, Pleasure/Ecstasy, Pride and Sourness. That set is not a chosen subset: the generator shards adapters across ranks with names[RANK::WORLD], and the adapter directory was still growing while the workers launched, so ranks duplicated each other and skipped others. Seventeen is where the job landed.

The design of that evaluation is the interesting part. A specialist adapter should do two things: make its own emotion stronger when it is asked for, and change nothing when it is not. So each adapter gets ten prompts naming its emotion at the intense band and ten prompts naming no emotion at all — the corpus's own neutral direction. A rise in the first is the adapter working. A rise in the second is the adapter leaking into plain speech.

So the adapters do push their emotion — and they push nearly as hard when nobody asked. They also cost about half again as much transcription error and about a fifth of the burst realisation. As they stand they are not a drop-in improvement. They are tinting, not control.

Word error rates here are for the emotion-naming (matched) prompts only. Different adapters draw different slices of the prompt pool, so cross-adapter comparison of the WER columns is not prompt-matched. The prompts are drawn from three source corpora — the run log's “three voices” label is a fallback string, not three verified speakers — so the design controls for corpus, not for speaker identity, which is weaker than it looks.

13.6 Is there enough data for a LoRA per bucket?

Counted over 3,144,739 rows, with 154 buckets (40 emotions, plus 57 VoiceNet dimensions × 2 tails):

Feasible — a median of 6,700 rows is comfortable for a rank-16 adapter — but badly skewed, and skewed by exactly the effect that motivated the gate. The four smallest emotion buckets are Sourness (2,663), Pleasure/Ecstasy (2,976), Contempt (3,105) and Awe (3,247), and their genuineness median in the top 1 % is 0.23–0.30: the most intense recordings of those four are almost entirely inauthentic, and the gate removes three quarters of them. Intoxication (15,725 rows, genuineness median 0.804) and Amusement (13,272, 0.774) have ample genuine material. A 40-adapter programme therefore needs either unequal training sets or a relaxation to the top 2 % for the four problem emotions.

One caveat on the VoiceNet half of that table, stated because it would otherwise mislead: its largest buckets are S_ASMR bottom (23,817) and S_CART bottom (18,934) — that is, “not ASMR” and “not cartoonish”. Those are the unremarkable default, not an expressive extreme, and their high genuineness (0.87) reflects that. Only the four two-sided affective axes have a meaningful lower tail.

13.7 The VoiceNet axis adapters, built today

The same bucket machinery was pointed at the VoiceNet axes on 26 August at 16:47. The policy is the one §13.2 established: both tails for AROU, VALN, ARSH and VALS; the top only for the rest — the survey found that the largest “bottom” buckets of the style axes are simply ordinary unremarkable speech, so there is no such thing as an extreme absence of ranting.

Twenty candidate buckets went in; three were dropped by a row-count floor of 1,500 applied after the top-1 % tail selection and the blend-and-genuineness quality gate, and 17 came out: 146,175 rows in total, median 8,216, minimum 1,922, maximum 22,849.

The three dropped axes are exactly the ones where the quality gate bites hardest: high pitch range keeps 1,147 of 31,453 candidates, whisper-talk 939 of 31,452, and cartoonish 680 of 31,450 — a 2.2 % survival rate for cartoonish, whose genuineness median in its own top 1 % is 0.109. That is the §13.1 finding again, in its sharpest form: the most cartoonish recordings in the corpus are almost entirely inauthentic.

Training of those 17 adapters started at 16:51 on 26 August (rank 16, α 32, 5 epochs, AdamW lr 1e-4, cosine with 10 % warmup, batch 4, non-DDP, one bucket per GPU stream) and was still running when this report was written. They are not yet published.

13.8 Every adapter is trained on SFT-3 alone, and here is why

14. What is published, and what the cards say

The round's artefacts went public on 26 August as four repositories under the laion organisation, and three more followed on 28 August: the preference adapter of §15, the public subset of the trajectory corpus (§17), and a listening page for the merge-weight ablation of §13.4. All four were uploaded from the login node (compute nodes have no route to the internet) and byte-verified after upload — the uploader re-reads the repository metadata with HfApi.repo_info(files_metadata=True), size-compares every file, and for the strictest of the scripts downloads and SHA-256-hashes anything the Hub reports no size for, exiting non-zero on a single disagreement. The reason is recorded in that script's docstring: “upload_folder has returned a success URL for a rejected upload in this project before, so nothing is trusted.”

The cards are unusually candid, and three of their statements are worth reproducing because they are the honest limits of what was shipped:

• From the base-model card: “Emotional intensity is not [solved]. Asked for percentile 0.90–0.98 of a named emotion, this model reaches about 0.35. The emotion adapters below are the only thing that has [moved it], and even they are not selective enough to merge in blindly.”
• From the emotion-adapter card: “The selectivity ratio is only about 1.4 : 1 … Do not merge one in permanently and expect neutral speech to survive.”
• From the voice-adapter card: identity was never gated during profile construction, only ranked, and no speaker-similarity metric was applied to the adapters at all — “whether an adapter reproduces its voice has to be judged by ear”. The base model's drift into a different voice at high intensity is not fixed by them.

The base-model card also carries a decoding warning that is worth repeating outside it: use proc.decode_audio_codes(..., return_stereo=False). The direct tokenizer path yields half-speed audio. In the card's words, “this project lost a whole corpus to that once” — see §19.6.

15. Two new contrastive pair families — and the model they produced

These are aimed directly at two measured defects — the leakage of §13.5 and an emotion-blind length signal — and they are the only intervention in this project that has raised the emotion percentile above the supervised baseline without costing word error rate.

15.1 What they are

Family A — emotion-contrastive, for selectivity. Every contrast built before this pits an intense clip against a mild one, so a model can win by being generically expressive — which is exactly the 1.4 : 1 tinting the per-emotion adapters showed. Family A pits intense against intense: same speaker, same length, emotion A against emotion B, with the instruction naming A. Turning the intensity up is then worth nothing; only reading the instruction is. The builder's own docstring states the motivation in one line: “A ratio of 1.4 : 1 is not control, it is tinting.”

The selection rule is strict on both sides. The chosen clip must sit at percentile ≥ 0.90 on the head being asked for and have that head as its own top emotion; the partner must be ≤ 0.50 on that head and ≥ 0.90 on its own top head. So both sides really are intense, on different emotions. Lengths are matched to within 10 % of frames. Each base pair is emitted twice, the mirror naming the partner's own top emotion. Because chosen and rejected are different recordings with different words and a DPO pair shares one prompt, the spoken words are removed, as in the CFG construction.

Family B — length-contrastive, conditioned on the emotion. The existing truncation and continuation families know nothing about emotion, so they teach “the right length is better” in the abstract. Family B keeps the speaker, the text and the emotion fixed and varies only the length: chosen is the clip at its true duration, rejected is the same clip cut to 50–75 % or run to 125–150 %. Here the words are in the prompt, with the full timed script, because the timing tags are exactly what the rejected side violates.

Both families take the emotion from the measured value, never from the prompt that generated the clip. Constants: HI_PCT 0.90, LO_PCT 0.50, FRAME_TOL 0.10, 8 emotion-contrastive and 6 length-contrastive pairs per cell, cut range 0.50–0.75, extension range 1.25–1.50, RNG seed 4711.

15.2 What was built

The base is the CFG corpus rather than the older one, and the mixer says why: because the CFG corpus produced the best checkpoint measured so far. Schema equality between the new rows and the base was checked field by field before the merge — 30 fields each, none missing, none extra — and all 342,284 referenced clips were found. An earlier build with a smaller per-cell budget produced 138,088 rows and was superseded.

15.3 What the run did

The run is dpo_sft3_p2: 32 ranks over 8 nodes, LoRA rank 64 (137.4 M trainable of 4.267 B, 3.220 %), β 30, chosen-NLL anchor 0.013, learning rate 1e-6, 10,331 steps per epoch with 1,033 warmup, 256 preference pairs per step, arm-balanced sampling. It ran at about 62 pairs per second and hit its six-hour wall clock at step 5,126 of 10,331 — half an epoch. Checkpoints were kept at 3906, 4132, 4355, 4579, 4800 and 5022.

Preference accuracy per family, aggregated from the step log (the trainer emits an instantaneous per-family accuracy for whichever families a batch happened to contain, so these are means over logged batches, not a single running statistic):

The model learned the harder contrast. Emotion-contrastive accuracy rises from 0.864 to 0.938 across the two halves and sits at 0.951 post-warmup, above the older CFG families' 0.82–0.84 at the corresponding cumulative point. That is the intended shape: a pair whose two sides are both intense can only be told apart by reading the instruction.

The length-contrastive family saturated. It reads 1.000 over the second half — the task is solved and it is no longer informative, which is consistent with timing having been solved since round 2. It is contributing nothing to the gradient by the halfway point and could be dropped from a future mix in favour of more emotion-contrastive material.

And the run failed the health metric, exactly as every previous DPO run has. The mid-run evaluation at step 2582 reported val loss 0.7277, preference accuracy 0.9805, rew_chosen −1.0015 and rew_rejected −38.851, and the trainer wrote "healthy": false with the annotation “policy suppresses the CHOSEN audio” (§11.8).

15.4 The result: a new best model

Both surviving checkpoints were evaluated on the standard harness. Step 5022 is now the best model this project has produced.

Because both this run and the CFG run use a global batch of 256 preference pairs, equal step numbers mean equal numbers of pairs seen — so step 5022 here and step 4912 there are a nearly matched comparison, and the difference is the corpus rather than the budget.

The adapter is published as laion/moss-va-sft3-dpo-lora-p2. The remaining half epoch was never run; whether it helps is unknown.

16. Activation forensics and steering: looking inside instead of at the output

Everything up to this point measures the model by generating audio and scoring it. That is the right way to select a checkpoint and it says nothing about why four objectives have failed on the same capability. A separate line of work therefore opened the model up: hidden states were extracted while it followed real recordings, probes were trained to read attributes out of them, and the resulting directions were injected back into generation to see whether the model could be steered. The full account is a companion page: Steering and probes. This section is the summary and the two findings that bear on the rest of the report.

16.1 What was measured

One teacher-forced forward pass per corpus row — the row's own audio on the assistant side, packed byte-identically to a training example — and the mean hidden state over the tail of the assistant audio span, kept at 38 taps: the embeddings, 35 layer outputs, the final norm, and the frame-slot state of the local transformer that the twelve audio heads actually read. 387,999 rows across four extractions, 75.5 GB of float16, on the SFT-3 export with the step-5022 adapter stacked. Five probing tasks were then trained per tap and scored by macro R² on a held-out, speaker-disjoint split.

Task A is under-specified by construction — the probe is not told which dimension it is being asked about, and 54.8 % of its target variance sits between bucket families it cannot see. Its 0.13 is a statement about the question, not about the representation, and it should not be quoted as a result. The companion page explains why in full.

All of it survives the controls: a fully speaker-disjoint split costs under 0.007 R² on four of five tasks, and stripping the emotion caption from the prompt before extraction costs at most 0.015 R² anywhere. The probes are reading the performance, not the speaker and not the prompt.

16.2 Two findings that bear on the rest of this report

16.3 Steering: a clean negative result

If a direction in activation space is the model's handle on an attribute, adding a multiple of it during generation should move that attribute. The injection is h ← h + α·(v/‖v‖)·‖h‖, so α is the fraction of the current hidden state's own magnitude added along the direction. 393 clip records over 27 grid cells: seven strengths on the dimension vector, a quality vector, a quality-orthogonalised dimension vector, multi-tap stacks, three single-tap probes, and — the control that makes the result interpretable — a random vector of matched magnitude.

At every non-zero strength, in both directions, the output collapses. Unsteered, the model transcribes at word error rate 0.094 and lands within 0.01 s of its requested length. At α = 0.5 the word error rate is already 1.14 — more errors than there are words. The emotion percentile does not rise at any positive strength. The probe directions are not merely noise: a random vector at α = 1.0 does about twice the damage. But no cell of the grid shows a dimension vector moving its own attribute in a way a matched random vector does not, and that is the result.

Three readings of the failure are consistent with the grid and none is tested: the intervention may be in the wrong place (a difference of window-averaged means is not an activation the model ever holds); the regimes may not match (extracted teacher-forced, injected during free generation, which is exactly the gap the extraction code warns about); or the discrete 13-channel audio-code output may simply be more brittle under perturbation than a text model's. The practical conclusion is narrower and firmer than any of them: activation steering is not currently a route to intensity control here.

One provenance note carried over from the companion page: the aggregated grid artefacts under report/ hold an eight-clip smoke test, not the executed grid, and the project's own protocol still quotes the smoke test's numbers. The tables above are aggregated from the 393 raw per-clip records. The protocol's conclusion — that the causal claim is not established — is unchanged and is if anything strengthened.

17. The trajectory corpus

Every measurement in this report is of one utterance in isolation. That is the right unit for “can this model perform contempt at the 95th percentile” and the wrong unit for almost everything a director actually asks for, because direction is about change: start polite and let it curdle, come down off it slowly, angrier with every sentence. None of those is a single target percentile, none can be scored by running an instrument on one clip, and nothing in this project has ever trained on or evaluated one.

A trajectory is the smallest object that makes that measurable: an ordered list of clips from one speaker in which one measured dimension moves monotonically. Because every clip in the corpus already carries the full 40-emotion and 57-VoiceNet annotation of §6, such sequences can be mined rather than recorded. The full account is a companion page: The trajectory corpus.

A row is a specification and contains no audio. Twenty-four columns: the ordered clip identifiers, the speaker they share, the dimension being walked, the distance covered and the per-step size, the monotonicity and consistency thresholds and the step count, quality bounds, total duration, whether the clips are adjacent in the source recording, and selection bookkeeping. Roughly 55 bytes per trajectory, which is why 7.6 million of them fit in 419 MB.

The two-sided family is the smallest and the most interesting. A single-dimension walk asks the model to do more of something; a two-sided walk asks it to trade — and every acting note about a turn, a realisation or a change of heart is a trade rather than an intensification.

Trajectories are short: two to five clips, and the step histogram sums exactly to the row total in both versions, so there are none longer. The public release is a clean removal of three source datasets — podcast, snippets and evasnippets — whose transcripts are not part of the open real-speech release. A trajectory row names exactly which segments were used, which is a weaker disclosure than publishing the transcripts but is not nothing. The exclusion of snippets in particular was a cautious default rather than a settled decision, and the companion page says so.

Published as laion/moss-va-trajectory-corpus (public subset, CC-BY-4.0); the unfiltered corpus is retained privately.

18. When a training metric lies

This project selects checkpoints, ranks adapters and decides whether a run worked by generating audio and scoring it, never by the training objective. That is more expensive — a 320-clip evaluation costs a GPU-hour or so — and it is not a stylistic preference. It is the conclusion of six separate cases in which a number was clear, consistent, and wrong — five of them training-time metrics, and one a measurement instrument saturating on audio that was not speech.

18.1 A preference-run “damage” indicator at its worst while the model generated best

Preference accuracy is the metric a DPO run naturally reports, and it rises monotonically in every run here — 0.865 → 0.981 in one, up to 0.994 in another. Over the same steps reward(chosen), the implicit reward on the preferred sequence, falls monotonically. The round-1 model card states the consequence outright: “preference accuracy is not a health metric here. It was highest exactly where the model was most degraded.”

And the inverse holds too, within a run. The classifier-free-guidance run of §11.5 ends with reward(chosen) at −1.12 — three times worse than at its first evaluation — while generating the best audio in the project at the time: lowest word error rate, highest quality, best bursts. Had that indicator been used to select, the best checkpoint would have been discarded.

One caveat that an earlier version of this section got wrong, corrected here rather than quietly: −1.12 is not the worst reward(chosen) in the project, and the compressed cross-run form of this story — “the worst indicator produced the best model” — is false. The corpus-v2 run ends at −2.30 and produced the fifth-best model. §11.8 gives all seven runs. The lesson survives in its within-run form, which is the form in which it was actually observed.

18.2 A validation loss that could not move, because the holdout was the wrong set

The three emotion adapters of §13.3 differ by a factor of four in trainable parameters and by 0.036 in emotion percentile on generation. Their validation losses all sat at 4.652–4.654, flat from step 1,225 onward and separated only in the fourth decimal. The reason is structural: the holdout is the general corpus holdout, not the extreme subset, so an adapter specialising in intense emotion cannot improve it. The loss was not noisy — it was measuring a different question.

18.3 A validation loss that was consistent, monotone, replicated ten times — and still wrong

The cleanest case is from the voice-profile rank ablation. Held-out validation loss separates the ranks monotonically — r16 < r8 < r4 — for every one of the ten voices, without exception. On loss alone you would ship rank 16 ten times out of ten, with clean, consistent, unanimous evidence.

The generation-based evaluation turns that 3.9 % into no measurable difference at all on 1,920 held-out clips per arm (p = 0.22 on speaker similarity, 0.94 on reward, 0.32 on WER). The two measurements agree on the ordering and disagree only on whether the remaining gap is worth paying for. Selecting on loss would have cost four times the adapter parameters to buy nothing measurable.

The honest limit of that claim is worth stating with it: the generation evaluation is itself a learned scorer, so what has really been shown is that the gap is below the resolution of every instrument that was pointed at it. No human was asked.

The manual reports the same failure a third time, from a different direction: at rank 64 the validation loss rose from 4.4237 at epoch 1 to 5.4369 at epoch 8, while listeners rated epoch 1 and epoch 8 identically (1.969 both).

18.4 A reward that rose because the reward function had changed

GRPO run 5 reported a training-time emotion term of 0.441 against run 4's 0.259 — a 70 % improvement. It was not one. Two of the four changes made for that run (§12.4) inflate that number directly: the ramp pays for partial progress that the strict yardstick does not count, and the curriculum asks easier questions. On the unchanged evaluation, run 5 is the worst of the four models on emotion. A training reward is only comparable across runs if the reward function did not change; when it does, the only honest comparison is on a yardstick that did not. This is why the evaluation here deliberately keeps the plateau band score even though training uses the ramp.

18.5 And one case where the comparison itself was on two scales

The round-3 supervised model was evaluated before the zero-point percentile correction of §6.7 and the preference and RL models after it. For a period the emotion column — and through it the composite reward — was on two different scales, and SFT-3's 0.4900 was inflated relative to the preference model's 0.4668. Every model was re-evaluated on the identical prompt set with the corrected scale; the superseded directory is kept under an explicit _UNCORRECTED_ECDF name so it cannot be quoted by accident. The two files are byte-identical on every field except four (§6.7), which is what makes the size of the distortion exactly measurable rather than estimated.

The rule this project now works to: ablate on the output, not on the objective — and when the objective changes, keep one yardstick that does not.

18.6 A quality scorer reporting a perfect score on audio that was not speech

The sixth case is not a training metric but an evaluation instrument, and it is the most extreme of the six. In the steering grid of §16.3, the condition that injects the quality direction at strength 2.0 reports 1.000 on both genuineness and vocal-burst blend — the top of both reward terms — and the highest emotion percentile of the entire grid. Every other cell in that column reports 1.000 as well, at every strength and every tap.

The same clips transcribe at word error rate 1.030 and run 13.76 seconds away from the length their own script asks for. The underlying raw scores are genuineness 1.31 out of 6 and blend 1.97 out of 10 — near the bottom of both scales. The percentile mapping that turns a raw score into a reward term was fitted on a corpus of speech; handed something that is not speech, it has no basis for a judgement and returns the top of its range.

19. Engineering failures worth publishing

Several of these cost node-days, and all of them are the kind of failure that looks like something else. They are recorded because the diagnosis, not the fix, is the transferable part.

19.1 Distributed rank divergence — a hang that looks like a hardware fault

In the GRPO loop each rank decided independently whether to skip a rollout group:

if len(recs) < 4:      skipped += 1; continue
if r.std() < 1e-6:     skipped += 1; continue
if nseq == 0:          continue      # skips opt.step() too

So one rank ran three backward passes while another ran two, and the gradient all-reduces went out of step. The watchdog fired after ten minutes: BROADCAST, NumelIn=104 … ran for 600066 milliseconds, exit code 134, and 35 GB of core files. The fix is to make the number of backward passes a constant of the configuration rather than of the data: a degenerate group is kept with advantage 0, every group is padded to the full group size, and padding entries are excluded from the KL term so they contribute no gradient. Runs 4 and 5 report skipped = 0 across 176 and 210 groups.

The same failure class had already appeared once as a heartbeat failure in a preference run, and was reproduced by hand a third time while rewriting the bucket-adapter trainer. The general form: any per-rank continue in a data-parallel training loop is a deadlock waiting for the right batch.

19.2 Weight tying makes the preference adapter unmergeable

The published DPO adapter ships unmerged. merge_and_unload() corrupts the model, because audio_lm_heads.N.weight and audio_embeddings.N.weight are the same tensor (they are tied at load), so merging the head delta also rewrites the embedding that produced it.

Measured: both tensors changed by exactly 6.103515625e−05 while the text embedding changed by 0.0, and a logit-equivalence probe reported hidden: max|diff| = 3.33, rms = 1.07e−01 against a signal rms of 2.52. That constant is 2⁻¹⁴ — the bf16 quantum at that magnitude — which is exactly the signature of a single shared tensor being written once and read twice.

Three independent corroborations sit on disk: the adapter configuration targets audio_lm_heads.0…11 and carries "ensure_weight_tying": false; every training log reports audio_lm_heads.{0…11}.weight | MISSING in the load report, because those tensors are not serialised separately; and at save time PEFT prints Removed shared tensor …audio_lm_heads.3.lora_A…. The adapters are stacked at inference instead, which is also why every adapter in §13 is trained on the supervised checkpoint rather than on the preference model.

19.3 A padding token that reaches the wrong embedding table — three times

Padding every channel with the text pad token puts an id around 151,643 into the audio channels, whose embedding tables hold 1,025 entries. The gather runs out of bounds and reports device-side assert triggered — vectorized gather kernel index out of bounds 19 seconds into the first rollout, taking the ranks down one by one and looking exactly like a hardware fault. Text and audio channels must be padded separately.

The identical bug was then reproduced twice more, in different files. Once by omitting the label masking when the batch packing was reimplemented — an 8-node job died in three minutes on every rank. And again this morning, when the bucket-LoRA trainer was launched for the first time: all 32 ranks died simultaneously at 08:51 with the same assert. It is now copied verbatim with a comment saying why it is not optional, and validated on a single-node smoke test before spending an 8-node allocation.

19.4 A finite loss with a non-finite gradient, and an adapter that reported DONE after zero steps

This one is new, from this morning, and it is the most instructive of the recent batch. After the padding fix, the bucket-LoRA jobs restarted and produced 1,305 loss=nan lines across two 8-node jobs before they were cancelled 25 minutes in. A single-GPU smoke test diagnosed it in six minutes — 21 non-finite losses in 1 steps — refusing to train on garbage — and a second smoke test caught the worse variant: a finite loss with a non-finite gradient, reported as non-finite grad norm at step 0, step skipped. That run then printed DONE emotion/Affection rows=5235 steps=0 17.9m. An adapter had been trained for eighteen minutes, taken zero optimizer steps, and reported success.

The root cause: the vendored model's own gradient-checkpointing path is broken by late binding — every layer's wrapper closes over the last layer, so the recomputed activations do not match the forward pass and the gradients come back inf or NaN. The fix is to disable the vendored path entirely and wrap each decoder layer with a non-reentrant checkpoint wrapper. This is the third distinct gradient-checkpointing failure in this project (§3.5 has the first).

Two guards now stand in the trainer as a result: a non-finite-loss counter that refuses to continue, and a non-finite-gradient check that skips the step. The second one is what turned a silent zero-step success into a visible failure — but note that it still printed “DONE”. A guard that skips is not a guard that reports.

19.5 The prompt contradicted the reward

Real-speech captions state the source clip's measured genuineness X/6 and vocal-burst blend Y/10 — exactly the two quantities the reward's quality term optimises. On a low-scoring source row, the model was therefore rewarded for contradicting its own instruction. Fixed by stripping those clauses in the RL prompt only; this is not out of distribution, because most voice-profile caption templates carry no such clause.

19.6 A shared-mode error that a consistency check could not see

The most instructive of the lot, and it predates the current round. The annotated corpus shipped for weeks with the invariant moss_frames == round(dur_s × 12.5) passing on 100 % of rows. It passed because both terms were doubled together: a decoder bug had been flattening stereo frames, producing half-speed audio, and the duration, the frame count, every emotion score, all 57 VoiceNet dimensions, the WER and the word timestamps were all computed on it. An earlier verification pass had reported that 100 % as proof of correctness.

It was found only by decoding the audio and comparing against an independent reader. The rule that came out of it is the one §6.8 and §9.1 both apply:

A consistency check cannot detect a shared-mode error. Assert the invariant on the emitted artefact, and make at least one leg of the check independent of the pipeline that produced it.

19.7 A checkpoint rotation that deleted the best model of its run

The DPO run on corpus v2 produced its best checkpoint at step 3216 — reward 0.4687, at the time the best model in the project. The trainer's --keep-last 2 policy then rotated it away. The evaluation directory survives, complete, which is why the number can still be quoted here; the weights do not. That checkpoint no longer exists and cannot be republished.

The lesson was applied afterwards: the CFG-DPO run has a sibling keep/ directory holding every checkpoint from step 212 through 4912, twenty-four of them, alongside the six the rotating directory kept — which is why step 4912 was still on disk to be published. The contrastive-families run (§15) rotated again, with six retained and no keep/ directory — it kept steps 3906 through 5022, the best of them survived, and the exposure was real rather than theoretical.

19.8 A module-cache race that cost 48 node-hours

128 ranks raced to populate the trust_remote_code module cache for a directory nobody had loaded yet. Some read a half-written source file (cannot import name MossQwen3Model); the survivors hung in the initialisation collective. Fixed by a single-process pre-warm before the parallel launch — three lines in the batch script. A related environment hazard on this filesystem: concurrent cold imports of large Python packages put every worker into uninterruptible sleep for up to 25 minutes with the GPUs idle, so launches are staggered by 45–150 s.

19.9 A run that trained on the wrong base, kept as a result

One preference run intended to start from the round-3 supervised checkpoint started from round 2 instead: the batch script reads its base from one variable, a different one was exported, and line 21 silently replaced it with the default. The log stated it plainly and that is where it was caught. The run is kept under the name dpo_on_sft2_WRONGBASE, because it is a valid experiment in its own right — preference tuning on SFT-2 under the round-3 prompt format. Its telemetry is the familiar shape: preference accuracy 0.709 → 0.818 while reward(chosen) falls −0.071 → −0.635 and reward(rejected) collapses to −8.34, with "healthy": false on every row. It was never run through the standard evaluation harness, so it contributes no reward or WER number to §13.3.

A second detail from it is worth keeping: it used 32 nodes, giving a global batch of 1,024 and only 184 optimizer steps for the same data, where an earlier run at 8 nodes got 864. The corrected run uses 8 nodes so the two are comparable.

19.10 Two silent evaluation bugs, and a library trap

• An empty speaker list that still produced results. The first attempt at the 17-adapter grid logged restricted to 3 voices: [''] — the speaker-extraction function had fallen through and produced a single empty string, and the job ran anyway on an 884-row pool. It was cancelled and re-run; the published comparison comes from the second job. Even in that second job, the “three voices” are three source corpora (other:emolia, other:mediathek, other:refvoice), not three verified speakers — the same fallback branch, one level less broken. The control is weaker than the label suggests, and §13.5 says so.
• A PEFT attribute that rejects a list. Assigning a list of adapter names to peft_model.active_adapter raises TypeError: unhashable type: 'list' from deep inside the library's activation check. Two generation jobs crashed on it today before the workaround — read and set model.active_adapters, plural — was found. This is the third documented trap in the same attribute (§4.13 has the other two).

19.11 A success message is not evidence

Observed seven times in three days, and worth reproducing as a table because each row is a different tool lying in a different way:

The remedy that works: process tables, repository metadata, parquet footers, decoded audio, controlled probes — never the status line. It is the reason the four repositories of §14 are byte-verified after upload rather than trusted, and the reason §19.4's “DONE … steps=0” was caught at all.

20. Application: dubbing under a duration budget

The capability that is solved has an obvious and immediate use, and it is worth spelling out because it is the part of this work that is ready now.

Dubbing has a hard constraint that ordinary TTS does not: the translated line must fit the time the original line occupied. A German utterance of 4.7 seconds has to become an English utterance of 4.7 seconds, including its pauses, or the picture and the sound come apart. Conventional pipelines fight this with post-hoc time-stretching, which audibly damages the voice, or by rewriting the translation until it happens to fit.

The timed-script format solves it at the source:

1. Measure the source. Force-align the German utterance to get per-word timings. That yields exactly the quantities the script format prints: per-sentence spoken durations, every silence of 0.2 s or more, and every vocal burst with its length and class (from the locator and detector of §6.6).
2. Re-render the script in the target language. Same tags, same numbers, translated words. The frame budget is floor(d × 12.5) of the original duration.
3. Clone the voice. Attach the reference recording, or a voice-profile adapter if one has been trained for that speaker.
4. Generate best-of-N. 32 candidates cost the same wall time as 16 on this architecture (§4.12), so the group is nearly free.
5. Rank and pick. Three independent criteria are available, and they answer different questions.

Add speaker similarity (ECAPA, speechbrain/spkrec-ecapa-voxceleb) as a fourth term, and apply §5's lesson: rank on it, do not gate on it — and if you must exclude, exclude below 0.40 rather than penalising.

A note on which VoiceCLAP. The audio–text contrastive models live in the laion organisation — voiceclap-commercial (768-d, CC-BY-4.0, the one the scoring stack uses), voiceclap-small-v2, and voiceclap-large-v2 (3584-d). The related artefact in the TTS-AGI organisation, audio-audio-clap-maestrino-sae-32x-k5, is a different kind of object: a sparse autoencoder trained on 155 M CLAP embeddings drawn from six audio corpora — 512-d in, 16,384-d hidden (32× expansion), top-k 5, with 14,127 of 16,384 features alive. It is an interpretability and retrieval tool over the embedding space, not the embedding model itself. For a dubbing ranker you want the embedder; the SAE is what you would reach for if you wanted to ask which few features a pair of clips differ on.

Two limits belong with this recommendation. First, the emotional-intensity ceiling of §13 means that if the source performance is at the top of the human range, the dub will land below it — the ranker will pick the closest of what the model produced, and the model does not produce extremes on command. Second, cross-lingual reference conditioning does not clone (§4.3, §8.1: mean ECAPA 0.188 against a 0.105 floor), so a German→English dub that must preserve identity needs a trained voice adapter for that speaker, not just a reference clip. Both of the newly published adapter families (§14) are relevant here: the 500 voice adapters give per-speaker identity, and the 40 emotion adapters at weight ~1.5 give the intensity lift, stacked rather than merged.

21. Running now, and what comes next

21.1 State on 28 August 2026, after a day of cluster downtime

The bucket-adapter trainer deliberately uses no distributed data parallelism. The work is embarrassingly parallel over buckets, so each of a node's four GPUs takes its own stream and trains independently. That removes the failure class that has cost this project the most (§19.1) and lets each process load the 8.3 GB base model once and attach / train / detach adapters in sequence instead of reloading per bucket.

21.2 What is next, in order of expected value

1. Finish the VoiceNet pilot and evaluate it. Two of seventeen adapters are still training. Once they are done, the set should be measured the way the emotion adapters were — matched against neutral prompts, against the same baseline, with a merge-weight sweep — and published as one repository. The burst adapter needs the same treatment. Neither has any measurement at all yet.
2. Finish the contrastive-families epoch. The best model in the project comes from a run that stopped at step 5,126 of 10,331 on a wall clock, and its best checkpoint is the last one taken, which is exactly the shape that means the run was still improving. Whether the second half helps is unknown and costs six more hours to find out. The same is true of the CFG and corpus-v2 runs.
3. A human baseline and a blind comparison. This is the single largest gap in the whole project, and every section above has had to be qualified because of it. Two things are needed: a real-audio ceiling control — the manual's own warning is that in one adapter study the best cell scored 1.969, real human broadcast commentary scored 1.775, and the un-adapted base model 1.750, i.e. the metric's ceiling was below real audio and the base model was already at it — and a blind pairwise comparison of this model against other TTS systems on identical prompts. Pairwise, not absolute: an absolute rating scale was measured to saturate, and a judge given “score 0–10” was shown to rate recency rather than listen (the agent's top-ranked take was chosen in 3 of 108 rounds against a 20 % chance rate, and the favourite was the 4th or 5th take presented in 71 of 81 rounds). Giving the judge concrete named criteria flipped that position correlation from +0.33 to −0.12.
4. Best-of-N supervised training on the RL harvest. The data already exists and is measurably what is wanted — median emotion percentile 0.641 at median WER 0.049 (§12.5). Scale the harvest before scaling the policy update.
5. Band-balanced supervised data. The untested hypothesis of §10.11: the current subset is 59.5 % extreme and 6.8 % faint, so the model has almost no contrast from which to learn that the intensity word controls anything. A band-balanced selection of the same size is feasible from the full corpus.
6. Train something on a trajectory (§17). 7.6 million published specifications exist and nothing has been rendered from one. The obvious experiment is to turn a trajectory into a single training example whose prompt states a movement rather than a band, and see whether that teaches what a static band evidently does not. It is the most direct available attack on §10.11.
7. Run Task A2 (§16). It is already written. It converts the one probing task that failed into a specified question, and the difference between A and A2 measures directly what conditioning is worth — the same quantity the entire intensity problem turns on. It needs no new extraction.
8. The three DPO follow-ups of §11.9, each of which is a cheap decisive experiment. The cheapest is the first: measure the absolute log-probability of held-out intense audio before and after a preference run. It costs one evaluation pass and it distinguishes “the objective cannot push production” from “the data has no headroom”. The run of §15 was the natural place to do it and it was not instrumented for it, so the question is still open after a fourth preference corpus.
9. Re-run the merge-weight sweep over all 40 emotion adapters, not the 31 that existed when it launched, and extend it past 2.0 to find where the derailment tail becomes a population rather than two clips. Also run it on German-only prompts, where the sweep showed the emotion gain at no intelligibility cost at all.
10. Fix the Jealousy_&_Envy key mismatch (§6.4) and re-score. The raw per-clip scores are stored, so this is an offline rescore, not a regeneration. It will move absolute emotion percentiles slightly and no rankings.
11. Complete the per-emotion adapter measurement — 40 adapters exist and 17 have been evaluated, and the 17 were a sharding accident rather than a sample. Selectivity, not lift, should be the acceptance criterion, and the measurement needs enough adapters to resolve a difference that is currently at t = 0.42.
12. A reward ablation rather than a longer RL run. Rank-within-group instead of an absolute band score; a ramp that is steeper near the band; the emotion weight raised only once the term actually varies. More nodes would multiply a signal that is currently near zero for half the prompts.
13. The voice-acting agent as an agentic system. The casting agent of §8 is the existing prototype and its measured limits are the specification for the next one: enforce identity rather than penalise it, anchor-plus-tail continuation rather than chaining, seeds unique per task across several sub-batches, whole-assembly ranking, and a supervisor prompt that names the failure modes it must check.

22. Everything, with URLs

Every URL below was checked on 26 August 2026. Repositories marked private returned HTTP 401 to an anonymous request and are listed because they are the authoritative location of an artefact this report cites, not because they can be opened.

22.1 The model line

22.2 Adapters

22.3 Measurement instruments

22.4 Listening pages

All audio on these pages is model-generated; none of it is source recording.

22.5 Datasets

22.6 Documentation and code

23. Limitations, and inconsistencies between sources

23.1 Limitations of everything above

• No human listening study. Stated in §1 and §6.8 and repeated here because it qualifies every number in this document. The project's own estimate of scorer–listener agreement on spoken material is ρ ≈ 0.21; on non-verbal material the manual reports that the sign is wrong.
• Two languages only — English and German — and the results differ sharply between them. The whole word-error story of §13.4 is an English-side story; German WER is flat across the entire merge sweep.
• Much of the training material is synthetic, generated by earlier versions of this same model family and filtered by these same scorers. The intensity ceiling that §13 keeps hitting may be partly a property of that loop. The emotion-LoRA line already had to cap its strongest synthetic source at 25 % because it was stamping its character on the output (§4.1).
• The 80-prompt evaluation is small. One prompt per emotion per language, four completions each. Differences of ±0.01 in the composite reward between arms are not resolvable, and this report tries not to lean on any.
• The 17-adapter selectivity result is not statistically resolved (t = 0.42 on the gap), rests on a mean that one outlier dominates, has medians pointing the other way, and covers 17 of 40 adapters chosen by a sharding accident. It is reported because it is what exists, with all four caveats attached.
• Single seed per arm in the adapter-rank comparison of §13.3, which is why the non-monotone ordering is reported as “not yet a capacity story” rather than as a finding.
• The emotion heads' own accuracy is unstated. The Empathic-Insight-Voice-Plus card publishes validation metrics for its four quality experts only. There is no published per-emotion MAE or correlation for the 40 heads on which this entire report's central axis rests.
• One of the 40 emotion heads is never actually read at evaluation time (§6.4), so every absolute emotion percentile in this report is biased slightly low. Rankings are unaffected.
• The published voice adapters have no speaker-similarity measurement at all. Their card says so: whether an adapter reproduces its voice has to be judged by ear.
• The trajectory corpus is entirely unmeasured. 7.6 million specifications are published (§17) and no model has been trained or evaluated on one. Its value is a proposal.
• The activation results are teacher-forced (§16). A probe answers “is this attribute linearly present while the model follows a real recording”, not “does the model use it when generating”. The steering grid is the direct evidence that those two come apart, and it is a negative result at every strength tested — so the internal findings of §16.2 should be read as suggestive of a mechanism, not as one.
• Every probe target is itself a scorer's output. A probe that reaches R² 0.82 on the VoiceNet task has learned to predict the VoiceNet regressor, including whatever the regressor gets wrong. The instrument caveats of §6.8 are inherited, not escaped.
• The VoiceNet taxonomy has no published paper — its definitions come from the model repository's own dimension table.

23.2 Inconsistencies found between sources, recorded rather than smoothed over

1. The size of preference corpus v1 was wrong in this project's own records. It is 236,166 pairs, not 236,656. The parquet shards and the build log agree, and the family counts force it — 118,328 on the emotion arm against 117,838 on the length arm. The wrong figure appears in the running protocol in two places and in an earlier draft of this report; it looks like a transposition that was then copied forward. Corrected in §11.3.
2. “247,049 rows after the round-3 selection” describes a different corpus. Round-3 supervised fine-tuning trained on corpus_x — 398,282 selected rows, materialised at 401,345 including the validation holdout. 247,049 is the emotion-LoRA corpus of §13.2, built the same evening by a different selection script. Corrected in §10.5.
3. The CFG corpus's family shares were reported slightly wrong, in a way that under-sells the result. Counted over all 128 shards, the CFG share is 20.38 %, not 20.5 %, and cfg_high and cfg_low are not “balanced to within 0.1 percentage points” — they are exactly equal at 237,209 rows each, which is what a correct flipped-role duplication must produce and is a stronger check than the approximate one. Corrected in §11.5.
4. “The DPO run whose rew_chosen was worst produced the best model” is false. It is a tempting compression of a real within-run effect and the seven runs refute it: the worst final rew_chosen (−2.3037) belongs to the corpus-v2 run, whose best checkpoint ranks fifth, while the best model comes from the run with the second least negative value. The supported claim is the within-run one. §11.8 gives both and says which is which.
5. Two of the five probing tasks were reported with the wrong split's numbers. An internal write-up gave task B as 0.4394 at h18 and task D as 0.7309 at h20; those are the uid-split figures, and the primary speaker-disjoint results are 0.4328 at h20 and 0.6685 at h23. The same page mislabels which of its two splits is the speaker-disjoint one, and states the uid-split test-set size for both tasks. Nothing downstream changes — the ordering and every conclusion stand — and the companion page records the correction in detail. A second internal record had the right numbers and was distrusted because it carried an unrelated caveat.
6. The aggregated steering artefacts describe a smoke test, not the executed grid. report/steer_grid.parquet holds one grid row and eight clips; the real 27-cell, 393-clip grid ran the following evening into raw per-clip records and was never re-aggregated. The running protocol still quotes the smoke test's numbers, which point the same way as the full grid and reach the same conclusion by a different route: the protocol's figures show a random vector raising the emotion metric more than the real direction, the full grid shows both falling below the unsteered control. Either way the causal claim is not established. §16.3 uses the full grid.
7. A pre-sidecar coverage figure could not be reproduced. The 11.9 % word-timing coverage attributed to preference corpus v1 is stated in the protocol and cannot be recomputed, because the v1 auxiliary table was overwritten by the v2 rebuild. The 99.55 % post-sidecar figure is reproducible and is reproduced in §11.4. The v1 → v2 emotion-arm growth (118,328 → 1,061,613) is independently confirmed and does not depend on it.
8. Two smaller bookkeeping disagreements inside the protocol. Its run table lists 32 nodes for the round-2 preference run where its own text and the run log say 8 (world size 32 ranks ÷ 4 GPUs). And it labels 803 dropped rows as vp_emotion_no_annotation, where the build's plan file calls that count vp_emotion_dur_missing and puts no_annotation at 2,981. §11.4 uses the plan file.
9. Two different prompt formats are both current. The public manual documents the shipped v2 surface, in which there are no duration tags and length is controlled only through the token budget. The research line described from §9 onward trains a timed-script surface with explicit per-sentence durations, pauses and burst lengths. Both are real; they belong to different checkpoints. A reader following the manual against the round-3 checkpoints, or the reverse, will get the format-collapse behaviour of §10.2.
10. The merge-weight question has two answers. §13.4 reports both. A single general emotion adapter shows no scaling trend at the intense band; 31 per-emotion adapters show a clear monotone rise at the extreme band. Neither measurement is partial — both are complete runs — and they are not the same experiment. Anyone quoting “the merge-weight sweep” must say which.
11. The DPO model card does not contain the merge warning this report's predecessor said it did. §14 records this. The warning is correct and the measurement behind it is on disk; it simply is not on the Hub, and the cards should be updated.
12. The published adapter-loading snippet does not work with the published adapters, because the DPO adapter is rank 64 and the others are rank 16 (§14). The cards contradict the project's own working practice, which is recorded in the study runner.
13. The emotion-adapter rank differs between generations, and both numbers are right. The shipped moss-emotion-loras-v3 set is rank 32 / α 64 throughout; the 40-emotion sweep that preceded it chose the best of ranks {16, 32, 64} per emotion; the new SFT-3 adapters of §14 are rank 16 / α 32. These are three different artefacts.
14. How v2 was made is described two ways that are compatible but sound different. Its model card says v2 = the previous public release plus further supervised fine-tuning on best-of-N-filtered synthetic voice-acting data, and gives the optimizer and augmentation settings. The run notes say v2 = a six-dataset full fine-tune of 185,503 rows in six epochs, and name the datasets. Reading them together, the our_* half of the six is the best-of-N-filtered self-generated material the card describes and the prep_* half is prepared external material, and the settings match line for line (8×GPU DDP, caption dropout, reference dropout, linear LR with warmup, channel-wise loss weighting). §3.3 presents the union. The one point the two do not settle jointly is whether the checkpoint the six-dataset run started from is byte-identical to the public v1 release or to its unmerged predecessor.
15. Bracket notation contradicts itself inside the manual. The prompt-notation chapter measures round brackets beating angle brackets (emotion 2.603 vs 2.535, blend 7.06 vs 6.77, genuineness 1.731 vs 1.600) and calls the angle form a scoring corruption — but the expressive chapter still writes bursts as <moan>, <gasp>. Follow the prompt-notation chapter.
16. The lower-case-tags rule is violated by the manual's own winning captions, which use (Scream), (Gasp), (Chuckle). The manual says why: the rule was written in a later round and nothing enforced it on the 808 texts that already existed. The same defect shipped into the 500-voice build, where 135,580 of 135,630 burst tags (99.96 %) are Title-Case — and Title-Case tags are spelled out letter by letter rather than performed.
17. Speaker-similarity thresholds were revised downward by a factor. The casting playbook uses 0.82 / 0.75 / 0.68; the voice-consistency chapter replaces them with 0.58 / 0.45 / 0.40 and says so explicitly. Any tooling built against the older numbers is over-rejecting.
18. The Empathic-Insight-Voice-Plus card's head count does not match its file listing. The prose says “54+ emotion/attribute experts” and tabulates 12 vocal/demographic/psychological dimensions; the underlying repository ships 55 checkpoints (40 emotions + 15 attributes), and the 15 include Authenticity, Arousal, Recording_Quality and Background_Noise, which the prose table does not list, while the prose lists Talking Speed, for which no checkpoint ships. The file listing was trusted.
19. The 40-emotion list is settled. The manual refers in one place to an “Empathic-Insight 42 emotions” profile used for a different scoring task. The instrument used throughout this report has 40 heads, confirmed against the repository file listing, the local emotion-name list and the 40 bucket names in the corpus statistics.
20. The polarity of two VoiceNet dimensions is counter-intuitive and was once implemented backwards corpus-wide (GEND runs feminine → masculine, BKGN runs noisy → clean). The numeric columns were always correct; only the generated prose inverted. Anyone who trained on the caption text from that period learned the opposite of the truth for those two attributes.
21. The audio–text contrastive models are in the laion organisation, not in TTS-AGI, which is worth stating because it is an easy assumption to make. TTS-AGI holds a sparse autoencoder trained over CLAP embeddings — a different object, described in §18.
22. Two references linked from the manual and from the emotion-adapter card are dead. LAION-AI/Voice-Acting-Pipeline-WIP and LAION-AI/voice-acting-swarm-artifacts both return 404. The model repository's default branch is master, not main, so raw links built with main will 404 as well.
23. One claim in the brief for this report could not be reproduced literally. The live preference run's per-family accuracy was described as rising 0.858 → 0.953. Those two exact figures appear nowhere in the log; the closest single split is 0.843 → 0.954 at a cut around step 3000, and §15.3 reports the several honest windowings instead. The shape of the claim — the emotion-contrastive family learning fastest and passing the older CFG families — is confirmed. Two related framings in the brief were also corrected against the files: the 17-adapter selectivity grid is 17 emotion adapters, not VoiceNet axes (the VoiceNet run also happens to produce 17 buckets, which is the likely source of the conflation), and the three dropped axes were dropped by a 1,500-row floor applied after the quality gate, in a bucket run four hours later than the grid.

23.3 Every claim removed from the previous version of this report

The previous draft contained assertions that could not be traced to a source. They are listed here rather than silently deleted, because a reader who saw that draft should know what changed.

23.4 The one-paragraph summary, for anyone who read only this section

A general-purpose 4.55-billion-parameter TTS model was turned into a voice-acting model by two full fine-tunes — one on distilled director-instruction data with a rank-256 adapter merged in, one on six datasets of prepared and self-generated material, with the released checkpoint chosen by scoring generated audio rather than by validation loss — and then surrounded by roughly a thousand LoRA adapters for emotions, characters, voices, timbre dimensions and vocal bursts. The current round gave that model a prompt format stating, per sentence, how long it may take, where the silences are and how long each vocal burst lasts. That worked: median duration error 0.08 s, essentially every clip within 0.5 s, and adding the format also fixed a word-error collapse (0.447 → 0.099) and improved burst realisation. The same round tried to make the model perform an emotion at a requested intensity, and four objectives — supervised fine-tuning on an extreme subset, preference tuning on contrast pairs, preference tuning on symmetric instruction-conditioned pairs, and group-relative reinforcement learning — all failed to move it. The reason for the RL failure is measured and structural rather than a matter of tuning: at the intensity being asked for, every completion in a group scores the same near-zero value, and a constant term contributes nothing to a group-normalised advantage. What has moved the number is data, dose and a better contrast: a small adapter trained on the most intense and most genuine 8 % of the corpus (0.3572 against a 0.3494 baseline); overdriving the per-emotion adapters to weight 1.5, which lifts emotion percentile from 0.408 to 0.471 across 31 adapters with the best word error rate in the sweep and no cost in timing; and a preference corpus whose contrasts pit intense against intense so that generic expressiveness wins nothing, which produced the best model in the project (0.3541, and the first preference tuning here that raises intensity while also lowering word error rate). All three remain far from the 0.90–0.98 that was asked for. Looking inside the model rather than at its output found two things consistent with the failure and neither sufficient to explain it: emotion is roughly half as linearly legible in the activations as voice quality is and loses 39 % of that legibility before the acoustic decoder, and the internal direction meaning “more emotional” runs close to opposite to the one meaning “more genuine”; steering along those directions destroyed the output at every strength tested. Meanwhile the older LoRA-merging agent produces more emotional performances than any of this, and is not reliable on character consistency. Every one of those statements is the output of a learned scorer, and no human has listened.

Written 26 August 2026 and expanded on 28 August from the project's running experimental protocol, the daily status records, the on-disk evaluation summaries, training logs, build logs and corpus shards on JUPITER, the model cards published on 26 and 28 August, the model cards of every external instrument (fetched and read for this report), the distilled run notes covering the pre-JUPITER fine-tunes, and the public voice-acting manual. Every figure quoted has a named source in one of those. Where the brief for this report and the sources disagreed, the sources won and the disagreement is recorded in §23.2 — eight such disagreements were found and corrected during the 28 August expansion, including two of this project's own published numbers. Companion pages: steering and probes · the trajectory corpus. Claims that could not be sourced were removed and are listed in §23.3. Contact: LAION.