Inside the model — what the layer forensics found

28 August 2026 · a plain-language report on four parallel investigations into the 4.55 B voice-acting model's internal representations. Every number here was measured on held-out data with speakers held out; nothing is an estimate. The technical artefacts, the code and the raw grids are in Voice-Acting-Pipeline-WIP under research-log-2026-08/layer-forensics/.

1. What was actually measured, in one page

The model is a 36-layer transformer that reads a director's brief and writes audio tokens. While it works, each layer holds a 2,560-number "hidden state" for every position. The question of this whole programme is: if you freeze the model and look at those numbers, what can you tell about the performance?

The procedure, concretely:

• Take a real recording, and let the model read it with the true audio already in place (teacher forcing) — so it is listening, not inventing.
• Average the hidden state over the last 20 % of the audio. That gives one 2,560-number vector per recording per layer.
• Keep 38 such taps: h00 the input embeddings, h01…h35 the outputs of the 35 transformer layers, h36 the final normalisation, and loc, the small "local transformer" that sits between the language model and the audio codec.
• Train a small predictor (a "probe") on those vectors to guess one attribute — say, how angry the clip is — and score it on recordings from speakers the probe has never heard.

The score is R²: 0 means the probe does no better than always guessing the average, 1 means perfect. A probe that scores 0.85 has essentially recovered the attribute; one that scores 0.20 has found a hint of it.

Scale: 217,583 recordings in the main extraction, 38 taps each, 2,560 numbers per tap — 75.5 GB of activations on disk, and roughly 26 GPU-hours of probing on top.

2. W1 — where each thing lives

One probe per layer, predicting all 99 attributes at once: 40 emotion scores, 57 voice-quality dimensions, "genuineness" and "vocal-burst blend". 38 fits cover the entire grid.

Read the two headline numbers together. 0.846 against 0.473. The model represents how a voice sounds almost completely, and what it feels about half as well. That is the same asymmetry every generation-side experiment in this project has run into — timing control is solved to a median error of 0.08 s, emotional intensity is not — and it is now visible in the representation itself, before any generation happens.

The easiest and the hardest things to read

The pattern is not random. What is legible is what a listener could name in the first second — gender, register, whether the voice is monotone. What is illegible is what needs the situation to be understood: envy, awe, gratitude. Those are the emotions you cannot identify from a second of sound without knowing what is being talked about, and the model does not encode them either.

Two findings that were not expected

Resolution: re-running a probe with a different random seed moves a score by 0.003 on average and 0.011 at the 95th percentile. Treat two layers whose scores differ by less than about 0.01 as tied — for most attributes the "best layer" is a formality, with a plateau many layers wide.

3. W2 — does reading several layers at once help?

If layer 12 knows the voice and layer 20 knows the emotion, does feeding a probe both beat feeding it either? The search: greedy, beam and random selection over all 38 taps, for ensembles of 2 to 6, on six attribute groups and two corpora — 4,888 fitted combinations, 13 GPU-hours.

The seed replicates are what make this table honest. The two attributes with the largest raw gains — blend at +0.071 and genuineness at +0.030 for a single seed — are also the two noisiest targets, and once you refit at four seeds their gains sit at or inside their own error bars. The two attributes with modest gains are the ones you can rely on. Emotion gains ten times its noise band; voice quality gains fifty times its own.

The layers chosen are informative in themselves

For emotion the winning six are h06 · h12 · h18 · h20 · h21 · h23. Note h06: on its own it ranks thirteenth and no sensible shortlist would have included it, yet it enters at size four and stays. And note that the best pair, h18+h21, are almost the same subspace — their representational similarity is 0.979 — and still gain +0.020 together. Redundancy between two layers does not predict whether combining them helps.

An open question that got closed: performance style vs. time-independent traits

The 57 voice dimensions split naturally into two families: things about the performance (ranting, whispering, dramatic delivery, ASMR) and things about the instrument (age, gender, timbre, resonance). The hypothesis was that these live in different parts of the network.

The one attribute that genuinely lives somewhere else is vocal-burst blend — the only group whose ensembles reach past the semantic middle into h32, h33 and the local transformer, and which loses 0.02–0.08 R² if forced onto another group's layers. It is also the noisiest target, so this is suggestive rather than settled.

4. W4 — the vocal bursts

A separate question with a separate method: a sigh, a gasp, a chuckle, a mumble. Can a layer tell that a burst is present, and can it tell which burst it is? Scored as AUC — 0.5 is coin-flipping, 1.0 is perfect.

Both plateaus sit in layers 13–23 — the same band as emotion (layer 20), not the band of voice quality (layer 12). Presence and identity are read out of the same place, and that place is the emotional one. The blend scalar, by contrast, peaks six layers deeper at 25.

Per class

Telling a burst from plain speech is close to solved (0.94–0.99). Telling one burst from another is genuinely harder and varies by class — laughter is nearly as separable as it is detectable (0.95), while "ahem" against another burst drops to 0.75. Twenty-three further classes had too few usable clips to report a number at all and are listed with their counts rather than given one.

Three controls back the headline up. Clips whose bursts are all outside the window still separate at 0.778 — that is the clip-level confound floor, 0.20 below the headline, and it is why the window restriction is not optional. Burst-bearing clips on both sides, matched on burst count, still separate at 0.968. Strictly same-speaker negatives give 0.969, so speaker identity is not the signal.

5. W3 — the steering vectors, and the negative result

If an attribute is legible at layer 20, the obvious next move is to write it there: take the average hidden state of the angriest clips, subtract the average of the middling ones, and add the difference during generation. That library now exists — 156 archetype vectors × 38 layers × 2,560 numbers, 61 MB, one per (attribute, tail): 40 emotions at their high end, 57 voice dimensions at both ends, plus genuineness and blend.

Verification first: all 97 pre-existing vectors were recomputed from the raw activations and match the earlier export bit for bit (maximum difference exactly 0 over 97 × 38 × 3 vectors).

The random control is what makes this a result. At α = 0.5 and above a random direction did as much damage as a real one, which is why the first read was that the perturbation, not its content, was doing the work. Below 0.3 the two separate cleanly. Paired over the same 24 prompts, the real direction beats the random one on emotion by +0.135 at α = 0.10 (t = 3.11, higher on 16 of 24 prompts) and by +0.193 at α = 0.15 (t = 3.76, 17 of 24). The random direction moves emotion by +0.014 at α = 0.10 (t = 0.48) — nothing.

α = 0.10 is free. Emotion rises 0.435 → 0.584 while the word error rate falls from 0.166 to 0.115 and the share of clean clips rises from 0.84 to 0.91. Nothing was paid for that except genuineness.

Above 0.3 everything in the original grid still stands: at α = 0.5 the WER is 1.139, at 2.0 the duration error is 7.66 s, and a random direction at α = 1.0 reaches WER 2.065. The error was not in those numbers, it was in reading a curve from its far tail. 264 generations, 11 configurations, 24 prompts × 4 samples each, same scorers and same prompt template as every other evaluation in this project.

Why the far tail behaves as it does — three measurements, not a story

• The two tails of an attribute are not opposite directions. "High arousal" and "low arousal", measured from the middle, are at a median cosine of −0.0004 — they are perpendicular, not opposed. Pushing towards one is not pushing away from the other.
• The attributes are not linear, and the neural-vs-linear probe gap says so on all 99 of them. Across the 57 voice dimensions the non-linearity correlates at −0.41 with how opposed the two tails are: the more non-linear an attribute, the more both of its extremes sit on the same side of the middle. A difference-of-means vector is exactly the object that assumes the opposite.
• Emotion and perceived quality are anti-correlated inside the representation. The cosine between an emotion direction and the "quality" axis runs −0.62 to −0.95. Moving a voice towards an emotion moves it away from what the quality scorers reward — in the model's own state, before any audio exists.

Also worth knowing before using the library: 32 of the 40 emotion heads have a "low" tail that is a tie at zero, and for 25 of them the low bucket and the middle bucket are the same clips. "Negative anger" is not defined by this library. 34 of the 156 rows carry a small-bucket warning flag.

6. The one thing that does not fit

7. What this changes in practice

1. Stop trying to fix emotional intensity with the training objective. Five GRPO runs and seven preference runs did not move it; the representation now says the information is thin where it would have to be used. The only lever that has ever moved it is more extreme training data — the emotion adapters trained on the top 1 % moved the score from 0.408 to 0.492 monotonically in merge weight.
2. If you probe, probe the middle, and probe several layers. Layer 12 for anything about the voice, layer 20 for anything about feeling, layers 18–19 for bursts. Six layers instead of one is worth +0.039 on emotion and it is not a capacity artefact.
3. The steering vectors are usable at α ≤ 0.15, and only there. Below that the effect is direction-specific and the words survive; above 0.3 the output collapses. Anything built on them has to carry the genuineness cost and the random-direction control with it.
4. The capture window needs to move before vocal bursts can be studied properly. 82 % of bursts are invisible to the current extraction, and which ones are invisible depends on the class.
5. Genuineness is the one attribute where speaker identity matters. Any measurement of it under a lax split is inflated by about 0.04–0.07.

8. Where everything is

9. Corrections made while writing this

Four things in the earlier write-ups were wrong and are now fixed in the repository. They are listed because a research log that only records successes is not one.

• The split labels were inverted. The column called split is the speaker-disjoint one (0 of 13,757 test voices leak); split_uid, despite the name, splits on clip id and leaks 4,510 of 16,827. The probe page had them the other way round. All four agents found this independently.
• Two probe results carried the wrong split's number. On the speaker-disjoint split the emotion probe reaches 0.4328 at h20 (not 0.4394 at h18) and genuineness 0.6685 at h23 (not 0.7309 at h20). The tell was that both splits showed an identical value.
• Two artefacts were described as the wrong thing. norm_stats.json is the 130,785,282-row annotation ladder, not the reward percentile scale (which is a separate 3,147,802-row object); lora_bucket_index.json holds one bucket, not the full list.
• "The preference run with the worst training reward produced the best model" holds within a run and not between runs — the worst final value belongs to a run that ranks fifth on generation quality.

Method note carried over from W1: nothing — not a mean, not a standard deviation, not a regularisation strength, not an early-stopping decision — was ever fitted on test data, and the split was held fixed across all 38 taps.