Prompting-style ablation & sample-size scaling
MOSS-TTS-8B voice-acting · structured two-part performance prompts · no reference audio · audio temp 0.8, rep-pen 1.1, text temp 0.7, top-p 0.95, top-k 25
CUT TO emotion change,
and a closing performance note — and ablate one thing: whether the quoted lines stay inside the
instruction (Variant A) or are stripped out so the instruction only describes how to perform while
the what is passed separately (Variant B). 25 prompts (5 from each of five pathways) × 2 variants ×
16 no-reference takes = 800 clips. Every clip is scored for transcription fidelity (Parakeet invWER
against the parsed spoken text), perceptual blend quality and genuineness (VoiceCLAP), and combined into
reward = invWER × (norm blend + norm genuineness), min–max normalized across all generations.
Two pathway groups: Character-consistent (a fixed voice/character) and Scenario-driven
(situation- or challenge-led).1 · Which prompting style is better?
Overall (all 25 prompts)
| Metric | Variant A (with quotes) | Variant B (no quotes) | Δ (A−B) | Winner |
|---|---|---|---|---|
| Mean combined reward | 0.564 | 0.536 | +0.028 | A |
| Best-of-16 combined | 1.161 | 1.091 | +0.070 | A |
| Mean genuineness | 1.175 | 1.174 | +0.001 | A |
| Best-of-16 genuineness | 1.949 | 2.054 | -0.105 | B |
| Mean blend (0–10) | 4.42 | 3.56 | +0.86 | A |
| Best-of-16 blend | 7.53 | 7.22 | +0.31 | A |
| Mean 1−WER | 0.633 | 0.652 | -0.019 | B |
| Best-of-16 1−WER | 0.984 | 0.988 | -0.004 | B |
| Selection gain (best16−mean) | 0.597 | 0.555 | +0.042 | A |
Paired by prompt (each prompt rendered under both variants): Variant A wins best-of-16 combined reward on 72% of prompts (mean paired Δ +0.070).
Character-consistent pathways
| Metric | Variant A (with quotes) | Variant B (no quotes) | Δ (A−B) | Winner |
|---|---|---|---|---|
| Mean combined reward | 0.587 | 0.610 | -0.023 | B |
| Best-of-16 combined | 1.149 | 1.114 | +0.035 | A |
| Mean genuineness | 1.099 | 1.212 | -0.113 | B |
| Best-of-16 genuineness | 1.899 | 2.123 | -0.225 | B |
| Mean blend (0–10) | 4.28 | 3.42 | +0.87 | A |
| Best-of-16 blend | 7.33 | 6.65 | +0.68 | A |
| Mean 1−WER | 0.690 | 0.773 | -0.083 | B |
| Best-of-16 1−WER | 0.997 | 1.000 | -0.003 | B |
| Selection gain (best16−mean) | 0.562 | 0.504 | +0.058 | A |
Scenario-driven pathways
| Metric | Variant A (with quotes) | Variant B (no quotes) | Δ (A−B) | Winner |
|---|---|---|---|---|
| Mean combined reward | 0.549 | 0.487 | +0.062 | A |
| Best-of-16 combined | 1.169 | 1.076 | +0.093 | A |
| Mean genuineness | 1.225 | 1.149 | +0.076 | A |
| Best-of-16 genuineness | 1.982 | 2.007 | -0.025 | B |
| Mean blend (0–10) | 4.50 | 3.65 | +0.85 | A |
| Best-of-16 blend | 7.67 | 7.60 | +0.06 | A |
| Mean 1−WER | 0.595 | 0.571 | +0.024 | A |
| Best-of-16 1−WER | 0.976 | 0.979 | -0.004 | B |
| Selection gain (best16−mean) | 0.620 | 0.589 | +0.031 | A |
2 · Sample-size ablation — best-of-k
k∈{4,6,8,16} takes without replacement (R=30 repeats, averaged)
and record the best take's reward / genuineness / blend, then average over groups.| Series | best-of-4 | best-of-6 | best-of-8 | best-of-16 |
|---|---|---|---|---|
| Variant A · reward | 0.894 | 0.989 | 1.054 | 1.161 |
| Variant A · genu | 1.635 | 1.736 | 1.808 | 1.949 |
| Variant A · blend | 6.442 | 6.839 | 7.122 | 7.531 |
| Variant B · reward | 0.897 | 0.970 | 1.008 | 1.091 |
| Variant B · genu | 1.670 | 1.786 | 1.877 | 2.054 |
| Variant B · blend | 5.811 | 6.316 | 6.604 | 7.222 |
3 · Quality vs sample size × prompting style
Quality rises with k for both styles but with diminishing returns: going 4→16 adds +0.267 combined reward for Variant A and +0.194 for Variant B, and most of that is banked by k=8 (the 8→16 step adds only +0.107 / +0.083). Best-of-8 captures the bulk of the best-of-16 selection gain; drawing 16 buys a little extra ceiling, mostly on genuineness. Thin dotted/dashed lines split the two pathway groups.
Throughput
Generation is the bottleneck. Sharded across 2× A100-80GB (no reference audio, batch-16 per generate call), the two GPUs sustained ~0.42 valid clips/s each (~0.8/s combined) over the full 800-clip run. The main throughput tax is over-generation: because the audio codec runs at ≈12.6 fps, takes that fail to emit an end token ramble on. Variant A (quoted lines left in the instruction) rambles markedly more than Variant B — mean clip 36s vs 26s, and 57% vs 34% of takes run past 40s — so A both costs more compute and needs more re-draws to fill a clean pool. We cap max-new-tokens to the expected two-part duration so a runaway is truncated and still scored (honestly penalised on WER) instead of silently wasting the batch.