The question. A DramaBox prompt mixes what to say (dialogue in "quotes") and how to say it (stage directions). MOSS-TTS takes a separate text field (what) and instruction field (how). We always put the extracted dialogue in the text field, and ablate the instruction two ways:
· with_dialogue — instruction = the complete DramaBox prompt, dialogue still quoted inside.
· no_dialogue — instruction = stage directions only, the quoted dialogue removed (it already lives in the text field).
Crossed with a reference-audio arm (none @ temp 0.8 vs a random emotion snippet @ temp 1.0), over 25 prompts (15 character-consistent + 10 single-emotion), 16 takes each. Ranked by reward = invWER · (norm blend + norm genu).
| style | groups | best-16 reward | mean reward | invWER | WER | blend | genu |
|---|---|---|---|---|---|---|---|
| with_dialogue | 50 | 0.921 | 0.471 | 0.686 | 0.355 | 3.088 | 1.547 |
| no_dialogue | 50 | 0.947 | 0.437 | 0.624 | 0.425 | 2.826 | 1.554 |
| pathway | style | best-16 reward | mean reward | invWER |
|---|---|---|---|---|
| cc | with_dialogue | 0.852 | 0.420 | 0.664 |
| cc | no_dialogue | 0.854 | 0.351 | 0.574 |
| noncc | with_dialogue | 1.025 | 0.548 | 0.720 |
| noncc | no_dialogue | 1.086 | 0.565 | 0.700 |
| reference | groups | best-16 reward | mean reward | invWER | genu |
|---|---|---|---|---|---|
| no_ref | 50 | 0.998 | 0.491 | 0.669 | 1.384 |
| emo_ref | 50 | 0.870 | 0.417 | 0.641 | 1.716 |
Adding a random emotion reference changes best-of-16 reward by -0.128 (26% of prompts better with the ref). Note the reference emotion is random w.r.t. the prompt, so this measures whether an unrelated emotional voice helps or fights the written direction. A full "does emotion follow the ref or the prompt?" attribution needs an emotion classifier on the outputs — flagged as follow-up.
| k | with_dialogue | no_dialogue | no_ref | emo_ref |
|---|---|---|---|---|
| 1 | 0.470 | 0.439 | 0.493 | 0.417 |
| 2 | 0.619 | 0.601 | 0.662 | 0.558 |
| 4 | 0.741 | 0.746 | 0.800 | 0.688 |
| 6 | 0.800 | 0.817 | 0.870 | 0.747 |
| 8 | 0.836 | 0.857 | 0.905 | 0.788 |
| 12 | 0.888 | 0.911 | 0.962 | 0.837 |
| 16 | 0.921 | 0.947 | 0.998 | 0.870 |
prompt-sim = VoiceCLAP-commercial cosine similarity between the direction text (the DramaBox prompt with the spoken dialogue removed) and the generated audio. It asks: does the voice actually perform "how to say it", separate from saying the right words? (These cosines sit around 0; the relative differences are what matter.)
| mean prompt-sim | with_dialogue / no_ref / cc | no_dialogue / emo_ref / noncc |
|---|---|---|
| by prompting style | -0.119 | -0.120 |
| by reference | -0.096 | -0.144 |
| by pathway | -0.096 | -0.155 |
| k | prompt-sim of reward-pick | prompt-sim of prompt-sim-pick (ceiling) |
|---|---|---|
| 1 | -0.119 | -0.119 |
| 2 | -0.117 | -0.059 |
| 4 | -0.115 | -0.009 |
| 6 | -0.113 | +0.017 |
| 8 | -0.111 | +0.034 |
| 12 | -0.110 | +0.057 |
| 16 | -0.108 | +0.072 |
Key result: reward and prompt-sim are almost uncorrelated (r = -0.041). Selecting best-of-k by reward barely lifts prompt adherence (-0.119 → -0.108), while selecting by prompt-sim could reach +0.072. The emotion reference lowers adherence (-0.144 vs -0.096 no-ref) — a random-emotion voice pulls the delivery toward the reference and away from the written direction. If prompt adherence matters, add prompt-sim into the reward — best-of-N on reward alone won’t get it.
Prompting: keep the dialogue in the text field either way; for the instruction, no_dialogue gave the higher best-of-16 reward (Δ 0.026). Stripping the dialogue from the instruction (directions-only) helped — the model isn't confused by seeing the words twice.
Reference audio: a random-emotion reference did not improve reward (Δ -0.128). Since it's emotion-random, prefer a reference whose emotion matches the prompt, or omit it and rely on the written direction.
Sample size: see the curve — most of the best-of-k gain is captured by k≈8; best-of-16 adds a little more. For bulk generation best-of-8 is a good quality/cost point.
Prompt adherence: the reward barely tracks how well the delivery matches the written direction (r≈0). If matching the performance direction matters, fold prompt-sim into the reward or select on it — best-of-N by reward alone does not improve it. And drop the random emotion reference: it measurably pulls the delivery away from the prompt.
1,597 scored takes · 25 prompts × 2 styles × 2 refs × 16 · no-ref temp 0.8 / rep 1.1 · emo-ref temp 1.0. Audio grids linked at top.