Voice-Acting Best-of-32 — can 32 takes beat the reference voice?

What is this?

We took 50 reference voice clips from a public emotion-attribute reference set — each clip is bucketed by a distinct emotional attribute (anger, grief, ecstasy, awe, longing, …) at meaningful intensity, and carries a natural-language voice caption + transcription. For every reference we wrote two fresh, emotionally-matched 3-sentence scripts — one English, one German (native scripts, not transliterated) — plus a delivery instruction. That is 50 refs × 2 languages = 100 prompts. For each prompt we asked our fine-tuned MOSS-TTS 8 billion-parameter model to voice-clone the reference and perform the script 32 times in a single batched call (32 independent takes / seeds). That is 3195 generated clips across 100 groups. Both languages reuse the same reference audio.

Every take — and every original reference — is scored by two independent VoiceCLAP reward heads: a blend quality head (0–10) and a genuineness head (0–6, “how real / human does this sound”, VoiceCLAP-commercial). Word-error-rate (WER) from Parakeet-tdt-0.6b-v3 gives invWER = max(0, 1−WER) to gate garbled takes.

Note: the reference clips are reference snippets used purely as voice/emotion anchors; provenance is kept generic. Each shown group embeds the top-8 + bottom-2 takes by combined reward — a subset of 32 for page size; every uplift number below uses all 32.

In plain words

The model is cloning each reference voice. The question: if we generate 32 takes and keep the best one, can that best take sound more genuine and better-blended than the real reference clip it was copied from? A positive Δ (uplift) means yes — the synthetic best-of-32 out-scores the original on that reward. This says the reward-guided best-of-N loop is not merely imitating the reference, it is improving on it on what these quality/genuineness heads reward.

Best-achievable uplift (ceiling) compares the reference to the single best-scoring take on each axis. We also report the combined-reward pick — the one take you would actually ship — and how often it beats the reference.

If you must ship ONE take chosen by the combined reward (the realistic scenario):

Bigger groups → higher ceiling: best-of-10 vs best-of-32

Intuition check: the more takes you draw, the higher the best-take you can find, so the achievable uplift ceiling rises as the group grows. Below, the mean best-achievable Δ using only the first 10 seeds vs all 32 seeds — the gap is the extra headroom that the larger batch buys you for free.