← Manual hub
MOSS Voice-Acting — Manual & Studies Hub
A measured, audio-backed usage guide for laion/moss-tts-local-transformer-4.55b-voice-acting-v2 (4.55B expressive voice-acting TTS). For any target it gives the best prompt, which LoRA at which merge %, the expected reward, and measured side-effects. This is the agent-readable mirror of the HTML hub.
Newest (Aug 2026) — if you want a vocal burst at a specific place: put the tag inline in the SCRIPT and merge that class's burst adapter at λ 0.75–1.0. Measured over 1,440 generations: the prompt alone lands the burst 23.6 % of the time, the emotion adapter alone lowers that to 16.7 % (below no-adapter), and the burst adapter at 1.0 raises it to 71.9 %. λ 0.25 is within noise of no adapter. Keep any emotion adapter at ≤ half the burst dose. Cost is blend, not genuineness. Beware the truncation wall — the model can produce the burst and stop; never put a silence beat straight after it, and verify the words after the burst survive into an ASR transcript. Listen · full chapter
Also new — building a style LoRA: on the sports-commentator task, training on real recordings only beat a synthetic+real mix in 11 of 12 matched configurations (paired Wilcoxon p = 0.0010), and transferred across a language boundary (German training → English prompts). Adding synthetic data of the target style diluted the register.
From the burst/merge/evaluation run: for a vocal burst inside a sentence, λ = 0.75 is the knee (91 % of full-merge presence at a much better blend) and burst@1.0 + emotion@0.5 is the best cell of 2,304 generations — but a half-merged burst adapter paired with a half-merged emotion adapter is the worst, because the two compete. Cross-lingual reference audio (Japanese anime → DE/EN) does not transfer the voice — similarity 0.188 over 1,280 generations against a 0.105 unrelated-speaker floor — but pitch still tracks the reference (r = 0.591, compressed toward the middle), so it is a cheap source of new character voices rather than a cloning method. And: always evaluate against a real-audio ceiling control; a sports-commentator sweep looked like a win until real human commentary scored the same as the un-adapted base. See the burst & evaluation page.
New (Aug 2026): reward is × (1 − WER) (stops over-acted, unintelligible takes); at high energy add stabilizers vn_ARSH_high@0.5 + vn_BRGT_high@0.5 (+vn_S_NARR_high for storytelling) to keep speech clear; a cheap API brain (gpt-5.6-luna) matches the local Gemma-4-31B (google/gemma-4-31B-it-qat-w4a16-ct) at cents/task. Favourite narrator recipe: vn_S_STRY_high@0.65 + vn_ARSH_high@0.20 + vn_BRGT_low@0.20 → S_STRY 0.998, ESTH 0.59, WER 0.0, ~11s.
🚀 Start here — conditioning manuals
How to drive the model: for a target (emotion, voice dimension, delivery), what prompt + which LoRA at which merge % — with measured side-effects. Each has an agent-readable Markdown mirror.
- 🎭 Emotion conditioning manual (40 emotions) · MD — Per emotion: the best prompting strategy with and without the emotion LoRA, the best merge λ, expected reward, and the top-3 ± correlations on other emotions / VoiceNet dims / genuineness / blend / quality. Start here to make a voice feel something.
- 🎚️ VoiceNet dimension manual (57 dims × high/low) · MD — Per speech dimension (tempo, warmth, arousal, roughness…): the dose-response across 25→125% merge, the best dose, the cost to genuineness/blend/quality, and the ± side-effects. The reference for how a voice sounds.
- 🔁 Reinterpretation guide (procedural, at scale) · MD — A deterministic rule to pick LoRAs + scales + prompt from a clip's scores without an LLM call, so you can reinterpret a huge corpus (e.g. 1M DramaBox clips) — including the LoRA hot-swap / batch-bucketing answer.
- 🔥 Expressive & NSFW recipe book (agent warm-start) · MD — Validated, ready-to-use LoRA+prompt recipes distilled from autonomous agent runs — per emotional target (desire, ecstasy, teasing, dominance, anger, pain…) and the explicitness LoRAs. Written for a fast warm-start (human or agent). 18+, synthetic.
- 🎭 Contained / masked emotion (holding a feeling back) · MD — Updated (Aug 2026 — corrected by a paired-audio re-run). How to make an intense emotion sound held back (cold controlled rage, forced-calm terror, swallowed grief) instead of merely mild — and how to measure it. Turning the emotion down FAILED (→ sounds weak) and VoiceNet tension does not separate contained from free; the mechanism is emotional masking — keep the emotion present and drop vulnerability (
vn_VULN_low, light + prompt-led over a char_genuine/human scaffold) so the feeling stays HIGH while VULN drops below a matched free take. The corrected headline: masking is not a general trick. With real paired ΔVULN it works where the signal is effort/arousal/social-surface (Fatigue, Embarrassment, Pain, and the cognitive/low-arousal states Emotional Numbness / Intoxication / Helplessness — best at intense), and reverses where vulnerability is the emotion (Sadness +1.29, Pride +1.09, Distress, Affection read more exposed held-back → ship the free take); Fear/Doubt erase rather than mask. The old "Anger works / moderate contains, intense flattens" story was the four-anchor + sparse-unpaired pass and does not survive paired measurement. Includes the decision rule up front, the {moderate,intense}×{free,suppressed} 2×2, the target-band fitness, validated cues, and the full 40-emotion + deep-re-run tables. - 💥 Vocal bursts, merge doses & how to evaluate a LoRA · MD — New (Aug 2026). The measured dose curve for vocal-burst adapters inside a sentence (presence 27 %→71 % from λ 0.25→1.0, but blend 5.73→3.97 — λ 0.75 is the knee, and Sexual_Lust/Pain get worse at 1.0), how burst and emotion adapters interact (
burst@1.0 + emotion@0.5 is the best of 2,304 generations; a half-merged pair is the worst), cross-lingual reference audio as a source of new character voices (similarity 0.188 vs a 0.105 floor — not a clone; pitch tracks at r = 0.591 but compresses toward the model's register), and the evaluation checklist — real-audio ceiling control, multiple-comparison correction, and why validation loss has now failed to rank checkpoints three times. - 🗣️ Voice consistency across clips while the emotion changes · MD — New (Aug 2026). Self-contained recipe for keeping ONE character's voice across several clips while the emotion changes, matched to a reference recording.
mode="continuation" with an anchor + 4 s tail prefix (chaining from the full previous clip collapses identity, 0.691 → 0.692 → 0.280; anchor+tail gives 0.777 → 0.787 → 0.835), identity enforced rather than ranked (regenerate < 0.82, VC < 0.75, reject < 0.68, sequence scored ½ mean + ½ worst), the four continuation API gotchas, how to make the emotion actually turn, the seed trap, and why none of it applies to screams. No acting-challenge context needed. - 🎭 Acting challenges: running an autonomous casting agent · MD — New (Aug 2026). The consolidated playbook from ~2,000 generations across nine casting challenges: continuation from an anchor + 4 s tail (chaining from the full previous part collapses identity, 0.691 → 0.692 → 0.280), identity enforced rather than ranked (resample below 0.82, reject below 0.68, score speaker as ½ mean + ½ worst part → 0.662 → 0.784), why seeds must vary per task (all nine challenges drew seed 7110 and repeated the same burst — a listener caught it), never open or close a part with a burst (it lands on the seam), why the ranker was selecting against burst blending (weight 0.00 vs genuineness 0.34, correlation −0.21), the narration index for "it read the line instead of acting it", the saturating tempo dial (asking 4 gets 2.8), and that non-verbal parts are a different regime — every remaining identity failure is a scream.
- 📚 Acting challenges — the FULL record (page tree) · MD — New (Aug 2026). The complete consolidated record behind the playbook above: the best recipe for each of the nine challenges with full verbatim captions, adapters, doses and sampling; version history with every cross-generation number; what worked / partly worked / failed; metrics; all 351 prompts verbatim; and the edge-case swarm. Headlines you will not find on the summary page: the 8-term casting score is effectively a WER detector ((1−WER) correlates +0.91…+0.98 with the final score, genuineness +0.00); three of the eight terms are dead (
min(emo_peak,1) clipped in 92 % of assemblies, arc at exactly 1.0 in 84 %, qual an unnormalised near-constant worth 31 % of the score); the Gemini supervisor was rating presentation order — it picked the 4th or 5th take presented in 71 of 81 rounds and the agent's own top take in 3 of 108 — until the v7 rubric fixed it; and one whole run (out_v5_real) has its results paired with the wrong plans (every res.json is 2.2 h older than its req.json). - 🔤 Prompt notation — brackets, capitals, line length · MD — New (Aug 2026). The four notation rules, each one learned by breaking it. Round brackets for delivery cues and vocal bursts
(screams); square brackets for pauses [pause]; never capitalise a tag ((SCREAM) is read out letter by letter); and ≥10 words per line, because at ~2.8 words/s a shorter line cannot fill three seconds. Includes the trap that makes this expensive: the WER reference strips \(...\), so an angle-bracket tag stays in the reference and every burst-tagged take is charged word-error for not pronouncing "sobs" — the ranker selecting against the takes the tags were added to help. Plus the publishing bug where audio and scores came from different runs. - 🏭 Scaling the voice-profile corpus to 500 voices · MD — New (Aug 2026). Self-contained build note and cost appendix for going from one reference voice to 500: the 832-condition matrix per voice (320 emotion + 456 VoiceNet + 28 edge + 24 character + 2 sports + 2 explicitness), the two subsets that make it useful (A = 32 candidates each on a different sentence from that condition's pool, EN/DE parallel; B = 16 candidates on sentences shared across all sub-conditions of a parent, for controllability fine-tuning), 19,963,500 candidates, 40,408 h of audio, 3.02 TB. Costed in GPU-hours off one complete measured build: 15,731 GPU-h audio + 146 GPU-h text = 1,143,120 core-h at 72 core-h/GPU-h, 1.66 % of REFORMO, 2.6 days on 64 nodes. Includes the verified defect that blocks the run — the currently published profile was built from a group list where 0 of 808 groups carry
cond_key, so every group took the self-contained early-return path and became 32 takes of ONE sentence (VoiceNet: 114 distinct texts for 456 groups) — plus a runnable pre-flight checklist with expected outputs (all 832 groups carry cond_key, A pools exactly 32 / B pools exactly 16, EN/DE parallel at 99.961 %, all 233 adapter keys resolve offline), the text pipeline (90 % FineTranslations paraphrase / 10 % hard-for-TTS domains at 1 % each, spoken form, length buckets, burst quotas, the minors-and-sexual-content rule), the ranking formula in use, WebDataset/parquet formats with the full 76-column list, the 150 s worker stagger, and an honest risk list (the 10-word floor is not enforced on generated text: 298 lines fail it). - 🏗️ Running the 500-voice build on a supercomputer · MD — New (Aug 2026). The execution note for the page above: what actually happened when the 500-voice plan was run on JUPITER. One voice per Slurm array element (not a slice — the unit of failure should equal the unit of re-queue; a bad node in an 8-voice slice orphans seven healthy voices),
0-499%160, sized against the 12 h QOS wall rather than the partition size. The 150 s worker stagger and why it is not optional — four concurrent model loads off a parallel filesystem put every worker on the node into uninterruptible sleep, and a 45 s stagger cost a dead 45-minute job. Three-layer resumability (shard markers / group-level resume from the data / a five-way completeness gate that is the only writer of VOICE.DONE), including the trap that cost three runs — a failed group writes a row carrying its gid too, so an unconditional done.add(gid) turns any transient failure into a permanent hole no restart can fill — and why check 5 (parquet key set == tar member set) is not implied by the two counts. Measured 1,405 gen/GPU-h and 28.65 GPU-h in-process per voice, but 31.2 GPU-h billed — report the billed number, the gap is 9 %. Plus three silent defects found in the corpus that had already shipped: 5,157–5,220 Title-Case burst tags per voice that the model spells out letter by letter instead of performing (and that the WER stripper then charges word-error for), and a burst density realising 33.3 % of a configured 0.50 because the placement step drops any burst with no interior position in both languages and logs nothing — a knob whose realised value comes from a lossy downstream step is a request, not a setting. And a third, which affects any corpus built through the same code path — the LoRA merge weight decaying as λ^k because the cache of "pristine" scalings is re-snapshotted, after dosing, every time the adapter LRU loads or evicts (λ = 1.0 is immune, eviction resets it, and nothing in any output shows it). - ✍️ Text-side recipes: make the WORDS carry the condition · MD — New (Aug 2026). 106-arm ablation. Disfluency ladder span 0.44 → 2.07 by writing the hesitation into the line; the explicitness dimension's adapter does nothing on its own (ladder span −0.006); chunking is a negative result; and merging a burst adapter into an edge case halves speaker identity for a marginal emotion gain — inline tags on laughter only, never the adapter.
- ✍️ Writing the condition into the line · MD — New (Aug 2026). Which dimensions get better when you write the condition into the TEXT, and which get worse. Disfluency gains 4.7x span; chunking is actively harmed; the explicitness dimension is completely flat without it (span -0.006); edge cases gain intensity but lose blend and words. 106 arms, 1,696 generations.
- 🎬 Multi-part performances & voice conversion · MD — New (Aug 2026). How to build a ~30 s performance with a dramatic arc out of 2–4 parts without it sounding like three different actors: part 0 chooses the voice, every later part is generated with it as
reference=[...] (the argument build_user_message has always had and the first run never passed). Plus Chatterbox voice conversion as a repair when conditioning is not enough — chained, and kept only if ECAPA similarity rises and DNSMOS does not fall — crossfaded seams, ranking whole assemblies, and why a non-verbal part collapses to 0.3 s if you size tokens by word count. - 🏷️ The caption system: what a procedural caption means, and what just changed · MD — New (Aug 2026). How the procedural captions that describe voices are derived, and two defects repaired across 165,516,420 corpus rows on 2026-08-23. The emotion clause used an absolute threshold across 40 heads that sit on different scales, so it named Interest on 90.6 % of all captions and Bitterness on 0.1 % — it was reporting the scale of the head, not the emotion of the clip; a top-10 %-per-emotion percentile gate over a 132,833,726-utterance ECDF takes Interest to 5.3 %, makes all 40 emotions occur, and lets a clip honestly say "no dominant emotion" (17.88 % do). And
GEND/BKGN were rendered backwards — high GEND is masculine (+0.842 with chest resonance; 89.2 % agreement against 379 design-spec genders) and high BKGN is cleaner (+0.786 with recording quality), so 83.1 M rows changed the gender word and 79.5 M changed the background word, ending captions that read "good recording, very noisy background". The numeric columns were always right — only the prose inverted, so if you trained on caption text you learned the opposite of the truth for those two attributes. Includes what to do if you train on captions (the old string is preserved verbatim in caption_general_v1), why you must never flip the words yourself, and why the two captioners are not interchangeable. - 🔊 Edge-case delivery (screams, crying, laughter…) — Reward-maximised recipes (prompt + λ + sampling) for hard non-speech-heavy deliveries — fear-scream, pain-scream/groan, sobbing, shivering, genuine laughter — evolved with and without the LoRA.
🔬 Interactive studies & audio grids
Listen to the effects. Every clip is labelled with its recipe and scores.
- ⭐ Emotion × Voice conditions — 6 cloned voices × 40 emotions × 4 delivery conditions (intense/moderate × free/contained), best-of-32 with a speaker-similarity floor. The flagship study — shows that intensity, not containment, breaks a cloned voice.
- 🧬 40-emotion evolution — Evolutionary search (10 generations, mean-of-8) for the best λ / prompt / sampling per emotion — the source of the per-emotion recipes used elsewhere.
- 🎛️ Prompt × LoRA strength study — Neutral vs prompt vs LoRA @50/100/150% for all 40 emotions, with a cross-steering heatmap — shows when the LoRA helps and when a good prompt alone wins.
- 💥 Edge-case evolution — The audio grid behind the edge-case recipes: screaming, crying, laughter, groaning, shivering, maximised with/without LoRA.
- 🎚️ VoiceNet LoRA evolution (dose 0/50/100%) — Per-dimension controllability + best prompting at three doses — 50% already delivers most of the shift for most dimensions.
- ↕️ VoiceNet high/low scaling grid — 57 dimensions × high/low × 5 merge doses (25→125%), 3 audio samples each — hear the full scaling sweep per dimension side by side.
- 🗣️ Character-voice clusters — VoiceNet-embedding clusters of the generated character voices with procedural captions — browse the space of distinct synthetic voices.
- ♻️ Character LoRA refinement grids — 120 character voices refined by SIDON self-distillation (generate → filter → restore → warm-start FT) — before/after quality per character.
🤖 Autonomous search agent
An agent (local or API LLM brain) that searches for merge + prompt recipes on its own, guided by the manuals above and scored by the full perceptual stack.
- 📊 Results & model comparison — Single-dimension optimisation, the 4-brain comparison (gpt-5.6-luna vs Gemma-4 31B/12B/MoE, side by side with audio), the supervised edge-case swarm, and the corrected per-dimension runs.
- 🏆 Which brain? (MODEL_COMPARISON.md) · MD — Gemini-judged, same-budget: gpt-5.6-luna (API) scored 9/9/10 — tied with the local Gemma-4 31B (9/10/9), ahead of 12B/MoE. Both independently found the stabilizer adapters.
- 💸 Cost estimates (COSTS.md) · MD — Measured token/$ figures: an API brain runs a search for cents per task (luna: $0.31 for 4 missions, no GPU), and supervision is ~$0.007/verdict — see the 100/1000-task projections.
- 🐝 Swarm design (SWARM_PLAN.md) · MD — The plan for scaling to a 32–64-agent self-improving swarm: two-tier hive-mind knowledge (raw experience vs. approved manual), task registry, supervisor loop, offline consolidation.
- 🧠 What the agents have learned · MD — The distilled, validated learnings folded into every agent's context: winner recipes, the stabilizer adapters, the (1−WER) reward, and the task-difficulty map (what the TTS model can and cannot do).
📖 Taxonomies (definitions)
The vocabularies everything is scored against.
🤗 Models & adapters (Hugging Face)
The checkpoints and LoRA collections referenced throughout.
- Base model — moss-tts…voice-acting-v2 — The 4.55B expressive voice-acting TTS model everything conditions on.
- Emotion LoRAs v3 (40) — One rank-64 LoRA per emotion (≤25% Got-Talent), the adapters behind the emotion manual.
- VoiceNet dimension LoRAs (114) — 57 dimensions × high/low, the adapters behind the VoiceNet manual and the high/low grid.
- German Mediathek HQ LoRAs (3) — Rank 16/32/64 trained on 43,612 segments / 185 h of high-quality expressive German public-broadcast speech (emotion half at EmoNet > 2.5 and class-balanced, plus an equal quality half ranked by vocal-burst blend + genuineness). Take
r64_epoch3. German adapter — English output runs long (10–22 s for a 13-word line vs 3.7–4.8 s for German). 🎧 7 emotions × DE/EN, with and without the emotion adapter - Vocal-burst LoRAs (64) — One adapter per non-verbal burst (chuckle, gasp, sob, scream, moan, whistle…), each confirmed by a listening audit against real recordings and a no-LoRA baseline. Read the merge-dose page before choosing λ.
- Sports-commentator LoRAs (6) — Use
real_r64_e8 — the default for energetic, real-sounding sports commentary, chosen by listening rather than by metric. Trained on real German broadcast segments only; the real-only run beat the mixed synthetic+real run in 11 of 12 matched configurations (paired Wilcoxon p = 0.0010). real_r32_e2 is the low-WER/half-size alternative. Ships with its own negative result: on this task the base model already matches real human commentary on the judge metric, so the adapters buy consistency and intensity, not capability. Side-by-side A/B. - Character LoRAs — refined (120) — 120 SIDON-refined from-scratch character voices (public mirror; a genuine-voices public repo also exists).