LAION · Expressive TTS · research log

MOSS-TTS-Local 4.55B Voice-Acting — Experiments & Grids Overview

Every experiment, grid, temperature sweep, best-of-N study and reward ablation run on this box with the 4.55B MOSS-TTS-Local transformer voice-acting model — the local-transformer (Qwen3-4B backbone, 12-codebook RVQ @ native 48 kHz) model published on Hugging Face and GitHub. This is the local 4.55B model, not the 8B "delay" sibling.

moss_tts_local Qwen3-4B backbone 12-codebook RVQ 48 kHz native MOSS-Audio-Tokenizer-v2 EN + DE vocal bursts

Live grid · production best-of-64 (40 emotions) Live · fast-inference write-up HF model GitHub repo

TL;DR — the load-bearing takeaways

Best sampling: audio_temperature 0.8 without a reference clip, 1.0 with a reference; fixed top_p 0.95, top_k 25, repetition_penalty 1.1, text_temperature 0.7. A no-reference temperature sweep (1000-take depth) confirmed t=1.0 is the quality-optimal point for best-of-N (invWER peaks 0.94); hotter sampling makes more bursts but worse intelligibility and does not produce better cherry-picked takes.
Best-of-N is the single biggest quality lever. Random take ≈ 0.30 blended reward → best-of-1000 ≈ 0.76 (2.5×). Diminishing returns knee at N≈25–50 (captures ~73%/90% of the 5→100 gain); premium runs go to 64–128.
Best reward for shipping: intelligibility-anchored. Production uses reward = invWER × (norm emotion + norm blend + norm genuineness). Of the V1–V6 rerank variants, V4 (1.2·10·invWER + blend) is the practical sweet-spot (cleanest WER 0.052, still blends), V3 (invWER·blend) maximises burst naturalness (blend 5.3/10).
Bursts: put the burst cue where training put it — as a (round-bracket) director note in the instruction, keep the spoken text clean. Writing bursts inline in the text ~5× the WER (the model reads the brackets aloud). The 100k inline-LoRA, bursts-in-instruction won the FAIR eval (reward 1.017 best-of-8, WER 0.067).
Voice cloning: no-reference generation beats code cloning ~2× on quality (BR 0.327 vs 0.136); cloning fidelity trades off against audio quality. For a consistent identity without a real reference, Chatterbox VC (s3gen v3) + Sidon imposes the target speaker at 95–100% objective win-rate.
Post-processing: this model is native 48 kHz — keep the output RAW; do not run Sidon (a 48→16→48 round-trip that softens it). Sidon is only for the 24 kHz 8B sibling.
Best checkpoint: the released model = V1 DramaBox full-FT + rank-256 LoRA (α=256) continued to 1M samples, merged (252 deltas). The 100k inline-burst variant is the dedicated burst model; the 100k full fine-tune is marginally the best raw burst-firer but loses on the intelligibility-anchored reward.

1 The model & checkpoint lineage 2 Temperature & sampling scaling 3 Best-of-N & reward variants (V1–V6) 4 VoiceCLAP-small "blend" scorer 5 FAIR eval — burst placement & checkpoint 6 Character / emotion / vocal-burst generation 7 Voice cloning & Sidon restoration 8 Production best-of-64 emotion grid + Fast-inference & VoiceCLAP pilot (support)

1The model & checkpoint lineage

A 4.55B parameter moss_tts_local model: a Qwen3-4B language backbone with a small per-frame local-transformer head that predicts 12 RVQ codebooks per 80 ms frame (12.5 Hz), decoded by MOSS-Audio-Tokenizer-v2 to native 48 kHz. Trained for expressive voice-acting (orc / dragon / fairy / goblin, shouting, whispering, vocal bursts) driven by natural-language director instructions, in English and German.

Checkpoint lineage (all 4.55B local; `/tmp/moss_train/output/`)

Stage	Checkpoint	What	Recipe
Upstream	MOSS-TTS-Local-Transformer-v1.5	OpenMOSS base (4.55B, 12-cb @48k)	—
V1 base	moss_dramabox_ft / checkpoint-last	full fine-tune on DramaBox voice-acting	LR 3e-5, 2 ep, global batch 128, channelwise loss 1:2.67, warmup 0.03 linear
1M LoRA	merged_1m (checkpoint-1m)	rank-256 LoRA → released model	1M samples / 10 ep, LR 1e-4
burst LoRA	merged_inline	100k inline vocal-burst LoRA (primary burst model)	100k / 1 ep, LR 2e-4
burst full-FT	moss_fullft_inline / checkpoint-last	100k full fine-tune (burst-in-text data)	all-param FT
burst adapter	moss_lora_r256_inline	the inline LoRA adapter (pre-merge)	r256, α256

LoRA config (all revisions): r=256, α=256 (scaling 1.0), dropout 0.05, targets q,k,v,o,gate,up,down _proj across 36 layers = 504 tensors; applied by merging B@A into each linear (252 deltas, ~30 s), non-destructive. Base loads with audio_lm_heads.{0..11} + text_lm_head reported MISSING — benign weight-tying.

Training data mix (50/25/25): 50% voice-acting subset (vocal-burst / strong-emotion filtered: dramabox_gemini_ft + emotional_va + gemini_dramabox), 25% Gemini-3.1-Flash-TTS audio-matched fixed prompts, 25% other burst-filtered (ears + German emolia_de + character_voices + gemini). 8×GPU DDP, bf16, gradient checkpointing.

Merged model (Apache-2.0) V1 base (moss-dramabox-ft) Repo · README + data recipe

2Temperature & sampling scaling

dirs: /tmp/vbtemp (temp ablation + depth), /tmp/vbexp (4-model burst grid), /tmp/vbexp2 (burst-recovery best-of-N)

What was tested. The temperature that maximises quality with vs without a reference clip, plus depth (codebook / pool depth) and top-1/3/5 variance. Method: fixed rep-penalty 1.1, top-p 0.95, top-k 25; deterministic per-sample seeds so temperature is a clean axis. Reward = blended reward BR = invWER × (blend/10) (blend = 0–10 MLP quality score). vbtemp: 6 burst-rich prompts, no reference — a breadth sweep (t 0.7–1.7 × 50 takes) → pick best temp → depth run of N=1000 takes at that temp (≈5,991 scored).

Best temperature: audio_temperature 0.8 without a reference, 1.0 with a reference (production recipe). The no-reference sweep independently found t=1.0 quality-optimal for best-of-N: invWER peaks 0.94 at t1.0 and degrades above; burst-detection rate rises monotonically with temperature (0.21 → 0.35 at t1.7) — hotter = more bursts but worse intelligibility.

temp	top10 BR	meanBR	invWER	burst rate	best blend
0.7	0.482	0.292	0.862	0.260	6.88
0.8	0.492	0.287	0.918	0.213	6.89
0.9	0.530	0.310	0.936	0.213	6.97
1.0 ◄	0.533	0.304	0.940	0.220	7.14
1.1	0.517	0.304	0.923	0.307	7.14
1.2	0.501	0.292	0.917	0.317	7.15
1.3	0.504	0.297	0.938	0.327	7.05
1.4–1.7	0.49–0.51	~0.30	~0.92	0.33–0.35	~7.1

Variance (top-1/3/5) & depth scaling

No "good variance" at high temperature. top-1, top-3, top-5 and top-10 blended reward all peak at t=1.0 (top-1 BR 0.686 → 0.61–0.63 by t1.7). The extra spread at high T is garbled takes, not occasional gems — the best cherry-picked take does not shift to a hotter temperature.

pool N (at t1.0)	5	50	100	500	1000
top-1 BR	0.554	0.686	0.713	0.752	0.759
top-5 BR	0.326	0.593	0.638	0.700	0.719
top-1 / random-mean	1.70×	2.25×	2.38×	2.48×	2.51×

Random single sample ≈ 0.30 BR → best-of-1000 ≈ 0.76 (2.51×). Breadth-then-depth compounds: top-5-of-50 at best temp (0.593) → top-5-of-1000 (0.719) = +21%. Depth ceiling is prompt-dependent (Primal-Fear best-of-1000 0.904; Orc-Shout caps ~0.61 — the model garbles the shout).

Token budget = pacing dial (not a limit). MOSS fills whatever frame budget you give it: tokens = clamp(words × 6.0, 96, 1400) frames (≈2.1 words/s) avoids mid-sentence cut-offs; words × 4.2 force-rushes and truncates 96.9% of slow emotional takes. max_new_tokens = 2.2 × tokens + 300.

3Best-of-N reranking & reward variants (V1–V6)

dirs: /tmp/vbrerank (V1–V6 rerank), /tmp/vbexp2 (best-of-N group-size + burst recovery)

What was tested. How large a candidate pool to generate, and which reward to rank it by. Setup: 24 groups (6 prompts × 4 reference voices) × 100 candidates = 2,400 clips from the 100k inline-burst LoRA, scored by Parakeet-WER, VoiceCLAP cosines, the "blend" MLP (0–10), and the vocalburst-locator detector. Base best-of-N reward: R = (cos_prompt + cos_nat + 2)·invWER.

Best-of-N group-size (diminishing returns)

25–50sweet-spot N (73% / 90% of gain)

2.134mean reward @ N=100

0.72→0.86burst-prob of best pick, N 5→100

N	5	10	15	25	50	100
mean reward (24 cells)	1.727	1.885	1.955	2.024	2.094	2.134

Reward and bursts are positively correlated (at t1.2 with the both strategy): selecting harder for reward picks more-bursty takes — burst-prob of the best pick rises 0.72→0.86 as N grows, and the burst-detection rate climbs from 40% overall to 66% within the top-25% by reward. The bursts↔intelligibility trade-off that plagues single takes effectively disappears under best-of-N.

Reward variants V1–V6 — definitions & which won

Each variant re-ranks the same 100-candidate pools; the "blend" term is the VoiceCLAP-small scorer (§4). Blended reward shorthand = invWER × (blend/10).

variant	definition	top-1 blend	WER	burst-det	role
V1	top-20% by R, re-order by blend	3.94	0.130	62%	balanced
V2	R2=(cosP+blend/20+2)·invWER	2.79	0.022	67%	blend under-weighted
V3 ◄ max-blend	R3 = invWER · blend	5.32	0.253	54%	most natural bursts
V4 ◄ ship	R4 = 1.2·(10·invWER)+blend	3.69	0.052	62%	cleanest WER + good blend
V5	R5 = (10·invWER)+blend	5.06	0.177	46%	equal-weight
V6	V5 on gated clips (dur≥mean & invWER>0.6)	3.53	0.041	67%	quality-gated

V4 is the recommended shipping reward — cleanest WER (0.052) while still blending and firing a real burst 62% of the time. V3 (invWER·blend) maximises burst naturalness (5.3/10) at the cost of prompt-fidelity and WER. Bigger pools help V1 most (blend 2.57→3.94 as G 15→100); V3 already extracts the best-blending clip from any pool.

Production reward family (recipe): reward_orig = invWER·(norm blend + norm genu), reward_all = invWER·(norm prompt_sim + norm blend + norm genu), reward_emo = invWER·(w·norm emotion + norm blend + norm genu) (w≈1.5). reward_orig and prompt_sim are near-uncorrelated (r≈−0.04) — if direction-adherence matters it must be in the reward. Optional hard gate WER<0.1 for intelligibility-critical corpora.

4VoiceCLAP-small "blend" scorer

dirs: /tmp/vbblend (dataset), /tmp/vbmlp (arch ablation), /tmp/vbscale (scale-up), /tmp/vbablate (grid + decontam), /tmp/vcblend_pkg (packaged)

What it is. A tiny MLP on frozen 768-d laion/voiceclap-commercial embeddings that predicts, 0–10, how naturally a vocal burst blends into the surrounding speech — a fast proxy reward for ranking best-of-N. Ground truth: Gemini-3.1-Pro ratings on a fixed rubric (0 = spliced/wrong-emotion, 5 = fits but "performed", 10 = totally organic).

Dataset → scale

Started at 1,500 clips (500 fresh burst-detector-positive LoRA takes + 1,000 inline-positive source clips), Gemini-labelled (0 errors), strongly bimodal (mean 5.37; 26% at 10, 15% at 0). Expanded with no-burst hard-negatives to 2,500, then scaled: Gemini-3.1-Pro annotated 12,998 more → final 15,843 train + 110 val (14,389 Gemini-labelled + 1,454 detector-confirmed hard-zeros).

Architecture ablation

small (768-d commercial) vs large (3584-d voiceclap-large-v2) × reg (Huber) vs cls (CE), ~20k-param budget. small-reg won (val MAE 2.166, corr +0.616); the 4.7× bigger large-v2 did not beat it — "most blend signal is already in the cheap 768-d embedding". Regression > classification.

Ablation grid + decontamination (the honest-eval story)

24-run grid: arch {small/base/big/deep} × dropout {0.2/0.33/0.5} × mixture {as_is/balanced}. Val-grid winner = big_d0.2_as_is (MAE 2.327). But the tiny 110-clip val was rewarding overfitting: on a fresh 275-clip decontaminated test (no shared source audio), scaling the data reversed the ranking.

config	train n	test MAE	test corr
base_d0.2_as_is · scaled ◄ default	15,568	2.332	+0.504
big_d0.2 · scaled	15,568	2.392	+0.503
base_d0.33 · OLD 1.6k data	1,630	2.701	+0.339
big_d0.2 · OLD 1.6k data	1,630	2.861	+0.337

Scaling helped — on the decontaminated test the scaled scorer (MAE 2.33, corr 0.50) clearly beats the old-recipe head (2.70 / 0.34). Mid-range blends (4–6) stay the least reliable; the frozen encoder is the accuracy ceiling. The final packaged default is base_d0.2 (Linear(768,25)→GELU→Drop0.2→Linear(25,1), ~19k params), shipped in /tmp/vcblend_pkg as a standalone scorer.

5FAIR eval — burst placement & checkpoint

dirs: /tmp/fair3 (2×2 burst-placement), /tmp/va_eval + /tmp/va_eval2 (harness evolution)

What was tested. A fair 2×2: model {100k inline-LoRA, 100k full-FT} × burst placement {in the instruction only (clean text) / written inline (…) in the spoken text}. 3 prompts × 5 ref voices × {EN,DE} × 8 takes = 240 clips/condition, temp 1.0. Reward = (voiceclap_cos + 1) × (1 − WER), WER from Parakeet-tdt-0.6b-v3.

condition	μ R_prompt	μ WER	best-of-8 R	best-of-8 WER
inline-LoRA · bursts in instruction (clean text) ◄	0.885	0.067	1.017	0.006
inline-LoRA · bursts inline in text	0.598	0.370	0.953	0.068
full-FT · bursts in instruction	0.754	0.202	0.998	0.005
full-FT · bursts inline in text	0.637	0.332	1.011	0.032

Winner: the 100k inline-LoRA with bursts in the instruction only (clean text) — highest reward (μ 0.885 / best-of-8 1.017) and lowest WER (0.067). Two robust effects: (a) writing bursts inline in the text roughly 5× the WER (0.067→0.370) because the model reads the bracketed cue as words; (b) the full fine-tune underperformed the LoRA on this intelligibility-anchored reward.

Harness evolution (va_eval → va_eval2 → fair3): va_eval used [square-bracket] bursts inline in the text — the opposite of the training convention — which a training-data audit exposed (bursts live in the instruction ~100% of the time, dominant syntax = round brackets (), 100% in dramabox_gemini_ft, 96% Gemini-3.1). Raw VoiceCLAP cosines proved too noisy to rank (all near 0); adding Parakeet-WER to the reward is what gave a clean signal.

6Character / emotion / vocal-burst generation

dirs: /tmp/vbchar (30-prompt char/emotion study), /tmp/vbexp (4-model burst grid), /tmp/vb_filter (training-data filter)

What was tested. 30 acting prompts (15 human emotional scenes + 15 fantasy archetypes) × 50 takes = 1,500 clips, no reference, "how to say it" in the instruction and clean text (only CAPS/!/?/…). temp 1.2. Metric = blended reward BR = invWER × (blend/10).

0.299 vs 0.213mean BR: human emotion > fantasy

0.256overall mean BR (invWER 0.932)

0.529→0.643best-of-10 → best-of-50

Quiet/whispered/grief-laden deliveries score far higher than loud shouting/cheerful/manic. Best conditions: zombie_whisper (BR 0.522), mourning_grief (0.456), heartbreak (0.441), primal_fear (0.403), orc_whisper (0.399, highest burst-prob 0.833). Worst: troll_cheerful (0.071), goblin_joyous (0.084), triumphant_pride (0.094). invWER stays high everywhere (0.85–0.98); the differentiator is the expressiveness (blend) score, which shouting kills.

The 4-model burst grid (/tmp/vbexp) & the burst problem

4 checkpoints (V1 / 1M-LoRA / inline-LoRA / full-FT) × 5 burst-placement strategies × 6 scenes, Gemini-judged burst_score /10. Findings: to get a burst at all the cue must be inline in the text (detect 37–61% vs 8–15% instruction-only) — but that raises WER; the 100k full fine-tune is the best raw burst-firer (Full-FT × both = 1.67/10 Gemini burst @ WER 0.146). Bursts remain unreliable (best ~1.7/10); the recommended fix is dedicated non-textual burst tokens + RL with reward detector_burst × (1−WER). The FAIR eval (§5) resolved the placement question in favour of instruction-cues + clean text for intelligibility.

Training-data burst filter (/tmp/vb_filter)

65,459 / 138,341 records kept (47%) by keyword-matching the instruction field against a ~200-term burst/voice-quality list. Keep rates: dramabox_gemini_ft 100%, char_voices 89%, emotional_va 84%, gemini_dramabox 76%, ears 57%, German emolia_de 25%, gemini 17%. Top triggers: breathy (23k), sigh (15k), high-pitched (13k) — timbre terms dominate over literal bursts.

7Voice cloning & Sidon restoration

dirs: /tmp/vbclone (reference-code cloning), /tmp/vcstudy (Chatterbox VC + Sidon)

What was tested (vbclone). Does a reference voice help? 17 reference voices + a no-reference path × 4 emotions × 2 languages × 50 takes = 7,199 clips (temp 1.0). Cloning = encode the reference to 12-codebook MOSS codes (refcodes.pt) passed as reference=[codes]; VoiceCLAP embedding used only to score speaker-similarity.

setting	mean BR (random)	best-of-50 BR
NO reference (invented voice) ◄	0.327	0.713
WITH reference (cloned)	0.136	0.391

No-reference generation beats code cloning ~2× on quality — imposing a (often monstrous/cartoon) reference timbre degrades intelligibility + expressiveness. Speaker-similarity trades off against quality: robot spk-sim 0.777 but BR 0.046; cartoon_gnome 0.752 / 0.059; best-cloning voice for quality = Old_noble_Dragon (best-of-50 BR 0.656, spk-sim 0.654). German slightly beats English on BR (0.160 vs 0.134). Empty instruction gives the highest speaker similarity.

Chatterbox voice conversion + Sidon (/tmp/vcstudy)

Impose a consistent identity onto no-reference clips without a real human reference: Chatterbox VC (s3gen v1 vs v3) keeps source content/prosody, imposes a target timbre, then Sidon (w2v-BERT-2.0+LoRA → DAC, 16→48 kHz) restores. 152 conversions, CAMPPlus x-vector cosine metric.

variant	cos→target	cos→source	target>source win-rate
v1 raw	0.847	0.619	98%
v3 raw ◄	0.838	0.610	100%
v3 + Sidon	0.788	0.618	95%

Identity is genuinely imposed (converted clips are far closer to the target speaker than to where their words came from), 95–100% objective win-rate, with s3gen v3 the more consistent engine. Sidon costs ~0.05 cosine for studio-quality restoration.

Policy for the 4.55B model: keep the output RAW, do NOT apply Sidon. This model emits native 48 kHz; Sidon is a 48→16→48 kHz round-trip that removes real 8–16 kHz detail and adds a vocoder sheen (neutral for ranking, Δreward≈0; verified on a 12.9k-clip paired ablation). Sidon is only a genuine upgrade for the 24 kHz 8B sibling.

8Production best-of-64 emotion grid ● published

published grid: production_best50.html · corpus: laion/moss-local-voice-acting-64x100

What it is. The flagship production run: 40 emotions × 100 prompts × 64 takes = 256k clips, generated with this 4.55B model, raw native 48 kHz, no post-processing. Every take scored with Parakeet WER, VoiceCLAP blend/genuineness/prompt-sim, and the Empathic-Insight-Plus 40-emotion + arousal/valence heads. Ranked by the equal-weight reward invWER × (norm target-emotion + norm blend + norm genuineness), min-max normalised over all 73,404 takes scored. The published page shows the top-3 takes of the 2 best groups per emotion.

256kclips (40×100×64)

0.97invWER (vs 0.83 for 8B)

~9–11 hwall on 8×A100 (raw, measured)

Example top take (Affection): reward 1.746, emotion 3.16, blend 6.18, genu 2.08, invWER 1.00, arousal 1.41. On identical prompts this 4.55B model reaches invWER 0.97 vs the 8B sibling's 0.83, at higher bandwidth (48 kHz raw) and smaller/faster — the basis for preferring it in side-by-side listening.

Open the live best-of-64 grid Corpus (part 1–4)

+Supporting studies: fast inference & VoiceCLAP pilot

Fast batched inference ● published

All measured on A100-80GB. Batch size is the lever, not flash-attn. bs=64 → 0.87 s/clip, bs=128 → 0.83 s (bs=64 is the sweet-spot for best-of-64 groups: batch==group → zero padding waste). SDPA + bf16 only — flash-attn 2.x has a broken reshape path in this checkpoint's remote-code attention. Batched bf16 ASR (Parakeet, batch 32) = 134× faster than naive — the single biggest pipeline win. Fused generate→score worker sustains ~1.1 s/clip/GPU end-to-end.

Open the live write-up

VoiceCLAP zero-shot pilot (/tmp/vclap_pilot)

Can VoiceCLAP do zero-shot VoiceNet-taxonomy scoring? 1,000 emolia clips, large-v2 (3584-d) vs commercial (768-d). Short discriminative anchors ≫ long rubric paragraphs. Works for high-level affective axes: Gender ρ −0.79, Arousal +0.57, Valence +0.52, Rec-Quality +0.50, Age +0.48 (large mean |ρ| 0.40 on emo-mapped dims); near-chance (|ρ|≈0.05) on fine-grained style dims. This is why the eval harnesses use VoiceCLAP only as a coarse signal and anchor rewards on Parakeet-WER.

Also probed: /tmp/moss_grid + /tmp/moss_grid_qc — a checkpoint QC grid (V1 / V2 / LoRA-100k / LoRA-1M, 7 fantasy scenarios ± reference, best-of-3). The intelligibility-only vc-intel metric could not separate the checkpoints (all ≈ +0.013…+0.017), so no expressive winner was declared by that metric; one QC signal is that the 1M-LoRA under-generates on short fantasy lines. The 4-model expressive comparison was resolved by the Gemini-judged /tmp/vbexp grid and the FAIR eval instead.

Built from artifacts under

/tmp/{vbtemp,vbexp,vbexp2,vbrerank,vbblend,vbmlp,vbscale,vbablate,vcblend_pkg,fair3,va_eval,va_eval2,moss_grid,vclap_pilot,vbchar,vbclone,vcstudy,vb_filter,moss_train}

and the published LAION-AI/laion-moss-local-1.5-voice-acting-4.55b repo (README · SYNTHETIC_DATA_RECIPE.md · docs/). Numbers are pulled from each study's *.json / *.log / built HTML. Scope: the 4.55B local-transformer model only — the 8B "delay" sibling is excluded.