A Sentence-Level Caption Prompt Format for Fine-Tuning MOSS-TTS-Local 4.55B

A concept and design document for a fine-tuning prompt format that gives the 4.55B MOSS-TTS-Local voice-acting model both a global speaker/scene description and per-sentence delivery control, mapped onto the model's two native input fields. The dynamic, sentence-by-sentence captions are derived automatically from audio using LAION's four voice scoring models and a deviation-from-baseline (z-score) recipe — the same procedural-caption method already prototyped in voicenet_v3grid/caption.py, extended here with worked, real examples.

1 Motivation & the two-field idea

The character-voice study on this model (rounds 1–5, see the 13-voice showcase) established that MOSS-TTS-Local responds strongly to a natural-language description of how the voice sounds placed in the instruction field. That work optimised a single, static instruction per character: one description for the whole clip. It is excellent for a fixed persona reading a line.

But a real performance is not static. Within one clip a grieving woman steadies herself, breaks, whispers, then chokes on a sob; a narrator lands a punchline, drops to a hush, then swells. A single global caption cannot say "this sentence is the one that cracks." For expressive fine-tuning — where we want the model to learn to move across a clip — we need per-sentence control. Yet we also do not want to throw away the thing that works: a stable, holistic description of the speaker.

The proposal is to keep both, and to split them by timescale: what is stable across the clip goes in instruction; what is dynamic per sentence goes inline in text, right next to the words it governs. Crucially, the per-sentence captions are not authored by hand — they are derived from the audio itself by scoring each sentence and describing only where it deviates from an ordinary-speech baseline.

2 The MOSS two-field model & verified inline markup

MOSS-TTS-Local's build_user_message(…) / the SGLang /v1/audio/speech API expose two text inputs plus a duration dial. From the model's own README and serving guide:

Verified markup facts (use exactly these)

Voice cloning (reference=[codes] from encode_audios_from_path(…)) is a different input path and orthogonal to this proposal — the caption format targets the reference-less instruction+text route, where an empty instruction otherwise gives the model no direction. It composes with cloning too (clone identity, caption delivery), but that is out of scope here.

The key observation this proposal builds on: pauses are already a native, inline text-channel construct. So the natural place to put a per-sentence marker — a (caption) before the sentence and a [pause Xs] after it — is exactly the channel MOSS already reads for timing markup. We are extending an existing inline convention, not inventing a new field.

3 The proposed format & grammar

The instruction field is unchanged in spirit from the character study: a single global description. The novelty is the text field, which becomes an interleaved sequence of per-sentence units.

The text field, labelled

text  =  (warm, storytelling, deeply genuine) Once, in a village at the edge of the frostwood, [pause 0.4s]
(hushed, intimate, slowing) there lived a girl who could hear the snow. [pause 0.9s]
(brightening, delighted, playful) "And oh, what stories the snow would tell!"
           \_______  ______/  \____________ _____________/  \___ ___/
                    \/                      \/                   \/
              per-sentence            the sentence's words     real silence
              (caption): HOW          (quotes optional)        between sentences

Grammar (EBNF-ish)

instruction  = global-speaker-caption ;   (* stable traits + overall register + stage dirs *)

text         = [ duration ] , unit , { sep , unit } ;
unit         = [ caption ] , words ;
caption      = "(" , phrase , { "," , phrase } , ")" ;   (* 0-5 short delivery keywords *)
words        = plain-text | '"' , plain-text , '"' ;   (* double-quotes OPTIONAL *)
sep          = pause | " " ;
pause        = "[pause " , seconds , "s]" ;   (* only when a real silence occurred *)
duration     = "${token:" , integer , "}" ;   (* optional pacing target, per clip or per unit *)
phrase       = { direction-word | emotion-word | style-word | burst-note } ;

Rules of the format

• Each sentence (from ASR sentence segmentation) is one unit: an optional (caption) immediately followed by its words.
• The (caption) describes only that sentence's dominant delivery — its salient emotions + speaking-style shifts + a genuineness/burst note when warranted. It is short (a few keywords), and empty/omitted for flat, unremarkable sentences.
• Between two units, insert [pause Xs] iff a real inter-sentence silence of ≈X seconds was measured in the source audio. No silence → no marker (just a space).
• Burst notes (e.g. (sigh), (soft laugh)) and all markup stay outside any double-quotes, so the model never reads brackets aloud.
• Double-quotes around the words are optional (see §10 — quotes-in-text raised WER in our earlier experiments, so bursts/markup stay outside them).
• ${token:N} may be placed once at the front (whole-clip pacing) or per unit (to teach sentence pacing on slow/fast deliveries).

4 Prompt templates

Template A — expressive / character narration (per-sentence dynamics)

instruction:
  <AGE + GENDER>, <TIMBRE: rough/smooth, breathy/full, bright/dark>,
  <RESONANCE: chest / throat / nasal / head>, <REGISTER + PITCH RANGE>.
  <overall emotional register of the whole clip>. <scene / stage direction>.
  High quality recording.

text:
  ${token:NN} (<dominant emotion>, <style shift>[, <genuine|burst note>]) Sentence one. [pause Xs]
  (<dominant emotion>, <style shift>) Sentence two. [pause Ys]
  A flat, unremarkable sentence with no caption. [pause Zs]
  (<dominant emotion>) Final sentence.

Template B — calm, even narration (sparse captions)

instruction:
  A <age> <gender> narrator, warm and full, <chest resonance>, even mid register,
  measured and unhurried. Calm, storybook, reassuring overall. Clean studio recording.

text:
  Sentence. [pause 0.4s] Sentence. [pause 0.5s]
  (gently emphatic) The one line that carries weight. [pause 0.6s] Sentence.
  // most units have NO caption because the delivery does not deviate from baseline

Template C — a fixed character voice (reuse a showcase archetype)

instruction:   // the stable persona = a round-5 champion instruction, verbatim
  A nasty little goblin voice, extremely nasal and pinched, buzzing high through the
  nose with a grating scratchy rasp, whiny and reedy. A weak, cowardly underling, no
  authority whatsoever. No chest, no warmth, no fullness. Gleeful, mean overall.

text:   // per-sentence emotion/energy shifts ride inline on the fixed goblin timbre
  (sly, taunting, low and conspiratorial) "You think you can stop me?" [pause 0.5s]
  (erupting, shrill, manic glee)(wicked cackle) "Nothing can stop me now!"

In Template C the timbre never changes (it lives in instruction), but the goblin goes from a conspiratorial hush to a manic shriek across two sentences — exactly what a static instruction could not express. The burst note stays outside the quotes.

5 Worked examples

Four full instruction+text pairs across different content. The captions shown are in the compact keyword style the pipeline (§6–7) produces.

text:
${token:210} (subdued, trembling, holding back) I keep expecting to hear his key in the door. [pause 0.8s]
(voice cracking, raw and vulnerable, breaking) And then I remember. [pause 1.4s]
(hushed, collapsing in energy, barely audible)(shaky breath) He's not coming back. [pause 1.1s]
(a fragile attempt at composure, distant) I should be stronger than this by now.

text:
The glacier moves less than a metre a day. [pause 0.5s]
To the eye, it is perfectly still. [pause 0.5s]
(quietly emphatic, slowing) But it has been moving for ten thousand years. [pause 0.7s]
Everything you see here, it carved.

text:
(sly, wheedling, conspiratorial hush) Ohhh, you found the treasure, did you? [pause 0.6s]
(mock-innocent, sing-song, taunting) Pity it's mine now. [pause 0.4s]
(erupting, shrill, manic spite)(wicked cackle) All mine, all mine!

text:
${token:96} (bright, high register, eager) It finally happened. [pause 0.3s]
(surging in energy, joyful, exclaiming)(delighted laugh) The wait is over! [pause 0.4s]
(warm, sincere, landing it) And it was worth every second.

6 The procedural-caption pipeline

The heart of the proposal: the per-sentence (captions) are not written by hand — they are derived automatically from the audio by four LAION voice models plus a deviation-from-baseline recipe. This makes captioning a large fine-tuning corpus cheap, consistent, and free of human labelling bias.

The four scoring models & their roles

Efficiency: one VoiceCLAP-commercial encode of the sentence feeds VoiceNet + genuineness + blend (three heads share the 768-d embedding); one BUD-E-Whisper encode feeds Empathic-Insight. Two encodes per sentence, four score vectors.

The six steps

1. ASR with sentence-level timestamps. Transcribe with word/segment-level timing, then aggregate to sentence spans. Two strong recommended options (either works; not a hard requirement):
   • nvidia/parakeet-tdt-0.6b-v3 — a TDT (Token-and-Duration Transducer) that natively predicts token durations, giving accurate word/token-level timestamps; multilingual (incl. EN + DE). We already use it in this project for invWER scoring, so it is a natural fit — its word timestamps are aggregated up to sentence spans.
   • microsoft/VibeVoice-ASR — provides good sentence / segment-level timestamps, well-suited to long-form transcription + alignment.
   Either yields each sentence's text and its [start, end] times, which feed the format two ways: (a) the span cuts the sentence segment for the four scoring models (next step); (b) the silence gap between one sentence's end and the next sentence's start is measured and, rounded, becomes the [pause Xs] marker (kept only ~50% of the time per the dropout policy in §6).
2. Cut each sentence segment [start, end] from the audio.
3. Score each segment through the four models. One VoiceCLAP-commercial encode → VoiceNet (57) + genuineness + blend; one BUD-E-Whisper encode → Empathic-Insight (42). Result: a per-sentence score vector of ~99 numbers + genuineness + blend.
4. Build an in-the-wild baseline. Run the same four models over a large, ordinary "in-the-wild" corpus and compute, per dimension/emotion, a robust median + spread (MAD or std). Most in-the-wild speech is flat and unremarkable, so this baseline captures "average delivery."
5. Per sentence, compute z-scores. z = (value − median) / spread for every dimension and emotion; take the top-3 to top-5 by |z| — the dimensions/emotions on which this sentence most stands out. These are its dominant delivery traits and dominant emotions.
6. Render the caption. Map each selected dim/emotion to a short phrase reflecting direction (above / below baseline) and intensity (from |z|); add the dominant EmoNet emotion word(s), a genuineness note if it deviates, and a burst note if blend is high. Flat sentences → short or empty captions (natural + cheap).

Format augmentation & annotation dropout

A fine-tuned model that only ever saw fully-marked prompts would require that markup at inference. But real users span a spectrum — from power-users who write per-sentence director notes and pauses, to novices who type a single bare line. To stay robust across all of them, the training set must vary its annotation richness on purpose. We therefore apply an augmentation / dropout policy: each sample is randomly rendered somewhere on a scale from fully-annotated to bare.

7 Real caption.py output — proof of concept

The VoiceNet half of step 5–6 is already implemented and running in voicenet_v3grid/caption.py. It is the concrete proof of concept for this proposal. Its design maps one-to-one onto the recipe above:

INTENSITY = [(2.0, "Extremely"),
             (1.5, "Very"),
             (1.0, "Notably"),
             (0.0, "Somewhat")]

DESC = {
 "FULL":  {hi:"full and cinematic",
           lo:"thin and paper-like"},
 "R_CHST":{hi:"rich in deep chest resonance",
           lo:"thin in chest resonance"},
 "VULN":  {hi:"raw and vulnerable",
           lo:"guarded and armored"}, ...}

caption(pred_dict, REFERENCE_STATS, topk=5) scores every dim against reference_stats.json (per-dim median/mean+std over 1000 real, stratified in-the-wild Emolia clips), sorts by |z|, and returns the top-k phrases. Below are real captions it produced when run on stored predictions in pred100_v3.json (each entry = one clip's 57 VoiceNet regression outputs). No GPU — pure inference on the stored score vectors.

Extremely raw and vulnerable.        [VULN  z=+3.08]
Extremely collapsing in energy.      [ARSH  z=-2.54]
Extremely audibly breathing/gasping. [RESP  z=+2.27]
Extremely meek and submissive.       [STNC  z=-2.26]
Extremely thin in chest resonance.   [R_CHST z=-2.01]

Extremely raw and vulnerable.        [VULN  z=+3.15]
Extremely noisy and aperiodic.       [HARM  z=-2.52]
Extremely darkening in mood.         [VALS  z=-2.21]
Extremely blurry and mumbled.        [CLRT  z=-2.20]
Extremely rough and raspy.           [ROUG  z=+2.19]

Extending the recipe to the emotion half (Empathic-Insight)

The same z-score-vs-baseline machinery extends verbatim from the 57 VoiceNet dims to the EmoNet emotions. Build a baseline median+spread per emotion over the in-the-wild corpus; per sentence, take the top emotions by |z|; emit the dominant emotion word(s) (EmoNet gives readable names — Grief, Amusement, Malice, Relief, Contempt, Awe…, but we expand each into a synonym cluster so the same concept is expressed with varied words — see §10.6), gated by intensity. Then two scalar heads add notes:

• Genuineness note. If genuineness deviates high, the emotion reads as sincere / genuinely felt; if it deviates low, as performed / theatrical. It also gates whether strong emotion words are added at all (see the dedicated §8).
• Burst note. If vocal-burst blend is high (as in ZH_B00039 at 8.55, FR_B00000 at 6.29), append a burst director-note — (soft laugh), (gasp), (sigh), (shaky breath) — matched to the dominant emotion, placed outside the words.

A full sentence caption is thus the concatenation, in a fixed order, of: [VoiceNet delivery phrases] + [EmoNet emotion words] + [genuineness note] + [burst note], each contributed only when it clears its |z| / value threshold, capped to a small keyword budget.

8 The genuineness gate — in detail

The single most important trust mechanism in the emotion half of the pipeline. It decides whether a salient emotion is described at all, and how strongly. Because it is the piece that keeps the caption→delivery mapping clean, it gets its own section.

Definition

The problem it solves

When we derive a per-sentence emotion caption from Empathic-Insight's top-|z| emotions, some high emotion scores are untrustworthy, in two distinct ways:

• (a) Artifacts. Noise, clipping, background music, or a bad ASR cut can make the emotion head spuriously fire — a burst of noise reads as Distress or Fear when nobody is distressed.
• (b) Flat / acted delivery. A sentence can score "sad" on the emotion head yet sound read, detached, or hammy — the words are sad, the delivery is not.

Captioning either case as "very sad" teaches the TTS a noisy caption→delivery mapping: it learns to attach a strong emotion word to audio that does not actually deliver that emotion. We want a strong emotion word only when the emotion is BOTH salient (high |z| on Empathic-Insight) AND genuinely embodied (high genuineness). Genuineness is the AND-term.

The concrete rule — median-anchored

"High genuineness" is made concrete the same robust way every other threshold in this pipeline is: against the genuineness baseline median computed over the in-the-wild corpus. The intensity a sentence's emotion word is allowed to reach is capped by where its genuineness sits relative to that median:

So extreme emotion words ("extremely", "very") are allowed only when genuineness is above-median; below-median genuineness caps the intensity, and well-below-median drops the emotion entirely. This is the same "only salient, above-baseline deviations become caption terms" principle from §6 — applied to the trust axis: a value far below its baseline median is discarded, not described.

Genuineness as a graded descriptor (its own ladder)

Genuineness is not only a gate — laddered against its own baseline it is also a describable trait, so it can be added to the caption directly (this is implementation 3 below):

Three implementations (pick per corpus)

if genuineness < 2.0 or quality < q_min:
    emotion_terms = []                 # suppress
    caption = acoustic_terms or ["flat, measured read"]
else:
    caption = acoustic_terms + emotion_terms

# f rises through 1.0 at the genuineness median (g_med): above-median -> full intensity
f          = clamp(genuineness / g_med, 0, 1)
eff_z      = abs(z) * f
adverb     = intensity_word(eff_z)     # Extremely / Very / Notably / Somewhat / (drop)

Worked example — same emotion, opposite treatment

Identical emotion salience (z = +2.5 in both); genuineness is the only difference, and it flips the caption from a full-intensity emotion phrase to a suppressed, honestly-labelled flat read.

Why it fits in-the-wild data

The two corpora we fine-tune on have opposite failure modes, and the gate fixes both:

• In-the-wild speech is mostly flat-but-genuine — low emotion, but what little there is is real. The gate lets those honest, mild emotions through (scaled), and correctly labels the many flat sentences as flat.
• Acted / performance data is high-emotion-but-sometimes-un-genuine — strong emotion scores, some of which are overacted or hammy. The gate withholds strong words from the un-genuine takes.

Net effect: the model learns emotion from real emotional moments, not from noise or overacting.

9 Deriving the general speaker caption

The global instruction caption is the aggregate of what is stable across the clip. After scoring every sentence, take the median over sentences of the speaker-identity dimensions — the ones that should not change within one clip:

Run the same caption() on this per-clip median vector (against the same baseline) to get the stable-trait phrases, then compose them into the instruction with the overall emotional register and any scene direction. Real example B above is exactly this shape when treated as a whole-clip aggregate: "warm, beautiful, full, narrator-like, elderly" → a global instruction like "An elderly man, warm and full, rich resonance, storybook narrator delivery."

10 Design recommendations — what I think

1. Deviation-vs-baseline beats absolute captioning

Caption only what is salient — the dims/emotions where the sentence departs from ordinary speech. This yields compact captions, keeps them on-topic, and makes flat sentences produce short or empty captions automatically. Absolute captioning would drown the signal in shared, uninformative traits and over-constrain the model.

2. Use robust statistics (median + MAD), not mean + std

In-the-wild score distributions are skewed and heavy-tailed (most speech is flat, a few clips are extreme). Median + MAD resists those outliers far better than mean + std, giving a more honest "typical voice" reference. (The current reference_stats.json ships mean/std; I recommend switching the production baseline to median + MAD, or at least winsorising before computing std.)

3. Keep captions short — a few keywords

Long captions over-constrain generation and, worse, drift from the data density: fine-tuning works best when the caption distribution the model sees at train time matches what it must interpret at inference. Cap each caption at top-3–5 dims + 1–2 emotion words + at most one note. Short, salient, consistent.

4. Put STABLE traits in instruction, DYNAMIC ones inline

This is the core architectural choice, and §9 gives a principled test for it: sentence-to-sentence variance. Low-variance traits (identity) belong in the global instruction; high-variance traits (performance) belong inline. Do not repeat the speaker's timbre in every sentence caption — it wastes the budget and teaches the model nothing new per sentence.

5. A genuineness gate on strong emotion words

Only add strong emotion words when genuineness (and overall quality) is high — otherwise the model learns to attach "grief-stricken" to unconvincing acting or to noise artifacts. This is important enough that it has its own section: see §8 — the genuineness gate for the definition, the three implementations (hard gate / soft scaling / own dimension + quality gate), and a worked example.

6. A controlled lexicon — with deliberate synonym variety

Draw every caption phrase from a controlled vocabulary — the VoiceNet rubric adjectives (caption.py's DESC) + the EmoNet emotion taxonomy + a small burst set. This is what lets a caption→delivery mapping be learnable at all: free-form LLM captioning would inject uncontrolled, inconsistent phrasing and blur the mapping.

But "controlled" must not mean "one canonical word per concept." If every sad sentence in training is captioned with the single token sad, the fine-tuned model learns to respond to that exact token — and an inexperienced in-the-wild user who types "he sounds really pissed off" or "melancholic" at inference gets nothing, because those words were never bound to the delivery. The fix is to vary the words within each concept, drawing from the EmoNet synonym clusters: each of the 40 emotions ships a cluster of near-synonyms. We rotate / sample among them across the dataset so the same measured emotion is expressed with varied surface words.

7. Use blend for burst director-notes

The vocal-burst-blend head cheaply flags laughs/gasps/sighs/moans. Turn a high blend into one matched burst note (the character study found one burst helps, a second usually hurts), placed as a (round-bracket) note outside the spoken words.

8. Real pauses teach real timing; per-sentence tokens teach pacing

[pause Xs] taken from the measured inter-sentence gaps teaches natural phrasing and the timing of emotion (the 1.4 s beat before "He's not coming back."). A per-sentence ${token:N} can additionally teach the model the pacing of a slow, heavy line vs a quick one. Both are free by-products of the alignment step.

9. Quality-control the corpus

Drop sentences with bad ASR (low alignment confidence, high WER on re-decode) or low recording quality (RCQL/BKGN off) before captioning — a wrong transcript or a noisy segment produces a misleading caption that teaches the model the wrong binding.

10. Double-quotes are optional — and markup stays outside them

Wrapping the words in "double quotes" is optional. In our earlier experiments, quotes-in-text raised WER, so if quotes are used, keep all bursts and markup outside the quotes (the model must never read a bracket aloud — inline bursts-in-text cost ~5× WER). The safest default is no quotes; the format supports both.

11 Open questions

• How many top-k? 3 vs 5 vs an adaptive cut (keep all dims with |z| > 1.5). Fewer is cleaner but may drop a real second signal; adaptive matches caption length to how unusual the sentence actually is.
• Caption length budget. A hard keyword cap vs a soft one; whether to merge redundant phrases (e.g. "high register" + "bright head resonance" often co-fire).
• Granularity. Sentence-level is proposed here, but should we also caption at phrase/word level for very expressive lines — or does that over-constrain and inflate the markup?
• Baseline choice. One global in-the-wild baseline vs per-language / per-domain baselines (a whisper corpus has a different "average" than a shouting corpus).
• Instruction vs inline overlap. When a trait drifts slowly across a long clip (e.g. a gradual build), does it belong in the instruction, inline, or both?
• Quotes. Whether to keep double-quotes at all, given the WER cost, or standardise on no-quotes.

12 Links & resources

The four scoring models

ASR / sentence-timestamping (recommended options)

The existing procedural-caption work & the model