Context (for dummies). our best-of-N pipelines generate tens of thousands of takes and score every one. this page documents every speed lever we measured on 8×A100-80GB for the two MOSS voice-acting models (8B delay @24 kHz and moss-local 4.55B @48 kHz), so future runs don't rediscover them.
| batch size | sec / clip | peak VRAM | speedup vs bs=32 |
|---|---|---|---|
| 32 | 0.998 | 20.6 GB | 1.0× |
| 64 | 0.868 | 24.2 GB | 1.15× |
| 128 | 0.83 | 31.5 GB | 1.2× |
measured: moss-local (4.55B, SDPA, bf16), ~15 s clips, one prompt repeated per batch. bs=64 is the sweet spot for best-of-64 groups (batch == group → perfectly homogeneous lengths, zero padding waste; this is also why length-bucketed batching adds nothing here — every sequence in the batch shares the same prompt and token budget. bucketing only matters if you batch across different prompts).
flash-attention 2.8.3: compiles and installs fine (~12 min, ninja -j62), and helps standard architectures — but the moss-local remote-code attention has a broken FA2 reshape path (shape [64,8,4,128] invalid for 258048 — the packed-QKV split doesn't match the model's GQA head layout). SDPA is the practical ceiling for this checkpoint. the MOSS-8B delay model runs SDPA too.
| stage | naive | optimized | speedup |
|---|---|---|---|
| ASR / WER (Parakeet-TDT-0.6b-v3) | 1.304 s (fp32, unbatched) | 0.0097 s (bf16, batch 32) | 134× |
| VoiceCLAP (blend + genuineness + prompt-sim) | 3 separate passes | 1 encode, 3 heads: 0.027 s | ~3× |
| Empathic-Insight-Plus (43 heads) | — | 0.24 s (one Whisper-encoder pass, all heads on the flattened states) | — |
| Sidon restoration (16→48 kHz) | 0.41 s (unbatched) | 0.34 s (batch 8, exact-length trim) | 1.2× (DAC-decode-bound) |
the single biggest win in the whole pipeline: batched bf16 ASR. the naive scorer made ASR ⅔ of total cost; batched, it is ~1%. corollary: after fixing ASR, generation dominates again and the EI-Plus encoder pass (0.24 s) is the next-largest scoring cost.
· fused worker — generate → sidon → score in one process per GPU: audio flows in memory, no second pass over disk, the GPU never idles between stages. measured end-to-end (moss-local + sidon + 43-head scoring + flac write): ~1.1 s/clip/GPU all-inclusive.
· merge LoRA into the weights once at load (252 deltas, ~30 s) instead of running through a peft wrapper — zero per-token adapter overhead.
· reference-code caching: encode each reference/voice wav once per worker, reuse across groups.
· resume markers at group granularity (.done files + parquet flush) — a crashed 10-hour run loses at most one group per GPU.
· token budget = pacing control: MOSS fills whatever frame budget you give it. words × 4.2 frames force-rushes (96.9% of takes hit the ceiling, truncations); words × 6.0 (~2.1 words/s) fixes cut-offs at ~40% more audio-seconds of cost. budget is a quality-cost dial, not a limit.
· what we did NOT do: vLLM-style continuous batching (custom audio-head architecture, real integration work, ~2× more; only worth it at 1M+ clip scale), fp8/int8 quantization (unvalidated audio-quality risk for a quality-selection pipeline).
| workload | measured |
|---|---|
| moss-local generation (bs=64, sdpa, bf16) | 0.87 s/clip/GPU |
| + sidon (batch 8) + full scoring + flac | ~1.1 s/clip/GPU |
| 40 emotions × 100 prompts × 64 takes = 256k clips | ≈ 78 A100-h ≈ 10 h wall on 8 GPUs |
| MOSS-8B delay for comparison (bs=100) | 0.67 s/clip gen; ~0.9 s/clip fused at G=16 |