Serving the 4.55B on SGLang-Omni (A100)

1 · What SGLang-Omni is, and why it is the fast path

SGLang-Omni is a serving framework for omni / speech models built on top of SGLang. It runs the model as a pipeline of stages and exposes them behind an OpenAI-compatible HTTP server. For this model the pipeline is:

# three stages, one GPU
preprocessing  # tokenize text, encode any reference audio with the ~1B causal codec
   -> tts_engine   # the 4.55B Qwen3 backbone + 12-codebook local-transformer, run by SGLang
   -> vocoder      # MOSS-Audio-Tokenizer-v2 decodes RVQ codes back to 48 kHz stereo PCM

Why this is the fast path for the 4.55B (versus a plain transformers generate loop):

• Continuous batching. The tts_engine stage is a real SGLang scheduler: independent requests are packed into a single running batch and admitted/retired token-by-token, so the GPU stays busy instead of idling between requests. This is the entire reason offline throughput scales to tens of times realtime.
• CUDA graphs. Both the backbone decode and the per-frame local-transformer step (1 + n_vq micro-steps with 13 seeded sampling passes per frame) are captured as CUDA graphs. Eager, that inner loop is kernel-launch-bound at ~22 ms/frame regardless of batch size; graphed, the per-frame cost drops and — crucially — a graphed batch of 64 costs barely more than a batch of 1, so throughput rises almost linearly with batch size.
• RadixAttention / prefix cache. Shared prompt prefixes (e.g. a fixed instruction or reference) are cached across requests.
• Native pipeline. The moss_tts_local architecture is implemented natively, including a seeded sampler that is reproducible at any batch size — a fixed seed gives the same audio at concurrency 1 or 128.

2 · A100 setup — step by step

2.0 · Assumptions

• GPU: one NVIDIA A100-80GB (this model needs ~58–68 GB at the throughput settings below; a 40 GB card forces a smaller KV pool and lower batch).
• Driver / CUDA: a recent driver exposing CUDA 12.x. Torch ships its own CUDA 12.8 runtime, so you do not need a system CUDA toolkit to serve. You only need a system nvcc if you enlarge the CUDA-graph batch sizes (see trap 7) — the stock config does not.
• Disk: ~25 GB free for the model shards + venv. Put the venv on a real disk, not /tmp (see trap 5).
• No flash-attn. This checkpoint runs on SGLang's default attention backend; flash-attn is not required and the remote-code attention actually has a broken FA2 reshape path. Do not install it.

2.1 · Create the venv (uv) and install SGLang-Omni

We use uv. Install it once (curl -LsSf https://astral.sh/uv/install.sh | sh), then:

# create the venv ON A REAL DISK (see trap 5) — /home, not /tmp
uv venv /home/$USER/sglang_moss_venv --python 3.12
source /home/$USER/sglang_moss_venv/bin/activate

# install SGLang-Omni (this pulls torch 2.9.x+cu128, sglang, flashinfer, transformers, soundfile, ...)
uv pip install sglang-omni
# soundfile is needed to decode base64 / data-URI reference audio
uv pip install soundfile

Key package versions this guide was validated against (yours may be newer):

2.2 · Download the model

# the LAION 4.55B voice-acting checkpoint (merged weights, Apache-2.0, public)
hf download laion/moss-tts-local-transformer-4.55b-voice-acting
# the v2 audio tokenizer / vocoder (pulled automatically on first run, or pre-fetch it)
hf download OpenMOSS-Team/MOSS-Audio-Tokenizer-v2

First run also JIT-compiles a flashinfer attention kernel into ~/.cache/flashinfer and captures CUDA graphs; expect the first startup to take a couple of minutes longer than later ones.

2.3 · The config file

SGLang-Omni selects the pipeline from a tiny YAML. The stock reference-less / voice-clone config is:

# examples/configs/moss_tts_local.yaml
config_cls: MossTTSLocalPipelineConfig
model_path: OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5
relay_backend: shm

The model_path inside the YAML only names the architecture family; the actual weights come from the --model-path you pass on the command line (the LAION checkpoint). The three stages (preprocessing → tts_engine → vocoder) are filled in by MossTTSLocalPipelineConfig's defaults, so you do not have to spell them out. relay_backend: shm uses shared memory to move audio between stages (see trap 3).

3 · The serve command (every flag explained)

cd /path/to/sglang-omni          # repo dir that holds examples/configs/
env -u LD_LIBRARY_PATH \        # TRAP 1 — strip a leaked cuDNN path or the 1st request SIGABRTs
  CUDA_VISIBLE_DEVICES=6 \        # pin to one GPU
  sgl-omni serve \
    --model-path laion/moss-tts-local-transformer-4.55b-voice-acting \  # the LAION weights
    --config examples/configs/moss_tts_local.yaml \                      # the pipeline
    --port 8000

When it is up you will see Uvicorn running on http://0.0.0.0:8000 and Application startup complete. Sanity-check:

curl -s http://localhost:8000/v1/models

Note (trap 6): /v1/models reports the base model id (OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5) — this is cosmetic. The LAION weights you passed with --model-path are the ones actually loaded and generating.

4 · Run it as a persistent background service

For a long-lived server, detach it and log to a file:

# start, fully detached, logging to a file
cd /path/to/sglang-omni
nohup env -u LD_LIBRARY_PATH CUDA_VISIBLE_DEVICES=6 \
  sgl-omni serve --model-path laion/moss-tts-local-transformer-4.55b-voice-acting \
    --config examples/configs/moss_tts_local.yaml --port 8000 \
  > ~/moss_server.log 2>&1 &
echo $! > ~/moss_server.pid          # remember the pid

# health / readiness
curl -s http://localhost:8000/v1/models >/dev/null && echo UP || echo "not ready"

# watch the log
tail -f ~/moss_server.log

# stop it cleanly (kills the stage subprocesses too)
kill "$(cat ~/moss_server.pid)"

Stopping gotcha. The server spawns one stage subprocess that holds the GPU memory. If you kill -9 only the parent, that child can survive and keep ~60 GB pinned. Always verify with nvidia-smi --query-compute-apps=pid,used_memory --format=csv and kill any orphan before relaunching, or you will hit a spurious CUDA out of memory on the next start.

5 · API reference (POST /v1/audio/speech)

OpenAI-compatible speech endpoint. Output is 48 kHz stereo 16-bit WAV — use it raw, no post-processing.

5.1 · Reference-less (no voice clone)

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "The lighthouse blinked twice before the storm rolled in."}' \
  --output out.wav

Python:

import requests
r = requests.post("http://localhost:8000/v1/audio/speech",
                  json={"input": "The lighthouse blinked twice before the storm rolled in."})
r.raise_for_status()
open("out.wav","wb").write(r.content)

5.2 · Style / director instruction & language

This is a voice-acting model: put the performance direction in instructions.

curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" -d '{
  "input": "You dare enter my lair?!",
  "instructions": "A furious ancient dragon, low and booming, building to a roar. Includes a growl.",
  "language": "English"
}' --output dragon.wav

5.3 · Voice cloning (reference clip + transcript)

Pass a reference clip in references (or the ref_audio/ref_text shorthand). audio_path may be a local path, an HTTP(S) URL, or a base64 data URI. Supplying the transcript (text) materially improves cloning.

curl -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" -d '{
  "input": "Get the trust fund to the bank early.",
  "references": [{
    "audio_path": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav",
    "text": "We asked over twenty different people, and they all said it was his."
  }]
}' --output clone.wav

Base64 data-URI form (decoded server-side with soundfile):

{"ref_audio": "data:audio/wav;base64,UklGR....", "ref_text": "Transcript of the clip."}

5.4 · Streaming (Server-Sent Events)

Set "stream": true to receive SSE. Each event is data: {"object":"audio.speech.chunk", "audio":{"data":"<base64 WAV>"}}; the final event has audio: null, then data: [DONE].

curl -N -X POST http://localhost:8000/v1/audio/speech -H "Content-Type: application/json" -d '{
  "input": "This sentence is streamed back as it is generated.",
  "stream": true
}'

Streaming granularity for this model. The moss_tts_local pipeline emits a coarse first chunk — for a one-sentence prompt the whole utterance arrives in the first audio event, so time-to-first-audio ≈ full generation time (see 7.1). The generic initial_codec_chunk_frames request field exists in the protocol but is only honored by other TTS pipelines (e.g. higgs), not by moss_tts_local — we tested values 2/4/8 and the first-chunk size and TTFA did not change. If you need true low-latency token-by-token streaming, that knob is not (yet) wired for this model.

5.5 · Duration control

The model conditions on a target codec-frame count. Set it inline with ${token:N} (stripped before synthesis) or with the token_count parameter (aliases duration_tokens, tokens). Larger N → longer audio. Omit it to let the model choose.

{"input": "${token:150}A sentence with an explicit duration target.", "ref_audio": "..."}

5.6 · Generation parameters

6 · Common traps & fixes

7 · Speed & tuning (measured)

All measurements: one A100-80GB, bf16, reference-less, a fixed pool of short (1-sentence) and medium (3-sentence) English prompts, mean clip ~9 s. Client is the async benchmark in this repo's serving notes. We define RTF = audio-seconds produced per wall-second — numerically equal to audio-hours generated per GPU-hour. Every run is preceded by a 5-request warm-up so CUDA-graph capture and lazy init don't pollute the numbers.

7.1 · Time-to-first-audio (TTFA) and how to lower it

Streaming, concurrency 1, 20 requests. TTFA = wall time from request send to the first audio SSE event.

What actually moves TTFA here. For a one-sentence prompt the pipeline emits the entire utterance as the first chunk, so TTFA ≈ full generation latency (~0.7–1.3 s for a short clip). The knobs:

• CUDA graphs (default ON) — the one that matters. They keep the per-frame decode fast; turning them off (disable_cuda_graph) reverts to the ~22 ms/frame eager path and roughly triples per-frame time, inflating TTFA. Leave them on.
• Streaming first-chunk size — not exposed for this model. initial_codec_chunk_frames is accepted by the API but ignored by the moss_tts_local vocoder (verified: first-chunk size and TTFA unchanged at 2/4/8). There is no config knob to make the first chunk smaller for this pipeline, so sub-500 ms TTFA on long inputs is not achievable today — you'd need the vocoder to implement chunked emission.
• torch.compile, mem-fraction, chunked-prefill: none measurably lower single-stream TTFA (the bottleneck is frames-to-generate, not prefill or memory). See 7.3.

Best TTFA achieved: ~0.74 s (short prompt, conc 1, stock CUDA-graph config). The lever was CUDA graphs, which are already on by default.

7.2 · Streaming throughput vs concurrency

Streaming, stock server (max_running_requests=16), 48 requests per point.

Streaming saturates around 16–17× realtime at concurrency 16–32. Past that, aggregate RTF flattens and TTFA balloons (requests queue), so for interactive streaming keep concurrency ≤16 per GPU. Streaming caps lower than offline because SSE framing + the coarse chunking add per-request overhead and the stock server's AR batch is capped at 16.

7.3 · Offline batch throughput (dataset generation) — the important one

Non-streaming, sweep the number of in-flight requests (SGLang continuous batching). Three server configs:

• Stock — max_running_requests=16, CUDA graphs bs≤16, mem_fraction=0.5.
• +MRR only — raise max_running_requests to 64 but leave graphs at bs≤16.
• +MRR +graphs (recommended) — max_running_requests=64 and capture CUDA graphs up to bs 64 (cuda_graph_bs=[1,2,4,8,16,24,32,48,64], cuda_graph_max_bs=64), mem_fraction=0.55.

The +MRR+graphs column at concurrency 64/96/128 is the rigorous steady-state run (384 requests per point, so start-up ramp and drain are negligible); the lower concurrencies use 96 requests. Peak is at concurrency 96 — concurrency 128 regresses (queue thrash / longer per-request latency for no extra throughput).

The lesson (this is the whole game):

• Raising max_running_requests alone does nothing — it even regresses. With CUDA graphs only captured up to bs 16, any running batch >16 falls back to the eager per-frame path (~22 ms/frame, kernel-launch-bound), which cancels the benefit of the extra concurrency. The "+MRR only" column plateaus at ~17–18×.
• You must raise the CUDA-graph batch sizes together with max_running_requests. Once batches of 32/48/64 are graphed, per-frame cost stays flat while 64 sequences advance per step, and throughput climbs from ~18× to ~35× realtime.
• Knobs that did NOT help / not needed: torch.compile (no measurable gain over CUDA graphs for this inner loop, and long compile), chunked-prefill (prompts are short; prefill isn't the bottleneck), pushing mem_fraction past ~0.6 on a single card (steals memory from the two codec instances → OOM risk for no throughput gain). flash-attn is not usable (broken FA2 path).

Peak: ~35× realtime = ~35 audio-hours generated per A100-hour, at concurrency 96 with the +MRR+graphs config, using ~68 GB VRAM and pinning the GPU at ~100% utilization (steady-state, 384-request run; concurrency 128 regresses to ~30×). Compared with a plain serialized transformers loop (~1× realtime, one clip at a time), SGLang-Omni's continuous batching + graphs is a ~35× speedup for bulk generation on the same card.

Applying the tuning: generate a config with the tts_engine stage's factory_args.server_args_overrides set to {max_running_requests:64, cuda_graph_bs:[1,2,4,8,16,24,32,48,64], cuda_graph_max_bs:64, mem_fraction_static:0.55} (and, optionally, the vocoder stage's max_batch_size:16), then launch with ninja + nvcc on PATH (trap 7). The stock YAML's typed runtime.sglang_server_args only exposes mem_fraction_static; the batch/graph knobs must go through factory_args.

8 · Recipes

(a) Lowest-latency interactive TTS

• Config: stock (CUDA graphs on, max_running_requests=16).
• Keep per-GPU concurrency ≤16; use stream:true.
• Expect TTFA ~0.7–1.3 s for a short line and ~16× aggregate RTF across up to 16 callers.

(b) Max-throughput offline dataset generation

• Config: +MRR+graphs — max_running_requests=64, cuda_graph_bs up to 64, mem_fraction_static=0.55 (launch with ninja+nvcc on PATH).
• Client: no streaming, keep ~96 requests in flight (a simple async pool with a semaphore); don't push past ~128 — it regresses.
• Expect ~35× realtime ≈ 35 audio-hours per A100-hour, ~3.9 req/s of ~9 s clips, ~68 GB VRAM.
• Set a fixed seed per item if you want reproducibility — it holds at any concurrency.

9 · Napkin math — how long / how much to make N hours of audio