An open, reproducible voice annotation pipeline

Reproduce the annotation pipeline

The output can be reproduced from saved sidecars without loading a model. Fresh audio runs use two isolated model environments.

Six explicit stages

  1. Decode audio. Read the original file, retain its SHA-256, convert to mono 16 kHz for inference.
  2. Score and align. Run Humaneness Ears Medium on the full recording/windows. Run Parakeet v3 to obtain words and endpoints, then score aligned sentences with a small context margin.
  3. Select cues and events. Use the pinned speech normalizer, compact cue vocabulary and event threshold. Keep raw outputs separate from these derived choices.
  4. Caption. Generate a whole-recording MOSS description. Switch to the event adapter and describe every selected interval using the same full audio and both endpoints.
  5. Compose. Combine the original global prose, local cues, recognized speech, event prose and pauses into the versioned GENERAL/SCRIPT format.
  6. Verify and save. Independently parse the text, check word/event retention and elapsed budgets, and save text plus canonical JSON.

Installation and the first reproducible example

See the repository README for the full two-environment installation. Python 3.11+ is required; composition uses only the standard library.

git clone https://github.com/LAION-AI/Humanness-Voice-Annotator.git
cd Humanness-Voice-Annotator
python -m pip install -e .
humanness-annotate from-saved --acoustic examples/acoustic.json   --moss examples/moss.json --output outputs/example.json
humanness-annotate verify outputs/example.json

The paired acoustic/MOSS sidecars reproduce a real demo annotation. They include predictions and provenance, with local file paths removed.

Fresh audio, including offline clusters

Transformers 5.14.1 supports the acoustic stack; the MOSS release uses 4.57.1. The two workers use separate interpreters. Stage the exact revisions with scripts/download_models.py on a connected machine. Then run GPU or CPU inference with the local-directory flags. Only one MOSS base is staged; the event release contributes its adapter.

.venv-acoustic/bin/python scripts/download_models.py --output models
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 humanness-annotate annotate clip.mp3   --language en --output outputs/clip.json   --acoustic-python .venv-acoustic/bin/python   --moss-python .venv-moss/bin/python --device cuda:0 --threads 4   --procedural-dir models/procedural   --humaneness-dir models/humaneness_ears --parakeet-dir models/parakeet   --voice-dir models/moss_voice --burst-dir models/moss_burst

On a connected machine, omitted local directories trigger pinned model/source downloads. Downloading large weights from GPU nodes is unsuitable for clusters without outbound networking. CPU execution uses --device cpu and enough memory for the MOSS base; it is much slower than the small score model alone. There is no API key or Gemini dependency in annotation inference.

Settings and limits

The frozen v1 configuration uses event threshold 0.60, minimum duration 0.08 s and suppression when overlap exceeds half the shorter interval. No additional selected-event cap is imposed; Humaneness Ears proposes at most 32 events per 30 s window. The threshold filters onset proposals rather than calibrated truth probabilities. More conservative thresholds may miss quieter events. Different clips may favor different settings.

MOSS uses greedy decoding at temperature 0, up to 512 global or 256 event tokens, with a 1,024-token retry if truncated. Local cues use up to two voice properties and one emotion with z ≥0.50. Explicit pauses begin at 0.20 s. The MOSS audio loader has its own maximum duration; use clips of at most five minutes and inspect the returned duration. The listening examples are approximately 30 s.

Caching, provenance and failures

Each worker writes an atomic JSON file. The cache key contains original audio SHA-256, exact source/model versions, language and settings. A completed acoustic stage can be reused if caption inference fails. Local directory overrides identify a path, not its contents; use the pinned artifacts and clear caches if you replace local weights in place.

The procedural source lock verifies downloaded source and the reference normalizer by SHA-256. Model revisions are frozen in assets/sources.json. The acoustic and MOSS stages must agree on original audio hash and duration before composition. Missing alignment, invalid endpoints, empty captions or failed format checks stop the run instead of fabricating a script.

The canonical sidecar retains all 192 raw scores, training-normalized and reference-normalized views, auxiliary targets, embeddings, frame probabilities, unfiltered proposals, selected events, MOSS texts and decoding metadata. Original ASR text is retained alongside the word-aligned text. A flag records whether their normalized word sequences agree.

Verified fresh inference

A complete offline-model smoke test passed on three English examples and one German example. Each recording ran on one GH200 GPU with four CPU threads, including acoustic inference, ASR, both MOSS adapters, composition and final verification. Observed cold-run times were 201–208 s per recording, including model loading and multiple event captions. This is a functionality check, not a bulk throughput benchmark. Read the verification record.

Bulk annotation

This CLI is a per-recording reference and reloads models between recordings. For scale, distribute recordings across GPU workers, initialize each model once per worker and use batched Humaneness Ears inference before scheduling the more expensive generative MOSS calls. The provided captioning releases expose their inference classes. Do not extrapolate the fast Whisper score-model throughput to the full generative captioning stack.