The models and how they were trained
A small joint acoustic model supplies scores and locations. A separate ASR model supplies words. Two language-model adapters describe what is heard.
Humaneness Ears Medium
Humaneness Ears Medium uses the Whisper Small encoder with task heads, approximately 97.1 million parameters. It has no Whisper ASR decoder. The joint heads predict 192 scalar targets, including 40 EmoNet emotion scores and 57 VoiceNet dimensions, AudioBox/DNSMOS and other voice/quality scores; auxiliary heads predict characters per second, burst count, Orange 128D timbre and 250D identity embeddings.
Temporal outputs include 20 ms frame probabilities, onset/duration event proposals and an event-local 53-class vocal-burst head. Top-three class softmax values are displayed as model outputs, not calibrated event-correctness probabilities. Longer recordings use 30 s windows with 5 s overlap and a center-ownership rule for event proposals.
The released model comes from the multi-task Whisper training program: score/embedding supervision is distilled from the original annotation models, with temporal supervision and later fine-tuning on the balanced annotation corpus. This orchestrator uses the released Medium checkpoint; it performs no further neural training. The detailed schedules, architecture, target list and checkpoint evaluations are in the model release and independent benchmark report.
Two MOSS LoRA adapters
Both adapters use MOSS-Audio-8B-Instruct, revision d4dc5a6d8cd79b43dcd82884c75e303b1ecd016d. Its frozen base has 9,052,463,488 parameters, combining an audio encoder (~12.5 audio tokens/s), an audio-to-language adapter, DeepStack features and a Qwen3 language backbone. Only language-transformer attention/MLP LoRA projections are trained; the audio encoder, audio adapter, mergers and original language weights are frozen.
| Training property | Whole-recording voice captioner | Interval burst captioner |
|---|---|---|
| Release with code and recipe | moss-voice-captioner | moss-vocal-burst-captioner |
| Training / validation / test examples | 58,380 / 3,239 / 3,544 | 80,773 / 4,649 / 4,820 |
| Epochs / optimizer updates | 2 / 1,826 | 2 / 2,526 |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 | 32 / 64 / 0.05 |
| Trainable parameters | 87,293,952 | 87,293,952 |
| Optimizer / peak LR | AdamW / 1e-4 | AdamW / 1e-4 |
| Schedule / weight decay | 5% warmup, cosine / 0.01 | 5% warmup, cosine / 0.01 |
| Effective batch | 64: 4 GPUs × 1 example × 16 accumulation | 64: 4 GPUs × 1 example × 16 accumulation |
| Precision / attention | BF16 / SDPA | BF16 / SDPA |
| Loss | Assistant-caption token cross entropy | Assistant-caption token cross entropy |
| Audio input | Full mono 16 kHz recording | Full mono 16 kHz recording + start/end seconds |
| Explicit audio time markers | Disabled | Every 2 seconds |
| Selected validation token loss | 1.290871 | 1.423833 |
Supervision consists of machine-generated caption text from the balanced S1 subset, LAION's Got Talent, balanced 40-emotion snippets and voice-annotation tasks. Train/validation/test splits are disjoint by exact source-audio SHA-256, and all events from a recording share its partition. Input does not include a reference caption, intended emotion, score vector or transcript. Token validation loss is not an acoustic caption-quality metric.
The 50 synthetic TTS listening examples are separate from the adapter-training corpus. In the new demo, saved predictions are reformatted; the neural captions have not been regenerated or edited for presentation. GENERAL comes from the voice adapter. Each selected event receives a separate burst-adapter inference call with full audio and both endpoints. The burst adapter describes an interval; it cannot fix an incorrect locator boundary by itself.
Parakeet v3 ASR and alignment
Parakeet TDT 0.6B v3 supplies recognized words and timestamps. It is a roughly 600M-parameter speech recognizer supporting 25 European languages, including English and German. The runtime uses its released Transformer implementation and the existing procedural word/sentence alignment helpers. Energy-based endpoint refinement does not constitute human alignment.
The Humaneness Ears normalizer includes Japanese, Korean and Chinese clips. This does not make the Parakeet checkpoint an ASR model for those languages; the CLI rejects unsupported language codes rather than invent timestamps.
Normalization, taxonomies and the original targets
The procedural speech reference contains 80,265 successfully predicted clips: 41,771 S10 and 38,494 multilingual Emilia. Arithmetic mean and population standard deviation are computed per dimension, with equal clip weight. Local cues select at most two VoiceNet properties and one emotion above the reference threshold. Content Enjoyment, predicted from AudioBox supervision, supplies pleasantness wording; VoiceNet aesthetics remains a raw target.
Explore the original EmoNet/emotion-annotation toolkit, VoiceNet and procedural captioning. Links and exact model revisions are in the downloadable source manifest. The score predictions are teacher-derived quantities, not direct human diagnoses, identity claims or probability estimates.