How to Fine-Tune Nemotron 3.5 ASR for Your Language, Domain, or Accent
NVIDIA's Nemotron 3.5 ASR is a 600M multilingual streaming speech model, and the post shows how to fine-tune it for a language, domain, or accent.
Intelligence analysis by GPT-5.4 Mini
The article introduces an open-weight ASR model for 40 language-locales and then walks through a fine-tuning recipe using tarred NeMo/Lhotse data, held-out evaluation, and latency knobs. In a Greek/Bulgarian example, fine-tuning reduced word error rate materially at an 80ms streaming setting.
It is like teaching a fast note-taker to understand more accents and special words. The article shows how to give it examples, test it, and make it make fewer mistakes without slowing it down too much.
Analysis
What the model is
NVIDIA's Nemotron 3.5 ASR is presented as a 600M-parameter, open-weights, streaming speech-to-text model for 40 language-locales. The post emphasizes that it transcribes in real time, includes punctuation and capitalization by default, and can be used without API dependencies or per-call billing. Its architecture combines a Cache-Aware FastConformer encoder with an RNNT decoder, so the model can stream audio without repeatedly recomputing overlapping chunks.
Why fine-tuning helps
The article argues that multilingual ASR systems often force teams into awkward tradeoffs: too many per-language models, too much latency, extra punctuation post-processing, and assumptions that the language is known in advance. Nemotron 3.5 ASR is designed to reduce those friction points, but the authors note that long-tail languages, specific domains, and unusual accents still have room to improve. Their worked example fine-tunes the base model on Greek and Bulgarian.
The recipe and results
The workflow is straightforward: prepare tarred speech data, fine-tune from the base checkpoint with the same streaming recipe, evaluate on held-out data at the deployment latency, add more data where needed, and export the new checkpoint. In the example, the training mix is about 2,000 hours drawn from public corpora, with each clip labeled by target language. Evaluation is done on held-out FLEURS splits at the strictest streaming setting.
The reported results show clear gains. At 80ms chunking, Greek WER drops from 35 to 24, and Bulgarian from 22 to 15. The article also says that adding roughly 2,000 more hours of parliamentary speech helps further, though gains are uneven and should be measured rather than assumed. The key practical lesson is that fine-tuning can substantially improve weaker languages, but the language tag, replay data, and deployment-latency evaluation all matter.
Key points
- Nemotron 3.5 ASR is a 600M open-weight streaming ASR model for 40 language-locales.
- The model combines a Cache-Aware FastConformer encoder with an RNNT decoder for low-latency transcription.
- It outputs punctuation and capitalization directly, reducing the need for a separate cleanup model.
- The article's fine-tuning example uses Greek and Bulgarian and reports large WER improvements on held-out FLEURS data.
- The same checkpoint can be run at different latency settings with att_context_size, so teams can choose their operating point at inference time.
If this works as described, teams can use one speech model for many languages and still adapt it to local accents or specialized fields. The same checkpoint can also be tuned for different speed-versus-accuracy settings, which makes deployment simpler.
The gains depend on having good in-language data and correct language tags, so weak training sets can limit the payoff. The article also warns that specializing for a target language can hurt other languages if teams do not use replay data and re-check overall performance.
