Gemini 3.5 Transcribe & Transcribe Live: Google Replaces Chirp with Dual Speech Models and Smart Normalization
Google has officially retired its legacy Chirp 3 speech pipeline in favor of two dedicated models built directly on Gemini’s multimodal foundations: gemini-3.5-transcribe for pre-recorded audio processing and gemini-3.5-transcribe-live for low-latency, bidirectional WebSocket streaming.
The release marks a significant departure from conventional acoustic speech-to-text models. By leveraging large-scale sequence modeling directly during decoding, the new engines do not merely map phonemes to text—they perform real-time semantic normalization, resolving mid-sentence self-corrections and stripping conversational disfluencies before emitting final tokens.
The Dual-Model Architecture
Rather than forcing a single model to compromise between batch accuracy and streaming latency, Google split the release into two distinct endpoints:
| Feature / Metric | gemini-3.5-transcribe (Batch) |
gemini-3.5-transcribe-live (Streaming) |
|---|---|---|
| Primary Use Case | Pre-recorded audio, meeting recordings, podcasts | Real-time dictation, voice agents, live captioning |
| Transport Protocol | REST / gRPC (v1beta/models) |
WebSockets (v1alpha/live) |
| Word Error Rate (WER) | 2.6% | 4.0% |
| Speaker Diarization | Native up to 3 speakers (speaker_0, etc.) |
Single-stream chunked processing |
| Word Timestamps | Millisecond-precision start/end bounds | Token-level interim hypothesis timing |
| Vocabulary Biasing | Supported via custom_vocabulary list |
Supported in initial session handshake |
| Language Coverage | 85+ languages with auto-detection | 85+ languages with auto-detection |
┌──────────────────────────┐
│ Incoming Audio Feed │
└────────────┬─────────────┘
│
┌────────────────────────┴────────────────────────┐
▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│ gemini-3.5-transcribe │ │ gemini-3.5-transcribe-live│
│ (Batch Engine) │ │ (Streaming Engine) │
├───────────────────────────┤ ├───────────────────────────┤
│ • REST / gRPC Payload │ │ • Full-Duplex WebSockets │
│ • 2.6% Benchmark WER │ │ • 4.0% Benchmark WER │
│ • 1-3 Speaker Diarization │ │ • 70% Faster Finalization │
│ • Word-Level Timestamps │ │ • Live Agentic Voice Loop │
└─────────────┬─────────────┘ └─────────────┬─────────────┘
│ │
└────────────────────────┬────────────────────────┘
▼
┌───────────────────────────────┐
│ Smart Transcription Layer │
├───────────────────────────────┤
│ • Strips "um", "uh", stutters │
│ • Resolves self-corrections │
│ • Truecasing & auto-format │
└───────────────┬───────────────┘
▼
┌───────────────────────────────┐
│ Clean Normalized Text / JSON │
└───────────────────────────────┘Smart Transcription: Semantic Normalization at Decode Time
The centerpiece of Gemini 3.5 Transcribe is what Google terms Smart Transcription. Traditional Automatic Speech Recognition (ASR) pipelines require downstream Large Language Models (LLMs) or complex regex pipelines to clean up spoken disfluencies and repair verbal self-corrections. Gemini 3.5 executes this normalization inside the decoding loop itself.
1. Conversational Self-Correction Resolution
When a speaker changes their mind mid-utterance, the model parses the semantic intent and outputs only the corrected proposition:
- Spoken Audio: “Let’s schedule the architecture review for Tuesday—actually no, make it Thursday at 3 PM.”
- Legacy ASR Output:
let's schedule the architecture review for tuesday actually no make it thursday at 3 pm - Gemini 3.5 Transcribe Output:
Let's schedule the architecture review for Thursday at 3 PM.
2. Disfluency & Filler Word Removal
Hesitation markers ("um", "uh", repeated false starts) are discarded by default:
- Spoken Audio: “We, um, we noticed that, like, the latency spiked after the deploy.”
- Gemini 3.5 Transcribe Output:
We noticed that the latency spiked after the deploy.
[!NOTE] Verbatim Mode Flag: For legal depositions, clinical notes, and compliance audits where exact verbatim transcripts are required, developers can pass
"transcription_mode": "VERBATIM"in the request configuration to disable semantic filtering and preserve all spoken utterances.
Real-World Performance & Benchmarks
Google reports a 70% reduction in time-to-final-transcription compared to Chirp 3. In enterprise evaluation across multi-accent English and multilingual datasets:
- Accuracy: Non-streaming batch WER dropped to 2.6%, outperforming Whisper large-v3 on noisy conversational benchmarks.
- Streaming Latency: The WebSocket stream achieves steady-state intermediate token delivery within 180ms, with final transcript reconciliation occurring in under 450ms post-utterance.
- Multilingual Auto-Detection: The model dynamically transitions between languages without requiring an explicit language code parameter, covering 85+ world languages with parity across Latin, Cyrillic, Arabic, and CJK scripts.
Developer Integration: Batch vs. Live API
1. Batch Transcription (gemini-3.5-transcribe)
For file-based processing, the API accepts standard audio binaries with diarization and custom vocabulary configuration:
POST /v1beta/models/gemini-3.5-transcribe:generateContent
Content-Type: application/json
{
"contents": [
{
"parts": [
{
"inline_data": {
"mime_type": "audio/mp3",
"data": "<BASE64_ENCODED_AUDIO>"
}
}
]
}
],
"generation_config": {
"audio_transcription_config": {
"diarization_enabled": true,
"max_speakers": 3,
"enable_word_timestamps": true,
"custom_vocabulary": [
"Kubernetes",
"eBPF",
"GraphQL",
"Model Context Protocol"
]
}
}
}The response returns speaker-segmented structures with millisecond boundaries:
{
"candidates": [
{
"content": {
"parts": [
{
"speaker_tag": "speaker_0",
"text": "Have we verified the Model Context Protocol endpoints?",
"start_time": "0.120s",
"end_time": "2.450s"
},
{
"speaker_tag": "speaker_1",
"text": "Yes, eBPF telemetry is active across all pods.",
"start_time": "2.600s",
"end_time": "4.800s"
}
]
}
}
]
}2. Streaming Transcribe (gemini-3.5-transcribe-live)
For real-time applications, clients open a WebSocket connection to the Gemini Live API, streaming PCM audio chunks and receiving interim JSON events:
import WebSocket from "ws";
const ws = new WebSocket(
"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=" + API_KEY
);
ws.on("open", () => {
// 1. Initial Handshake & Setup
ws.send(JSON.stringify({
setup: {
model: "models/gemini-3.5-transcribe-live",
generationConfig: {
responseModalities: ["TEXT"]
}
}
}));
});
// 2. Stream 16kHz Linear PCM chunks
function streamAudioChunk(pcmBuffer) {
ws.send(JSON.stringify({
realtimeInput: {
mediaChunks: [{
mimeType: "audio/pcm;rate=16000",
data: pcmBuffer.toString("base64")
}]
}
}));
}
// 3. Receive Streamed Transcript
ws.on("message", (data) => {
const response = JSON.parse(data.toString());
if (response.serverContent?.modelTurn?.parts) {
const text = response.serverContent.modelTurn.parts[0].text;
console.log("Transcribed:", text);
}
});Production Ecosystem & Client Adoption
Google has already deployed gemini-3.5-transcribe-live into production client surfaces:
- Android Rambler: Powers continuous system-level voice dictation with zero-latency visual feedback.
- Gemini App on macOS: Handles real-time voice queries and multi-modal audio input.
- Autonomous Voice Agents: Integrates into full-duplex agentic loops without requiring separate Whisper transcription servers.
Migration from Chirp 3: What Teams Need to Know
- Deprecation of Chirp 3: Google Cloud Speech-to-Text v2 users relying on Chirp 3 should plan migration paths to the
gemini-3.5-transcribeendpoint to take advantage of the 70% latency improvement and lower WER. - Unified Billing: Gemini 3.5 speech models are billed under the Gemini API token and compute model rather than per-minute Cloud Speech quotas.
- Diarization Limit: The native diarization engine is tuned for up to 3 concurrent speakers. Workflows requiring large-panel meeting diarization (4+ speakers) will still require dedicated acoustic clustering or multi-pass processing.