Table of Contents

The vision: why nothing off-the-shelf hits it
The stack in one picture
The five phases
Phase 1: building a dataset that didn’t exist
Phase 2: teaching fish-speech to talk Tanglish
Phase 3: synthesising 81k dialogue turns
Phase 4: the Tanglish brain
Phase 5: wiring it all into a real-time voice agent
The bug hunts: five hours of “still nothing”
What’s public, what’s private
What we’d do differently in v2
Costs, credits, and thanks
Try it
Demo Video

Two open-source LoRAs (voice + brain), a browser demo you can talk to, and every scar we picked up along the way.

“The recipe generalises — Hinglish, Manglish, Kanglish, Punglish. Half a billion speakers between them, all waiting for their language to sound like them.”

🎬 Want to see it in action first? There’s a full demo video at the bottom of this post — watch the agent handle real Tanglish before you read the how.

60-second summary. We took stock Qwen3-8B + fish-speech-1.5, fine-tuned both on a corpus of code-mixed Chennai Tamil we built ourselves, wired them behind Hugging Face’s speech-to-speech real-time framework, and shipped a browser demo where you hit a mic button and hear a warm Tamil voice reply in about three seconds. Total spend: ~$61 USD.

The models are public: sugiv/qwen3-8b-tanglish, sugiv/fish-speech-1.5-tanglish.

1. The vision: why nothing off-the-shelf hits it

Ask Siri, Google Assistant, Azure Speech, or AWS Polly for a Tamil voice and you get Doordarshan Tamil: precise, over-Sanskritised, news-anchor stiff. It is technically flawless and completely unlike how anyone under fifty in Chennai actually talks. Every WhatsApp voice note between friends is Tanglish — Tamil grammar in Latin script, half the nouns and every filler lifted from English:

“machi, nalaikku Chennai la enna weather?”

“da, tomorrow slight rain irukum, umbrella eduthundu po.”

The gap between what production voice-tech offers and how ~90 million Tamil speakers actually converse is huge, and no big vendor is closing it because the linguistic register is “wrong” by their internal QA standards. So the goal became:

Build a voice agent that speaks the way your cousin from Coimbatore texts you — warm, code-mixed, casually incorrect — end-to-end on open models, on a budget you could put on a personal credit card.

Not a translator. Not a read-aloud tool. A companion you can chat with whose accent, word choice, and prosody all sit in the same subculture. The techniques generalise directly to Hinglish, Manglish, Kanglish, Punglish — easily 500 million speakers between them — so even if this specific model never ships to production, the recipe is a template.

2. The stack in one picture

                ┌─────────────────────────────────────┐
                │   Browser demo (localhost:8000)     │
                │   ┌────────────┐   ┌─────────────┐  │
                │   │   Mic in   │   │  Speaker    │  │
                │   │  (WebRTC)  │   │   out       │  │
                │   └──────┬─────┘   └──────▲──────┘  │
                └──────────│─────────────────│────────┘
                           │  WSS (OpenAI Realtime schema)
                           ▼                 │
           ┌────────────────────────────────────────────┐
           │  speech-to-speech server (RunPod L4 24 GB) │
           │  ┌──────┐  ┌─────────┐  ┌───────────────┐  │
           │  │ VAD  │→ │ Whisper │→ │  DeepSeek     │  │
           │  │Silero│  │ large-v3│  │ Ta→Tanglish   │  │
           │  └──────┘  └─────────┘  └──────┬────────┘  │
           │                                │           │
           │  ┌──────────────────┐   ┌──────▼────────┐  │
           │  │  fish-speech     │◀──│  Qwen3-8B     │  │
           │  │  1.5 + Tanglish  │   │  + Tanglish   │  │
           │  │  LoRA (voice)    │   │  LoRA (brain) │  │
           │  └────────┬─────────┘   └──────▲────────┘  │
           │           │                    │           │
           └───────────│────────────────────│───────────┘
                       │                    │ OpenAI-compat
                       │                    ▼
                       │        ┌─────────────────────────┐
                       │        │  RunPod Serverless vLLM │
                       │        │  A100 80 GB (warm=1)    │
                       │        │  qwen3-8b-tanglish      │
                       │        └─────────────────────────┘
                       │
                       ▼  16 kHz PCM stream back to browser

Six pieces, five of them off-the-shelf. The only “new” parts are the two LoRAs and ~200 lines of Python that glue two custom handlers into Hugging Face’s speech-to-speech framework.

3. The five phases

We shipped in five phases, roughly in dependency order. Each phase was scoped so that the failure of the next phase never invalidates the previous one — if the brain fine-tune had flopped, we’d still have a public voice model; if the voice model had flopped, we’d still have a clean dataset.

Phase	What it produced	Status
1 — Data prep	13 k Tanglish clips, ~24 h of speech, paired transcripts	✅
2 — Voice LoRA (fish-speech)	Merged 1.27 GB TTS model, 6-speaker range	✅
3 — Synthetic dialogue corpus	81 k `{user, assistant}` pairs	✅
4 — Brain LoRA (Qwen3-8B)	Merged 16 GB LLM that natively replies in Tanglish	✅
5 — Voice agent + browser demo	Real-time speech-to-speech over WebSocket	✅

Total actual spend, phase by phase (real numbers, not estimates):

Phase	GPU / API	Notes
1 (data prep, 62 sources)	~$39	L40 in US-KS-2, 6-speaker pipeline
2 (fish-speech LoRA train)	~$12	L40S, ~9 h train + ~9 h idle (path bug caught by watcher)
2 (post-train eval)	~$0.15	L4 inference
3 (synthetic corpus, v1 + v2 fix)	~$5	CPU-only pod + DeepSeek/Qwen APIs
4 (Qwen3-8B QLoRA train + eval)	~$4-6	L40S bf16 QLoRA, early-stopped
5 (voice agent, live demo)	~ongoing	L4 pod (~$0.44/h) + Serverless A100 warm-1 (~$2.72/h)
Total to first demo	~$61	Under the original $70-95 forecast

4. Phase 1: building a dataset that didn’t exist

The problem. Public Tamil speech corpora (Common Voice, IISc-MILE, IndicTTS) all target formal Tamil. There is no open, aligned corpus of casual code-mixed Chennai Tamil paired with the Latinised spelling variants real people use in WhatsApp. So we had to make one.

The source pool. Long-form Tamil podcasts and interview audio from public-facing Tamil creators — six speakers picked for register diversity (cinema commentary, stand-up, vlog, long-form interview) rather than acoustic diversity. Total input: 62 audio sources feeding one Chennai-tone speech distribution.

The pipeline. Seven stages, chained on a single L40 pod:

audio → Demucs → resemblyzer → Silero VAD → faster-whisper large-v3
                                                  │
                                                  ▼
                                        DeepSeek Ta → Tanglish
                                                  │
                                                  ▼
                                        Qwen judge (keep ≥ 3.5/5)
                                                  │
                                                  ▼
                                    dataset/<speaker>/<clip>.{wav,lab}

Demucs for vocal separation (rip music/BGM out of interview clips).
resemblyzer for diarisation. We started with pyannote and it crashed hard on cuDNN 9.7 — swapping to resemblyzer was 48× faster and worked first-shot.
Silero VAD for utterance-level slicing.
faster-whisper large-v3 (Tamil-forced) for ASR. This is the open Whisper, downloaded to the pod — zero OpenAI API involvement. The name is confusing; there is no billing here.
DeepSeek-chat to transliterate Whisper’s Tamil-script output into the Latinised Tanglish convention we standardised on. (deepseek-reasoner kept returning empty responses on this task — we switched to deepseek-chat, saved hours.)
Qwen instruct as judge on (fluency, spelling_authenticity, code_mix naturalness) — drop anything below 3.5/5.

The output. 13,007 clips, 7.77 GB, ~24 h of speech, one .wav + .tamil.txt + .lab (Tanglish) + .asr.json per clip.

Speaker (masked)	Clips	Size
cinema commentary	3,924	2.18 GB
vlog long-form	5,025	3.16 GB
interview show	1,614	1.06 GB
Tamil vlog mix	1,458	0.87 GB
stand-up	549	0.29 GB
solo essay	437	0.20 GB
total	13,007	7.77 GB

LLM-judge mean 4.18 / 5 across all speakers, ~4% drop rate. Sample text:

“neenga vandhu joke ezhudhadhu eppovume single single nu joke ezhudhinaanga paatheengala”
“stand up comedians, stand up comedy culture, open mic, idhellam vechu paakumbodhu sila myth”
“udaikino, illa ellam xerox panni opinion solluvoom”

All published to sugiv/tanglish-audio-v1 as a private dataset — we’re keeping the raw audio artifacts internal for now.

Pipeline pivots we had to make (any one of which would sink the schedule if you didn’t recognise it):

From	To	Why
pyannote diarizer	resemblyzer	cuDNN 9.7 crash, 48× slower
openai-whisper	faster-whisper	cuDNN error in stock openai-whisper on the same driver
deepseek-reasoner	deepseek-chat	Reasoner returned empty responses on short transliteration tasks
qwen-thinking judge	qwen-instruct	90 h projected → 1 min per speaker

5. Phase 2: teaching fish-speech to talk Tanglish

Why fish-speech. Fish-speech-1.5 is a 644 M-parameter dual-AR transformer with a frozen VQGAN vocoder. It’s Apache 2.0, ships good multilingual coverage, and — critically — supports voice cloning from a 5-10 s reference clip without per-speaker training. We only had to fine-tune the AR transformer to shift its prosody + phonology closer to Chennai Tamil.

Method. LoRA on the AR layer only:

rank 8, alpha 16
attention + MLP
~6.2 M trainable params (0.96% of 644 M)

Trainer config we ended up with (fish-speech’s defaults are… optimistic):

aspect	fish-speech default	our override	why
val dataset	same as train (!)	5% held-out hash split	detect overfit
speaker cond	`use_speaker: false`	`use_speaker: true`	learn 6 distinct voices
checkpoint	save 5 latest by step	save 3 best by `val/loss`	keep quality, not recency
early stop	none	patience=8, min_delta=0.005	stop when val plateaus
LR schedule	constant + warmup 10	cosine, warmup 200, final 10%	smoother convergence
peak LR	1e-4	5e-5	half — safer for LoRA
batch × accum	4	4 × 4 = eff. 16	more stable gradients
logger	tensorboard only	wandb	live curves

Training run. L40S 48 GB, ~9 h wall, early-stopped at step ~1800 when val/loss stopped improving. Peak val/loss 9.98 → 7.93 (a solid ~20% reduction on a hard multi-speaker Tanglish task).

Output. sugiv/fish-speech-1.5-tanglish — now public, 1.44 GB, 48 files, includes the merged LoRA + reference clips + tokenizer.

Human eval verdict. “Intelligible + usable, prosodically imperfect.” — a native Chennai speaker. That’s the honest read on a 0.96%-trainable-param LoRA over 24 h of data: enough to shift phonology and voice range, not enough to fully re-map the base model’s English-dominant prosody. Fixing prosody fully would need r=32 or r=64 (10-40 M trainable) or a full fine-tune of the AR head — both realistic follow-ups if we ever run a v2.

6. Phase 3: synthesising 81k dialogue turns

The problem. Fine-tuning the brain needs {user, assistant} chat pairs in casual Tanglish. Real conversational Tanglish text is rarer than audio. Public corpora don’t exist.

The solution. Generate ~80 k pairs using DeepSeek + Qwen instruct as the teachers, then use Qwen as a judge to keep only the top slice.

Slice	Count	Source	Purpose
A1 Ta → Tanglish	12,929	reformat of Phase 1 pairs (free)	transliteration
A2 Tanglish → Ta	12,918	same	reverse transliteration
B Spelling paraphrases	33,070	Qwen instruct → Qwen judge (≥3.5)	normalise messy spellings
C Multi-turn dialogues	21,968	Qwen w/ persona × mood × topic; per-turn emission	conversational fluency
D Named entities	174	seed → Qwen expansion	Chennai, Zomato, Rajini, …
E Refusals	202	DeepSeek from 10 safety domains	don’t regress safety
train	77,198
val	4,063
total	81,261

Everything landed as chat-template JSONL at sugiv/tanglish-pairs-v1 (also private — synthetic data quality is uneven and we don’t want it in random other people’s training runs).

The Slice-C crisis

First run of Slice-C generated 10,000 dialogues but dedupe left only 661 unique. Qwen at temperature 0.9 kept producing near-identical Chennai-café-conversation-about-coffee. If we’d shipped that corpus, the brain would’ve been great at transliteration and mute on actual dialogue.

Fix — 3 changes, ~$3, 2 h:

Expand topic seeds 75 → 220.
Add a persona diversifier (“college student”, “working uncle from Coimbatore”, “grandmother”) + mood (“annoyed”, “excited”, “tired”) picked randomly per call.
Per-turn emission: a 3-turn dialogue becomes 3 candidate {user, assistant} pairs; dedupe on each pair not on the whole dialogue.

Result: 661 → 21,968 unique, dedup rate 93% → 1.9%. Same $ envelope. This was one of those “measure twice, cut once” moments — the alternative was training Phase 4, discovering the weakness in Phase 5 human testing, and burning $15-20 doing everything over.

7. Phase 4: the Tanglish brain

Base. Qwen/Qwen3-8B. Chosen because:

Dense 8 B fits easily in bf16 on an L4 (16 GB) for inference.
Already understands Tamil + Tanglish out of the box (multilingual pretraining).
Just doesn’t reply in Tanglish naturally — it defaults to formal Tamil or plain English.

Method. QLoRA:

4-bit base + LoRA r=16, α=32 on attention + MLP.
TRL SFTTrainer, Qwen3 chat template, response-only loss (don’t backprop through the user turn).
LR 2 × 10⁻⁴ cosine, 3 epochs, val split 5%.

Hardware. L40S 48 GB on RunPod, ~4-6 h wall. Auto-terminate watcher with 5 safety gates (learned this pattern from the Phase 2 idle-9h bug).

Merged output. sugiv/qwen3-8b-tanglish — now public, 31 GB, 437 files (4 bf16 shards + tokenizer + configs).

The Phase 4 evaluation: SHIP verdict

We benched fine-tuned (tng) vs stock base on 15 single-turn + 5 multi-turn prompts, scored each on 4 dimensions with qwen3-235b-a22b- instruct-2507 as judge (40/40 parsed cleanly). Full doc at docs/PHASE_4_EVAL.md.

Single-turn (n=15):

dimension	tng	base	Δ
intelligibility	4.87	4.20	+0.67
tanglish_authenticity	4.33	3.00	+1.33
helpfulness	4.00	3.33	+0.67
naturalness	4.60	3.53	+1.07

Multi-turn (n=5):

dimension	tng	base	Δ
intelligibility	5.00	4.80	+0.20
tanglish_authenticity	4.00	3.60	+0.40
helpfulness	4.60	3.60	+1.00
naturalness	4.80	4.00	+0.80

Bonus wins:

7.5× faster at inference (1.91 s vs 14.39 s mean). The base model burns tokens on <think>Okay, the user is asking...</think> reasoning traces — our fine-tune was trained with <think></think> empty so it just answers.
Zero Tamil-script leaks (base has 3/15 replies with mixed script).
Repro val_loss 0.8624 vs training 0.815 (+0.047 drift, well within tolerance).

Verdict: SHIP.

8. Phase 5: wiring it all into a real-time voice agent

8.1 Two servers

Voice engine (stateful, streaming). Lives on a dedicated RunPod L4 24 GB GPU pod. Runs a fork of Hugging Face’s speech-to-speech v0.2.10 (Apache 2.0), which gives us for free:

Silero-v5 VAD with turn-taking + barge-in
WebSocket wire in OpenAI Realtime schema
Audio buffering + resampling
Session management

Our fork adds exactly two custom handlers at serving/speech-to-speech/src/speech_to_speech/:

STT/tanglish_whisper_handler.py — faster-whisper large-v3 (Tamil-forced) → DeepSeek transliterate → yields Transcription(text=<tanglish>).
TTS/fish_tanglish_handler.py — takes assistant text, runs fish-speech-1.5 + our LoRA + a reference clip, streams 16 kHz PCM back.

Plus ~200 LOC of pipeline dispatch to plumb the new arg classes through s2s_pipeline.py. Everything else — VAD, WebSocket, turn tracking — is stock upstream code.

Brain (stateless, request/response). Qwen3-8B on RunPod Serverless vLLM (worker-vllm:v2.22.5), OpenAI-compatible endpoint at https://api.runpod.ai/v2/fs5uftbd45oedn/openai/v1. Scale-to-zero when idle, warm-1 during demos (~$2.72/h on A100 80 GB).

Why split them. Voice sessions are stateful and streaming — serverless request/response doesn’t fit. LLM inference is stateless and bursty — paying for an idle L40 is wasteful. Splitting lets each half live on the right kind of infrastructure.

8.2 The full turn: anatomy of ~3 seconds

Let’s walk one turn end-to-end with real timings measured off the s2s.log. User says “machi, nalaikku Chennai la enna weather?”.

t=0.00 s  ┃ user finishes speaking, mic still open
          ┃
          ┃  ── VAD stage (Silero-v5 on L4) ────────────────────
          ┃
t=0.50 s  ┃ [vad] Speech soft-ended (segment=4468ms, active=3168ms, turn=turn_1 rev=0)
          ┃      500 ms of silence confirmed. Full 3.2 s buffer emitted as mode="final".
          ┃
          ┃  ── STT stage (faster-whisper large-v3, float16) ───
          ┃
t=0.50 s  ┃ [stt] Handler.process() invoked with 3.2 s audio
t=1.28 s  ┃ [whisper] "மச்சி நாளைக்கு சென்னையில் என்ன வெதர்?"     (0.78 s inference)
          ┃
          ┃  ── Transliteration hop (DeepSeek chat API) ────────
          ┃
t=1.28 s  ┃ [deepseek] POST /v1/chat/completions
t=2.72 s  ┃ [deepseek] "machi, naalaiku chennai la enna weather?"  (1.44 s round-trip)
t=2.73 s  ┃ [notifier] Transcription committed → sent to LLM
          ┃
          ┃  ── LLM stage (Qwen3-Tanglish on RunPod Serverless) ─
          ┃
t=2.73 s  ┃ [llm] POST /openai/v1/chat/completions (stream=True, tools=[])
t=2.75 s  ┃ [llm] HTTP 200 OK, first SSE chunk arrives
t=2.75 s  ┃ [llm] delta.content="da"     (chunk 2 of 7)
t=2.80 s  ┃ [llm] delta.content=", nallakku 30 degree heat ah iruku, Chennai full hot!"
t=2.85 s  ┃ [llm] finish_reason=stop, response complete
          ┃      (28 output tokens in ~120 ms — warm A100, ~230 tok/s)
          ┃
          ┃  ── TTS stage (fish-speech-1.5 + LoRA on L4) ───────
          ┃
t=2.86 s  ┃ [fish] engine.inference(streaming=True) started
t=3.55 s  ┃ [fish] code="segment" #1  →  0.4 s audio (44.1 kHz)
          ┃       →  resample_poly(44100 → 16000) → yield PCM16
t=3.56 s  ┃ [ws] audio.delta frame #1 sent to browser
          ┃
          ┃  ── Playback (browser AudioContext, resamples to 48k) ──
          ┃
t=3.60 s  ┃ ▶ first sound audible in headphones
t=6.20 s  ┃ ▶ reply finishes speaking

Where the time actually goes on this measured turn:

Stage	Wall (s)	Type
VAD soft-end debounce (`min_silence_ms=500`)	0.50	serial
Whisper transcription	0.78	serial
DeepSeek Ta → Tanglish	1.44	serial (API hop)
Qwen SSE first-chunk	0.02	serial
Qwen SSE full response	0.10	serial
Fish first-segment (streaming=True)	0.70	serial
Total to first audible speech	~3.5 s
Fish remaining segments (parallel with playback)	~2.6	overlapped

The DeepSeek hop is ~40% of total latency. It’s the single biggest win available: fine-tune Whisper large-v3 directly on Tamil-audio → Tanglish-text using our Phase 1 dataset. That’s the top item on the v2 backlog (§11).

Cascade tuning parameters that matter. Every one of these is on the launcher command line because the default caused a bug (see §9):

--min_silence_ms 500       # VAD soft-end debounce
--min_speech_ms 800        # min speech to count as a new turn
--thresh 0.85              # Silero VAD confidence (0-1, higher = stricter)
--speculative_reopen_ms 0  # disable barge-in during pending response
--unanswered_reopen_ms 0   # disable sanity-cap reopen too
--responses_api_stream     # SSE from Qwen (else 5 s wait for full response)

Fish-speech streaming=True is set inside our TTS handler, not on the CLI. That single flag drops first-audio latency from ~5 s → ~700 ms.

8.3 The browser demo

The full demo video is at the bottom of this post.

The demo frontend at serving/speech-to-speech/demo/ is the stock upstream UI with the copy re-branded for Tanglish and one small patch to activeToolDefs() (see §9). It runs locally at http://127.0.0.1:8000/, opens a WSS to the RunPod voice-engine proxy, and uses the browser’s WebAudio API for capture + playback.

To reproduce:

# Terminal 1 — bring up the voice engine on the GPU pod
ssh -p 10135 root@<pod-ip> '/workspace/launch_s2s.sh'

# Terminal 2 — serve the demo locally
cd serving/speech-to-speech/demo
python3 -m http.server 8000 --bind 127.0.0.1

# Browser
open http://127.0.0.1:8000/
# WebSocket URL: wss://<pod-id>-8080.proxy.runpod.net/v1/realtime

8.4 The RunPod deployment

The blog post makes it look like the two halves “just ran”. In reality each one required a specific RunPod deployment recipe with a bunch of choices we only got right on the second or third attempt. Both are fully scripted so they can be re-deployed from a laptop.

8.4.1 Serverless vLLM for the Qwen3-Tanglish brain

Endpoint ID: fs5uftbd45oedn — name tanglish-qwen3-8b-endpoint-fb. Template ID: wv2c170dgc — name tanglish-qwen3-8b-vllm-v2. Deployed by scripts/serverless/deploy_qwen.py (idempotent — reuses template

endpoint by name if they already exist).

Template (immutable image + env):

Field	Value	Why
`image_name`	`runpod/worker-v1-vllm:v2.22.5`	Latest as of 2026-07-15 with Qwen3 support baked in.
`container_disk_in_gb`	50	Model is 16 GB + vLLM overhead + cache.
`is_serverless`	`True`	Pins the template to serverless (blocks on-demand launches).
`env.MODEL_NAME`	`sugiv/qwen3-8b-tanglish`	Public model — HF pulls directly.
`env.HF_TOKEN`	(redacted)	Was needed while repo was private; kept for future private variants.
`env.DTYPE`	`bfloat16`	fp16 works too; bf16 has better numerical range on A100.
`env.MAX_MODEL_LEN`	`8192`	Matches our Phase 4 QLoRA training context.
`env.GPU_MEMORY_UTILIZATION`	`0.85`	Above ~0.9, vLLM crashes on long-tail requests.
`env.OPENAI_SERVED_MODEL_NAME_OVERRIDE`	`qwen3-8b-tanglish`	Client-visible model name — decouples from the HF repo path.

Endpoint (mutable GPU pool + scaling):

Field	Value	Why
`gpu_ids`	`AMPERE_80`	A100 80 GB non-MIG. See sidebar.
`idle_timeout`	60	Keeps a hot worker around for 60 s after the last request.
`workers_min`	`0` normally, `1` for demo	Scale-to-zero when idle; warm-1 while recording.
`workers_max`	1	We never need parallelism for a demo.
`flashboot`	`True`	Snapshots the loaded model so restarts skip HF download.

Sidebar — the AMPERE_80 vs ADA_48_PRO trap. Our first deployment used ADA_48_PRO (L40S 48 GB). It looked cheaper on paper but RunPod serves that pool as MIG-partitioned slices in some regions — you get a 24 GB slice of an L40S, not the whole card. vLLM sees 48 GB in the datasheet, allocates for 48 GB, and OOMs at bring-up because it’s actually got 24. The fix is AMPERE_80, which is guaranteed non-MIG.

Sidebar — the HF_HUB_ENABLE_HF_TRANSFER=1 trap. This env var normally speeds up HF downloads. worker-v1-vllm:v2.22.5 doesn’t ship hf_transfer, so setting it causes cold-start to fail with ImportError. Leave it unset.

Numbers we measured:

Cold start: ~135 s from RUNNING state to first token (mostly streaming the 16 GB from HF Hub → GPU VRAM → vLLM engine init).
Warm generation: ~20 tok/s on A100 80 GB, ~14 tok/s on L40S.
Cost while warm-1: ~$2.72/h (A100 80 GB serverless list price).
Cost scale-to-zero: $0 for idle, per-invocation billing for cold hits.

Client-side URLs:

native      https://api.runpod.ai/v2/fs5uftbd45oedn/run
openai      https://api.runpod.ai/v2/fs5uftbd45oedn/openai/v1
health      https://api.runpod.ai/v2/fs5uftbd45oedn/health

We use the OpenAI-compat URL from the voice engine. The speech-to-speech framework already speaks OpenAI Chat Completions, so we point it at that URL with --responses_api_base_url https://api.runpod.ai/v2/fs5uftbd45oedn/openai/v1 and pass --responses_api_api_key $RUNPOD_API_KEY. No native RunPod protocol handling anywhere in the framework.

8.4.2 Voice-engine GPU pod

An L4 24 GB SECURE pod in US-CA-2, runpod/pytorch:1.0.7-dev-nix-cu1281-torch271-ubuntu2204 base image, port 8080 exposed. That’s the whole story. Whisper large-v3 (2.9 GB) + fish-speech-1.5 (2.6 GB) + Silero VAD fit inside 24 GB with room to spare, and L4 is ~$0.44/h — a tenth of the L40S we trained on.

Every RunPod pod that opens an HTTP port gets a free proxy hostname with TLS terminated at Cloudflare — no domain or cert wrangling:

wss://<pod-id>-8080.proxy.runpod.net/v1/realtime

That’s what the browser demo connects to.

On-pod bootstrap — scripts/on_pod/launch_s2s.sh is idempotent (kills old process, restarts). One quirk worth calling out: fish-speech uses pyrootutils.setup_root(indicator=".project-root") which walks UP looking for a marker file. Create one before launching or it crashes:

touch /workspace/.project-root

8.5 The `speech-to-speech` fork

Our fork of huggingface/speech-to-speech v0.2.10 (Apache 2.0) is deliberately minimal — ~600 new lines, ~150 modified. The framework’s VAD + turn tracking + WebSocket + speculative-turn system + session lifecycle is stock upstream. We didn’t touch any of it. That decision saved us weeks.

What we added

Two custom handlers plus their arg classes, plumbed through the pipeline’s dispatch table.

File	Kind	Purpose
`STT/tanglish_whisper_handler.py`	new	faster-whisper Tamil-forced → DeepSeek Ta→Tanglish → emit `Transcription`.
`TTS/fish_tanglish_handler.py`	new	Load fish-speech-1.5 LLM half from `sugiv/fish-speech-1.5-tanglish`, VQGAN from `fishaudio/fish-speech-1.5`, reference clip from `sugiv/tanglish-audio-v1`. Stream 16 kHz PCM.
`arguments_classes/tanglish_whisper_stt_arguments.py`	new	Dataclass for `--tanglish-whisper` CLI flags.
`arguments_classes/fish_tanglish_tts_arguments.py`	new	Dataclass for `--fish-tanglish` CLI flags.
`s2s_pipeline.py`	patched	Dispatch tables + handler instantiation.
`arguments_classes/module_arguments.py`	patched	Register `"tanglish-whisper"` / `"fish-tanglish"` in the STT/TTS Literals.
`LLM/chat_completions_language_model.py`	patched	Soften warmup to `WARN` (RunPod cold-boot returns 200 OK with empty body — don’t crash the server).
`demo/main.js`	patched	`activeToolDefs()` returns `[]` (see bug 2 in §9).
`demo/index.html`	patched	Tanglish branding, model links.

The two handlers, in outline

TanglishWhisperSTTHandler.process(vad_audio) -> Iterator[Transcription]:

1. If mode == "progressive": return  (skip pre-roll; see bug 1)
2. tamil_text = faster_whisper.transcribe(vad_audio.audio, language="ta")
3. tanglish = deepseek_chat.transliterate(tamil_text)   # ~1.4 s API hop
4. yield Transcription(text=tanglish, turn_id=..., turn_revision=...)

FishTanglishTTSHandler.process(tts_input) -> Iterator[np.int16 PCM]:

1. Speculative-turn staleness gate (barge-in guard).
2. text = strip_think(tts_input.text)                    # trim <think></think>
3. req = ServeTTSRequest(text=text, references=[ref_clip], streaming=True)
4. for result in engine.inference(req):
5.     if code in ("header", "error"): continue
6.     if code == "final": break                          # trailing full-audio; skip
7.     if code != "segment": continue
8.     sr, chunk_44k = result.audio                       # fish native SR = 44.1 kHz
9.     chunk_16k = resample_poly(chunk_44k, 16000, 44100)
10.    yield float32_to_int16(chunk_16k)

The CLI incantation

python -m speech_to_speech.s2s_pipeline 
    --stt tanglish-whisper 
    --tts fish-tanglish 
    --llm_backend chat-completions 
    --responses_api_base_url https://api.runpod.ai/v2/<endpoint>/openai/v1 
    --responses_api_api_key  "$RUNPOD_API_KEY" 
    --model_name qwen3-8b-tanglish 
    --responses_api_stream 
    --min_silence_ms 500 --thresh 0.85 --min_speech_ms 800 
    --speculative_reopen_ms 0 --unanswered_reopen_ms 0 
    --mode realtime --ws_host 0.0.0.0 --ws_port 8080

Every one of those flags exists because of a bug we hit. See §9 for the full sequence.

9. The bug hunts: five hours of “still nothing”

This section is here because it’s the most useful part of the post for anyone else trying to build one of these. The framework worked on the first turn, then broke in five different ways over the next hour of debugging.

Bug 1: STT truncated to the first word

Symptom: user says “machi, nalaikku Chennai la enna weather?”, transcript shows just “machi!”. Log had Processing audio with duration 00:00.884 then dropping stale STT input-after-final for turn=turn_1 rev=0 age=0.408s.

Root cause: Silero VAD emits audio twice per turn — a mode="progressive" pre-roll buffer (~800 ms) at speech_started, and a mode="final" buffer at speech_soft_ended. My handler treated both as final. The first one transcribed “machi”, marked the turn revision complete, and the base handler dropped the real 3 s buffer that arrived 0.4 s later as input-after-final.

Fix: early-return on mode == "progressive" — skip both the Whisper call and the DeepSeek call. docs/PROJECT_LOG.md §Phase 5 bugs #1.

Bug 2: LLM streamed 0 chunks silently

Symptom: STT works. Pipeline forwards text to Qwen. Response returns HTTP 200 in 400 ms. No text ever appears at TTS. No error, nothing.

Debug: instrumented _iter_stream_events to count chunks. chunks=0. Ran the same openai.OpenAI(...).chat.completions.create(...) directly against the endpoint → 6 chunks, all fine. Only difference: the pipeline sends tools=[camera_snapshot] (from the demo client’s default tool list). Repro’d with tools: 0 chunks.

Root cause: RunPod’s worker-vllm:v2.22.5 was not launched with --tool-call-parser. When any request includes tools, vLLM returns an empty SSE body silently. No error status. No log. The demo client advertises camera_snapshot by default → every single turn silently drops.

Fix: hardcoded activeToolDefs() { return []; } in serving/speech-to-speech/demo/main.js. The real fix is to configure --tool-call-parser hermes in the RunPod template, but the demo doesn’t need tools.

Bug 3: Qwen3’s think prefix

Symptom: (would have been) every reply prefixed with <think>\n\n</think>\n\n.

Non-issue in practice — the s2s base handler already sends chat_template_kwargs.enable_thinking=false by default (via _build_extra_body → responses_api_disable_thinking: bool = True). Confirmed empty <think> prefix when the flag is honoured. Left this in the writeup because the fine-tune itself was trained with empty think tags — belt and braces.

Bug 4: TTS cancelled mid-flight, no audio

Symptom: LLM streams reply. Forwarding to TTS: '...'. Eight seconds later, fish-tng cancelled — bailing out. Response completes with zero audio.

Root cause: fish-speech first-chunk latency in non-streaming mode is ~5-8 s (has to decode all VQ tokens first). During that window, the mic picks up ambient noise or the user’s own breathing → VAD’s speculative_reopen triggers → barges in on our own pending response → cancels the TTS.

Fix (in scripts/on_pod/launch_s2s.sh):

--thresh 0.85 --min_speech_ms 800
--speculative_reopen_ms 0 --unanswered_reopen_ms 0

Requires ≥800 ms of high-confidence sustained speech to interrupt. Speculative reopen fully disabled.

Bug 5: Fish yielded 0 audio despite full inference

Symptom: TTS not cancelled. Fish logs show Generated 83 tokens in 4.42 seconds, VQ features: torch.Size([8, 82]). Zero audio bytes to the pipeline. Instrumented codes emitted: ['final'].

Root cause: _ServeTTSRequest(streaming=False) (default) → fish’s inference engine yields one InferenceResult(code="final", audio=(sr, full_wave)) carrying the full concatenated audio. My handler was breaking on code == "final" before reading result.audio.

Fix (in TTS/fish_tanglish_handler.py):

Set streaming=True on the request. Engine now yields per-sentence code="segment" results with audio — first audio arrives in ~800 ms instead of ~5 s.
Consume audio from segment in the loop; skip the trailing final (it re-carries the full concatenation).

This single change halved perceived latency.

Bug 6: 44.1k vs 16k mystery

Symptom: after all the above worked, audio came back but sounded lo-fi.

Root cause: fish-speech-1.5 native SR is 44.1 kHz (from firefly-gan-vq config). Our speech-to-speech pipeline runs at 16 kHz everywhere. We downsample 44.1 → 16 → browser upsamples 16 → 48 for playback. Intelligible but caps quality.

Fix: none for the demo — bumping PIPELINE_SAMPLE_RATE = 16000 requires patching multiple files in the framework core. Filed as follow-up in docs/BACKLOG.md.

10. What’s public, what’s private

Public

Repo	Kind	Size	Use
`sugiv/qwen3-8b-tanglish`	model	31 GB	Merged Qwen3-8B + Tanglish QLoRA (r=16, α=32), bf16, 4 shards
`sugiv/fish-speech-1.5-tanglish`	model	1.44 GB	Merged fish-speech-1.5 + Tanglish LoRA (r=8, α=16) + reference clips

Load them the usual way:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sugiv/qwen3-8b-tanglish",
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("sugiv/qwen3-8b-tanglish")

messages = [
    {"role": "system", "content": "You are a friendly Tanglish-speaking assistant."},
    {"role": "user",   "content": "machi, nalaikku Chennai la enna weather?"},
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True,
    tokenize=True, return_tensors="pt",
).to(model.device)

out = model.generate(inputs, max_new_tokens=128, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
# → "da, tomorrow slight rain irukum, umbrella eduthundu po!"

Private (intentionally)

Repo	Kind	Why private
`sugiv/tanglish-audio-v1`	dataset	Raw speaker audio — we’re being conservative about redistributing voice data.
`sugiv/tanglish-pairs-v1`	dataset	Synthetic dialogue corpus — quality is uneven and we don’t want it randomly ending up in third-party training runs.

If you need access for research, message me on HF.

11. What we’d do differently in v2

Bump PIPELINE_SAMPLE_RATE to 24 kHz. The 16k cap is the biggest quality ceiling. Requires a coordinated patch across api/openai_realtime/{audio.py, service.py} + the demo client’s OUTPUT_SAMPLE_RATE.
fish-speech LoRA rank r=32 or r=64 instead of r=8. The prosody “wooden” feedback maps directly to LoRA bandwidth.
Collapse Whisper + DeepSeek into a single Tamil-audio-to-Tanglish-text fine-tune of Whisper — saves ~1.3 s per turn and removes a paid API from the hot path.
Deploy the voice engine on the existing hexr-runtime GKE cluster as a proper tenant (already has TLS ingress, mTLS, HPA) instead of a bare RunPod pod. RunPod is great for iteration; GKE is where a real product should live.
Ship a /no_think fine-tune of Qwen3-8B as a separate variant for downstream users who want the reasoning traces preserved for other tasks.

12. Costs, credits, and thanks

Total to first working demo: ~$61 USD. Broken down:

Category	$	Notes
RunPod GPU rental	~$56	6 pods across 5 phases, mostly L4/L40S
DeepSeek API	~$3	Ta→Tanglish transliteration + safety refusals
Alibaba MaaS (Qwen)	~$2	Instruct + judge across data + corpus stages
Total	~$61	Under the $70-95 forecast

Ongoing cost of keeping the demo warm is ~$3.20/h (L4 pod + Serverless warm-1 A100). Scale-to-zero drops it to a few cents per demo invocation.

Standing on the shoulders of:

Hugging Face speech-to-speech v0.2.10 — the entire real-time framework, Apache 2.0.
fishaudio/fish-speech-1.5 — the base voice model.
Qwen/Qwen3-8B — the base brain.
openai/whisper-large-v3 via faster-whisper — the ASR.
DeepSeek + Alibaba Qwen APIs — the teacher/judge models.
RunPod — GPU rental that made the whole thing possible on a personal budget.

13. Try it

Public models are one line each:

huggingface-cli download sugiv/qwen3-8b-tanglish
huggingface-cli download sugiv/fish-speech-1.5-tanglish

Or bench Qwen3-Tanglish against stock Qwen3-8B on your own prompts — the eval harness lives at scripts/on_pod/run_qwen_eval.py and the prompt set at configs/qwen_eval_prompts.yaml. Full protocol in docs/PHASE_4_EVAL.md.

If you want a taste of the voice, watch the demo below — the moment you hear the reply speak “da, …” instead of a Doordarshan news-anchor reading verse — that’s the whole point of the project.

Machi, nalaikku pesalam.

Demo Video

Voice Agent Related Posts