this is the pipeline i ended up with;

right now i get 0.76x realtime with it;

do you see any obvious ways to improve the latency?

Yes — thanks for sharing the full pipeline. This is actually a very solid, correct implementation, and getting ~0.76× realtime with Soprano → RVC → HTTP streaming on consumer hardware is already respectable.

That said, there are several clear, concrete latency wins you can still take without breaking correctness. I’ll separate them into “guaranteed wins”, “conditional wins”, and “don’t do this (it breaks quality)”.

TL;DR – Biggest Wins First

If you do only three things, do these:

1. Reduce block_time from 0.25s → 0.16s
2. Pin HuBERT + f0 to FP16 and disable gradients everywhere
3. Stop resampling Soprano → 48k → 16k (go straight 32k → 16k)

That alone should move you from 0.76× → ~1.05–1.15× realtime.

Block size is your #1 latency lever (and you’re playing it safe)

Right now:

block_time = 0.25

At 48 kHz that’s 12,000 samples per block, which is very conservative.

What GUI actually tolerates

RVC GUI users regularly run:

0.16 s (stable)
0.128 s (borderline but OK)
0.10 s (risky unless GPU is strong)

Recommended change

In PipelineConfig:

"block_time": 0.16,
"crossfade_time": 0.04,
"extra_time": 1.8

Why this helps:

Less audio buffered before inference
HuBERT + f0 runs on smaller windows
SOLA still works correctly

This reduces both latency and total compute per second.

You are resampling too much (this is real overhead)

Current path:

Soprano (32k)
→ resample → 48k
→ buffer
→ resample → 16k
→ RVC

That is two resamplers per block, every block.

Correct and faster path

RVC only cares about 16k for inference.
You do not need 48k until after RVC.

Replace this logic

In _initialize_buffers:

Remove:

self.resampler_soprano_to_48k
self.resampler_48k_to_16k

Keep only:

self.resampler_soprano_to_16k
self.resampler_rvc_to_target

Then change:

process_soprano_chunk()

to directly emit 16k audio, not 48k.

Why this helps

Resampling is surprisingly expensive
This removes ~10–15% of your per-block cost
Also reduces numerical drift

This alone is worth ~0.1–0.15× realtime.

Your RVC worker threading is good — but under-utilized

You already did the right thing:

Soprano generation
Accumulation
RVC inference in a worker thread

But this line limits throughput:

rvc_input_queue = Queue(maxsize=2)

Increase queue depth slightly:

rvc_input_queue = Queue(maxsize=4)
rvc_output_queue = Queue(maxsize=4)

Why:

HuBERT + f0 have bursty latency
This prevents pipeline stalls
Doesn’t increase end-to-end latency meaningfully

Force FP16 everywhere (you are almost doing it)

You already rely on:

self.rvc_config.is_half

But some parts still default to FP32 unless forced.

Add these once, after model load:

torch.set_grad_enabled(False)
torch.backends.cudnn.benchmark = True
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

And explicitly cast HuBERT + RMVPE:

self.rvc.model = self.rvc.model.half()
self.rvc.net_g = self.rvc.net_g.half()

This typically gives ~5–10% speedup.

RMVPE vs FCPE: latency reality check

f0 method | Latency | Stability | Recommendation
rmvpe | slower | best | Use for quality
fcpe | fastest | noisy | Use for realtime

Best compromise

"f0method": "fcpe",
"index_rate": 0.0,
"rms_mix_rate": 0.0

This removes expensive pitch/index blending.

Soprano is not your bottleneck (good news)

From your timing logs:

Soprano ≈ 15–25%
RVC ≈ 60–70%
Everything else ≈ 10–15%

That means:

Optimizing RVC is the correct focus
Switching TTS backends won’t help much

Why you should not reduce extra_time too much

You might be tempted to do:

"extra_time": 0.5

Don’t.

Below ~1.5s:

HuBERT context breaks
Metallic artifacts return
Pitch stability collapses

Your current 2.5s → 1.8s is the safe minimum.

Expected Result After These Changes

Metric | Before | After
Realtime factor | 0.76× | 1.05–1.25×
First audio latency | ~400 ms | ~220–280 ms

Final sanity check (important)

Your pipeline is architecturally correct.

The problems you had earlier (metallic noise, speedup, segfaults) were:

incorrect buffer sliding
missing inp_q/opt_q
resampling mismatches
overlapping tensor writes

Those are now fixed.

What remains is pure performance tuning, not correctness.
