Building a pipeline that dubs a voice

At Speax.ai I led engineering for an AI dubbing platform. A user uploads a video in one language and gets it back in another, in something close to their own voice. The models get the headlines. Most of the engineering is the plumbing around them.

Here is what that plumbing looks like when you design it for the way AI services actually behave: slowly, expensively, and not every time.

Draw the stages before the services

A dub is a chain of transformations. Each one takes files in and puts files out:

1. Ingest — normalise the upload, extract the audio track.
2. Separate — split the voice from music and background.
3. Transcribe — speech to timed text.
4. Translate — text to text, keeping the timing.
5. Synthesise — the translated lines in a cloned voice.
6. Mix and render — the new voice back over the original background.

Writing it down this way does two things. It shows where the expensive steps are, and it makes every arrow between stages a place where you can stop, inspect and resume.

Every stage is a job, every output is a file

Each stage reads its inputs from object storage and writes its outputs back, under a key derived from the inputs. The stage itself keeps no state. That gives you three properties for free:

• Idempotency. Run a stage twice with the same inputs and you get the same key. If the file is already there, skip the work.
• Resumability. A crash in step five does not cost you steps one to four.
• Debuggability. When a dub sounds wrong, you can listen to the output of every stage and find the one that broke.

We built a small S3 file service for exactly this, and it was one of the best-value pieces of the system.

Concurrency is a budget, not a setting

A ten-minute video becomes hundreds of segments, and it is tempting to fan them all out at once. Don’t. GPUs, rate-limited APIs and your cloud bill all have a ceiling. Give each stage its own limit and let the queue absorb the rest. In Go, errgroup does most of the work:

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(cfg.SynthesisWorkers) // the budget for this stage

for _, seg := range segments {
    g.Go(func() error {
        return synthesise(ctx, seg)
    })
}
return g.Wait()

The first error cancels the context, the siblings stop early, and the job is retried as a unit. Different stages get different budgets because they burn different resources.

Retry the weather, not the bug

AI services fail in two ways. Some failures are weather: a timeout, a 429, a busy GPU. Retry those with backoff and jitter. Others are bugs: a corrupt file, a language you don’t support, a segment that is all silence. Retrying those only spends money. Classify errors at the adapter boundary, and make “permanent” an explicit type so the pipeline can stop and say why.

Progress is a feature

A dub takes minutes, not milliseconds. Each stage emits events — started, progress, finished, failed — and the UI subscribes. Users forgive a slow job they can watch. They don’t forgive a spinner.

Leave room for a human

Machine translation of a joke, a name or a brand is often wrong. Because every stage output is a file, a person can open the translated script, fix one line and re-run only what depends on it. The pipeline should make that cheap, not heroic.

None of this is specific to dubbing. Any product built on slow, probabilistic services ends up as a pipeline. Design the pipeline first and the models become replaceable parts.