How to Get Transcripts from Video Files with NestJS Pipelines

Why transcription deserves a background workflow

Video is the default format for product demos, onboarding sequences, webinars, internal training, support walkthroughs, and campaign assets. The problem is asymmetric: the information inside those files is rich, but it sits inside an audio stream that search indexes, support bots, and language models cannot read. A transcript breaks that lock. Once you have text, you can index it, translate it, summarize it, generate captions, pull clips by topic, measure content performance, and satisfy accessibility requirements that many organizations and jurisdictions now expect.

The naive implementation looks tempting: accept a file in one controller, call a speech-to-text API, return the text. It works in a demo and collapses in production. Audio extraction is CPU-heavy, engine calls take minutes rather than milliseconds, uploads arrive in bursts at unpredictable times, and any network hiccup leaves a half-finished job with no way to resume. A synchronous handler turns every one of those realities into a timeout for your users.

NestJS is a good host for this workload because it gives you structure without ceremony. Modules keep provider adapters swappable, dependency injection makes the pipeline testable with fakes, interceptors and guards handle authentication and throttling in one place, and integrations with queue libraries and ORMs mean you are composing familiar pieces rather than inventing infrastructure. The rest of this guide describes a production-shaped design: decoupled intake, clean audio preparation, a criteria-based engine choice, durable job states, useful storage, enrichment, and observability.

Architecture: decouple intake from processing

The most important decision in the whole system is that the upload endpoint must never do the transcription. The HTTP layer accepts a file or a reference to one, validates it cheaply, writes a job record, and returns an identifier immediately. Everything expensive happens somewhere else.

The canonical flow has five steps. A client either requests an upload target or posts the file directly to an API endpoint. The API validates the file, stores it, and creates a job in a queued state. A worker process picks up the job, extracts audio, calls a speech-to-text engine, and normalizes the result. The normalized transcript is written to durable storage and indexed for search. The client learns about completion through polling, a webhook, or a server-sent event stream.

Design job states that survive reality

Vague states produce vague bugs. Define an explicit state machine and record every transition with a timestamp: queued, preparing, transcribing, post processing, completed, failed. The preparing state covers container validation and audio extraction. Transcribing holds a provider-side job reference so a crashed worker can reconcile on restart. Post processing covers speaker separation, punctuation cleanup, and derivative generation. Failed is terminal and must carry a machine-readable error code plus an attempt count.

Give every job an idempotency key derived from the file checksum combined with the engine, language, and speaker-labeling settings. If the same file arrives twice, you either return the existing finished transcript or return a conflict. What you never do is pay for the same audio twice or ship two transcripts that disagree with each other.

Pick a queue backend with your eyes open

For most teams, a Redis-backed queue such as BullMQ is the pragmatic default. It integrates cleanly with NestJS, supports delayed jobs, priorities, rate limiting, and repeatable schedules. Teams already standardized on managed infrastructure may prefer SQS or RabbitMQ. Teams with long-running, human-in-the-loop steps may prefer a workflow engine such as Temporal. A queue table in Postgres is also viable when you want transactional guarantees without adding another datastore.

Write down your decision criteria before committing. Ordering and concurrency: do you need strict per-tenant ordering, or is global parallelism fine? Visibility: can you inspect waiting, active, delayed, and failed counts from dashboards you already run? Retry semantics: do you need exponential backoff with jitter and a dead-letter queue? Operational cost: a second datastore is a second thing to monitor, back up, patch, and secure.

Prepare audio that speech models read accurately

Speech-to-text engines are dramatically more accurate on clean, correctly formatted audio. If you hand an engine a container with a six-channel surround track and variable frame rate video, you are asking the model to solve problems you could have solved in a second with a conversion step.

Probe before you transcode

Run a probe pass first. A tool such as ffprobe returns container format, duration, stream list, codecs, sample rate, channel layout, and bitrate. Reject files with no audio stream, files that exceed your duration limits, and codecs you have decided not to support. Store the probe output next to the job record. Months later, when someone asks why a specific file failed, that JSON is the answer.

import { spawn } from 'node:child_process';

function probe(filePath) {
  return new Promise((resolve, reject) => {
    const proc = spawn('ffprobe', [
      '-v', 'error',
      '-print_format', 'json',
      '-show_format',
      '-show_streams',
      filePath,
    ]);
    let out = '';
    proc.stdout.on('data', (chunk) => { out += chunk.toString(); });
    proc.on('error', reject);
    proc.on('close', (code) => code === 0
      ? resolve(out)
      : reject(new Error('ffprobe failed')));
  });
}

Normalize to a transcription-friendly format

Most engines perform best with mono, 16 kHz, PCM audio. One extraction command handles the common case:

ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav

For long recordings, prefer FLAC or a high-bitrate Opus file so storage stays reasonable without hurting accuracy. Keep the original media untouched: extracted audio is a derived artifact and can always be regenerated, which means you can delete it aggressively once a job is done.

Chunk long files on silence, not on fixed offsets

Engine limits vary, but practical ceilings often sit between thirty minutes and a few hours. Cutting a file at an arbitrary byte offset splits words and confuses punctuation models. Detect silence instead:

ffmpeg -i audio.wav -af silencedetect=noise=-35dB:d=0.8 -f null -

Use the detected boundaries as cut points, then offset every timestamp by the chunk start time when merging. Overlap chunks by one or two seconds so a word straddling a boundary survives in at least one chunk, and deduplicate overlapping segments during the merge. Keep the merge map in storage so alignment bugs are debuggable rather than mysterious.

Choose a speech-to-text engine on criteria, not hype

Engine selection is where projects get locked in unnecessarily. Define an interface, write two adapters, and keep the decision reversible. Candidates worth evaluating include Whisper and its hosted variants, Deepgram, AssemblyAI, Speechmatics, Amazon Transcribe, Google Speech-to-Text, and Azure Speech. They differ in ways that matter.

Criteria that actually change outcomes:

• Word error rate on your audio. Studio narration and noisy conference audio behave completely differently. Test with your own files, not with vendor samples.
• Speaker separation quality. Does the engine return speaker turns, and how does it handle crosstalk and overlapping speech?
• Timestamp granularity. Word-level timing enables karaoke captions and precise clip extraction. Segment-level timing is enough for a readable transcript.
• Language coverage and code-switching. Multilingual content often switches mid-sentence, which a long language list does not guarantee.
• Deployment model. Hosted API, self-hosted model, or hybrid. Data residency and retention policies sometimes make the decision for you.
• Throughput and latency. Batch endpoints cost less; real-time streaming only matters for live captioning.
• Cost shape. Per minute, per hour of audio, or per compute-second if you self-host. Model the cost at your expected volume before you commit.

A practical division of labor: use a hosted API for delivery speed and strong speaker separation, and keep a self-hosted Whisper-class model as a fallback for sensitive files or provider outages. The abstraction costs an afternoon and saves a migration later.

interface SttEngine {
  name: string;
  transcribe(input: TranscribeInput): AsyncResult;
}

The interface forces you to describe what you actually need, such as audio path, language hint, speaker flag, and timeout, instead of leaking one vendor's SDK shape into every service in your codebase.

Build the NestJS transcription module

Keep the module focused: a controller for uploads and status, a service for orchestration, a processor for queue consumption, and one adapter per engine.

@Module({
  imports: [BullModule.registerQueue({ name: 'transcription' })],
  controllers: [TranscriptionController],
  providers: [
    TranscriptionService,
    TranscriptionProcessor,
    HostedSttAdapter,
    LocalSttAdapter,
  ],
  exports: [TranscriptionService],
})
export class TranscriptionModule {}

Inject a token, not a concrete SDK

Bind the engine interface to a token and select the implementation from configuration. In tests you swap in a fake that returns a fixture transcript instantly; in production you pick the adapter by environment. That single choice is the difference between a test suite that runs in two seconds and one that needs network access and a funded account.

Keep the processor thin

The processor should orchestrate, not implement. It updates progress, calls the engine, normalizes the response, persists segments, and returns a small summary. Anything more, including retries, formatting, and indexing, belongs in dedicated services so it can be tested and reused.

@Injectable()
export class TranscriptionProcessor extends WorkerHost {
  constructor(
    @Inject(STT_PROVIDER) private readonly engine: SttEngine,
    private readonly repo: TranscriptRepository,
  ) { super(); }

  async process(job) {
    await job.updateProgress(10);
    const raw = await this.engine.transcribe({
      audioPath: job.data.audioPath,
      language: job.data.language,
      diarize: job.data.diarize,
    });
    await job.updateProgress(80);
    const normalized = normalize(raw, job.data.offsetMs);
    await this.repo.save(job.data.jobId, normalized);
    return { segments: normalized.segments.length };
  }
}

Move cross-cutting concerns into the framework layer

Authentication, tenant resolution, request size limits, and rate limiting belong in guards and interceptors, not in the transcription service. This keeps the pipeline reusable: the same queue and processor can serve an internal admin tool, a public API, and a batch import job without duplicating policy code.

Reliability: idempotency, retries, and dead letters

Network calls fail, rate limits happen, and workers restart mid-job. Design for those events rather than hoping they never occur.

Classify errors by whether a retry can possibly help. Timeouts, throttling responses, server errors, and transient storage failures are retryable. Unsupported codecs, missing audio streams, authentication failures, and content rejected by policy are terminal, and retrying them wastes capacity while delaying the real fix.

Use exponential backoff with jitter, cap attempts at three to five, and route exhausted jobs to a dead-letter queue with the full error context attached. A poison job that cycles forever is worse than a failed job you can inspect, because it quietly consumes worker slots and hides the underlying problem.

Idempotency deserves the same rigor on the delivery side. If your engine supports webhooks, verify an HMAC signature and ignore duplicate deliveries. If you poll a provider job reference, store that reference on the job row so a restarted worker resumes instead of resubmitting. When a job completes, write the transcript in a transaction that also flips the job state, so you never have a finished transcript attached to a queued job.

Finally, cap concurrency per worker. Audio extraction and transcription both saturate CPU and network. Four to eight concurrent jobs per worker instance, scaled horizontally, behaves far better than fifty concurrent jobs on one box that spends its time swapping memory.

Store and serve transcripts that stay useful

A transcript is not a single string. Storing only plain text throws away timestamps, speaker labels, and confidence values that make the data valuable six months later.

A schema that survives feature requests

A workable relational shape has four tables. A jobs table holds the tenant, source file reference, state, engine name, language, error code, and timestamps. A transcripts table holds the job id, detected language, duration, model version, and creation time. A segments table holds the transcript id, index, start and end times in milliseconds, speaker label, text, and confidence. An optional words table holds word-level timing for caption generation and clip extraction.

Store the raw engine payload in object storage as JSON alongside the normalized rows. You will want it when normalization logic changes, when you need to regenerate captions without calling the engine again, or when someone disputes a line of text.

Search and retrieval

Postgres full-text search with a generated tsvector column covers keyword search cheaply and well. Vector embeddings over segment text enable semantic search and question answering. The strongest pattern is hybrid: full-text for exact product names and acronyms, embeddings for conceptual queries. Index asynchronously after the transcript is saved so a slow embedding call never blocks job completion or inflates perceived latency.

Deliver formats that clients actually need

Return JSON for applications and generate derivatives on demand: WebVTT and SRT for captions, plain text for copy-paste, and a chaptered Markdown export for documentation. Cache derivatives with an ETag tied to the transcript version so a regenerated transcript invalidates stale captions automatically. For long jobs, expose a status endpoint plus a server-sent event stream so the interface shows progressive updates instead of polling every second.

Speaker labels, metadata, and enrichment

Speaker separation answers the question every reader asks: who said this? Good output is a list of speaker turns with time ranges. Bad output flips labels mid-sentence and destroys trust in the whole transcript. Mitigations that work: prefer engines that return turns rather than per-word labels you must reconstruct; merge adjacent turns from the same speaker when the gap is under 300 milliseconds; relabel anonymous speakers with a stable convention such as Speaker 1 and Speaker 2, then let users map real names afterwards; and flag low-confidence segments for human review instead of silently shipping them.

Beyond speakers, enrich with data that is cheap to compute now and expensive to reconstruct later: detected language per segment, punctuation and casing cleanup, chapter boundaries derived from topic shifts, entity extraction for names and product terms, and a duration-weighted word count that feeds analytics. Each of these is a small addition to the normalization step and a large improvement to everything downstream.

Security, quotas, observability, and the mistakes to avoid

A transcription pipeline handles user media, which makes it a security surface. Guard every endpoint, scope every query to a tenant, and serve media and audio artifacts through short-lived signed URLs. Verify webhook signatures and reject unsigned payloads. Enforce maximum file size, maximum duration, and allowed container formats at the edge, before anything is written to storage. Track audio minutes per tenant per period and reject or queue beyond the agreed allowance, because a runaway retry loop on a large file can burn a month of budget in an hour. Define retention separately for source media, extracted audio, raw engine payloads, and final transcripts. Extracted audio is usually safe to delete the moment a job completes.

Observability deserves equal attention. Emit structured logs that carry the job id and tenant on every line. Track queue depth, job duration percentiles per engine, failure rate by error code, and audio minutes processed per day. Alert on queue depth growth and failure-rate spikes rather than on individual job failures. Sample finished transcripts periodically and compare them against a small human-reviewed set to compute word error rate. That is the only honest signal that accuracy held steady after a model or configuration change.

Common mistakes worth naming explicitly:

• Transcoding inside the request cycle. The fastest route to an unreliable API.
• Skipping idempotency. Retrying clients are normal; duplicate provider charges are not.
• Discarding timestamps. Plain text is a dead end for captions, chapters, and clip extraction.
• Hard-coding one vendor SDK. A vendor change becomes a rewrite instead of an adapter swap.
• Unbounded worker concurrency. Saturation looks like an outage and costs more than it saves.
• Testing only with studio audio. Build fixtures with crosstalk, music beds, accented speech, and phone-quality recordings.
• Ignoring silence. Long pauses inflate duration-based cost, so trim them where the content allows.

Frequently asked questions

Can I transcribe video without extracting audio first?

Some engines accept video containers directly, and for a quick prototype that is fine. Extracting audio yourself gives you control over sample rate, channel layout, and chunking, makes caching and retries cheaper, and lets you detect broken or missing audio tracks before you spend on an engine call. In a production pipeline, extraction is the safer default.

How long should a transcription job take?

For batch endpoints, budget roughly a quarter to a half of the media duration plus queue wait. A ten-minute video typically finishes in three to six minutes. Self-hosted models on a GPU can be faster, but they require capacity planning and a strategy for handling bursts.

Do I need speaker separation on every job?

No. Enable it for interviews, panels, and meeting recordings. For single-narrator content, it adds cost and occasional label noise without improving the transcript.

How do I handle a three-hour recording?

Chunk on silence boundaries, overlap by one or two seconds, transcribe chunks in parallel with a concurrency cap, then merge with timestamp offsets and deduplicate overlapping segments. Store the merge map so alignment issues can be diagnosed later.

Should I self-host a Whisper-class model?

If you have GPU capacity, strict data residency requirements, or very high steady volume, self-hosting can be the better trade. If you need strong speaker separation and low operational overhead, a hosted API usually wins. The adapter pattern lets you support both and switch per file.

How do I keep accuracy stable over time?

Version every transcript with the engine and model identifier, keep the raw payload, and re-run a sample through human review after any model or configuration change. Accuracy is a maintenance property, not a one-time setting.

What about live streams?

Live captioning is a different architecture: streaming audio frames, partial hypotheses, and a caption buffer that smooths revisions. Build it as a separate service that shares storage and schema with the batch pipeline, rather than bending the batch worker into a streaming role.

Where should transcription live in a larger product roadmap?

Treat it as shared infrastructure from the start. One pipeline that serves uploads, imports, and internal tooling is cheaper to run and easier to harden than three bespoke scripts, and the transcript store becomes a durable asset that search, analytics, and content teams can all build on.