Building an AI Video Generation Pipeline in NestJS Backends

Why Video Generation Breaks Ordinary Web Backends

A standard REST endpoint is built on a quiet assumption: the work finishes in tens or hundreds of milliseconds. Parsing a body, validating a payload, writing a row, and returning JSON all fit comfortably inside a load balancer timeout. Video generation violates that assumption at every level. A single clip can occupy a GPU for twenty seconds or six minutes, produce a file measured in tens of megabytes, fail halfway through, and require moderation review before it is ever shown to a user.

When teams bolt that work onto a controller, the failure pattern is remarkably consistent. The client waits. The proxy in front of the API gives up at thirty or sixty seconds and returns a 504. The browser or mobile app retries automatically. The API, which never learned that the first request succeeded, launches a second render. Now there are two GPUs busy, two output objects in storage, and one confused customer who eventually receives three near-identical clips and a support ticket asking why they were billed for work they did not request.

The fix is architectural rather than cosmetic. Treat generation as a workflow with durable state: accept intent, persist it, enqueue work, process it asynchronously, publish status, deliver a signed asset. NestJS is a good coordinator for that workflow because TypeScript, modules, dependency injection, guards, and interceptors give you clear seams between transport concerns and business logic. It will not generate a single frame for you, and that is exactly the point. The framework should orchestrate providers, storage, and workers without knowing which model is fashionable this quarter.

A useful mental model is a print shop. The front desk takes an order. The order book records it. The queue is the line of jobs waiting for a press. The presses are GPU workers. The loading dock is object storage. The notification system tells the customer the print is ready. Once you picture the system that way, the module boundaries become obvious, and swapping a press manufacturer no longer requires rebuilding the front desk.

Map the Pipeline Before You Write a Single Module

Before touching decorators, write down the stages a request passes through. A workable list looks like this: intake, validation, persistence, enqueue, dispatch, generation, moderation, post-processing, storage, notification, delivery, and cleanup. For each stage, answer four questions.

First, what is the unit of work? A job is the user's intent. An attempt is one execution of that intent. Keeping those separate is the difference between a system that can retry safely and one that creates duplicates whenever a network hiccup occurs.

Second, which stages are retryable? Validation failures are terminal. A moderation rejection is terminal. A provider rate limit is transient. A GPU out-of-memory error is arguably a configuration problem that deserves a different resource class rather than an immediate retry.

Third, what does failure look like at each stage? A failure during generation should surface a clear error code. A failure during post-processing should leave the raw output intact so you can reprocess it without paying for another render. A failure during notification should never mark the underlying video as failed.

Fourth, where is the source of truth? The database is authoritative. Provider dashboards, queue state, and storage listings are all secondary. Any reconciliation routine must compare those sources back to your own records rather than trusting them.

Teams that skip this exercise usually discover the gaps in production, at which point the cost of adding an attempt table or a cancellation flag is measured in weekend hours rather than a design conversation.

Designing the HTTP Contract: Intake, Status, and Delivery

Intake and validation

Keep the creation endpoint narrow. Something like POST /v1/video/jobs should accept a prompt, a duration, an aspect ratio, an optional style preset, optional source asset identifiers, and an optional seed. It should not accept a provider name, an internal model identifier, or a raw file upload.

Validation belongs in DTOs backed by a schema validator. Check prompt length against the limits your providers actually enforce, not the limits you wish they enforced. Check that the requested duration and aspect ratio exist in your capability registry. Check that every source asset identifier belongs to the authenticated tenant and that the corresponding upload reached a completed state. Attach tenant identity through an auth guard rather than trusting a tenant field in the body, because a body field will eventually be forged or copy-pasted from a stale request.

Large media should never pass through the API process. Issue a signed upload URL, let the client write directly to object storage, then submit the object key with the job. This keeps memory consumption predictable and prevents a 200 MB upload from occupying a Node.js worker that should be handling status checks.

Idempotency is not optional

Require an idempotency key on creation requests and enforce uniqueness per tenant. When the same key arrives again, return the existing job with its current status instead of creating a second render. This single constraint eliminates the most common category of duplicate work in generation systems, and it costs one unique index.

Be strict about scope. A key that is unique globally invites collisions between tenants who happen to use timestamp-based identifiers. A key scoped to the tenant and paired with the endpoint path is safer and easier to reason about.

Status, progress, and notification

Return 202 Accepted with a job identifier, a status URL, and an honest estimate of how long the work will take. Expose a status endpoint that reports a stage rather than a fake percentage. Stages such as queued, generating, post-processing, storing, and complete are truthful and useful. A progress bar that leaps from four percent to ninety percent teaches users to distrust your interface.

For server-to-server integrations, publish signed webhooks. Include the job identifier, terminal status, output reference, error code when relevant, a timestamp, and an HMAC signature over the payload. Consumers should verify the signature, reject stale timestamps, and fetch the authoritative job record before acting on the payload. For browser clients, polling with exponential backoff and a Retry-After hint is simpler to operate than a persistent socket, and it survives deploys far more gracefully.

Always provide a cancellation endpoint. Cancellation should set a cooperative flag that workers check between stages, then clean up partial assets and record the reason. Killing a process mid-render leaves orphaned GPU allocations and files nobody will ever find again.

Queue Topology and Worker Pools

Use a dedicated queue rather than background promises inside the API process. A promise-based approach loses work on every deploy and gives you no visibility into backlog. BullMQ on Redis is a common choice in the NestJS ecosystem; managed queues are equally viable if your team prefers to buy operational simplicity.

Split queues by resource profile rather than by feature name. Text-to-video, image-to-video, upscaling, audio generation, and moderation have different hardware demands and different acceptable latencies, so they deserve separate worker pools. A single queue with one concurrency setting will eventually let a batch of forty archival clips starve an interactive user who is watching a spinner.

Add priority lanes for interactive and batch work, and cap concurrency per tenant. Without a per-tenant cap, one customer submitting a large batch consumes every slot while everyone else waits. Fair scheduling can be as simple as weighted round-robin across active tenants.

Retries need a taxonomy. Transient network errors, provider throttling, and temporary storage failures deserve exponential backoff with jitter. Validation errors, unsupported formats, and policy rejections should fail immediately. Set a maximum attempt count and send exhausted jobs to a dead-letter queue that preserves the original payload, the last error, and the attempt history. A dead-letter entry is a signal for a human or a reconciliation routine, not a graveyard.

Watch out for visibility timeouts. If your queue considers a job abandoned after sixty seconds but generation takes four minutes, workers will be handed the same job repeatedly. Either extend the lock with periodic heartbeats or configure a timeout that matches the real work duration.

Provider Adapters and the Capability Registry

One interface, many vendors

No controller should import a vendor SDK directly. Define a provider interface with methods to submit, poll, cancel, and normalize a result. Everything vendor-specific, including authentication style, polling cadence, status vocabulary, and aspect ratio encoding, lives inside the adapter.

A capability registry then maps a model key to a set of facts: maximum duration, supported aspect ratios, whether image conditioning is available, whether audio is generated, supported output containers, region, latency class, and price band. The API asks for capabilities, and a router chooses a compatible model. When a model is retired, you update one registry entry and one adapter instead of hunting through feature modules.

This structure also makes testing tractable. A fake provider that returns deterministic results lets you exercise the full workflow in continuous integration without spending anything on external inference.

Prompt normalization and versioning

Prompts are product data, not disposable strings. Store a template version and the resolved prompt on every job. Normalize negative prompts, style presets, and duration constraints before dispatch so that two providers interpret the same creative request as closely as possible. If one vendor expects a different aspect ratio enum, translate it inside the adapter rather than sprinkling conditionals through business logic.

Routing decisions and fallbacks

Automatic fallback is useful and dangerous in equal measure. Route to a secondary model only when the user experience remains acceptable, and record the decision on the job so support staff can see why a clip looks different from the one a colleague generated an hour earlier. Never silently reduce resolution, shorten duration, or drop audio. For brand-sensitive work, an honest failure with a retry option beats a quiet downgrade.

Track success rate, median generation time, and retry rate per model and region. Routing without data is superstition. A weekly review of those numbers tells you which routes to retire long before customers start complaining.

Data Model, Storage, and Lineage

Store jobs, attempts, assets, and events as separate records. The job holds intent and current status. Each attempt records the provider, model key, dispatch time, completion time, and error code, which makes retries auditable. Events form an append-only log of status transitions that is invaluable during incident review.

Enforce status transitions in the service layer. A cancelled job must never drift back to running because a stale worker finished its work. Compare-and-set updates on the status column, guarded by the expected previous value, prevent that class of bug cheaply.

Video files belong in S3-compatible object storage, not in the relational database. Key objects predictably by tenant, job, attempt, and asset type, and store checksums and MIME types alongside them. Generate short-lived signed URLs for delivery instead of exposing buckets publicly. Set lifecycle rules so temporary uploads and partially written objects expire automatically, and run a nightly sweep that deletes orphaned assets older than a grace period.

Lineage is the feature most teams wish they had built earlier. For every generated asset, record the prompt version, resolved prompt, model key, model revision if the provider exposes one, seed, duration, aspect ratio, source asset identifiers, and post-processing steps. That metadata enables reproducibility, debugging, rights review, and a genuinely useful product feature: comparing two generations produced from the same source material.

Never overwrite the original when you create a cropped or upscaled variant. Create a new asset with a parent reference so the family tree stays intact, and apply retention rules to the whole tree rather than to individual files.

GPU Workers and Container Runtimes

API pods and GPU workers should deploy separately. Worker images should pin CUDA versions, driver assumptions, Python dependencies, and model weights so that a rebuild does not silently change output quality. Keep a small warm pool if cold starts dominate interactive latency, and cache weights on a fast node-local volume rather than downloading them per job.

Autoscale on queue depth and job wait time rather than CPU. A GPU worker can sit at twelve percent CPU while fully occupied. Health checks should verify that the runtime can actually load a model and allocate memory, not merely that the process is alive and accepting sockets.

If you use spot capacity for batch traffic, workers must handle interruption gracefully: stop accepting new work, finish or requeue the current attempt, and drain. On the isolation side, give every job its own scratch directory and clean it up afterward, mount secrets at runtime instead of baking them into the image, and restrict egress so a compromised worker cannot reach arbitrary destinations. Sensitive tenants can be routed to dedicated node pools when contractual requirements demand it.

Observability: Metrics That Explain Waiting

Instrument the whole chain with a trace that connects the API request, the enqueue operation, worker pickup, the provider call, the storage write, and the webhook delivery. Log with job identifier, tenant identifier, model key, and attempt number. Avoid logging full prompts verbatim when they may contain personal or confidential material; log a hash or redacted excerpt instead.

The metrics that matter most are queue wait time, dispatch latency, generation duration by model, post-processing duration, failure rate by error code, retry rate, stuck-job count, and webhook delivery success. Split dashboards between interactive and batch traffic, because a two-minute median is excellent for one and unacceptable for the other.

Alert on user-facing symptoms rather than on machine statistics. A p95 completion time breach or a rising failure rate is a real incident. A single CPU spike is usually noise. Define a simple objective such as ninety-five percent of interactive jobs completing within four minutes, and let that number drive your paging rules.

Failure Playbook and the Mistakes That Cause It

Distributed pipelines drift. A worker can crash after the provider finishes but before the database is updated. A provider can report success while the webhook is lost. Build a reconciliation routine that finds jobs stuck in a running state beyond a threshold, queries the provider directly, and finalizes or requeues them. Make it idempotent and log every intervention, because silent repairs are indistinguishable from data loss during an audit.

Symptom	Likely cause	First action
Duplicate clips for one request	Missing idempotency constraint	Add the unique index, then deduplicate storage
Requests time out on creation	Provider called synchronously	Move dispatch into a worker
Interactive jobs wait behind batch	Single shared queue	Split queues and add priority lanes
Jobs reprocessed repeatedly	Visibility timeout shorter than work	Add heartbeats or raise the timeout
Outputs exist with failed status	No reconciliation step	Run a provider status sweep
Silent quality regressions	Unpinned worker images	Pin weights and runtime versions

The recurring mistakes are predictable. Storing video blobs in the database makes backups enormous and queries miserable. Hard-coding model names in business logic turns every provider change into a refactor. Faking progress erodes trust faster than an honest spinner. Logging only errors makes it impossible to explain a slow render after the fact. And unbounded retries on a permanently failing job waste GPU hours while hiding a bug that should have been fixed on day one.

A useful decision rule is to optimize for clarity before cleverness. A simple queue with strong status tracking and a working reconciliation job beats an elaborate orchestration graph that nobody on the team can debug at two in the morning.

A Practical Rollout Checklist

• Creation returns 202 Accepted and never waits for a render.
• Every job carries a tenant identifier, an idempotency key, and explicit status transitions.
• Uploads travel directly to object storage through signed URLs.
• Queues are separated by resource profile, with priority lanes and per-tenant caps.
• Workers classify errors and retry only transient failures.
• Vendor SDKs sit behind adapters and a capability registry.
• Outputs live in object storage with signed delivery URLs and lifecycle rules.
• Prompt version, model key, seed, and source assets are recorded for lineage.
• Traces connect API, queue, worker, provider, and storage.
• Reconciliation and dead-letter replay are exercised before launch, not after.

FAQ

Should a NestJS controller ever call a generation provider directly?

Not for long-running video work. The controller should validate, persist, enqueue, and return a status URL. Direct calls belong in worker processes that can retry, time out, and scale independently. A fast synchronous moderation check with a hard timeout is a reasonable exception, provided you accept that it can occasionally reject a valid prompt during an outage.

How many queues does a video pipeline actually need?

Start with one queue per resource profile and one priority split. Text-to-video, image-to-video, upscaling, and moderation usually diverge quickly because their hardware and latency expectations differ. Add queues when isolation or scaling demands it, and remember that every additional queue is another dashboard, another alert, and another thing to configure on a new environment.

Where should generated videos be stored?

In S3-compatible object storage, with metadata and lineage in your relational database. Use signed URLs for delivery, checksums for integrity verification, and lifecycle rules for cleanup. Storing large media as database blobs complicates backups, bloats replication, and rarely delivers the performance you hoped for.

How do you stop duplicate renders when clients retry?

Require an idempotency key on creation, scope its uniqueness to the tenant, and return the existing job for repeated keys. Workers should also verify whether an output already exists before writing a new one, using the job identifier and attempt number in the storage key. Between those two guards, duplicates become rare rather than routine.

What should be monitored first?

Queue wait time, completion time, failure rate by model, retry rate, and stuck-job count. Those five signals tell you whether the bottleneck sits in the API, the queue, the provider, or the GPU layer. Add detailed traces and per-tenant dashboards once the basic customer-facing signals are trustworthy.

How should provider outages be handled?

Detect them through error rates and timeouts, pause the affected routes, and keep jobs queued where the wait is tolerable. If a compatible fallback exists and quality remains acceptable, route to it and mark the job clearly so support can explain any visual differences. Replay failed jobs after recovery, and communicate delays rather than pretending nothing happened.

Is a separate attempt table really necessary?

Yes, once you retry anything. Without attempt records you cannot distinguish a slow first render from a thrashing third attempt, you cannot audit which model produced which output, and you cannot safely replay a failed job without guessing at its history. The table is small, the queries are trivial, and the incident-review value is immediate.

How do you keep the system from becoming unmaintainable?

Keep vendor knowledge inside adapters, keep status transitions inside the service layer, and keep transport concerns inside controllers. Every time vendor-specific logic leaks into business code, future model changes get more expensive. Review new features against one question: would this still work if the underlying provider were replaced tomorrow?