Building a Resilient AI Video Job Pipeline in NestJS

Why long-running video jobs break naive backends

A video generation request looks simple from the outside: someone types a prompt, picks a style, and presses a button. Inside the system, that button press starts a chain of work that can run for minutes, touch several external services, produce large binary files, and fail in a dozen different ways. Teams often describe this category of work as video integration jobs: long-running, resource-heavy tasks that stitch together generative models, object storage, media processing, and user-facing status updates.

The difficult part is rarely the model call itself. It is everything around it. Keeping job state consistent when a worker crashes mid-transition. Preventing duplicate renders when the same callback arrives twice. Avoiding runaway spend when a retry loop misfires against a provider that keeps returning a soft error. Giving users honest progress instead of a spinner that lies for six minutes and then disappears.

This guide walks through a practical backend architecture for that problem using NestJS and TypeScript. It assumes you are comfortable with the framework basics and have a rough idea of how a queue works. The emphasis is on structure, decision criteria, worked examples, and the mistakes that cost the most time to unwind six months later.

One framing helps before diving in: treat the video job pipeline as a product surface, not as plumbing. Users judge the feature by how predictable it feels, how clearly it reports failure, and whether their finished clip is still there next week. Every architectural decision below is downstream of those three promises.

A module map that survives provider churn

Generative video providers change constantly. A model that was best in class last quarter becomes a legacy option, and a new one arrives with different input parameters, different latency, and a different callback format. If the codebase treats each provider as a special case sprinkled through controllers, every new integration turns into a refactor.

Define a module map where volatile parts are isolated and stable parts stay boring:

• Jobs module — intake, validation, state transitions, status queries
• Providers module — one adapter per external model or rendering service
• Media module — uploads, transcoding, thumbnails, signed URL delivery
• Queue module — queue registration, worker bootstrapping, concurrency policy
• Entitlements module — usage metering and plan checks
• Webhooks module — inbound callbacks, signature verification, replay protection
• Common module — shared DTOs, error taxonomy, logging context

Each module owns its own data. The jobs module never reaches into media tables directly; it asks the media service for an asset handle. That single rule prevents most of the accidental coupling that makes these systems painful to test and terrifying to deploy.

Intake stays thin and strict

The controller accepts a request, validates it against a DTO, checks entitlements, writes a job row in a pending state, and enqueues work. It does not call the model. It does not upload anything. It returns a job identifier immediately, because the client will poll or subscribe for the result anyway.

@Post('video-jobs')
async createJob(@Body() dto: CreateVideoJobDto, @CurrentUser() user: User) {
  await this.entitlements.assertCanRender(user, dto.estimatedSeconds);
  const job = await this.jobs.createPending(user.id, dto);
  await this.queue.enqueueRender(job.id);
  return { id: job.id, status: job.status };
}

Validation deserves more attention than it usually receives. Model parameters are not uniform: one provider wants a seed and a negative prompt, another wants a reference image plus a motion strength value, a third wants a duration in seconds and an aspect ratio from a fixed list. Rather than exposing every provider knob to every client, define a normalized request shape and let each adapter translate it. Your public API stays stable while the integrations churn underneath.

Adapters have one job

An adapter is a class with a narrow, predictable contract: submit a render, describe how to interpret a callback, and normalize an error. Nothing else. Keep it narrow so adapters stay cheap to write and easy to delete — you will delete some.

Two design choices matter. First, adapters never write to your database. They return data, and the jobs service decides what to persist. Second, adapters translate provider errors into your own taxonomy: rate limited, content rejected, invalid input, provider unavailable, quota exhausted. Retry logic reads that taxonomy rather than raw HTTP status codes, because providers do not agree on what a 422 means.

Media owns binaries

A single ten-second clip can dwarf every text record in your database combined. The rule is simple: binaries live in object storage, metadata lives in the database, and everything the client sees is a short-lived signed URL. The media module owns three responsibilities — accepting uploads for image-to-video inputs, pulling finished renders from provider URLs into your own bucket, and generating derived assets such as thumbnails and preview clips. Copying the render into your own bucket matters more than it looks. Provider URLs expire, and a link that worked during testing may be dead by the time a user opens their library next week.

Modeling the job lifecycle as a state machine

Status handling is where most pipelines rot. A boolean completion flag becomes a nullable timestamp, then a separate failure flag, then an error message column, and eventually nobody can answer the question "what is actually happening with this job?"

Use an explicit state machine with a small set of states and legal transitions:

From	To	Trigger
pending	submitted	Provider accepted the request
submitted	processing	First progress signal received
processing	completed	Render downloaded and verified
submitted	failed	Terminal provider error
processing	failed	Timeout or unrecoverable error
failed	queued for retry	Retry budget still available
completed	expired	Retention window passed

Every transition goes through one function that validates the move, appends an audit row, and emits an event. Anything that mutates status outside that function is a bug waiting to happen. The audit trail is not optional. When a user insists their render vanished, a transition log answers the question in seconds instead of an hour of log archaeology.

There is a second benefit that teams discover later: a strict state machine makes partial failure survivable. If the download step fails after the provider already produced the clip, you can model that as a separate retryable stage instead of throwing the whole job away. The provider work is the expensive part; never discard it because a later step hiccuped.

Queue topology and concurrency policy

Long-running renders need a queue, but not all queue work is alike. Submitting to a provider is a fast HTTP call that can run at high concurrency. Polling for status is cheap but frequent. Downloading and transcoding a finished video is bandwidth and CPU heavy. Put these on separate queues with separate concurrency settings, or a burst of downloads will starve submissions and make the whole system feel slow.

Practical starting settings:

• Submission queue — high concurrency, short timeout, aggressive retry on rate limit errors
• Polling queue — moderate concurrency, scheduled with a tapering interval
• Media queue — low concurrency, long timeout, sized by CPU cores per worker
• Cleanup queue — low priority, scheduled, deletes expired assets and abandoned uploads

Cap per-user concurrency as well. One account launching twenty renders will happily consume every worker slot you have, and everyone else experiences it as an outage. A simple cap of two or three concurrent renders per account, with a clear queue position message, converts a mysterious slowdown into an understandable wait.

Retry budgets and backoff

Retries should be bounded by both attempt count and elapsed time. A job that has been retrying for forty minutes is not going to succeed; it is going to generate a support ticket. Retry only errors that are plausibly transient, and never retry content rejections or validation failures, because they will fail identically every single time.

Exponential backoff with jitter is the default. Fixed intervals synchronize workers into a thundering herd against provider rate limits, and jitter is the cheapest possible fix. Write the retry policy down as configuration, not as scattered constants, so you can tune it per provider without a code review debate.

Idempotency and duplicate protection

Queues deliver at least once, and providers send duplicate callbacks. Both mean handlers will run twice for the same logical event. Guard against it:

1. Store a request fingerprint derived from a deterministic hash of user, prompt, and settings, and reject duplicate submissions inside a short window.
2. Make transitions conditional — update only when the current state matches the expected previous state.
3. Keep a processed-callback table keyed by the provider event identifier.

Conditional updates are the single most valuable habit here. An update that flips a row from processing to completed returns zero rows the second time it runs, which tells you the event was already handled without any extra bookkeeping. If that update returns zero rows when you expected one, treat it as a signal worth logging rather than an error to swallow.

Webhooks, polling, and callback safety

Most providers notify you when a render finishes. That endpoint is public, untrusted, and will be probed by automated scanners within hours of going live. Treat it accordingly.

• Verify the signature against the raw request body, not the parsed JSON. Re-serializing changes bytes and breaks comparison.
• Reject stale timestamps to blunt replay attacks.
• Respond quickly and do the real work asynchronously. Providers retry on timeouts, and a slow handler manufactures duplicate traffic.
• Never trust callback payloads for authorization. Look up the job by your own identifier and confirm it belongs to the account you expect.

A pattern that holds up well: the webhook controller verifies the signature, writes the raw payload to a durable inbox table, enqueues a processing job, and returns 200. A consumer then performs the state transition. This decouples delivery from processing and leaves a replayable record of everything a provider ever told you — invaluable when a provider's dashboard disagrees with your job list.

Polling without hurting anyone

Not every provider supports callbacks. For those, polling is fine as long as you taper the interval: every two seconds for the first thirty seconds, then every ten seconds for two minutes, then every thirty seconds until a hard deadline. After the deadline, mark the job failed with a clear timeout reason so the user can act instead of waiting forever.

Keep polling metadata out of the hot path. A simple column recording the next poll time is enough; a dedicated scheduler that scans for due polls every few seconds is far cheaper than one scheduled task per job.

Data model, storage, and retention

PostgreSQL is a sensible default for job metadata. Keep the schema small and opinionated:

• jobs — identifier, account, provider key, normalized input as JSONB, status, timestamps, error code, attempt count
• job_events — append-only transition log with actor and payload
• assets — storage key, mime type, byte size, checksum, owner
• callback_inbox — raw provider payloads with processing status
• usage_records — metered quantities per job for reporting

Index according to query patterns: account plus creation time for library views, status plus update time for reaper jobs, and a partial index over active statuses to keep sweeper queries cheap. JSONB is convenient for provider-specific inputs, but keep any field you filter or sort on as a real column. Querying deep into JSONB on a hot path is a performance trap that surfaces at the worst possible moment.

Migrations should be forward-only and reviewed like code. When you introduce a new state, backfill existing rows inside the same migration rather than leaving ambiguity in production. Retention deserves an explicit policy from day one: originals for thirty days, thumbnails for as long as the library entry exists, and a documented export path for users who want to keep a finished clip permanently. Deleting assets is easy; explaining to a customer why an old project no longer opens is not.

Cost control and usage accounting without surprises

Generative video has a real marginal cost per request, so meter at the job level rather than at invoice time. Record an estimate at submission and a reconciled actual at completion, and keep both. When a provider changes rates or you migrate to a different model, historical reports stay meaningful because the estimate and the actual are stored independently of the current rate card.

Guardrails worth building early:

• Reject jobs whose estimated cost exceeds a per-request ceiling, with an actionable message rather than a generic failure.
• Cap daily spend per account and surface the cap in the interface before users hit it, not after.
• Record failed jobs separately, since you often pay for partial work.
• Reconcile provider invoices against usage records monthly. Discrepancies are usually a callback you never processed.

Reserve entitlement at submission and release it on terminal failure rather than deducting afterward. Reservation prevents the classic race where ten parallel requests all pass an affordability check that only one should have passed. A reservation also gives you a natural place to store the estimate, which keeps the reporting model consistent.

Testing, observability, and rollout

You cannot debug a six-minute pipeline with console logs. Attach a request identifier to every job and propagate it through queue payloads, adapter calls, and webhook processing. Then log structured events at transition points only — noise is worse than gaps because it hides the signal you actually need.

Metrics that earn dashboard space:

• Jobs created, completed, and failed per provider, with p50 and p95 durations
• Queue depth and oldest-job age per queue
• Webhook verification failures
• Download and transcode duration
• Retry rate grouped by error code

Alert on oldest-job age rather than queue depth. A deep queue that is moving quickly is healthy; ten jobs stuck for twenty minutes is not.

A fake provider pays for itself

Test in three layers. Unit test adapters against recorded provider payloads. Integration test the state machine against a real PostgreSQL instance with transactional rollback. Then run end-to-end tests against a fake provider that can be instructed to succeed, fail, rate limit, time out, or send a duplicate callback. That fake catches more regressions than any mock, because it exercises the real queue, the real transitions, and the real storage path.

Ship behind a flag

Roll out a new provider behind a feature flag and route a small percentage of traffic to it. Compare duration and failure distributions before widening. Keep the previous path warm for at least a release cycle so rollback is a flag flip rather than a deployment at an inconvenient hour. Track quality signals too: how often users regenerate the same prompt after a render, which is the clearest hint that an engine is producing disappointing output.

Mistakes that cost the most time to unwind

Storing renders in the database. Binary columns feel convenient until backups take four hours and replication lags behind every export.

Treating provider status as truth. Providers report completion before a file is actually retrievable. Verify with a HEAD request and a checksum before marking a job complete.

One queue for everything. Mixed workloads create head-of-line blocking that looks like random slowness and resists diagnosis.

Unbounded retries. Always pair a retry policy with a deadline and a terminal state.

Silent failures. A job that fails without a user-visible reason converts a technical problem into a support problem. Persist a human-readable error code and show it.

Skipping the reaper. Every pipeline needs a scheduled job that finds stuck work and resolves it, because no amount of careful coding eliminates crashes mid-transition.

Faking progress percentages. Show discrete stages — queued, submitted, rendering, downloading, ready — and only advance when a real signal arrives. Users tolerate honesty far better than a bar that stalls at ninety percent.

Ignoring per-account limits. Without concurrency caps, a single enthusiastic user can simulate a platform-wide outage.

FAQ

Do I need NestJS specifically, or would Express work?
Express works fine for a small pipeline with one provider. NestJS pays off once you have several providers, background workers, and a genuine need for dependency injection and module boundaries that stay enforced as the team grows. The architecture advice here transfers to any structured framework; the framework is not the interesting decision.

Should I use BullMQ, RabbitMQ, or a cloud queue?
BullMQ on Redis is the quickest path if you already operate Redis and want scheduling plus repeatable jobs. Cloud queues are excellent for durability and elastic scale but require more glue for delayed retries and visibility into scheduled work. Choose based on operational appetite, not benchmark charts.

How long should a render be allowed to run?
Set a hard deadline from the observed p99 for that provider plus a margin. Ten to fifteen minutes covers most short-form generation. Anything past the deadline is marked failed with a timeout reason so the user can decide whether to retry.

How do I show accurate progress?
Do not invent a percentage. Map provider signals to named stages and advance monotonically. If a provider gives no intermediate signal, show an estimated time range and a cancel option instead of a spinner with no meaning.

What about multi-tenant isolation?
Enforce tenant scoping in the service layer, not only in controllers, and include an ownership check in every query that touches jobs or assets. Add per-tenant concurrency caps so one customer cannot monopolize workers.

When should media processing move off the API process?
Immediately. Any CPU-heavy work in the request path eventually causes timeouts under load. Transcoding, thumbnailing, and checksum verification belong on dedicated workers with their own scaling policy.

How do we handle a provider that silently stops sending callbacks?
Run a reconciliation sweep that queries the provider for jobs still marked processing beyond a threshold. It catches both missed callbacks and provider-side incidents before users notice.

What is the first thing to build if the team is small?
The state machine and the audit log. Everything else — queues, adapters, dashboards — can be simple at first, but unclear job state is the one thing that becomes exponentially harder to fix once real users depend on it.

A video job pipeline is less about any single model and more about disciplined state handling, tight module boundaries, honest observability, and respect for the fact that generation is expensive. Build those first, and swapping in the next generation engine becomes a small pull request rather than a quarter-long project.