Generative video is not a normal API call. A single request can occupy a GPU for several minutes, spawn intermediate artifacts, trigger a moderation pass, wait on a human review step, and then fan out into transcoding jobs for three different aspect ratios. If you try to model that as a synchronous request/response cycle, everything falls apart: gateways time out, load balancers drop connections, browser tabs hang, and your users refresh the page until your database fills with duplicate renders.
Event-driven architecture solves this by inverting control. Instead of the caller waiting for the work to finish, the caller submits a job, receives an acknowledgement, and a worker notifies the system later when something meaningful happens. That notification is a webhook. The pattern is simple in theory and surprisingly subtle in practice, because everything that makes distributed systems hard — ordering, duplication, partial failure, retries, clock skew, secret rotation — shows up the moment you stop assuming the network is reliable.
This guide is a practical walkthrough of building an open-source webhook layer for an AI video workflow. It covers trigger models, payload design, infrastructure selection, a reference architecture end to end, and the operational habits that keep a pipeline healthy at scale. It is written for engineers and technical producers who already generate video programmatically and now need the plumbing to hold up under real traffic.
You do not need a commercial webhook platform to do this well. The open-source ecosystem is mature enough that a small team can run signed delivery, retries, dead-letter queues, replay tooling, and delivery analytics on infrastructure they already understand.
The first decision is whether you need webhooks at all. Polling is simpler: the consumer asks the producer "is it done yet?" on a fixed interval. It requires no public endpoint, no signing, no delivery logs. For a prototype with one job per hour, polling is the correct answer and building anything else is over-engineering.
The calculus changes as volume and latency sensitivity grow. A render that takes four minutes polled every five seconds costs roughly forty-eight wasted requests per job, most of them returning "still processing." Multiply that by a few thousand jobs and you are burning API quota and database connections to learn nothing. Webhooks collapse that to a single request that arrives exactly when state changes.
| Dimension | Polling | Webhooks |
|---|---|---|
| Latency to detect completion | Up to one interval | Near-instant |
| Request overhead | High at scale | Low, proportional to events |
| Consumer requirements | Outbound network only | Public HTTPS endpoint, verification |
| Failure visibility | Implicit | Explicit delivery logs and retries |
| Complexity to build | Low | Moderate |
The pragmatic answer for most video teams is a hybrid. Use webhooks as the primary trigger for speed and cost, and run a low-frequency reconciliation sweep — every ten or fifteen minutes — that queries for jobs stuck in a non-terminal state. The sweep catches deliveries lost to an outage, a misconfigured firewall, or a deploy that briefly returned 502s. Webhooks give you responsiveness; the sweep gives you a guarantee.
Also decide which events deserve a webhook. Not every state change matters. A useful event set for video looks like: job accepted, job started, progress milestone, artifact ready, job failed, job cancelled, and review required. Progress milestones should be throttled — emitting on every one percent of progress is a denial-of-service attack on your own consumer.
Payload design is where teams accumulate technical debt fastest, because the first version is always shaped around a single client's convenience rather than around the domain. A good webhook payload is small, self-describing, versioned, and stable across model changes.
Start with a schema_version field. When you eventually need to rename or restructure fields, the version tells consumers which shape to expect without breaking existing integrations. Pair it with an event_type string and a unique event_id that consumers can use for deduplication.
A practical video-job payload includes:
Two rules keep payloads manageable. First, never inline base64 video. It bloats delivery, breaks log tooling, and forces consumers to buffer megabytes before they can verify a signature. Store the artifact in object storage and send a signed URL with a short time-to-live. Second, do not send the full prompt verbatim if it may contain sensitive user input unless the consumer is trusted; a hash plus a retrieval endpoint is often enough.
Batching matters too. If a single project produces twenty clips, sending twenty separate webhooks is fine, but sending one event per progress tick per clip is not. Consider a batch envelope with an array of event objects and a shared signature, which keeps verification cheap while giving consumers a single delivery to process.
The open-source landscape splits into four layers. You rarely need all four, but knowing which problem each solves prevents you from bolting together a system that is more fragile than a plain background worker.
A gateway sits between your producers and your consumers. It handles signing, retries, exponential backoff, delivery logs, and a replay interface. Open-source options worth evaluating include Svix's self-hosted core, Convoy, Hook0, and WebhookX. Their value is not just retry logic — it is the operator console that lets a support engineer answer "did we send that event?" in ten seconds instead of an hour.
If you build your own, the minimum viable feature set is: persistent delivery queue, per-endpoint circuit breaking, exponential backoff with jitter, a dead-letter store, and an authenticated replay endpoint.
For the internal hop between your producer and the gateway, you need durable buffering. Redis Streams and BullMQ are excellent when you already run Redis and your throughput is moderate. RabbitMQ gives you mature routing, dead-letter exchanges, and per-consumer acknowledgements. NATS JetStream is a strong middle ground with low operational weight. Apache Kafka is the right choice when you need replayable logs, multiple independent consumer groups, and long retention — but it is a poor fit for a five-person team generating a few thousand clips a day.
Video pipelines are workflows, not single tasks. Temporal, Cadence, and to a lesser extent Apache Airflow model multi-step processes with durable state, retries per activity, and timeouts that survive a process restart. Temporal in particular handles the "render for twelve minutes then notify three systems" pattern with far less custom code than a home-grown state machine.
Score candidates on six axes:
Here is a concrete four-stage pipeline that scales from a hundred to a hundred thousand jobs without changing its bones.
An API endpoint accepts the generation request. It validates the schema, checks quota and entitlement, normalizes parameters, computes an idempotency key from the request body plus tenant, and writes a job row with status accepted inside a database transaction. In that same transaction it inserts an outbox record. Committing the job and the outgoing event together is what prevents the classic bug where the job exists but the notification never fires, or the notification fires for a job that rolled back.
A publisher process reads the outbox and pushes the event to the queue. A dispatcher worker pulls jobs, checks GPU pool capacity, and acquires a slot. If capacity is full, the job waits — this is backpressure, and it is far better than letting the model server queue hundreds of requests and time out unpredictably. Once a slot is free, the worker submits the render to the inference service and records the external job reference.
For long renders, the inference service emits progress callbacks. Your system throttles them, updates the job row, and forwards a milestone event to consumers at most every thirty seconds. Failures are classified: transient errors (worker preemption, network blip) are retried with exponential backoff up to a cap; deterministic errors (content policy rejection, malformed prompt) are terminal. Never retry a policy rejection — you will simply generate the same rejection five times and burn compute.
When frames are ready, the artifact is uploaded to object storage, a checksum is computed, and a CDN or signed URL is generated. Only then does the artifact.ready event fire. Downstream consumers — transcoders, subtitle generators, thumbnail services, approval UIs — each receive the same event through their own subscription. A final assembly step concatenates clips, applies loudness normalization, and marks the project published, emitting a project.completed event.
The key discipline is that each stage emits its own event rather than one giant "everything is done" notification. Granular events let consumers subscribe only to what they care about, and they make debugging dramatically easier.
Assume every event will be delivered at least once, and possibly out of order. Two consumer-side patterns make this safe.
First, deduplicate on event_id. Store processed event ids in a table with a unique constraint, or in Redis with a TTL long enough to cover your retry window. When a duplicate arrives, acknowledge it and move on. This one habit prevents double-rendering, double-charging, and duplicate publish actions.
Second, make state transitions monotonic. Give each job a status enum and a rule that rejects backwards transitions — a job that reached completed cannot return to processing. If your database enforces that invariant, an out-of-order event becomes a harmless no-op instead of a corrupted record.
For the producer side, the transactional outbox is the pattern that pays for itself fastest. Instead of publishing to a broker inside a request handler — where the broker call can succeed while the database write fails — you insert the event into an outbox table in the same transaction. A relay process then drains it. This guarantees at-least-once publication and keeps your request latency predictable.
Finally, choose your source of truth deliberately. Many mature teams treat the webhook as a notification and the REST API as the source of truth. When a webhook arrives, the consumer calls GET /jobs/{id} to fetch authoritative state, so the payload can stay thin and evolve without breaking anyone.
An unauthenticated webhook endpoint is an open door: anyone who learns the URL can inject fake "render complete" events. The baseline defence is HMAC signing. The producer computes a signature over the raw request body plus a timestamp using a shared secret, and sends it in a header such as X-Signature along with X-Timestamp and a key identifier.
On the consumer side, verify in this order:
Rotation should be boring. Support two active secrets per endpoint — one primary, one previous — and give consumers a window of at least a week to migrate. Publish rotation events through a status page or a dedicated key.rotated webhook so nobody is surprised.
Beyond signing, apply standard hygiene: HTTPS only, no secrets in URLs, IP allowlisting when the consumer has static egress, strict request size limits, and a content-type check. If your consumers are internal services on a private network, mutual TLS is a reasonable additional layer, though it complicates onboarding for external partners.
A webhook system without delivery logs is impossible to operate. The first question during any incident is "did the event leave our system?" — and the answer must be one query away.
Log every delivery attempt with: event id, event type, endpoint id, attempt number, response status, response time, and a truncated response body. Correlate everything with a request id that also appears in application logs. Structured JSON logs make this searchable without heroics.
The metrics that actually predict trouble:
Build a replay tool before you need it. When a consumer has an outage for two hours, you want a button that re-sends all failed events for that endpoint within a time range — not a Python script written under pressure. Replays must preserve the original event_id and signature semantics so consumers deduplicate correctly.
Finally, add a synthetic canary. A scheduled job that emits a harmless ping event to every registered endpoint every fifteen minutes will detect a broken consumer long before a customer does.
Video generation is a queueing problem with a hard physical constraint: GPUs. Your webhook layer should reflect that reality rather than pretend capacity is infinite.
Implement concurrency caps per tenant and per model. When the caps are reached, accept the job but keep it in queued state, and tell the user honestly how deep the queue is. Priority tiers — interactive, standard, batch — let a product team guarantee fast turnaround for premium workflows without building a separate pipeline.
Set generous timeouts. A render that normally takes ninety seconds may take six minutes under contention. A timeout tuned for the happy path will cause spurious retries and duplicate GPU spend. Prefer long timeouts plus heartbeats: the worker emits a progress heartbeat every thirty seconds, and the orchestrator only declares a job dead after three missed heartbeats.
Make long jobs resumable where the model allows it. If a render is split into segments, a failure in segment nine should not discard segments one through eight. Track segment-level state and emit a job.partial event so consumers can show meaningful progress instead of an indefinite spinner.
Watch your egress costs too. Signed URLs to object storage are cheap; proxying gigabytes of rendered video through your application servers is not. Send URLs, not bytes.
These recur across almost every team that builds this layer.
Emitting events inside the request transaction. If the broker publish fails after commit, the event is lost forever. Use the outbox pattern.
Signing the parsed JSON instead of the raw body. Whitespace and key ordering differences break verification in ways that are maddening to debug.
No deduplication. At-least-once delivery is the norm, and consumers that assume exactly-once will double-charge, double-publish, or double-notify.
Unthrottled progress events. Emitting on every percentage point generates hundreds of deliveries per job and trains consumers to ignore notifications.
Retrying deterministic failures. A content policy rejection will fail identically every time. Classify errors before you retry.
No dead-letter queue. Failed deliveries without a terminal resting place vanish silently, and you discover the problem from a customer complaint.
Hardcoding secrets in consumer configs. Rotation then becomes an outage. Support two active secrets from day one.
Treating the webhook payload as the source of truth. Payloads evolve; an API fetch keeps consumers correct across schema changes.
How many events should a single video job emit? Typically four to eight: accepted, started, one or two throttled progress milestones, artifact ready, and a terminal completed or failed event. More than that usually means you are emitting internal telemetry rather than integration events.
Is at-least-once delivery good enough? For almost all video workflows, yes, provided consumers deduplicate on event id and enforce monotonic state transitions. True exactly-once delivery across a network boundary is not achievable; exactly-once effects are, and that is what matters.
Should I build my own delivery service or self-host an existing one? If your only requirement is "POST a JSON body and retry on failure," a small worker is fine. The moment you need per-endpoint delivery logs, replay, rotation, and circuit breaking, self-hosting an established open-source gateway is cheaper than building those features yourself.
How long should I retain delivery logs? Thirty days is a practical default — long enough to investigate a customer report filed weeks later, short enough to keep storage predictable. Keep aggregate metrics far longer than raw payloads.
What about consumers that are temporarily offline? Exponential backoff with jitter over twenty-four hours, then dead-letter. Alert the consumer owner on repeated failures rather than on the first one, and expose a self-service replay.
Do I need a workflow orchestrator if I already have a queue? Only when the process has multiple durable steps with distinct retry policies. A queue moves messages; an orchestrator remembers where a long-running multi-step job was when the process crashed. Video pipelines usually reach that point quickly.
How do I test the whole pipeline? Run a contract test suite that replays recorded payloads against each consumer, plus a staging environment with a synthetic tenant that generates short low-resolution clips. Fire a full workflow on every deploy and assert that all expected events arrive in order.

Alexander
Learn how to integrate Synthesia into a repeatable AI video workflow, from scripting and avatars to review, localization, security, and scale.
Learn how to combine Sora, ChatGPT, and supporting AI tools into a reliable video workflow, from brief to storyboard, generation, review, and delivery.
A practical guide to the video metrics that actually signal platform and channel growth, plus a repeatable AI video workflow for measuring, benchmarking, and improving output.