Black Friday Offer: Get 50% OFF Starter & Basic Yearly Plans 🎉

Open-Source Webhook Management for AI Video Pipelines

Oct 1, 2026

Why Event-Driven Automation Fits AI Video Production

Generative video is not a normal API call. A single request can occupy a GPU for several minutes, spawn intermediate artifacts, trigger a moderation pass, wait on a human review step, and then fan out into transcoding jobs for three different aspect ratios. If you try to model that as a synchronous request/response cycle, everything falls apart: gateways time out, load balancers drop connections, browser tabs hang, and your users refresh the page until your database fills with duplicate renders.

Event-driven architecture solves this by inverting control. Instead of the caller waiting for the work to finish, the caller submits a job, receives an acknowledgement, and a worker notifies the system later when something meaningful happens. That notification is a webhook. The pattern is simple in theory and surprisingly subtle in practice, because everything that makes distributed systems hard — ordering, duplication, partial failure, retries, clock skew, secret rotation — shows up the moment you stop assuming the network is reliable.

This guide is a practical walkthrough of building an open-source webhook layer for an AI video workflow. It covers trigger models, payload design, infrastructure selection, a reference architecture end to end, and the operational habits that keep a pipeline healthy at scale. It is written for engineers and technical producers who already generate video programmatically and now need the plumbing to hold up under real traffic.

You do not need a commercial webhook platform to do this well. The open-source ecosystem is mature enough that a small team can run signed delivery, retries, dead-letter queues, replay tooling, and delivery analytics on infrastructure they already understand.

Webhooks vs Polling: Choosing the Right Trigger Model

The first decision is whether you need webhooks at all. Polling is simpler: the consumer asks the producer "is it done yet?" on a fixed interval. It requires no public endpoint, no signing, no delivery logs. For a prototype with one job per hour, polling is the correct answer and building anything else is over-engineering.

The calculus changes as volume and latency sensitivity grow. A render that takes four minutes polled every five seconds costs roughly forty-eight wasted requests per job, most of them returning "still processing." Multiply that by a few thousand jobs and you are burning API quota and database connections to learn nothing. Webhooks collapse that to a single request that arrives exactly when state changes.

Dimension Polling Webhooks
Latency to detect completion Up to one interval Near-instant
Request overhead High at scale Low, proportional to events
Consumer requirements Outbound network only Public HTTPS endpoint, verification
Failure visibility Implicit Explicit delivery logs and retries
Complexity to build Low Moderate

The pragmatic answer for most video teams is a hybrid. Use webhooks as the primary trigger for speed and cost, and run a low-frequency reconciliation sweep — every ten or fifteen minutes — that queries for jobs stuck in a non-terminal state. The sweep catches deliveries lost to an outage, a misconfigured firewall, or a deploy that briefly returned 502s. Webhooks give you responsiveness; the sweep gives you a guarantee.

Also decide which events deserve a webhook. Not every state change matters. A useful event set for video looks like: job accepted, job started, progress milestone, artifact ready, job failed, job cancelled, and review required. Progress milestones should be throttled — emitting on every one percent of progress is a denial-of-service attack on your own consumer.

Designing the Payload: What a Video Job Actually Needs

Payload design is where teams accumulate technical debt fastest, because the first version is always shaped around a single client's convenience rather than around the domain. A good webhook payload is small, self-describing, versioned, and stable across model changes.

Start with a schema_version field. When you eventually need to rename or restructure fields, the version tells consumers which shape to expect without breaking existing integrations. Pair it with an event_type string and a unique event_id that consumers can use for deduplication.

A practical video-job payload includes:

  • Identifiers: event id, job id, workspace or tenant id, idempotency key from the original request.
  • Status: current terminal or non-terminal state, plus a monotonically increasing sequence number for ordering.
  • Job parameters: model name and version, prompt hash (not always the raw prompt), duration, resolution, aspect ratio, seed, style preset.
  • Artifacts: URLs rather than embedded data, with content type, byte size, checksum, and an expiry window.
  • Timing: created, started, completed timestamps in UTC with explicit offsets.
  • Errors: a machine-readable error code, a human-readable message, and a retryability flag.
  • Signature metadata: algorithm, key id, timestamp.

Two rules keep payloads manageable. First, never inline base64 video. It bloats delivery, breaks log tooling, and forces consumers to buffer megabytes before they can verify a signature. Store the artifact in object storage and send a signed URL with a short time-to-live. Second, do not send the full prompt verbatim if it may contain sensitive user input unless the consumer is trusted; a hash plus a retrieval endpoint is often enough.

Batching matters too. If a single project produces twenty clips, sending twenty separate webhooks is fine, but sending one event per progress tick per clip is not. Consider a batch envelope with an array of event objects and a shared signature, which keeps verification cheap while giving consumers a single delivery to process.

Picking Open-Source Webhook Infrastructure

The open-source landscape splits into four layers. You rarely need all four, but knowing which problem each solves prevents you from bolting together a system that is more fragile than a plain background worker.

Delivery gateways and relays

A gateway sits between your producers and your consumers. It handles signing, retries, exponential backoff, delivery logs, and a replay interface. Open-source options worth evaluating include Svix's self-hosted core, Convoy, Hook0, and WebhookX. Their value is not just retry logic — it is the operator console that lets a support engineer answer "did we send that event?" in ten seconds instead of an hour.

If you build your own, the minimum viable feature set is: persistent delivery queue, per-endpoint circuit breaking, exponential backoff with jitter, a dead-letter store, and an authenticated replay endpoint.

Queues and brokers

For the internal hop between your producer and the gateway, you need durable buffering. Redis Streams and BullMQ are excellent when you already run Redis and your throughput is moderate. RabbitMQ gives you mature routing, dead-letter exchanges, and per-consumer acknowledgements. NATS JetStream is a strong middle ground with low operational weight. Apache Kafka is the right choice when you need replayable logs, multiple independent consumer groups, and long retention — but it is a poor fit for a five-person team generating a few thousand clips a day.

Workflow orchestrators

Video pipelines are workflows, not single tasks. Temporal, Cadence, and to a lesser extent Apache Airflow model multi-step processes with durable state, retries per activity, and timeouts that survive a process restart. Temporal in particular handles the "render for twelve minutes then notify three systems" pattern with far less custom code than a home-grown state machine.

Selection criteria

Score candidates on six axes:

  1. Operational burden: does it need a dedicated database, cluster, or on-call rotation?
  2. Delivery semantics: at-least-once is almost always the realistic answer — confirm it, then design for it.
  3. Replay and audit: can you resend a specific event to a specific endpoint without a script?
  4. Signing support: does it implement standard HMAC signatures with timestamp tolerance?
  5. Observability: are delivery metrics exported in a format your monitoring stack consumes?
  6. Exit cost: how hard is it to migrate away in two years?

Reference Architecture: From Prompt to Published Cut

Here is a concrete four-stage pipeline that scales from a hundred to a hundred thousand jobs without changing its bones.

Stage 1 — Intake and validation

An API endpoint accepts the generation request. It validates the schema, checks quota and entitlement, normalizes parameters, computes an idempotency key from the request body plus tenant, and writes a job row with status accepted inside a database transaction. In that same transaction it inserts an outbox record. Committing the job and the outgoing event together is what prevents the classic bug where the job exists but the notification never fires, or the notification fires for a job that rolled back.

Stage 2 — Dispatch and queueing

A publisher process reads the outbox and pushes the event to the queue. A dispatcher worker pulls jobs, checks GPU pool capacity, and acquires a slot. If capacity is full, the job waits — this is backpressure, and it is far better than letting the model server queue hundreds of requests and time out unpredictably. Once a slot is free, the worker submits the render to the inference service and records the external job reference.

Stage 3 — Progress, retries, and partial results

For long renders, the inference service emits progress callbacks. Your system throttles them, updates the job row, and forwards a milestone event to consumers at most every thirty seconds. Failures are classified: transient errors (worker preemption, network blip) are retried with exponential backoff up to a cap; deterministic errors (content policy rejection, malformed prompt) are terminal. Never retry a policy rejection — you will simply generate the same rejection five times and burn compute.

Stage 4 — Delivery and post-processing

When frames are ready, the artifact is uploaded to object storage, a checksum is computed, and a CDN or signed URL is generated. Only then does the artifact.ready event fire. Downstream consumers — transcoders, subtitle generators, thumbnail services, approval UIs — each receive the same event through their own subscription. A final assembly step concatenates clips, applies loudness normalization, and marks the project published, emitting a project.completed event.

The key discipline is that each stage emits its own event rather than one giant "everything is done" notification. Granular events let consumers subscribe only to what they care about, and they make debugging dramatically easier.

Idempotency, Ordering, and State Management

Assume every event will be delivered at least once, and possibly out of order. Two consumer-side patterns make this safe.

First, deduplicate on event_id. Store processed event ids in a table with a unique constraint, or in Redis with a TTL long enough to cover your retry window. When a duplicate arrives, acknowledge it and move on. This one habit prevents double-rendering, double-charging, and duplicate publish actions.

Second, make state transitions monotonic. Give each job a status enum and a rule that rejects backwards transitions — a job that reached completed cannot return to processing. If your database enforces that invariant, an out-of-order event becomes a harmless no-op instead of a corrupted record.

For the producer side, the transactional outbox is the pattern that pays for itself fastest. Instead of publishing to a broker inside a request handler — where the broker call can succeed while the database write fails — you insert the event into an outbox table in the same transaction. A relay process then drains it. This guarantees at-least-once publication and keeps your request latency predictable.

Finally, choose your source of truth deliberately. Many mature teams treat the webhook as a notification and the REST API as the source of truth. When a webhook arrives, the consumer calls GET /jobs/{id} to fetch authoritative state, so the payload can stay thin and evolve without breaking anyone.

Security: Signing, Replay Protection, and Secret Rotation

An unauthenticated webhook endpoint is an open door: anyone who learns the URL can inject fake "render complete" events. The baseline defence is HMAC signing. The producer computes a signature over the raw request body plus a timestamp using a shared secret, and sends it in a header such as X-Signature along with X-Timestamp and a key identifier.

On the consumer side, verify in this order:

  1. Read the raw body bytes before any JSON parsing. Re-serializing the object will change whitespace and break the signature.
  2. Reject requests whose timestamp is outside a tolerance window — five minutes is typical — to blunt replay attacks.
  3. Recompute the HMAC with the shared secret and compare using a constant-time function.
  4. Check the key id against your active secret set, so you can rotate without downtime.
  5. Optionally record the signature in a short-lived cache to reject exact replays within the tolerance window.

Rotation should be boring. Support two active secrets per endpoint — one primary, one previous — and give consumers a window of at least a week to migrate. Publish rotation events through a status page or a dedicated key.rotated webhook so nobody is surprised.

Beyond signing, apply standard hygiene: HTTPS only, no secrets in URLs, IP allowlisting when the consumer has static egress, strict request size limits, and a content-type check. If your consumers are internal services on a private network, mutual TLS is a reasonable additional layer, though it complicates onboarding for external partners.

Observability and Debugging a Failing Pipeline

A webhook system without delivery logs is impossible to operate. The first question during any incident is "did the event leave our system?" — and the answer must be one query away.

Log every delivery attempt with: event id, event type, endpoint id, attempt number, response status, response time, and a truncated response body. Correlate everything with a request id that also appears in application logs. Structured JSON logs make this searchable without heroics.

The metrics that actually predict trouble:

  • Delivery success rate per endpoint, as a rolling five-minute window.
  • p95 and p99 delivery latency, which spike before failures become visible to users.
  • Queue depth and oldest message age, the earliest signal of backpressure.
  • Retry ratio, which creeps up when a consumer endpoint degrades.
  • Dead-letter volume, the number you want at zero and alert on above a handful per hour.

Build a replay tool before you need it. When a consumer has an outage for two hours, you want a button that re-sends all failed events for that endpoint within a time range — not a Python script written under pressure. Replays must preserve the original event_id and signature semantics so consumers deduplicate correctly.

Finally, add a synthetic canary. A scheduled job that emits a harmless ping event to every registered endpoint every fifteen minutes will detect a broken consumer long before a customer does.

Capacity, Backpressure, and Long-Running Jobs

Video generation is a queueing problem with a hard physical constraint: GPUs. Your webhook layer should reflect that reality rather than pretend capacity is infinite.

Implement concurrency caps per tenant and per model. When the caps are reached, accept the job but keep it in queued state, and tell the user honestly how deep the queue is. Priority tiers — interactive, standard, batch — let a product team guarantee fast turnaround for premium workflows without building a separate pipeline.

Set generous timeouts. A render that normally takes ninety seconds may take six minutes under contention. A timeout tuned for the happy path will cause spurious retries and duplicate GPU spend. Prefer long timeouts plus heartbeats: the worker emits a progress heartbeat every thirty seconds, and the orchestrator only declares a job dead after three missed heartbeats.

Make long jobs resumable where the model allows it. If a render is split into segments, a failure in segment nine should not discard segments one through eight. Track segment-level state and emit a job.partial event so consumers can show meaningful progress instead of an indefinite spinner.

Watch your egress costs too. Signed URLs to object storage are cheap; proxying gigabytes of rendered video through your application servers is not. Send URLs, not bytes.

Common Mistakes and How to Avoid Them

These recur across almost every team that builds this layer.

Emitting events inside the request transaction. If the broker publish fails after commit, the event is lost forever. Use the outbox pattern.

Signing the parsed JSON instead of the raw body. Whitespace and key ordering differences break verification in ways that are maddening to debug.

No deduplication. At-least-once delivery is the norm, and consumers that assume exactly-once will double-charge, double-publish, or double-notify.

Unthrottled progress events. Emitting on every percentage point generates hundreds of deliveries per job and trains consumers to ignore notifications.

Retrying deterministic failures. A content policy rejection will fail identically every time. Classify errors before you retry.

No dead-letter queue. Failed deliveries without a terminal resting place vanish silently, and you discover the problem from a customer complaint.

Hardcoding secrets in consumer configs. Rotation then becomes an outage. Support two active secrets from day one.

Treating the webhook payload as the source of truth. Payloads evolve; an API fetch keeps consumers correct across schema changes.

FAQ

How many events should a single video job emit? Typically four to eight: accepted, started, one or two throttled progress milestones, artifact ready, and a terminal completed or failed event. More than that usually means you are emitting internal telemetry rather than integration events.

Is at-least-once delivery good enough? For almost all video workflows, yes, provided consumers deduplicate on event id and enforce monotonic state transitions. True exactly-once delivery across a network boundary is not achievable; exactly-once effects are, and that is what matters.

Should I build my own delivery service or self-host an existing one? If your only requirement is "POST a JSON body and retry on failure," a small worker is fine. The moment you need per-endpoint delivery logs, replay, rotation, and circuit breaking, self-hosting an established open-source gateway is cheaper than building those features yourself.

How long should I retain delivery logs? Thirty days is a practical default — long enough to investigate a customer report filed weeks later, short enough to keep storage predictable. Keep aggregate metrics far longer than raw payloads.

What about consumers that are temporarily offline? Exponential backoff with jitter over twenty-four hours, then dead-letter. Alert the consumer owner on repeated failures rather than on the first one, and expose a self-service replay.

Do I need a workflow orchestrator if I already have a queue? Only when the process has multiple durable steps with distinct retry policies. A queue moves messages; an orchestrator remembers where a long-running multi-step job was when the process crashed. Video pipelines usually reach that point quickly.

How do I test the whole pipeline? Run a contract test suite that replays recorded payloads against each consumer, plus a staging environment with a synthetic tenant that generates short low-resolution clips. Fire a full workflow on every deploy and assert that all expected events arrive in order.

Alexander

Alexander