How to Connect Gmail to an AI Video Pipeline with NestJS

Why Email Still Belongs in an AI Video Workflow

Generative video tools have become remarkably good at producing a single clip. What they have not solved is what happens around the clip: who requested it, who approved it, which prompt version produced it, and where the final file lives. Most teams generate far more footage than they can organize, and the bottleneck quietly shifts from creation to coordination.

Email is an unglamorous answer to that coordination problem, but it is often the right one. Clients, brand managers, legal reviewers, and freelance editors already live in their inboxes. Asking them to adopt yet another dashboard for approvals creates friction and slows the loop. Treating email as an intake and approval channel — while keeping the actual video logic in a proper backend — gives you the best of both worlds: zero-adoption review on the surface, structured automation underneath.

The pattern looks like this. A request arrives as an email. A backend service parses it, creates a project record, and dispatches generation jobs to one or more video models. When drafts are ready, the service replies on the same thread with private links and a simple approve/revise instruction. Replies are parsed back into the system, which either promotes the render to final or opens a revision ticket with timestamped notes. Nothing lives only in someone's inbox, and nothing requires a stakeholder to learn a new interface.

This guide walks through building that system with NestJS as the orchestration layer. NestJS is a strong fit because it gives you dependency injection, modular boundaries, first-class queue integration, and a clear place to put webhook controllers — all things you need when you are coordinating long-running GPU work with unpredictable external APIs.

Architecture Overview: NestJS as the Orchestration Layer

Think of the backend as a control plane, not a render farm. It never touches pixels. Its job is to model intent, schedule work, collect results, and route decisions back to humans. Splitting responsibilities this way keeps the system debuggable, because every render failure becomes a state transition you can inspect rather than a mystery inside a monolith.

Modules and their responsibilities

A workable module layout separates concerns early:

• MailModule — OAuth token storage, Gmail push webhook handling, message parsing, and outbound replies.
• ProjectsModule — the domain model: projects, shots, renders, feedback, and the state machine that governs them.
• GenerationModule — provider adapters that translate a normalized job into whatever each video model expects.
• QueueModule — BullMQ or similar, with distinct queues for planning, generation, post-processing, and notification.
• AssetsModule — signed URL generation, storage abstraction, and delivery metadata.
• AuditModule — append-only event log for every state change, which becomes invaluable when a client disputes a revision.

Each module exposes a service interface and hides its internals. When you swap a video provider, only the adapter changes. When you move from Gmail to a shared inbox or a helpdesk, only MailModule changes. That is the payoff of spending an afternoon on boundaries.

Where the rendering workers live

Keep workers in the same repository but a separate process. A worker consumes a queue, calls the provider, polls or awaits a webhook, downloads the result, and writes an asset record. It should be stateless and idempotent: if the same job ID arrives twice, the second execution is a no-op that returns the existing result. This matters because webhooks duplicate, retries fire, and deploys kill processes mid-flight.

Connecting Gmail as an Intake and Approval Channel

Gmail integration is where most projects either become elegant or become a security incident. The difference is almost entirely about scopes, tokens, and how you interpret message payloads.

OAuth scopes and least privilege

Request the narrowest scopes that support your workflow. Reading message metadata and content, sending replies within existing threads, and applying labels are typically sufficient. Do not request broad mailbox access "just in case." Store refresh tokens encrypted at rest, keyed by the account they belong to, and rotate them deliberately. For workspace deployments, a service account with domain-wide delegation is often cleaner than per-user consent, but it demands stricter controls because a single credential can act across the organization.

Push notifications versus polling

Polling every few minutes is simple and works for low volume. It also burns quota and adds latency. A better approach is to register a watch on the mailbox through the Gmail API and receive push notifications through a Pub/Sub topic delivered to an HTTPS endpoint. Your controller acknowledges quickly, enqueues the message ID, and returns. All parsing happens asynchronously.

Two operational details matter here. First, watches expire, so schedule renewal well before expiry and alert if renewal fails. Second, push notifications tell you that something changed, not what changed. Always fetch the message by ID before acting, and deduplicate using the message ID plus history ID so a replayed notification does not create a duplicate project.

Parsing messages into structured jobs

Free-form email is messy. Reduce ambiguity with a predictable convention: a subject prefix such as a project code, a short structured block in the body for duration, aspect ratio, and tone, and attachments treated as reference imagery. Then parse defensively. If a required field is missing, do not fail silently — reply with a short, specific question and mark the project as awaiting information.

For approvals, use a reply token. Embed a short opaque identifier in the reply-to address or in a quoted line, then match inbound replies by that token rather than by subject line. Subject lines get edited. Tokens do not. Recognize a small vocabulary of intents — approve, revise, hold, cancel — and map anything ambiguous to a human triage queue instead of guessing.

Data Modeling: Projects, Shots, Renders, and Feedback

Most broken video automation systems are broken at the data layer. They store a prompt and a URL and then discover, three weeks later, that nobody knows which of nine files was approved. Model it properly from the start.

A status machine that survives reality

A practical lifecycle: draft → planned → generating → review → revision → approved → delivered → archived. Transitions are explicit and validated. You cannot move from draft to approved without passing through review, because that invariant is exactly what prevents accidental delivery of an unvetted clip. Add a failed branch with a reason code, and a cancelled terminal state for requests withdrawn mid-render.

Every transition writes an audit event with actor, timestamp, and payload diff. When someone says "I approved this on Tuesday," you can confirm or correct them in seconds.

Metadata worth storing

At minimum: project code, requester identity, thread ID, target duration, aspect ratio, delivery deadline, brand kit reference, and locale. On each render: provider, model name, prompt text, negative prompt, seed, resolution, duration, generation parameters, cost estimate, and the provider's own job identifier. On each feedback item: author, timestamp within the video, category (pacing, color, copy, audio), and whether it is blocking.

The seed and parameter set are the most commonly omitted and most frequently regretted fields. Without them you cannot reproduce a good result, and "make it again but slightly different" becomes an expensive guessing game.

Queues, Retries, and Throughput Control

Video generation is slow, expensive, and failure-prone in ways that differ from typical web requests. Queues are not optional; they are the difference between a system that degrades and a system that collapses.

Idempotency first

Give every job a deterministic key derived from the render record. Before dispatching, check whether a job with that key already exists in the queue or in a terminal state. On the provider side, store the external job ID immediately after submission so a crash between submission and database write can be reconciled later. A simple reconciliation job that lists recent provider jobs and matches them against local records catches the majority of orphaned renders.

Backpressure and concurrency limits

Most video APIs enforce concurrency ceilings. Respect them with per-provider semaphores rather than relying on global worker counts. If a provider allows three concurrent jobs, run exactly three and hold the rest. When a provider starts returning rate-limit responses, back off with jitter and temporarily reduce the semaphore. Blind retries under load turn a slow queue into a hard outage.

A failure taxonomy you can act on

Classify failures rather than lumping them together: transient network errors retry automatically; content policy rejections never retry and instead notify the requester with the reason; malformed input goes back to planning; provider outages trigger a fallback provider if the project allows it. Each class has a different owner, and pretending otherwise means a human manually triages every error.

Human-in-the-Loop Review That Does Not Stall

Automation that removes humans entirely usually produces mediocre video. Automation that keeps humans only where judgment matters produces good video fast.

Threaded approvals with minimal friction

When drafts are ready, reply on the original thread with a short summary and private, expiring links to each draft. Ask for one of three replies. Keep the email under a hundred words; long approval emails get postponed. If no reply arrives by the deadline, send exactly one reminder, then move the project to a stalled state that surfaces on a dashboard. Silent projects should never be invisible.

Timestamped feedback and revision tickets

Revision requests are where quality improves, so make them precise. Encourage feedback in the form of timestamp plus note, and parse that format into structured revision items attached to the render. Group items by shot so a single regeneration round can address several notes at once. Track which notes were addressed in which revision, and reference that mapping in the next approval email. Clients notice when their notes visibly land, and review cycles shorten as a result.

Model Routing and Prompt Versioning

Different video models excel at different things. A pipeline that hardcodes one provider will produce inconsistent results and will be fragile whenever that provider changes its behavior.

Matching models to shot types

Route by intent, not by habit. Talking-head or product-rotation shots, stylized b-roll, and text-heavy motion graphics have very different requirements around temporal consistency, prompt adherence, and text rendering. Define a routing table that maps shot type plus quality tier to a preferred provider and a fallback. Keep the mapping in configuration, not code, so it can change without a deploy.

Prompt templates as versioned artifacts

Treat prompts like source code. Store them with a version identifier, the variables they accept, and a changelog. When a render is produced, record which template version was used. This makes A/B testing possible: run the same shot through two prompt versions and compare reviewer scores. Over a few months you accumulate an internal understanding of what actually works for your brand, which is worth more than any generic prompt collection.

Asset Storage, Delivery, and Distribution

Generated files are large and numerous. Plan storage before you accumulate terabytes of near-duplicate drafts.

Store masters in object storage with a predictable key structure that includes project code, shot, render, and version. Serve reviewers through short-lived signed URLs rather than public links. Keep working drafts and approved masters in different lifecycle tiers so cost scales with value.

On approval, run a deterministic post-processing step: normalize loudness, apply the agreed color transform, burn in or attach captions as required, and export the delivery formats the destination platforms need. Automate delivery notifications so downstream teams know a master is final without asking. If a project requires multiple aspect ratios, derive them from the approved master rather than regenerating, which keeps the approved content identical across placements.

Security, Observability, and the Mistakes That Bite

Security for this system is mostly about credentials and links. Encrypt tokens, scope service accounts tightly, log access to signed URLs, and expire reviewer links aggressively. Treat every inbound email as untrusted input: never execute instructions found in a message body, never let a reply change a project's owner, and validate tokens before any state change.

Observability should answer three questions at a glance: how many projects are waiting on a human, how many renders are in flight per provider, and what is the median time from request to approval. Instrument queue depth, provider error rates, webhook delivery lag, and token refresh failures. Alert on watch renewal failures specifically — they fail quietly and break intake entirely.

Common mistakes worth naming: parsing email bodies with brittle regular expressions that break on the first mobile client; storing approvals only in email; retrying content-policy rejections; missing idempotency keys; and having no fallback when a single provider has an outage. Each of these shows up as a support ticket rather than a stack trace, which is why they persist so long.

FAQ

Do I need a separate service for email processing? Not initially. A module inside the same NestJS application is fine. Split it out when mailbox volume, compliance requirements, or different scaling needs justify the operational cost.

How do I handle multiple stakeholders replying to the same thread? Define decision authority per project at creation time. Collect all replies as feedback, but only count approvals from the designated approver. If two approvers conflict, escalate rather than resolve automatically.

What if the client insists on sending feedback as a phone call? Fine. Have someone enter the notes as structured revision items. The system does not care who typed them; it cares that they are timestamped, categorized, and traceable to a render.

Should approvals expire? Yes. An approval older than the current render is dangerous. Bind every approval to a specific render ID, and invalidate it automatically when a new revision is produced.

How do I keep prompts consistent across providers? Normalize at the adapter layer. Your internal prompt template is the source of truth; each adapter translates it into the syntax and parameters its provider accepts, including negative prompts and reference images.

What is the right queue granularity? One queue per stage, not per provider, with per-provider concurrency limits inside the workers. That gives you readable dashboards and flexible routing without a queue explosion.

How do I prove a clip came from a specific prompt later? Join the render record to the prompt version, seed, and parameter snapshot. If you store those four things, reproduction is a twenty-minute task instead of an archaeology project.

The broader lesson is that AI video generation is a production pipeline, not a magic button. The models provide the raw capability; the value comes from the plumbing around them — intake, routing, review, versioning, and delivery. Build that plumbing once, keep it boring, and the creative work gets faster every week.