AI Video Story Planning: Turn a Brief Into a Shot List

Why Story Structure Decides Whether an AI Video Is Watchable

Most people working with generative video tools discover the same uncomfortable truth within a few attempts: the technology can produce a beautiful shot, but it cannot decide what the film is about. You can generate a rain-slicked neon alley, a slow push-in on a face, a drone reveal over a coastline — each clip impressive on its own — and still end up with something that feels like a demo reel instead of a story.

The gap between "cool clips" and "a video someone watches to the end" is almost never a rendering problem. It is a planning problem. Shots are cheap now; coherence is expensive. The work that used to happen before anybody touched a camera — interpreting a brief, breaking it into scenes, deciding what each shot must accomplish, targeting the prompt at the right model — is still the work. It has just moved from a storyboard binder into a set of scripts, tables, and prompt templates.

This guide walks through a practical, tool-agnostic approach to turning that planning layer into something repeatable: how to parse a brief into scenes, translate scenes into machine-ready shot specs, hold visual consistency across separate generations, choose the right model per shot, mix modalities, and keep costs and render times under control. It closes with a troubleshooting table and an FAQ for the situations that trip people up most often.

The Planning Layer Behind Every Good AI Video

The phrase "AI director" gets used loosely, but in a working pipeline it means something specific: an orchestration layer that sits between your intent and the generative models. It is not a chatbot bolted onto a video generator. It is the component that decides what to generate, in what order, with which model, and how to check the result.

Its responsibilities break down into four jobs:

1. Interpretation. Read the brief (a paragraph, a script, a client email) and extract the dramatic spine: protagonist, want, obstacle, turn, resolution. Even a 15-second product spot has a spine.
2. Decomposition. Break the spine into scenes, then into shots, then into the atomic describable units a model can render. Each unit needs a subject, an action, a camera behavior, a lighting state, and a duration.
3. Specification. Convert each unit into a structured brief — a shot contract — that a text-to-video or image-to-video model can consume without ambiguity.
4. Arbitration. Decide which model and modality handles each unit, in what sequence, with what fallbacks if the output misses.

The critical insight is that jobs one and two are language work, and jobs three and four are systems work. Teams that skip straight to job three get technically valid prompts for shots that do not belong in the film. Teams that skip job four waste enormous time re-rolling a model that was never suited to the shot in the first place.

From Brief to Beat Sheet: Parsing Intent First

Before writing a single prompt, convert the brief into a beat sheet. This is the highest-leverage step in the entire process, and it takes about twenty minutes for a short piece.

A workable beat sheet has one row per beat and five columns:

• Beat — what changes in the story ("she realizes the letter is unsigned").
• Purpose — why the beat exists ("establish doubt, set up the reversal").
• Emotional temperature — the feeling the audience should have here.
• Duration budget — seconds allocated.
• Visual anchor — the one image that carries the beat.

Here is a before-and-after. A vague brief might read: "Make a moody video about a photographer returning to their hometown." That sentence contains four beats if you look for them:

Beat	Purpose	Temperature	Budget	Visual anchor
Arrival	Reintroduce the place as foreign	Unease	6s	Empty platform at dusk
Recognition	Bridge past and present	Warmth, then ache	8s	Hands turning an old print
Confrontation	Introduce the unresolved thing	Pressure	10s	Two figures across a table
Departure	Resolve without explaining	Release	7s	Gate closing, subject walking away

Notice how much of the shot design is now determined. You know you need a wide establishing shot, a macro insert, a two-shot with tension, and a wide with negative space. You know the duration budget constrains each shot. You know the temperature curve, which dictates lighting and color treatment across the piece. A prompt writer who starts from the original one-line brief has none of this and will generate expensive but arbitrary material.

Separating Narrative Beats from Shot Beats

A common mistake is treating a beat as a shot. They are different granularities. One narrative beat usually needs two to four shots: a wide to place the audience, a medium to attach them to the subject, a close-up to register the internal change, and sometimes an insert for a physical detail that carries meaning. If your beat sheet has one shot per beat, the edit will feel abrupt. If it has eight, you will not finish.

A practical rule for short-form work: one narrative beat equals two to four shots, and one shot equals one generation (plus retries). That math tells you the real scope of the job before you start paying for it in time.

Writing Shot Contracts: The Bridge to Generative Models

A shot contract is the artifact that makes the planning layer useful. It replaces the loose prompt with a structured record that both humans and orchestration code can read.

A shot contract typically includes:

• shot_id — stable identifier so retries and versions stay traceable.
• subject — who or what is on screen, described concretely (age range, wardrobe, distinguishing feature) but not over-specified.
• action — the single physical verb and its direction ("she lifts the lid, then freezes").
• camera — framing, height, movement, and lens character ("eye-level medium, slow dolly in, 35mm feel").
• lighting and palette — time of day, key direction, color temperature, contrast.
• duration — target seconds, plus acceptable range.
• continuity anchors — everything that must match the previous shot: wardrobe, props, screen direction, weather.
• acceptance criteria — what "good enough" means ("hands articulate clearly, no extra fingers, background plates stay constant").
• negative constraints — what must not appear (text artifacts, logos, crowds, reflections of the camera crew).

The value of separating these fields is that failures become diagnosable. When a generation misses, you can ask which field was underspecified rather than re-rolling blindly. Usually the answer is action (too many verbs in one shot) or continuity anchors (the field said nothing about the coat that is now a different color).

Turning Prose into Structured Prompts

A reliable pattern for converting a shot contract into a model prompt is a fixed ordering:

1. Subject and appearance
2. Action and its progression
3. Camera framing and movement
4. Lighting and time of day
5. Environment and background specifics
6. Render style or stock reference
7. Negative constraints

Consistent ordering matters more than elegant phrasing. It gives you a diffable format: when version two of a shot looks worse than version one, you can see exactly which clause changed. It also makes prompt templating feasible — you can store a per-project style clause and concatenate it into every shot, which is the cheapest consistency trick available.

If you are working inside a single interface, this same structure applies to the native prompt boxes on text-to-video and image-to-video surfaces, which accept the same subject-action-camera-language logic.

Directing Scene by Scene: A Worked Example

Abstract advice is easy to nod at and hard to apply. Here is a full pass on a 40-second piece, start to finish, the way a small team would actually execute it.

The brief: a 40-second spot about a ceramicist preparing a kiln at dawn, alone, for a piece that matters to them.

Step 1 — beat sheet. Four beats: waking the studio (8s), preparing the piece (10s), the waiting (12s), the opening (10s). Temperature curve: quiet, focused, anxious, release.

Step 2 — shot list. Twelve shots total, which fits the budget of 40 seconds with an average 3.3 seconds per shot. That is aggressive; consider ten shots at 4 seconds each instead, with one held shot at the end.

Step 3 — contracts. The "waiting" beat is the hardest, because nothing happens. Its contract leans on camera and light rather than action:

• shot_id: 07
• subject: ceramicist in work apron, mid-thirties, sleeves rolled, no jewelry
• action: sits still, exhales once, looks toward the kiln door
• camera: static wide, subject left of frame, kiln doorway right of frame, 50mm feel
• lighting: pre-dawn blue from window, warm spill from kiln seam
• duration: 5s
• continuity anchors: same apron, same clay-dusted forearms, window on left
• acceptance: no camera drift, exhale visible but subtle, no door opening
• negatives: no text, no visible equipment branding, no additional people

Step 4 — generation plan. Shots with complex human action get a strong motion model; static atmospheric plates get a cheaper, faster model; inserts of texture get an image model for the still and a light image-to-video pass. More on that selection logic below.

Step 5 — assembly logic. Because the "waiting" beat is deliberately static, the edit carries the tension. Cut the two opening shots a beat early, hold shot 07 three frames longer than comfortable, then accelerate into the final beat. That rhythm was decided in the beat sheet, not in the timeline — which is why the beat sheet matters.

Keeping Characters, Wardrobe, and World Consistent

Consistency across separate generations is the single most common complaint about multi-shot AI projects. The problem is structural: each generation is independent, so it re-invents everything you did not pin down.

Five techniques, in order of effort-to-impact:

1. Anchor images. Generate one canonical still per character and per location. Approve it, then use it as the seed image for every shot in which that character or place appears. This alone eliminates most drift.

2. Explicit continuity fields. Wardrobe, hair, props, and screen direction should be written once in a project-level continuity sheet and copied verbatim into every applicable contract. Never paraphrase your own description; paraphrasing is how a green jacket becomes an olive one.

3. Locked style clause. A single paragraph of style language — lens character, film stock feel, contrast, grain, palette — appended unchanged to every prompt. Change it once and you change the whole film in a controlled way.

4. Reference-consistent techniques. Image-to-image and reference-driven video strengthen the pull toward your anchors. When a new generation drifts, lower the change strength instead of rewriting the prompt, which is usually the real fix.

5. Named asset inventory. Keep a table of approved stills keyed by character_id and location_id. Orchestration code can then attach the right references automatically, and you stop relying on memory.

When to Accept Drift

Perfectionism here is a trap. Deliberate shot-level variation — a different angle, a different distance, a changing time of day — is not drift; it is coverage. Drift is unwanted change in identity, wardrobe, props, or geography. Audit for the second, and stop re-rolling the first. A scene that looks identical in every shot reads as static, not consistent.

Choosing the Right Model for Each Shot

Model selection is a decision problem with roughly four inputs: what the shot needs, what the models are good at, how long the render will take, and how many attempts you can afford. Build a simple routing table and follow it rather than improvising per shot.

Routing works best when it is written down before the session starts. Make the table once, keep it in the project folder, and update it as you learn which approaches fail. A living routing table turns model selection from a mood into a decision.

Shot type	Preferred approach	Why
Hero human performance	Motion-focused video model, driven from an approved still	Best articulation and continuity from a reference
Static atmospheric plate	Fast text-to-video model	Cheap, quick, motion fidelity is not the priority
Texture or product insert	Image model to still, then a short image-to-video pass	Still quality dominates; motion is minimal
Complex camera move	Video model with explicit camera language	Native camera control reduces re-rolls
Text-heavy or graphic frame	Composite in an editor, not a generator	Text rendering remains unreliable

Two operational habits make routing work. First, prototype the hero shot first, not last: if the pivotal shot cannot be achieved, the whole plan changes. Second, cap retries per shot — three attempts is a reasonable ceiling. If a shot misses three times, the contract is the problem, not the model. Rewrite the contract, simplify the action, or split the shot into two.

Using the Still as a Contract Enforcer

Rendering the still first is the cheapest form of quality control in the whole pipeline. A still exposes bad framing, wrong wardrobe, weak lighting, and nonsensical background detail in seconds and at minimal cost. Fixing those in a still is trivial; fixing them in a five-second render is not. Treat the still as the approval gate, and only animate approved frames.

Mixing Modalities Without Losing the Thread

Modern pipelines mix text, image, and video generation in one project, and the mixture is usually what makes an output look finished. Text models handle ideation, beat sheets, contract drafting, and alternate lines. Image models handle stills, key art, texture, and the reference frames that anchor video work. Video models handle motion, performance, and camera behavior.

The rule that keeps this from becoming chaos: material flows in one direction through the pipeline. Text produces contracts, contracts produce stills, stills produce shots, shots produce an edit. When you find yourself generating a still to reverse-engineer copy, or prompting video to discover what a scene is about, you are solving a problem in the most expensive place possible.

A Useful Division of Labor

• Text layer: beat sheet, shot list, contracts, negative constraint lists, alt copy.
• Image layer: character anchors, location plates, inserts, color tests, thumbnails.
• Video layer: moving shots only, always driven by an approved reference.
• Editor: assembly, timing, sound, transitions, titles.

Sound deserves a note because it is consistently underplanned. Decide at the beat sheet stage whether each beat is silent, scored, or diegetic. Silence in a tense beat is a choice you can only make if the timing was designed for it.

Tuning the Pipeline: Speed, Cost, and Quality

Three levers control throughput, and they trade against each other. Making the trade explicit stops it from being accidental.

Resolution and duration. Doubling duration usually costs far more than double, because the model must hold coherence for longer. Prefer more short shots over fewer long ones, and reserve long takes for moments that need unbroken performance.

Iteration depth. Prototype at low fidelity, approve composition, then finalize. Groups that skip low-fidelity passes tend to approve composition late, when it is expensive to change.

Batch timing. A render queue submitted all at once will hide its own failures until everything finishes. Stagger submissions so early results inform later contracts.

A simple monitoring habit: log per-shot retry counts. If one shot type routinely needs four attempts, the contract template for that type is wrong. Fix the template and every future project gets faster.

What "Good Enough" Looks Like

Define acceptance criteria before generating, not after. A practical checklist for approval: identity matches anchor, wardrobe matches continuity sheet, action reads in a single viewing, background matches the location plate, no visible artifacts in the visual center, and duration matches the edit rhythm. Six binary checks beat a subjective "does it feel right," because they can be delegated and repeated.

Troubleshooting the Common Failure Modes

Symptom	Likely cause	Fix
Shots look great but the piece drags	No beat-level rhythm or duration budget	Rebuild the beat sheet with timing allocations
Character changes appearance between shots	No anchor image, paraphrased wardrobe description	Approve one anchor still; copy continuity text verbatim
Motion is mushy or rubbery	Action field contains several verbs	Split into separate shots, one clear action each
Faces distort mid-shot	Model asked to hold identity through fast movement	Shorten the shot, reduce movement, drive from a stronger reference
Camera does nothing	Camera clause missing or vague	Specify framing, height, movement, and lens feel explicitly
Style feels incoherent across the film	Style clause varied between prompts	Lock one style paragraph and reuse it unchanged
Text or signage renders as nonsense	Generation used for typography	Composite text in the editor instead

Frequently Asked Questions

How much planning is enough before generating? For anything over 30 seconds, a beat sheet plus shot contracts pays for itself in avoided re-rolls. Under 15 seconds, a short shot list and a locked style clause is usually sufficient. The threshold is not length, though — it is whether the piece has a narrative turn. If it turns, plan it.

Should I write the script before or after choosing models? Before. Story decisions should not be constrained by what a particular model happens to do well this month. Choose the story, then route shots to the best available approach, then simplify only if a specific shot proves impossible.

How do I keep a series looking like one series? Freeze the style clause, the character anchors, and a shared continuity sheet across episodes. Consistency in a series comes from shared assets, not from re-describing the same look each time.

Do I need a different tool for every modality? No, but almost every serious pipeline ends up using at least one image model and one video model, because their strengths differ. What matters is that a single planning layer owns the contracts and asset inventory, so tools stay interchangeable.

What is the fastest way to raise perceived quality? Sound and shot length. Tightening the edit and adding intentional sound design improves perceived production value more than upgrading render resolution, and both are free.

When should I stop retrying a shot? After three attempts with the same contract. Change the contract, split the shot, or replace it with a static frame or an insert. Persistence with an unchanged contract almost never converges.

Where Storytelling Skill Still Wins

The tools will keep improving. Motion fidelity will rise, identity consistency will tighten, and prompt surfaces will get simpler. None of that removes the planning layer, because the planning layer is where intent lives, and intent is not something a generator can infer from a paragraph.

What separates watchable AI video from forgettable AI video is unglamorous: a beat sheet with honest timing, shot contracts with verbatim continuity, approved anchor stills, a routing table for model selection, and a retry ceiling that forces you to fix the contract instead of gambling. Teams that build that scaffolding can absorb new models the day they ship. Teams that do not will keep generating beautiful shots that add up to nothing.

Start with the next piece you make: write four beats, allocate seconds, write one contract per shot, and approve a still before you animate anything. The improvement in the final cut will be obvious, and the process only gets faster from there.