Why AI Video Projects Fall Apart Between Script and Shot
Most disappointing AI video projects do not fail because the model is weak. They fail at the handoff. A text-to-video engine can produce a gorgeous five-second clip — believable motion blur, convincing skin texture, sunlight that looks like it cost money. What it cannot do is decide that this clip belongs after the wide establishing shot, that the character is holding the letter in her left hand because she held it in her left hand two scenes ago, or that the kitchen is supposed to be the same kitchen we saw ninety seconds earlier.
Those decisions belong to a director. In traditional production, the director is supported by people whose entire job is protecting continuity: the script supervisor, the storyboard artist, the first assistant director, the editor. An AI workflow compresses all of those roles into one operator — you — which is why the work often feels heavier than the tooling promises. Generation is fast. The connective tissue between shots is still manual.
The symptoms are easy to recognize:
- Faces drift between shots, so the same character looks like three different people by the end of a scene.
- Wardrobe, hair, and props change without reason.
- Lighting flips from warm interior to cold exterior inside a single conversation.
- Camera energy is chaotic: a locked-off wide, then a handheld close-up, then a drone shot, with no logic connecting them.
- The cut feels like a collection of clips rather than a scene.
The fix is not a better model. It is a directing layer: a structured way to translate a written screenplay into a visual plan, and then into generation instructions that can be repeated. This guide walks through that layer end to end.
The Three Layers of an AI Directing Workflow
Every AI video project, whether it is a 15-second product spot or a 12-minute narrative short, moves through three layers. Problems almost always trace back to a layer that was skipped or left vague.
The narrative layer
This is the script: logline, tone, characters, beats, scenes, dialogue. It answers what happens and why. The artifact here is a screenplay or a beat sheet. Keep it short and specific. A two-page script with sharp beats outperforms a twenty-page draft full of mood that never resolves.
The visual layer
This is the translation of prose into images: shot list, storyboard frames, lookbook, color palette, lens language, blocking. It answers how it looks and how the camera behaves. The artifact is a shot list plus reference stills.
The generation layer
This is the machine-facing side: prompts, reference images, seeds, model settings, aspect ratio, motion strength, negative constraints. It answers what we actually type and press. The artifact is a reusable prompt block per shot.
The critical rule: changes should flow downward, never upward. If a shot looks wrong, fix the prompt. If the prompt is fine but the shot is wrong for the scene, fix the shot list. If the shot list is technically correct but the scene is boring, fix the script. Rewriting the script because a model keeps rendering the wrong jacket is a sign you are working in the wrong direction.
Turning a Script into a Machine-Readable Shot List
A shot list is not a formality. It is your interface between human intent and machine output. Make it a spreadsheet or a structured document with consistent columns, and fill it in for every shot before you generate anything.
| Field | Purpose | Example |
|---|---|---|
| Scene / Shot ID | Ordering and naming | S02-04 |
| Duration | Target length in seconds | 5s |
| Shot size | Framing | Medium close-up |
| Camera | Movement and angle | Slow push in, eye level |
| Action | What happens on screen | She opens the envelope, reads, looks up |
| Dialogue / VO | Spoken content | VO: "I never sent it." |
| Lighting | Key direction and mood | Warm window light from camera left |
| Continuity | What must stay identical | Grey coat, red scarf, chipped mug |
| References | Assets to attach | char_elena_v3, kitchen_wide_a |
| Status | Pipeline state | Keyframe approved |
The two mistakes people make here are granularity and coverage. On granularity: AI video models behave best in four-to-eight second units, so break action into shot-sized pieces rather than writing a thirty-second camera move. On coverage: generate more angles than you think you need for a scene. A close-up you never use costs you one generation; a scene with only one usable angle costs you the whole scene.
Also decide your render budget per shot up front. A practical default is three to five attempts for hero shots and one to two for connecting shots. Writing that number next to each shot prevents the classic spiral of infinite regenerating on a clip nobody will notice.
Character, Location, and Prop Consistency Across Shots
Consistency is the single hardest problem in AI video, and it is solvable with discipline rather than luck.
Start with a character sheet: three to five canonical reference images of each character from different angles, plus a written description block. That block should be frozen — same adjectives, same order, every time. Models weight early tokens heavily, so a fixed phrase order genuinely improves stability.
Then lock the wardrobe as a separate slot. Do not let the model improvise clothing. Write "grey wool coat, red scarf, dark jeans" as a fixed string and repeat it verbatim in every shot where the character wears it. If a scene changes wardrobe, note the change in the shot list so you don't accidentally revert.
Build a location bible the same way: one establishing image per location, a fixed lighting description, and a fixed palette. If the kitchen is "late afternoon, warm tungsten fill, oak cabinets," that string should appear in every kitchen shot.
Use reference conditioning wherever your tools support it. Image-to-video, character reference, style reference, and seed locking are all continuity tools, not shortcuts. Locking a seed can hold texture and lighting stable across a run of shots; changing the seed is a deliberate creative decision, not a random one.
Finally, treat continuity notes as a living document. When you discover that the mug is chipped in shot three, every later shot featuring the mug inherits that detail.
Writing Prompts as Directing Notes
A prompt is a directing note, not a wish. Write it the way you would brief a cinematographer: subject, action, framing, lens, lighting, mood, and constraints — in that order.
A workable template:
[Canonical character block]. [Action verb]. [Shot size and camera behavior]. [Lens and depth of field]. [Lighting direction and quality]. [Mood and color]. [Style reference]. [Negative constraints].
Example:
Elena, 30s, dark curly hair, grey wool coat, red scarf. She opens the envelope slowly and looks up. Medium close-up, slow push in, eye level. 50mm, shallow depth of field. Warm window light from camera left, soft falloff. Quiet, tense, muted teal and amber. Photoreal, film grain. No text, no extra characters, no camera shake.
Three rules make prompts behave:
- One camera move per shot. "Push in while orbiting and tilting" produces mush.
- Motion verbs matter more than adjectives in video models. "Opens," "turns," "walks away" give the engine something to animate.
- Avoid contradictions. "Soft daylight" and "harsh neon" in the same line produce an average that satisfies neither.
Choosing Tools for AI-Assisted Directing
There is no single best stack. There are two viable philosophies, and the right one depends on how much control you need.
All-in-one suites keep the script, shot list, storyboard, and generation in one place. You trade some peak quality for a coherent pipeline, fewer file transfers, and less context loss. Good for solo creators, fast-turnaround content, and teams that hate tab-switching.
Best-of-breed pipelines combine a script/breakdown tool, an image model for keyframes, a video model for motion, and a traditional editor for assembly. You gain quality and flexibility per stage, and you pay for it in friction: naming conventions, asset libraries, and manual handoffs become your responsibility.
Evaluate any option against these criteria:
- Script-to-shot breakdown: can it structure scenes and shots, not just store text?
- Reference conditioning: does it accept character and style references for consistency?
- Model choice: can you switch video models per shot rather than per project?
- Length and resolution: do the limits match your longest hero shot?
- Assembly: is there a timeline, or do you export everything to an editor?
- Versioning: can you keep and compare multiple takes per shot?
- Cost predictability: do you know what a full project costs before you start?
- Licensing and rights: are you clear on commercial usage for every model in the chain?
If you are starting out, pick the simplest stack that covers your script, your shot list, and one good video model. Add complexity only when a specific shot fails for a specific reason.
Step-by-Step: From Logline to Assembled Cut
- Lock the logline and tone. One sentence, one emotional register. Everything downstream inherits it.
- Write in beats, then in scenes. Keep scenes short: one location, one objective, one turn.
- Break the script into shots. Aim for four to eight seconds each. Note framing and camera behavior.
- Build the character sheet and location bible. Gather or generate reference stills before you touch video.
- Generate keyframes first. Approve stills for every shot before animating. Fixing composition in a still is cheap; fixing it in motion is expensive.
- Write the prompt block per shot. Paste the canonical character and location strings verbatim.
- Generate motion. Three to five attempts for hero shots, one to two for connectors. Save everything.
- Review at low resolution. Check continuity: face, wardrobe, light direction, prop state, screen direction.
- Assemble on a timeline. Cut for rhythm, not for completeness. Kill any shot that does not advance the scene.
- Add sound, then re-cut. A cut that feels slow without sound often feels right with it, and vice versa.
Common Mistakes and How to Fix Them
Over-prompting. Long prompts dilute attention. Strip any clause that does not change the frame.
Mixing models mid-scene. Different models render skin, motion, and color differently. If you must switch, do it at a scene boundary, not mid-conversation.
Regenerating everything when one thing is wrong. If a clip is 90% right and the coat color is wrong, try an edit or a mask before a full regeneration.
Ignoring aspect ratio. Vertical and horizontal framing change how close a close-up feels. Decide the delivery format before you shoot.
No coverage. Every scene should have at least one wide, one medium, and one close-up option.
Neglecting screen direction. If a character walks left to right, keep them moving left to right across the scene's shots, or the geography will read as broken.
Forgetting sound. Silence makes even good AI footage feel synthetic. Room tone, footsteps, and a single music bed do most of the work.
Quality Control and Iteration Loops
Run three passes over every scene. The first pass checks technical continuity: face, wardrobe, props, lighting direction, exposure. The second checks narrative continuity: does the action follow logically, and does the emotional beat land? The third checks rhythm: does the cut breathe, and does it end where it should?
Keep a selects bin and a naming convention that includes scene, shot, and version (S02-04_v3). Never overwrite a take you liked. When a scene fails twice in a row, stop regenerating and change something upstream: simplify the action, shorten the shot, or split it into two.
FAQ
Do I need a storyboard before generating video?
You need approved stills for any shot where composition matters. That is a lightweight storyboard. For B-roll and transitions, a written prompt can be enough.
How long should each AI-generated shot be?
Four to eight seconds is the sweet spot for most current models. Longer shots tend to drift in detail and motion quality.
How do I keep a character's face consistent?
Freeze a written character block, keep three to five reference images, repeat wardrobe strings verbatim, and use reference conditioning or seed locking consistently across the scene.
Should I use one video model for the whole project?
For narrative work, yes, where possible. Consistency of rendering style matters more than a marginal quality gain from switching models per shot.
How do I handle dialogue and lip sync?
Write dialogue into the shot list, keep speaking shots short and frontal, and treat lip sync as a post step. For talking-head-heavy scenes, shoot plates of the performer and process them, rather than generating speech from scratch.
What about sound design and music?
Budget as much time for sound as for generation. Ambience, foley, and a restrained music bed cover more AI artifacts than any color grade.
How do I keep a long project manageable?
Work scene by scene to a finished state — picture, sound, and color — instead of generating the whole film and assembling at the end. Finished scenes give you a template and a reality check for everything that follows.
An AI director's real advantage is not speed at the model level. It is the ability to hold intent steady across dozens of small decisions, so that sixty generated clips resolve into one coherent piece.
