How to Direct Your First AI Video: A Complete Workflow

Why AI Video Direction Is a Craft, Not a Prompt

Generative video tools have removed the hardest logistical barrier in filmmaking: you no longer need a crew, a location permit, or a camera package to get a moving image onto a screen. What they have not removed is the reason a film works. Audiences do not remember renders. They remember decisions — who was in frame, what the camera chose to hide, when the cut landed.

That shift moves the bottleneck. It used to be "can I shoot this?" Now it is "should this shot exist, and in what order?" A director does three jobs, and all three survive the transition to AI production:

1. Decide what the audience should feel next. This is story structure, and no model does it for you.
2. Translate that intention into concrete visual instructions. Shot size, camera move, subject action, lighting direction, duration.
3. Protect continuity across dozens of separately generated clips. Characters, wardrobe, props, geography, light, and color.

The most important mental adjustment is this: generation is non-deterministic. Every render is a roll of the dice. You cannot rely on luck twice in a row, so direction becomes the deliberate design of constraints — reference images, locked descriptions, reusable setups, and a shot list written to survive randomness. Think of yourself as a storyboard artist, a continuity supervisor, and an editor who happens to hand the actual photography to a machine.

This guide walks through that entire job, end to end, in the order you will actually do it.

Pre-Production: Write the Story Before Touching Any Tool

AI video rewards preparation more than any traditional format, because every unplanned detail becomes a re-render. Before you generate a single clip, produce three documents.

The logline and the emotional target

One sentence: who wants what, what stands in the way, and how it ends. Then one more sentence about the feeling you want in the final ten seconds. Every shot decision gets tested against those two sentences. If a beautiful shot does not serve them, it is a distraction, not a bonus.

Example: A night-shift station cleaner finds a lost child on the last train and must decide whether to break protocol to help. The ending should feel like reluctant warmth, not triumph.

That second sentence already tells you the final shot should be small and quiet — a hand on a shoulder, not a wide hero frame.

The beat sheet

Break the logline into six to ten beats. A beat is a change: information arrives, a decision is made, a relationship shifts. Write each beat as one line. For a 90-second short, eight beats is generous; for a three-minute piece, twelve to fifteen.

The shot list

Only after the beats are locked do you allocate shots. A practical ratio: one to three shots per beat, 12–25 shots for a 90-second piece, average duration 3–6 seconds. Longer shots are harder to generate cleanly; shorter shots hide small inconsistencies and give you editorial flexibility.

For each shot, write five fields: beat number, shot size, subject action, camera behavior, duration. Keep it in a spreadsheet. You will live in it.

A common beginner mistake is writing the shot list in prose. Prose hides gaps. A table exposes them: two consecutive medium shots with no cut in energy, a missing establishing frame, an action beat with no reaction shot.

Storyboard With Stills Before You Generate Motion

Motion generation is the expensive, slow, unpredictable part of the pipeline. Still generation is fast, cheap, and easy to iterate. Use that asymmetry.

Generate a keyframe for every shot first. Not a polished image — a compositional decision. You are answering: where is the subject in frame, what is in the foreground, where is the light coming from, what is the horizon doing?

Then do two passes:

• Pass one, composition. Generate 4–6 variations per shot at a wide aspect ratio if you plan letterboxed delivery. Pick one.
• Pass two, look. Once the composition works, push style: film stock, contrast, palette, texture. Lock the look for the whole scene, not per shot.

Two practical rules from experience:

Rule one: lock the look with 3–5 reference frames per scene. These become your style anchors. Every subsequent generation in that scene references them, which is the single most effective consistency technique available.

Rule two: storyboard the cut, not just the frame. Place your picked keyframes side by side in order. If two adjacent frames feel like they belong to different films, fix it now — before you have generated 200 seconds of unusable motion.

Keep a folder structure that mirrors the shot list: /scene01/shot01/refs, /scene01/shot01/renders, /scene01/shot01/selected. Sounds tedious; saves hours.

Choosing the Right Generation Approach for Each Shot

Not every shot should be made the same way. Directors who get consistent results match the method to the requirement.

Shot requirement	Best approach	Why
Brand-new action, no existing reference	Text-to-video	Full creative freedom, least control
Character must look identical to prior shots	Image-to-video from a locked keyframe	Identity comes from the still, motion from the model
Existing footage needs restyling	Video-to-video	Preserves motion and timing
Slow, deliberate camera move on a still subject	Animated still / parallax	Far more stable than full generation
Dialogue close-up	Image-to-video plus separate audio and lip sync	Keeps face stable while audio drives timing
Establishing landscape, no characters	Text-to-video, long duration	No continuity risk, forgiving subject

Decision criteria, in priority order:

1. Does a human face need to be recognizable? If yes, start from a locked image.
2. Does the shot need motion the model must invent, or motion you can add? Inventing motion is where artifacts live.
3. How long is the shot? Anything past six seconds invites drift. If you need eight, generate two overlapping segments and cut on action.
4. How many times can you afford to retry? Reserve text-to-video for shots where you can tolerate variation.

The strategic takeaway: treat image-to-video as your default and text-to-video as your exception. Beginners invert this and then wonder why their characters change faces every three seconds.

The Consistency Problem: Characters, Wardrobe, and Light

Continuity is the single largest difference between an amateur AI short and one that reads as intentional filmmaking. Fix it with documentation, not hope.

Build a character sheet

For each recurring character, create one canonical reference image plus a written description that never changes. The written description should be specific and stable:

woman, late 30s, close-cropped dark hair, faint scar above left eyebrow, olive work jacket with rolled sleeves, gray undershirt, no jewelry

Write it once, paste it into every prompt, and never improvise synonyms. "Cropped hair" and "short hair" will drift apart after five generations.

Lock wardrobe and props per scene

Wardrobe changes are continuity events. If a jacket comes off in shot 12, it stays off for the rest of the scene unless the script says otherwise. Keep a props list with location per scene; a missing lamp or an extra cup on a table reads as a mistake to viewers even when they cannot name it.

Control light direction

Establish one dominant light source per scene and name its direction in every prompt: "window light from frame left, cool." Inconsistent light direction is the most common invisible error — viewers feel it as a scene that does not cohere.

Maintain a color script

Decide the palette per act. Act one cool blues and greys, act two warmer amber, act three a single saturated accent for the emotional peak. Then check that each generated shot fits the palette before you accept it. Rejecting a shot for color is faster than color-correcting it later.

Keep the same base model within a scene

Switching generation models mid-scene changes rendering texture, motion style, and often facial structure. If you must switch, switch on a scene boundary and accept a slight visual reset.

Upscale last

Do all consistency work at native resolution, then upscale the selected takes as a final pass. Upscaling earlier multiplies fix time.

Directing the Camera: Shot Language That Models Understand

You are writing instructions for a system that has absorbed a lot of cinematography vocabulary. Use the vocabulary explicitly, and put the most important instruction first.

Shot size

Use the standard ladder: extreme wide, wide, full, medium wide, medium, medium close-up, close-up, extreme close-up. Name one per shot. Vague requests produce mushy framing that is neither wide nor close.

Lens and depth

"35mm, deep focus" and "85mm, shallow depth of field, background blurred" produce visibly different images. Choose lens language per scene and keep it consistent, because switching focal length mid-scene without a reason reads as an error.

Movement

Name one movement per shot, not three. Slow push in, slow pull out, lateral tracking left, handheld follow, static tripod, crane up, orbit. Combined movements confuse the model and produce wobble.

Blocking and action beats

Describe action in sequence with clear verbs, and describe only what happens within the shot duration:

Medium close-up, 85mm, shallow depth. Woman lifts the folded coat, pauses, sets it down on the bench, looks off frame right. Window light from frame left. Slow push in. 5 seconds.

Notice what is absent: no backstory, no emotions the model cannot render, no camera moves that fight the action.

Prompt ordering that survives revision

A reliable order: subject → appearance → wardrobe → action → environment → light → camera → lens → duration. Once you standardize the order, you can edit one field at a time when a render fails, which turns troubleshooting into a controlled experiment instead of guesswork.

Negative instructions

Be specific and short: "no text overlays, no extra fingers, no lens flare, no camera shake." Long negative lists dilute each other.

Sound, Voice, and Rhythm: The Half of the Film People Forget

A silent AI short feels like a tech demo. Sound is what converts a sequence of clips into a film, and it is also the cheapest consistency tool you have: ambience glues together visually mismatched shots.

Score the structure before the final cut

Use a temporary music bed while you assemble. Music sets pacing, and pacing tells you which shots are too long. You will cut faster and more decisively with temp music than without.

Layer ambience per scene

One continuous ambience bed per scene — room tone, rain, station hum, wind — covers small visual discontinuities. Do not change ambience mid-scene; that draws attention to the cut.

Dialogue and lip sync

Record or generate dialogue first, then time the visuals to the audio, not the reverse. Generate the performance shot from a locked keyframe, then apply lip sync as a separate step. Keep dialogue shots slightly longer than the line so you have handles in the edit.

Foley and emphasis

Add a small number of deliberate sounds: a door latch, a cup on wood, fabric movement. Two or three well-placed foley hits per scene read as high production value. Full foley coverage on AI footage often fights the visuals.

Mixing basics

Dialogue around -12 to -6 dB, music 6–10 dB below dialogue, ambience 12–18 dB below. Then check the whole film on a phone speaker. If dialogue disappears there, it disappears for most of your audience.

A Worked Example: A 90-Second Short, End to End

Assume the night-station story from earlier. Here is how the workflow actually sequences.

Step 1 — Beats (8). Cleaning routine. Sound of a train. A child appears. Cleaner notices. Child will not speak. Cleaner checks the schedule. Last train leaves. Cleaner chooses to stay. Small moment of warmth.

Step 2 — Shot list (18 shots). Roughly two shots per beat plus three establishing frames. Wide of empty platform. Close on mop and bucket. Medium of cleaner. Insert of departure board. Wide of child at the far end. Reverse medium of cleaner.

Step 3 — Keyframes (18 stills, 4 variations each = 72 images). This takes an evening. You select the 18 that cut together.

Step 4 — Look lock. Pick five frames across the film as color and texture anchors: cool blue for the first act, amber fluorescent for the middle, one warm frame for the final shot.

Step 5 — Motion. Fifteen shots generated as image-to-video from locked keyframes. Three shots — the wide establishing frames and one insert — generated as text-to-video or animated stills, because no character identity is at risk.

Step 6 — Assembly. Cut to temp music. Expect to drop three to five shots here because they repeat information.

Step 7 — Audio. Ambience bed (station hum), two foley moments (mop bucket, door latch), a short line of dialogue, and a music cue that enters at beat six.

Step 8 — Finish. Color match, upscale selected takes, export at delivery resolution and aspect ratio, plus a vertical crop if you need a social version.

The recurring lesson: about 60% of the total time goes to steps 1–4, and that is correct. The generation phase is the fastest part when pre-production is finished.

Common Mistakes and How to Fix Them

• Generating before writing a shot list. Fix: no motion generation until the beat sheet and shot list are approved.
• Text-to-video for character shots. Fix: lock a keyframe, then animate it.
• Changing the descriptive wording mid-scene. Fix: build a prompt template per character and paste it unchanged.
• Shots longer than six seconds. Fix: split into two segments and cut on action.
• Multiple camera moves in one shot. Fix: one movement per shot, and only if it serves the beat.
• Inconsistent light direction. Fix: name the light source and direction in every prompt for the scene.
• No negative instructions. Fix: a short, specific negative list per project.
• Scoring the film last. Fix: temp music during assembly, final score after picture lock.
• Accepting a shot because it looks good. Fix: reject any shot that does not move the beat forward.

Quality Control Checklist and FAQ

Pre-delivery checklist

• Character described identically across every shot in a scene
• Wardrobe and props consistent within scenes, with deliberate changes documented
• One light direction per scene
• Palette per act verified across all shots
• No shot longer than six seconds without a narrative reason
• Cut rhythm alternating between shot sizes
• Dialogue mixed and legible on a phone speaker
• Ambience continuous per scene
• Aspect ratio and export settings matched to the delivery platform
• Final pass done at full resolution for text, titles, and any overlays

FAQ

How many shots do I need for a first project? Roughly 12–18 shots for 90 seconds. Fewer if you hold shots longer and use sound to carry tension.

Do I need a script if I am generating visuals? You need beats and a shot list. Formal screenplay formatting is optional, but improvisation during generation is expensive.

How do I keep a face consistent across many shots? One canonical reference image, one frozen written description, image-to-video for every shot that includes the face, and no model switching within a scene.

What if a shot keeps failing? Change one variable at a time — duration first, then camera movement, then action complexity. If it still fails after three attempts, redesign the shot rather than fight it.

How do I deliver for both widescreen and vertical? Generate at a wide aspect ratio, protect the center of frame during composition, and create the vertical version with a deliberate reframe rather than a blind crop.

How long should a first AI short be? Sixty to ninety seconds. It is long enough to have structure and short enough that continuity stays manageable.

When should I bring in music? Temp music during assembly, final score after picture lock so the edit and the music agree.

The through-line across all of it: you are not prompting for footage, you are directing. The tools change every few months; the decisions — what the audience should feel next, what the camera should show, and how the last shot connects to the one before it — do not.