Directing AI Short Videos: Scene, Lighting, and Pacing

Why AI Video Direction Is a Different Skill From Prompting

Two creators open the same text-to-video tool, choose the same vertical format, aim for the same thirty-second runtime, and end up with wildly different results. One produces a short that holds attention to the final frame. The other produces eight pretty clips stitched together with hard cuts and a music bed that starts and stops at random.

The gap is almost never the model. The gap is direction: the deliberate work of deciding what each shot is for, how one shot relates to the next, and how light, motion, and sound accumulate into a feeling across time. A prompt produces a clip. Direction produces a sequence.

Think of a generative video model as an extremely fast, extremely literal camera crew. It will do exactly what you describe, and it has no memory of your intentions. If you do not specify eyeline, it invents one. If you do not specify screen direction, it flips it between shots. If you do not state that the character is carrying a red umbrella in shot one, the umbrella disappears in shot two, or turns blue, or grows a second handle.

That is why a director mindset matters more than prompt cleverness the moment you move past single clips. The work breaks into five layers:

1. Intent — what the short is about in one sentence, and what the viewer should feel by the end.
2. Shot list — the minimum set of shots that delivers that feeling, in order.
3. Composition and continuity — framing, blocking, props, wardrobe, screen direction.
4. Light and motion — the mood and rhythm that turn stills into a scene.
5. Edit and sound — the pacing decisions that account for most of the perceived quality.

Everything below is a practical way to run those five layers without a studio, a crew, or a budget line for either.

The Pre-Shot Phase: From Idea to Shot List

Define the promise in one line

Before any tool opens, write one sentence. For example: A night-shift nurse walks home through a city that slowly turns into a garden. That sentence is your filter. Every shot either serves it or gets cut. Shorts fail most often not because individual shots are bad, but because the shots are unrelated to each other and to the promise.

Then sketch the emotional curve: what the viewer should feel at second zero, at the midpoint, and on the last frame. For a thirty-second short, three beats is plenty. Five is the maximum before the piece starts feeling like a trailer for something that does not exist.

Build a shot list that fits your runtime

A useful rule of thumb: a thirty-second short with narration can carry six to ten shots; a thirty-second short driven purely by music and visuals usually wants ten to sixteen, because each shot carries less narrative weight. Average shot length is your pacing dial. Two to three seconds per shot feels energetic and social-first. Four to six seconds feels cinematic and slower.

Your shot list should specify, per row:

• Shot number and rough duration
• Subject and action (one verb per shot, no exceptions)
• Shot size (wide, medium, close-up, insert)
• Camera motion, or explicitly locked off
• Light and time of day
• Continuity notes: wardrobe, props, screen direction

That last column is what separates a professional-looking sequence from a collage. Continuity notes take two minutes to write and hours to fix retroactively.

Collect references before you generate anything

Pull six to twelve stills that represent your intended look: color, contrast, lens character, composition. Keep them in one folder you can flip through. When you write prompts for keyframes, describe your reference rather than reaching for a vague style word. Cinematic means nothing to a model. Overcast daylight, soft shadow edges, muted teal and rust palette, forty-millimeter perspective, shallow depth of field means something you can reproduce.

Scene Composition: Blocking, Framing, and Continuity

Blocking in generated scenes

Generative models handle precise spatial instructions poorly, so reduce the amount of spatial information per shot. Give each shot one clear arrangement: subject left with negative space right; subject centered with a corridor receding behind them; two figures facing each other across a table. If you genuinely need a complex arrangement, generate a keyframe image first and animate it. Image-to-video holds composition far better than text-to-video.

Place characters consistently on one side of the frame. The classic rule: if a character moves left to right in shot A, they should keep moving left to right in shot B unless a cut deliberately reverses it. Breaking this is the single most common reason AI sequences feel disorienting even when every individual clip looks fine.

Framing for vertical video

Vertical 9:16 changes composition rules in ways that trip up creators who learned on widescreen:

• Keep faces in the upper-middle third. Platform interfaces cover the bottom, and often the right edge.
• Avoid wide establishing shots with tiny subjects. In 9:16 they read as empty. Establish instead with a medium shot that includes an environmental cue — a sign, a window, weather, a reflection.
• If you generate in 16:9 and crop, do not crop to the center by default. Crop to the subject's eyeline and check how the headroom feels on a phone.

If you plan to reuse the same footage in other formats, generate or matte a wider framing and keep safe areas generous at the top and bottom.

Continuity for AI footage

Maintain a continuity bible: a short text file containing the character description, wardrobe, hair, key props, location descriptors, time of day, and palette. Paste the relevant lines into every prompt in that scene. Consistency comes from repeating the same descriptors verbatim, not from inventing fresh phrasing that might drift a little each time.

Decide screen direction once. For example: the protagonist always moves toward camera-right. Write it in the bible, then check every shot against it before you approve a take.

Lighting and Mood: Getting Predictable Results

Describe light as a setup, not a vibe

Models respond well to the vocabulary of physical lighting. Instead of moody lighting, specify:

• Source: window light, practical lamp, neon signage, overcast sky, firelight, screen glow
• Direction: front, side, back, top, under
• Quality: hard or soft
• Contrast ratio: high contrast, or flat and even
• Color temperature: warm tungsten, cool daylight, or deliberately mixed

A prompt fragment like lit by a single cool window on frame left, soft shadow falloff on the right side of the face, dark background, slight atmospheric haze is repeatable. Dramatic mood is a coin flip.

Lock a palette per scene

Choose two or three dominant hues and one accent. Then enforce them by naming them in every prompt: palette of dusty ochre, deep green, and warm cream. Consistent palettes make separate clips feel like they belong to one film, and they let you grade in the edit without fighting the source footage.

Use light to mark time and progress

Light is your cheapest narrative device. If a short spans an evening, let the light change across shots: golden hour, then blue hour, then artificial interior light. Viewers read that progression instantly, and it makes a thirty-second piece feel like a story rather than a montage.

Camera Motion and Pacing: Making Movement Mean Something

Choose motion with intent

Each camera move carries a psychological effect:

• Slow push in — rising tension, intimacy, realization
• Pull out — isolation, reveal of context, resolution
• Lateral tracking — journey, momentum, following a subject
• Handheld drift — immediacy and documentary realism
• Locked off — formality, stillness, observation

Do not stack motions. Slow push with slight handheld and a crane rise produces mush. One motion per shot.

Match motion energy across cuts

Two adjacent shots flow when their motion energy is compatible. A fast lateral track cutting into a locked-off medium shot feels like a gear change — sometimes that is exactly what you want, and often it is jarring. Smooth sequences typically alternate: a moving shot, then a beat of stillness, then movement again.

Let sound set the rhythm

Cut on musical beats only where you want emphasis. Cutting on every beat for thirty seconds flattens all dynamics and makes the piece feel mechanical. A structure that works across many genres:

• 0–3s: hook, one strong image, cut on the downbeat
• 3–12s: setup, medium shot lengths, steady rhythm
• 12–22s: development, shot lengths shorten as tension builds
• 22–27s: peak, either the longest or the fastest shots depending on the payoff
• 27–30s: resolution, hold the final frame or a slow pull out

Because generated shots rarely carry usable sync sound, build the audio bed first and cut visuals to it. This is consistently faster than the reverse, and it prevents the common trap of building a visual sequence that no track can support.

A Repeatable Shot-to-Sequence Workflow

Step 1: Write the beat sheet

Three to five beats, one line each, under a hundred words total. This is not a script. It is a map.

Step 2: Build the continuity bible and shot list

One page. Character, wardrobe, props, locations, palette, screen direction, and the per-shot rows. Keep it open on a second monitor while you generate.

Step 3: Generate keyframes as stills

Use an image generator for the first frame of each shot. Iterate on stills — they are fast and inexpensive relative to video. Approve composition, light, and wardrobe here, before you spend time on motion.

Step 4: Animate with image-to-video

Feed the approved keyframe plus a short motion prompt: subject action, camera motion, speed, and any continuity reminder. Keep motion prompts to twenty to forty words. Longer prompts dilute the parts that matter.

Step 5: Generate alternates and pick ruthlessly

Generate two to four takes per shot. Score each on four criteria: composition match, motion quality, artifact level, and continuity. Keep the best take and do not sink time into rescuing a broken one — regenerate instead. Repair time is the hidden cost that turns a two-hour project into an eight-hour one.

Step 6: Assemble a rough cut with audio

Drop all selects onto a timeline at approximate durations with the audio bed underneath. Watch it without fixing anything. If the story does not read at this stage, no amount of polish will save it. Go back to the shot list and cut shots, not just seconds.

Step 7: Repair, then polish

Repair pass: remove flicker, stabilize drift, fix hand and face artifacts, hide seams with short transitions or by cutting on motion. Polish pass: color grade to unify shots, add grain or halation for cohesion, mix audio with a duck under narration, and add subtitles.

Choosing Tools for Each Stage

Different stages reward different tools. A practical split:

Stage	What to optimize for	Practical choice
Ideation and beat sheet	Speed, no cost of trying	Any text assistant or a notebook
Keyframe stills	Composition control, style consistency	Image generators with reference or character features
Shot animation	Motion realism, prompt adherence	Image-to-video models with strong camera control
Character consistency	Identity retention across shots	Reference-image conditioning or trained character tokens
Voice and narration	Natural prosody, easy re-recording	Dedicated text-to-speech with voice options
Assembly and grade	Timeline control, reliable exports	A standard editor; AI features are a bonus
Upscaling and cleanup	Artifact removal, detail retention	Video upscalers and frame interpolation

Decision criteria worth weighing before you commit to a stack:

• Control versus speed. Models with more parameters and more setup steps give better adherence but slow you down on simple shots.
• Consistency features. If your short has a recurring character, identity conditioning is more valuable than raw resolution.
• Clip length limits. Longer native clips reduce stitching, but a stitched sequence of animated keyframes often looks more intentional than one long wandering generation.
• Cost predictability. Pick tools where a normal working session has a predictable footprint, and prototype at low resolution before committing to full-quality renders.
• Export flexibility. You want clean, high-bitrate exports at your target aspect ratio, without watermarking surprises or forced re-encoding.

A reasonable approach is to standardize on one image generator, one image-to-video model, one voice tool, and one editor. Tool-hopping mid-project is the fastest way to lose continuity and time.

Common Mistakes and How to Fix Them

Too many ideas per shot. Fix: one verb, one subject, one motion. If a shot needs a comma, split it.

Prompt drift between shots. Fix: paste the same descriptor block verbatim, and generate shots for a scene in one session so settings and phrasing stay aligned.

Every shot the same size. Fix: alternate wide, medium, and close. Insert shots of hands, objects, and environment are cheap and effective transitions.

Fighting the model on hands and text. Fix: avoid close-ups of hands doing fine work, and avoid on-screen text entirely. Add text in the edit where you control it.

No clear screen direction. Fix: define it in the bible, then regenerate violating shots rather than mirror-flipping them, which often breaks asymmetrical details.

Cutting on every beat. Fix: hold at least one shot through two full beats.

Silent-first editing. Fix: lock the audio bed before placing visuals.

Over-grading. Fix: match each shot to your reference still rather than to the shot next to it. Use scopes or a consistent LUT instead of eyeballing each clip.

Ignoring the first second. Fix: open on your strongest image, not on a logo, a title card, or a slow establishing shot.

Skipping subtitles. Fix: burn in or add captions. A large share of viewers watch muted first, and captions also improve retention on rewatches.

Quality Control: Reviewing Footage Like an Editor

Run a four-pass review before you export anything:

1. Story pass — watch once at normal speed with sound on. Does the sequence communicate the premise without explanation?
2. Continuity pass — watch with the bible open. Wardrobe, props, screen direction, time of day, palette.
3. Technical pass — scrub frame by frame. Look for flicker, warping backgrounds, extra fingers, melting faces, and text artifacts.
4. Perceptual pass — watch muted, then watch on a phone at arm's length. If it works muted and small, it works.

Keep a running defect log per shot instead of fixing problems the moment you notice them. Batch repairs. Context switching between generation, editing, and grading is the biggest hidden time sink in AI production.

A Short End-to-End Example

Premise: A courier delivers a letter to a lighthouse at dawn. Twenty-eight seconds, no dialogue, one music bed.

Beat sheet: (1) the courier runs through wet streets in the dark; (2) climbs the cliff path as the sky lightens; (3) hands the letter over at the top; (4) the lighthouse lamp switches off as day breaks.

Shot list: eight shots. Two wides (the cliff path, the lighthouse exterior), three mediums (running, climbing, the handover), two close-ups (the envelope in hand, the lamp), and one insert (footsteps in a puddle). Screen direction: left to right throughout.

Continuity bible: red waterproof jacket, black satchel, short dark hair, wet asphalt, palette of deep blue, warm amber, and grey-green. Light progresses from street lamp glow to blue dawn to amber sunrise.

Audio first: a twenty-eight-second ambient track with a soft build around eighteen seconds. Cut visuals to it. Grade with cool shadows and amber highlights, light grain. Add subtitles for the single line of narration at the end.

Realistic production time for a creator on their second or third project like this: two to four hours, most of it spent iterating on keyframes and repairing small artifacts. The first attempt will take longer, and the shot list is what keeps it from taking all night.

FAQ

Do I need generative video at all, or can I animate stills?
For slow, moody shorts, animated stills with subtle parallax, drift, and particle overlays often look more controlled than full generative motion. Use generative video where you need believable subject movement; use stills and motion graphics where you need precision.

How many takes per shot is reasonable?
Two to four. If a concept needs eight takes, the prompt is doing too much. Simplify the shot before you generate more.

What if my character's face changes between shots?
Anchor identity with reference-image conditioning, keep the descriptor block identical word for word, and favor medium shots over extreme close-ups, which expose small identity differences most clearly.

Should I generate in widescreen and crop, or generate directly in vertical?
Generate directly in your target ratio whenever possible. You get better composition and fewer awkward crops. Generate wide only when you need multiple deliverables from the same footage.

How do I keep one visual style across an entire series?
Save a series bible with palette values, lens character, lighting defaults, and the reusable descriptor block. Reuse it and change only the story. Style consistency is a documentation problem more than a model problem.

Where should I spend the most time?
Keyframe composition and the audio bed. Those two choices drive most of the perceived quality of the finished piece. Motion settings matter less than most creators expect.

Is a long AI short harder than a live-action short?
Differently hard. Continuity is more brittle and small errors are more visible, but iteration is far cheaper. You can produce ten versions of a scene and keep the best one, which is a luxury live-action production rarely has.

How do I stop a sequence from feeling like unrelated clips?
Repeat three things: a consistent palette, a consistent screen direction, and a recurring visual motif — a prop, a gesture, or a light source. Motifs are what make an audience believe the shots belong together.

The Takeaway

Directing AI video well is mostly ordinary film craft applied to an unusually literal collaborator. Write the premise, list the shots, lock continuity, describe light physically, give each shot exactly one motion, cut to sound, and review in passes. Do that consistently and the model stops being a novelty and starts behaving like a crew you can direct — one that never gets tired, never argues about call times, and never asks for a second take you cannot afford.