Why the Production Math Changed
Not long ago, a polished sixty-second brand video meant a crew, a location, a permit, and a post-production schedule measured in weeks. Today a single creator with a laptop can generate dozens of variations of the same concept before lunch. The bottleneck has moved from can we afford to shoot this? to which take is actually worth finishing?
That shift has three practical consequences:
- Idea velocity beats equipment. The person who can test twelve hooks in one afternoon learns faster than the person with the better camera.
- Taste becomes the differentiator. When generation is cheap, the scarce skill is judging which output feels intentional rather than merely impressive.
- Iteration is the workflow. Experienced creators do not generate once and hope. They generate in passes, review quickly, and keep only what survives.
Most frustration with AI video comes from treating it like a vending machine: type a sentence, expect a finished scene. It behaves more like a film crew with a very short memory. You must give precise instructions, review the result immediately, and re-shoot without sentiment.
This guide is deliberately platform-neutral. Tool names appear as examples of capability, not as endorsements, and every technique works whether you are generating on Runway, Kling, Luma, Pika, Veo, Sora, or a local ComfyUI pipeline.
The Four Layers of an AI Video Workflow
A reliable process separates four layers. Skipping any one of them is the most common reason a project stalls halfway.
Layer one: concept and hook
Everything starts with a single sentence that names the promise of the clip. "A barista explains why your espresso tastes sour" is a concept. "Coffee video" is not. Write the hook first, because every later decision — shot length, pacing, music — exists to support it.
Deliverable: a one-line premise plus the first three seconds of narration or on-screen text.
Layer two: shot list and storyboard
A short-form video usually needs four to eight shots. Write them as a table with columns for shot number, duration, subject, action, camera movement, and lighting. If a shot does not advance the hook, delete it before you spend time generating it.
Storyboards do not need to be drawings. A grid of reference images and rough frames is enough to align a team or keep yourself honest.
Layer three: generation
This is the layer people obsess over, but it is only as good as the two above it. Generate in two passes: a fast exploratory pass at low resolution to check composition, then a quality pass on the shots that worked.
Layer four: assembly and sound
Editing, sound design, colour, and captions are where generated footage stops looking like a demo and starts looking like content. Budget at least as much time here as you do on generation. In practice, a 60-second clip needs roughly 20 minutes of concept, 30 minutes of shot planning, 60 to 90 minutes of generation and selection, and 60 minutes of finishing.
Choosing the Right Model for Each Shot
There is no single best model. There is a best model per shot, and learning to match them is the core technical skill of this craft.
Text-to-video versus image-to-video
Text-to-video is fastest for establishing shots, landscapes, abstract textures, and anything where exact composition does not matter. Image-to-video gives you control: generate or photograph a still, then animate it. If a shot needs a specific product angle, a specific face, or a specific location, start from an image.
Motion-heavy versus motion-light shots
Models handle different kinds of movement with different reliability:
- Simple motion (a person walking, steam rising, fabric moving) is the most reliable category and works well on almost any model.
- Complex interaction (hands manipulating objects, two characters touching, sports actions) degrades quickly. Break it into shorter clips or use image-to-video with a locked first frame.
- Camera-driven motion (a slow dolly, a drone push-in, an orbit) is usually better achieved by describing camera motion than subject motion.
Dialogue, lip sync, and performance
If a shot needs a talking head, generate the visual performance first and add the voice separately, then use a lip-sync pass. Trying to get both right in one generation is where quality collapses. Generate a neutral, well-lit, front-facing performance with a stable camera, then align audio in post.
Upscaling, interpolation, and finishing
Generation rarely ends the shot. A typical finishing chain looks like this:
| Stage | Purpose | When to use |
|---|---|---|
| Upscale | Increase resolution, restore texture | Final selected shots only |
| Interpolate | Smooth motion, double frame rate | Fast action, slow motion |
| Stabilise | Remove micro-jitter | Handheld or drone simulations |
| Grade | Unify colour across shots | Always, before export |
Interpolation is not a cure-all. If motion looks wrong at the source, smoothing it makes it look smoothly wrong.
Writing Prompts That Survive Generation
Prompt writing is not poetry. It is a specification.
The five-slot skeleton
Use five slots in a fixed order, every time:
- Subject — who or what, with two or three identifying details.
- Action — one clear verb phrase, present tense.
- Camera — shot size, angle, and movement.
- Light — source, direction, quality, and time of day.
- Style — film stock, lens character, colour palette, reference era.
Example: "A ceramic cup of espresso on a steel counter, thin steam curling upward, medium close-up at a slight low angle with a slow push-in, warm morning window light from the left with soft shadows, shallow depth of field, muted editorial colour palette."
Negative constraints and continuity anchors
Most models accept some form of exclusion list. Keep it short and specific: no text, no watermark, no extra fingers, no camera shake, no people in background. Long negative lists dilute each other.
Continuity anchors are phrases you repeat across every prompt in a sequence — the same lens description, the same colour language, the same time of day. Consistency between shots is largely a prompt discipline problem before it is a model problem.
Prompt hygiene
Change one variable at a time. If you alter subject, camera, and lighting simultaneously, you learn nothing about which change caused the improvement. Keep a simple log: prompt, model, seed, settings, verdict. After twenty projects, that log becomes your most valuable asset.
Keeping Characters and Locations Consistent
Consistency is the hardest part of AI video, and the part viewers notice instantly.
Character sheets and reference frames
Build a character sheet before you build scenes: one neutral portrait, one three-quarter view, one full-body frame, all in the same lighting. Use those images as starting frames rather than describing the face in words. Words cannot hold a face steady; images can.
First-frame locking and seeds
When a model supports a locked first frame or a seed value, use both. The first frame controls composition and identity; the seed reduces random drift in texture and lighting. For recurring locations, keep a hero image of the space and animate from it every time.
A practical continuity checklist
- Wardrobe: same colours and materials in every shot.
- Hair: length, part, and styling match.
- Light: same direction and colour temperature across the sequence.
- Lens: consistent focal length feel unless a shot intentionally breaks it.
- Props: the same objects appear in the same places.
- Grade: run one LUT or colour pass across the entire timeline.
When a shot refuses to match, the fastest fix is usually not more prompting. Regenerate it from a reference frame that already matches the rest of the sequence.
Cinematic Controls That Read as Professional
Audiences cannot name why one clip feels premium and another feels amateur, but the signals are consistent.
Lens and depth of field
Specify focal length language: wide, normal, or telephoto. Telephoto compresses space and flatters faces; wide lenses add energy and context. Shallow depth of field isolates a subject, but too much of it in every shot makes a sequence feel like a product reel rather than a story.
Camera movement vocabulary
Use precise terms instead of "dynamic camera":
- Push in — building tension or revealing importance.
- Pull out — context, isolation, or a reveal.
- Truck left or right — parallax and spatial understanding.
- Orbit — product hero shots and character introductions.
- Handheld drift — documentary realism.
- Static lock-off — comedy timing and dialogue.
One movement per shot. Two competing movements read as chaos.
Light, contrast, and grade
Describe light like a gaffer would: source, direction, hardness, and colour temperature. "Soft window light from camera left, warm 3200K, gentle falloff" gives a model far more to work with than "nice lighting." In post, pick a single grade and apply it everywhere. Unifying colour is the single fastest way to make separate generated clips feel like one film.
A Sixty-Second Short-Form Build, Step by Step
Here is the process applied end to end, using a hypothetical product story.
Step 1 — Lock the hook
Write the first three seconds as text: on-screen or spoken. If the hook does not work as a sentence, no amount of visuals will rescue it.
Step 2 — Build a six-shot spine
Establishing shot, problem shot, product or solution shot, detail shot, human reaction shot, closing shot with a call to action. Six shots at roughly 4–8 seconds each, with cutaways filling the rest.
Step 3 — Generate rough passes
Low resolution, two or three variations per shot. Do not judge quality yet; judge composition and whether the idea reads.
Step 4 — Select and regenerate
Pick the best of each and regenerate the weak ones with one variable changed — usually camera or lighting. Discard ruthlessly. A sequence is only as strong as its worst shot.
Step 5 — Assemble a radio edit
Cut picture to a scratch voiceover or music bed before polishing anything. If the story works with sound alone, the visuals will only improve it.
Step 6 — Add sound
Voice first, then music, then effects. Sound design is what makes generated footage feel physical: cloth, footsteps, room tone, subtle whooshes on cuts.
Step 7 — Caption and crop
Add burned-in captions, then check the safe zones for vertical formats. Keep text away from the bottom 15% and the top 10% where interfaces sit.
Step 8 — Export variants
Export a vertical cut, a square cut, and a horizontal cut from the same timeline. One build, three placements.
Editing, Sound, and Captions
Cut rhythm
Short-form editing rewards variation. Use fast cuts in the opening three seconds, then slow down. A constant rhythm feels mechanical; contrast creates the perception of pace.
Voice, music, and effects
Generate narration with a voice model, but always review pacing — synthetic voices rush sentence endings. Music should sit under dialogue at roughly minus 18 to minus 22 dB. Add three to five effects per minute of finished video, not more.
Captions and readability
Two lines maximum, six to eight words per line, high contrast, consistent position. If your captions move or change style, keep the change to one moment rather than every cut.
Aspect ratios and export settings
Vertical, square, and horizontal versions should be framed separately, not cropped blindly. Export at high bitrate — a compressed master will look soft no matter how good the source generation was.
Common Mistakes and How to Fix Them
Generating before planning. Fix: write the shot list first, always.
Overloading prompts. Fix: five slots, one idea per slot, no adjectives doing the same job twice.
Judging at first generation. Fix: always produce at least two variations per shot before deciding.
Ignoring the first frame. Fix: if a composition matters, generate or source a still and animate it.
Inconsistent grade. Fix: one colour pass across the whole timeline, applied last.
Too many camera moves. Fix: one movement per shot, and cut on the movement if possible.
Weak sound. Fix: spend as much time on audio as on visuals; viewers forgive soft images far more readily than bad sound.
No iteration loop. Fix: after publishing, note which hook, pacing, and length performed best, and carry that learning into the next build.
FAQ
How long should a generated shot be?
Three to eight seconds is the practical sweet spot. Anything longer draws attention to small inconsistencies, and anything shorter makes the sequence feel frantic.
Do I need multiple models?
Yes, in practice. Most creators keep one model for realistic motion, one for stylised or animated looks, and one image generator for reference frames.
How do I stop characters from changing between shots?
Animate from a consistent reference frame, lock the seed where possible, and keep wardrobe, hair, and lighting descriptions identical across prompts.
Can I use AI video for client work?
Yes, provided you check licensing terms for each tool, keep a record of source assets, and set expectations about revision limits before the project starts.
What is the biggest quality lever?
Lighting language in the prompt and colour grading in post. Together they do more for perceived production value than resolution.
How many variations should I generate?
Two or three per shot for exploration, then two more for the finalists. Beyond that, returns drop sharply.
Should I write prompts in English?
Most models respond most predictably to English, even when the interface supports other languages. Write the prompt in English and the captions in your audience's language.
How do I keep a series visually consistent?
Create a style guide with fixed lens language, colour palette, caption style, and music genre, then reuse it across every episode. Consistency in a series is a template problem, not a talent problem.
The creators who get the most from these tools are not the ones with the longest prompt. They are the ones with the tightest loop: plan, generate, judge, refine, publish, and learn.
