Start With the Deliverable, Then Map Every Shot
Most failed AI video projects begin with a model and end with a folder of disconnected clips. The teams that ship consistently do the opposite: they define the final deliverable first, then work backwards into the footage.
Lock these before you generate anything:
- Aspect ratio and platform (9:16 for shorts, 16:9 for web and YouTube, 2.39:1 for cinematic cuts)
- Total runtime and target shot count
- Frame rate (24 fps for a cinematic feel, 25 or 30 for broadcast, 60 for slow-motion source material)
- Caption and lower-third style
- Colour, grain, and contrast treatment
- Emotional register: documentary, commercial, dreamlike, clinical
These choices cascade. A dreamlike register tolerates minor temporal artefacts and rewards long, slow camera moves. A clinical product demo does the opposite: it punishes flicker and needs locked-off, repeatable framing. Deciding the register early saves you from generating beautiful footage you cannot use.
Then build a shot inventory. One row per shot, with columns for duration, subject, action, camera movement, lighting, and a short note about what the shot must accomplish in the edit. This is the single highest-leverage document in the entire workflow. It converts a vague creative brief into a list of discrete, testable generation tasks, and it lets you start production on shot 12 while shot 3 is still being revised.
A finished forty-second piece usually needs roughly 12 to 20 generated shots plus a handful of title cards, insert shots, and transitions. Budget your effort accordingly: about three of those shots are hero shots that carry the piece, and the rest are connective tissue. Hero shots deserve five to ten attempts. Connective tissue deserves one or two. If a transition shot is not working after two tries, redesign the transition instead of burning an afternoon on it.
Choosing the Right Model for Each Shot Type
No single generative model wins at everything. Treat model selection as a casting decision, not a loyalty decision. You would not cast the same actor for a quiet dialogue scene and a stunt sequence, and you should not expect one video model to handle a talking head, a product turntable, and an aerial landscape equally well.
Fast draft models
Fast, inexpensive models are for planning, not delivery. Use them to test framing, pacing, and blocking at low resolution. Speed matters far more than fidelity here because you will throw almost all of it away. Render a rough version of the entire piece before you perfect a single shot. An animatic built from rough generations tells you more about whether the concept works than any number of polished individual clips.
A useful rule: never spend more than fifteen minutes on a shot before you have seen the whole sequence at draft quality. Sequence problems are cheap to fix. Shot problems are expensive.
High-fidelity hero models
For the shots that carry the piece, use the strongest photoreal model available to you. Tools such as Runway, Kling, Veo, Sora, and Luma Dream Machine handle skin, fabric, water, and complex lighting far better than earlier generations of generators. They are also slower and less predictable, which is exactly why you should reserve them for the three or four shots where quality is visible on screen.
Specialised models
Niche models quietly outperform generalists in narrow domains: animation and stylised illustration, product turntables, architectural fly-throughs, character performance and lip-sync, and nature footage. If your piece includes a talking presenter, a dedicated performance or lip-sync model will beat a general video generator almost every time. If it includes a stylised mascot, an animation-tuned model will hold line quality that a photoreal model smears into mush. The pragmatic move is to keep a short list of three or four specialists and match each shot to the one that has already proven itself on that shot type.
Open-weight models for control
Open-weight video and image models, including Stable Video Diffusion, AnimateDiff, and the ControlNet family, matter when you need reproducibility, structural control, or fine-tuning. They run locally, they accept depth maps, pose skeletons, and edge maps as conditioning, and they let you train a custom adapter on your own footage. The trade-off is engineering time and hardware. For a studio with a recurring character or a signature look, that trade is often worth it. For a one-off campaign, it rarely is.
A four-question test for any shot
- Does this shot need to look real? If not, a stylised model will be faster and more consistent.
- Does it involve complex physical motion such as hands, liquids, or crowds? If yes, expect more attempts and prefer models known for strong temporal consistency.
- Does it need to match a previous shot exactly? Prioritise models that expose seeds, camera parameters, and image conditioning.
- How many attempts can the schedule absorb? A four-attempt budget points to a reliable mid-tier model; a twelve-attempt budget lets you chase a premium one.
Answer those four questions per shot and you will stop making model choices by vibes.
Reference Images, Style Bibles, and Character Consistency
Consistency is the hardest problem in generative video, and it is almost always solved before generation starts rather than after.
Begin with a character sheet. Generate or photograph a clean reference set: front view, three-quarter view, profile, full body, and three expressions (neutral, speaking, reacting), all in flat, neutral lighting. This sheet becomes the conditioning input for every shot that features the character. Text-only prompts describing a person will drift within three shots; image conditioning keeps identity anchored.
Then write a style bible. One page is enough. It should specify:
- Palette and contrast curve
- Preferred lens choices (wide, normal, telephoto) per scene type
- Grain and texture treatment
- Two or three reference film stills for tone
- Camera movement vocabulary you will and will not use
The style bible prevents the slow slide into visual incoherence that happens when different shots are generated days apart with slightly different wording.
For environments, reuse seeds. If a hallway looked right, keep the seed and change only the action. For performance, use fresh seeds and accept variation, because identity is being carried by the reference image anyway.
When something drifts, change one variable at a time. If a character's jacket turns from navy to grey, do not rewrite the whole prompt. Adjust the wardrobe wording, regenerate, and compare. Debugging two variables simultaneously is how a twenty-minute fix becomes a two-day rabbit hole. Keep a note of which wording reliably produces which result; that note becomes your prompt library.
Writing Prompts That Survive Generation
Prompt quality is not about length. It is about removing ambiguity from the things that matter and leaving room for the model to handle the rest.
A reliable structure for video prompts:
Subject and wardrobe, action and motion, environment, camera framing and movement, lens character, lighting, mood, and constraints.
Keep it between forty and ninety words. Shorter prompts let the model improvise details you did not want. Longer prompts contain contradictions that the model resolves arbitrarily.
A workable example:
A woman in a charcoal trench coat walks slowly through a rain-slicked alley at night, camera tracks alongside her at chest height, 35mm lens, shallow focus on her face, neon sign reflections in puddles, cool blue key light with warm rim light, restrained and tense mood, steady motion, no camera shake, no on-screen text.
Notice what is doing the work. The camera instruction is explicit because tracking versus handheld changes the entire feel. The lens instruction controls distortion. The constraints at the end remove the two failure modes that ruin the most shots: unwanted camera shake and garbled text.
Additional practices that pay off:
- Use motion verbs in the present tense and avoid stacking three actions in one shot.
- Name the shot duration you expect. Models that support longer clips respond better to prompts describing a single continuous action.
- Keep a negative prompt list for recurring artefacts: warping faces, extra fingers, floating limbs, dissolving textures, flickering signage.
- Version your prompts. Save each iteration with a number so you can roll back.
- Separate style prompts from content prompts. When a client asks for a different look, you want to swap one block, not rewrite everything.
Building a Review Loop That Catches Failures Early
Review is where most time is lost, so make it systematic. Watch every clip once at normal speed first. Your eye catches rhythm and continuity problems instantly that frame-by-frame inspection hides. Only after a clip passes the normal-speed watch should you scrub it.
A practical checklist for each clip:
- Face and eye stability across the full duration
- Hand morphology, especially when hands enter frame
- Limb intersections and body proportions
- Any text, signage, or logos, which are usually corrupted
- Background elements that bleed, morph, or duplicate
- Flicker on flat surfaces such as walls and skies
- Lip-sync offset on spoken lines
- Edge ghosting during fast pans
- Colour and exposure match with the previous shot
Log every failure by category rather than by shot. If six clips fail on hands, the problem is your prompt template, not your luck. If six clips fail on colour, your style bible is too vague.
Approve in batches of five to ten clips so you can judge continuity as a group. Continuity errors are almost invisible when you review one clip at a time against memory, and almost impossible to miss when you review ten in sequence.
Finally, keep a fix list with an owner and a status. The three states that matter are draft, approved, and locked. Once a clip is locked, it does not get regenerated unless the edit demands it, because regeneration is how previously solved continuity problems come back.
When Fine-Tuning a Custom Model Is Worth It
Custom training is the most over-requested and under-used technique in AI video. It is powerful, and it is often the wrong answer.
Fine-tuning earns its cost when at least one of these is true:
- You have a recurring character appearing across many episodes or campaigns
- You need a signature visual style that general models keep flattening into generic output
- Your subject is niche and poorly represented in training data, such as a specific product, uniform, interior, or medical illustration
It is usually not worth it for a single campaign, a one-off music video, or anything where the look can be achieved with a strong style bible and disciplined reference conditioning.
If you do proceed, the workflow looks like this. Collect fifteen to sixty minutes of clean footage, or two hundred to two thousand carefully curated still frames. Caption them accurately and consistently, because caption quality matters more than dataset size. Hold back a small evaluation set that the model never sees. Train an adapter rather than a full model where possible, since adapters are faster, cheaper to iterate on, and easy to swap between projects.
Then evaluate honestly. Generate the same ten test prompts with the base model and the tuned model, shuffle the outputs, and judge them blind. If the tuned model wins on identity but loses on motion quality, you have overfitted. Reduce training steps, add more varied footage, and try again. A custom model that nails a face but cannot animate a walk is not production-ready.
Audio, Voice-Over, and Final Assembly
Audio is not the last step. It is the step that determines whether anyone forgives your visual compromises.
Build the picture cut first with rough audio, then record or generate voice-over to picture. Writing narration against finished timing produces tighter scripts and fewer awkward pauses. For voice, tools such as ElevenLabs and Descript handle scratch tracks and some final work well, but for hero pieces a real human voice recorded on a decent microphone still outperforms synthetic narration on emotional range.
Sound design hides temporal artefacts. A whoosh over a hard cut, a room tone bed, a music swell placed under a morphing background, and a well-timed impact all reduce how much the viewer notices imperfect motion. Budget real time for this. Fifteen minutes of sound design can save a clip that took an hour to generate.
In assembly:
- Cut on motion rather than on stillness, so transitions feel motivated
- Use L-cuts and J-cuts to smooth pacing
- Match colour and grain across clips in a proper grading tool such as DaVinci Resolve or Premiere Pro
- Keep a consistent grain layer over the whole piece so model-to-model differences disappear
- Add captions for accessibility and for sound-off viewing
- Export at platform-appropriate bitrates and check the final file on a phone before delivery
Scaling Up: Batching, Versioning, and a Week-Long Rhythm
Once a workflow works, the next gain comes from repetition and hygiene.
Use a naming convention that encodes everything: project, scene, shot, version, and seed. For example, brandx_s02_sh07_v04_seed88213. Six months later, that filename is the difference between reusing a shot and regenerating it.
Keep two shared documents: a prompt library and an asset registry. The prompt library stores prompt blocks that reliably produce a given look. The asset registry lists every approved clip, its source model, its seed, its prompt version, and its status. Both are plain and boring and both are worth more than any single generation tool.
A calm week-long rhythm for a sixty-second piece:
- Script, shot inventory, and deliverable specs locked
- Character sheet and style bible produced and approved
- Full animatic at draft quality, sequence reviewed end to end
- Hero shots generated and iterated
- Connective shots, transitions, and pickups
- Assembly, sound design, voice-over, grading
- Review pass, fix list, export, archive
If you have API access to your chosen models, batch your generation. Queue shots overnight, review them in the morning, and keep expensive high-fidelity renders for the shots that survived the draft review. Overnight queues are the cheapest form of patience available to a small team.
Common Mistakes That Quietly Kill AI Video Projects
Switching models mid-project. Every model has a different colour response, motion feel, and grain. Switching halfway means re-achieving consistency from scratch. Lock your model choices per shot type before production starts.
No locked style bible. Without it, shot twenty looks like a different film from shot two, and no amount of grading fully repairs it.
Judging stills instead of motion. A frame can look gorgeous while the clip around it wobbles, warps, and breathes. Always judge the moving image.
Over-prompting. Six adjectives about mood compete with the two words that actually control the shot. Cut anything that does not change the image.
Ignoring audio until the end. Problems that sound design would have hidden get discovered in the final week when there is no time left to hide them.
Skipping rights checks. Confirm that your chosen models permit commercial use, that your reference images are licensed, and that music and voice assets are cleared. This is unglamorous and non-negotiable.
Treating editorial problems as prompt problems. If a shot does not work in the edit, no regeneration will save it. Cut the shot, restructure the sequence, and only then decide whether you need new footage at all.
FAQ
How many generated shots do I need per minute of finished video?
For fast-paced commercial work, plan on fifteen to twenty-five shots per minute, many of them short. For narrative or documentary pacing, ten to fifteen is closer. Count your shots in the edit rather than in the generator, and expect roughly a third of generated clips to be unusable for reasons you cannot control.
Do I need expensive hardware?
Only if you plan to run open-weight models locally or fine-tune your own adapters. For hosted generation, a mid-range laptop with a stable connection is enough for planning and review. For local diffusion work, a modern GPU with substantial video memory makes the workflow tolerable rather than painful.
Can I mix models in one project?
Yes, and most experienced creators do. Match each shot to the model that handles that shot type best, then unify the result with a shared grade, grain layer, and aspect ratio. The risk is visual inconsistency, so test one shot per model before committing an entire scene.
How do I stop a character from changing between shots?
Use image conditioning from a fixed character sheet, keep wardrobe and lighting wording identical across prompts, reuse seeds for environments, and avoid changing more than one variable at a time when debugging drift.
What resolution should I generate at?
Generate at draft resolution for planning and at the highest practical resolution for hero shots. Upscale the final selections rather than the whole batch, and check for artefacts after upscaling, since upscalers can sharpen a warp into something more visible than the original.
How long does a one-minute piece take?
A solo creator with a clear shot list can complete a one-minute piece in four to seven working days, with most of the time spent on review and assembly rather than generation. The first project takes longer because you are building your prompt library and style bible at the same time.
Is fine-tuning always better than prompting?
No. Prompting plus reference conditioning solves most consistency problems. Fine-tuning is for recurring subjects and signature styles where you will amortise the training effort across many projects.
How should I handle on-screen text?
Do not generate it. Text inside generative video is unreliable, so add titles, captions, and logos in your editor where you control kerning, spelling, and legibility.
A workflow beats a model. The creators who produce consistent, client-ready AI video are not using secret tools. They are locking deliverables, mapping shots, matching each shot to the right model, holding a style bible, reviewing in motion, and treating audio as part of the craft. Build those habits once and every future project gets faster.
