Start With the Edit, Not the Model
Most AI video projects fall apart for a boring reason: the creator opens a generator before knowing what the finished sequence requires. They produce a gorgeous eight-second clip, fall in love with it, then discover it cuts with nothing else. Three days later there are forty clips and no film.
The better habit is to work backwards. Write the sequence as a shot list first, even a rough one, and annotate each shot with what it has to accomplish. A shot either establishes place, advances action, reveals character, or buys you a transition. If it does none of those, it is not a shot, it is a test render.
Once shots have jobs, model selection becomes a much easier question. You stop asking "which generator is best?" and start asking "which generator is best for a slow push-in on a static subject with a locked background?" That is a question you can actually answer, and it often has a different answer than the one your favourite tool would give.
This guide lays out a model-agnostic workflow for AI video: how to plan, how to choose between model families, how to prompt consistently, how to handle audio, how to troubleshoot artifacts, and how to decide when a shot is finished. Tool names change every few months. The workflow does not.
Map the Pipeline Before You Generate Anything
The five stages
Every AI video project moves through the same five stages, whether it is a fifteen-second social ad or a ten-minute documentary short:
- Development — script, beat breakdown, shot list.
- Previsualisation — style frames, animatics, look tests.
- Generation — turning prompts and reference images into moving footage.
- Assembly — rough cut, pacing, temp audio, revisions.
- Finishing — upscaling, stabilisation, colour, sound mix, delivery.
Most people collapse stages 1 and 2 into "idea" and then spend 80% of their time in stage 3. That ratio is backwards. Generation should be the fastest part of the process, because it is the part you can repeat.
Build a shot ledger
A shot ledger is a spreadsheet with one row per shot and a handful of columns: shot ID, duration, purpose, model family, prompt, seed, reference frame, status, and notes. It sounds bureaucratic. It saves hours.
The ledger does three things. It stops you regenerating shots you already solved. It lets you hand a project to a collaborator without a thirty-minute briefing. And it makes model comparisons honest, because you can see that model A solved twelve of your shots on the first try while model B solved four.
Define "done" per shot before you render it
Write the acceptance criteria next to the shot. "Locked background, no camera drift, character keeps the same jacket, hands out of frame." Criteria like this are far more useful than a vague sense that a clip feels right, because they are checkable. If a shot fails one criterion, you know exactly what to change in the prompt instead of rerolling blindly.
Matching Model Families to Shot Types
Model families are not interchangeable, and treating them that way is the single biggest source of wasted render time. Here is how to route shots to the right family.
Text-to-video: establishing shots and abstract B-roll
Text-to-video models shine when the shot is about atmosphere rather than identity. Wide landscapes, city aerials, textures, weather, abstract backgrounds for titles—these tolerate the drift and reinterpretation that pure text prompting introduces. Use them for the first and last shots of a sequence, where you need mood more than continuity.
Image-to-video: character and product consistency
When a face, a costume, or a product has to stay recognisable, start from an image. Generating or selecting a strong still first gives you control over casting and wardrobe that text alone cannot provide. The video model then only has to animate, which is a much narrower problem. This is the workhorse family for narrative work and for any commercial spot where a specific object must not mutate.
Camera-control and motion-transfer models
Some models accept explicit camera instructions—dolly in, crane up, orbit left—or transfer motion from a source clip onto a new subject. Use these for shots where the movement itself is the point: a reveal move, a whip pan into a title, a dance reference applied to an animated character. Because the motion is specified rather than inferred, results are more predictable, which matters when the shot has to cut precisely against music.
Enhancement passes: upscaling, interpolation, cleanup
Treat enhancement as a separate family rather than an afterthought. Upscalers add resolution and, on good models, plausible detail. Frame interpolation smooths motion when a clip was generated at a low frame rate. Cleanup and inpainting tools remove watermarks, stray objects, or a boom mic that wandered into frame. Plan for at least one enhancement pass on every hero shot; budget none for connective tissue.
A quick routing heuristic
- Need a mood? Text-to-video.
- Need a person, pet, or product to stay itself? Image-to-video.
- Need a specific move? Camera control or motion transfer.
- Need it bigger, smoother, or cleaner? Enhancement.
Writing Prompts That Survive a Model Swap
The six-slot skeleton
Write every prompt as six separable slots: subject, action, camera, lens and format, lighting, and mood. For example: "a courier in a rain-soaked jacket (subject) running across a rooftop (action), handheld tracking shot from behind (camera), 35mm anamorphic, shallow depth of field (lens and format), overcast dusk with practical neon spill (lighting), tense and breathless (mood)."
Separable slots matter because different models weight different parts of a prompt. When a clip fails, you can change the lighting slot without touching the subject slot, and you keep a record of what actually caused the improvement.
Keep prompts portable
Avoid model-specific incantations unless you are committed to one tool for the whole project. Words like "cinematic" and "8K" are interpreted differently by every model, and some ignore them entirely. Concrete nouns and camera vocabulary travel better than hype adjectives.
Negative prompts and failure tags
Keep a shared list of negatives for the project: extra fingers, text artifacts, watermark, duplicate limbs, jittery background, oversaturated colours. If a model does not support negative prompts, encode the constraint positively instead: "hands resting at her sides, sleeves visible" works better than "no hands" in most engines.
Finally, version your prompts. Save prompt v1, v2, v3 in the ledger alongside the seed. Two weeks later, when you need one more shot in the same look, you will be grateful.
Keeping Visual Style Consistent Across Shots
Lock a look bible
Assemble a one-page look bible: a colour reference (three to five stills), a contrast reference, a grain reference, and a lens reference. Every prompt should be written so it could plausibly belong to that page. If a new shot needs a different palette, that is a creative decision you should make consciously, not accidentally discover in the edit.
Seeds, references, and repeats
Seeds give you repeatability within a single model. Reference images give you repeatability across models. Use both: pin a seed while you are iterating on a shot, then, once approved, archive the exact prompt, seed, and reference frame so the look can be recreated later.
Fix it in the grade, not the generator
Colour, contrast, grain, and vignette are cheap to apply in post and expensive to chase in prompts. Generate shots that are internally consistent in composition and lighting direction, then unify the palette in a single colour pass. Chasing colour consistency across ten generations is a good way to burn a week.
A Step-by-Step Workflow From Script to First Cut
1. Break the script into beats
Write the script, then split it into beats of roughly three to eight seconds. Each beat becomes one or two shots. Mark which beats carry story weight and which are connective.
2. Generate selectively, not evenly
Hero shots—close-ups, reveals, anything the audience will remember—deserve three candidates minimum. Connective shots deserve one. This keeps your render queue focused on the shots that actually determine whether the piece works.
3. Assemble a rough cut with temp everything
Drop the clips on a timeline with scratch music, scratch voice-over, and rough sound effects. Do not wait for finished assets. The rough cut is the first honest test of whether your shot list was right, and it will tell you which shots are missing, too long, or redundant.
4. Replace weak shots rather than patching them
If a shot does not work after two revisions, the problem is usually conceptual, not technical. Regenerate it from a different angle, a different model family, or a different moment in the action. Patching a fundamentally weak shot with effects is the most expensive decision in the whole workflow.
5. Run a finishing pass on locked shots only
Once the cut locks, upscale, stabilise, colour, and mix. Enhance before colour, colour before audio, audio before delivery. Finish in that order so you never re-render an expensive pass because a shot changed underneath it.
Dialogue, Voice, and Lip Sync Without Uncanny Results
Speech is where AI video most often tips into the uncanny valley, so treat it as its own pipeline.
Start with clean source audio. Generate or record the voice line first, then animate to it rather than the other way around. If you animate first and match audio later, you will spend hours stretching phonemes to fit a mouth that is already committed to different words.
Keep on-camera dialogue shots short. Two to four seconds of speaking is convincing; twelve seconds rarely is. Cut away to reaction shots, hands, or environment before the illusion thins out. If a line must run longer, cover it with a wide shot where facial detail is lower and the viewer has less to scrutinise.
For non-English dialogue, use a native speaker to check stress and rhythm, not just pronunciation. Bad intonation reads as fake even when every phoneme is technically correct.
Finally, mix dialogue with intention. A little room tone, a slight high-frequency roll-off, and consistent loudness across shots do more for believability than any amount of extra generation.
Troubleshooting Common Artifacts
Flicker and texture swimming
Flicker usually means too much detail in the prompt or too high a motion setting. Simplify the scene description, reduce motion strength, and generate a shorter clip. Long clips accumulate error; two clean four-second clips often beat one eight-second clip.
Morphing hands and faces
Keep hands out of frame, behind objects, in pockets, or small in the composition. For faces, use image-to-video with a strong reference still and reduce the amount of movement. A talking head that barely moves looks better than one that gesticulates and dissolves.
Warping on fast motion
Fast action is where most models break. Either slow the action in the prompt ("a measured sprint") or fake speed in post with a ramp on a slower, cleaner clip. Motion blur added in editing hides a remarkable amount of imperfection.
Identity drift within a shot
If a character changes across a single clip, shorten the clip and regenerate the back half from the last clean frame. Chaining short clips from extracted frames keeps identity stable without retraining anything.
Lighting mismatches between shots
Batch shots that share lighting conditions. Generate all the dusk shots in one session using the same prompt skeleton and reference frame, then move to daylight. Session batching reduces the small stylistic shifts that individual generations introduce.
Decision Criteria: Resolution, Runtime, and Budget
When two models can both plausibly shoot a scene, decide with a short list of criteria.
| Criterion | What to check |
|---|---|
| Motion fidelity | Does complex movement stay coherent, or does it smear? |
| Identity retention | Does the subject stay itself over four to eight seconds? |
| Camera control | Can you specify a move, or is the camera decided for you? |
| Native resolution | Will you need a full upscale pass, or a light one? |
| Clip length | Can it hold eight seconds, or does quality collapse after four? |
| Consistency across runs | Do repeated prompts produce similar results? |
| Turnaround | How long does a typical shot take at your target settings? |
| Licence and usage | Are commercial uses permitted for your distribution? |
Rank these by what your project actually needs. A social ad with a single hero product shot should weight identity retention and resolution. An abstract title sequence should weight motion fidelity and clip length. A documentary insert should weight camera control and turnaround, because you will generate many variations.
Also decide your ratio between generated footage and real footage early. Hybrid projects—real background plates with generated elements composited in—often look more grounded than fully synthetic sequences, and they let you spend generation time only where it adds something.
Pre-Delivery Checklist and FAQ
Checklist
- Every shot has a job in the edit and survives the rough cut.
- Prompts, seeds, and references are archived per shot.
- No identity drift on recurring characters or products.
- Resolution and frame rate are consistent across the timeline.
- Colour and grain are unified in a single finishing pass.
- Dialogue loudness is consistent and room tone is present.
- Motion artefacts are hidden, not merely tolerated.
- Captions are burned in or supplied as a separate file.
- Licensing for every model used is confirmed for your distribution channel.
FAQ
Do I need to master one tool or learn many?
Learn the workflow deeply and keep two or three model families at hand. Depth in a single generator is limiting; breadth without a system is chaos.
How long should a generated clip be?
As short as the edit allows. Four seconds of clean footage beats eight seconds of drifting footage almost every time, because you can always extend with a cutaway.
How do I avoid the "AI look"?
Reduce contrast, add grain, slow the camera, cut on motion rather than on stillness, and let some shots be imperfect. Polish applied uniformly reads as synthetic; texture and small asymmetries read as photographed.
Can I mix models in one project?
Yes, and you usually should. Unify the look in colour and sound, not by forcing every shot through one engine.
What is the biggest time sink?
Regenerating shots that were never properly specified. Ten minutes writing acceptance criteria saves hours of rerolling.
When should I stop iterating?
When the shot passes its criteria and survives in the edit. A shot that is perfect in isolation but cut in the first assembly was never worth the extra renders.
Once the workflow is in place, the interesting decisions become creative ones: which beat deserves the hero shot, where the cut should land, and what the audience should feel. That is the point of building a system at all—it moves your attention from fighting the tools back to directing the story.
