Why text-to-video is harder than it looks
Every few months a new text-to-video model arrives with a demo reel that looks like a feature-film trailer. The clips are cinematic, the camera moves are fluid, and the lighting looks as though it came from a real set. Then you try it yourself and get a woman with three hands walking through a wall.
The gap between a ten-second showcase clip and a sixty-second finished video is where most projects quietly die. A showcase clip is selected from dozens of attempts and judged on its own merits. A deliverable has to work as a sequence: every shot present, every character recognizable, every cut motivated, and a runtime that holds attention. That is a workflow problem far more than it is a model problem.
This guide gives you a repeatable pipeline: how to plan shots, build references, generate cheap first passes, lock selects, and finish a video that looks intentional. Along the way you get decision criteria for choosing a model per shot type, a prompt template you can reuse, a consistency toolkit, and a quality-control checklist to run before you export.
The advice is model-agnostic on purpose. Tools change every quarter; the pipeline below is what survives those changes.
The four failure modes that break AI video projects
Almost every disappointing result traces back to one of four root causes. If you name them before production starts, you can design around them instead of discovering them in the edit.
1. Shot drift and character drift
Text-to-video models do not remember your character between generations. Each prompt is a fresh interpretation, so hairstyle, eye shape, jacket color, and facial proportions shift from shot to shot. Audiences notice immediately, even when they cannot say what is wrong.
The fix is to stop relying on words alone. Generate a character sheet first (front, three-quarter, profile), then drive each shot with an image reference or an image-to-video mode where available. When you must use pure text, keep every prompt identical except for the one variable you are changing, and log the seed when the tool exposes one.
2. Unstable motion and temporal artifacts
Hands, eyes, teeth, on-screen text, crowds, liquids, and fast lateral camera moves are where models still struggle. The failures are not random: they cluster around occlusion, small features, and rapid change in the frame.
Design shots that avoid those conditions. Keep the action slow, keep the camera move single and deliberate, avoid showing readable text, and cut away before a hand needs to do something precise. Three restrained shots beat one ambitious shot that took twenty attempts and still looks melted.
3. Unpredictable iteration cost
Every generation costs something, and iteration multiplies that cost by the number of attempts per shot. When you discover that a shot needs a fundamentally different approach, the attempts you already spent are gone.
The fix is sequencing: storyboard thoroughly, generate at draft quality first, and only spend on high-quality passes after picture lock. Decide before you start how many attempts a shot is allowed. If it exceeds that number, change the approach rather than the parameters.
4. Access, rights, and commercial readiness
Availability windows, region gating, watermark rules, unclear commercial terms, and training-data questions all create delivery risk. A clip you cannot license for a client is not a clip.
Confirm usage terms in writing before production begins, keep a record of every asset and its source, and maintain a fallback model that can reproduce the same shot at acceptable quality. Never build a client deliverable around a single tool you cannot control.
A five-stage text-to-video workflow
This pipeline is deliberately front-loaded. Most of the value is created before the first render.
Stage 1: Script to shot list
Break the script into shots of three to six seconds. Each shot carries one idea, one action, and one camera move. Write every shot as a single sentence plus two constraints: one that must be present and one that must not appear.
A useful shot entry looks like this:
- Shot 07, 4 seconds: woman in a mustard raincoat steps onto a wet crosswalk as a tram passes behind her.
- Must have: rain on the pavement, shallow depth of field.
- Must avoid: readable signage, hands in frame, fast camera movement.
That structure forces clarity. If you cannot describe the shot in one sentence, the shot is doing too much.
Stage 2: Reference building
Before generating any video, build an asset library. You need character sheets, three to five style frames that establish the look, location plates for recurring environments, and a color script mapping the emotional arc.
Generate these with an image model, or shoot them with a phone. The origin does not matter, only that they are consistent and reusable. These references become the anchor for everything that follows and are the single biggest lever on final quality.
Stage 3: Cheap first passes
Generate at draft resolution and the lowest settings that still communicate the shot. Run two or three variants per shot. The goal here is coverage, not beauty.
Cut the drafts into an assembly immediately. Pacing problems, missing coverage, and unclear story beats are all visible in an assembly, and fixing them costs nothing compared with fixing them after expensive renders. This is the stage where the project actually gets made.
Stage 4: Lock selects and refine
Once the edit works with ugly footage, lock it. Only then regenerate the approved shots at higher quality, with upscaling or frame interpolation where the motion needs smoothing.
A common mistake is polishing a shot before deciding whether it belongs in the cut. That is how teams end up with three beautiful shots and a video that does not hold together.
Stage 5: Edit, sound, and finish
Cut to music, then layer sound design: footsteps, room tone, cloth movement, whooshes on transitions, and a subtle ambience bed under every exterior. Add a light grade to unify color across shots, since models drift in white balance and contrast.
AI video arrives silent and slightly inconsistent. Sound and grade hide more problems than extra generations ever will, and they are far cheaper.
A reusable prompt structure
Vague prompts produce vague footage. Use six slots, in this order:
- Subject: age, wardrobe, distinguishing detail, emotional state.
- Action: one verb, physically observable, present tense.
- Environment: location, weather, time of day, background activity.
- Camera: framing (wide, medium, close), lens feel, movement (static, slow push, tracking).
- Lighting: source, direction, quality (soft window light, hard noon sun, neon practicals).
- Style and technical: film stock or look, color palette, aspect ratio, grain level.
Example: A woman in her thirties wearing a mustard raincoat walks slowly across a wet city crosswalk, medium shot, 50mm, gentle tracking left, overcast daylight with soft reflections, muted teal and amber palette, 16:9, fine grain.
Three rules matter more than the template. First, avoid negation: models handle must-avoid constraints poorly, so remove the unwanted element from the scene description instead of naming it. Second, keep one action per prompt; two actions in four seconds reads as chaos. Third, log every prompt, seed, and output so a successful result is reproducible rather than lucky.
Choosing a model per shot
There is no single best model, only a best model for a shot. Evaluate candidates against a fixed set of criteria:
- Realism and surface detail, especially skin, fabric, and water.
- Motion fidelity: how well it handles walking, crowds, and camera movement.
- Maximum duration and whether the clip is extendable.
- Control inputs: image conditioning, video-to-video, masking, camera controls.
- Aspect ratio and resolution options, including vertical formats.
- Commercial terms and watermark policy.
- Generation speed, which determines how many iterations fit in a day.
- Cost per attempt, which determines how brave you can be.
Then map shot types to tools. Product macros reward crisp texture and shallow depth of field. Talking heads reward facial stability and lip-sync accuracy. Landscapes reward motion smoothness and long durations. Stylized animation rewards strong aesthetic adherence. Crowd scenes and complex VFX elements are usually better composited from stills and stock than generated from scratch.
Mixing three tools in one video is normal and invisible to the audience, provided you unify color and sound in the final pass.
Consistency toolkit
Consistency is not a single setting; it is a set of habits.
- Reference-first generation: always start from an image when the character or location recurs.
- Minimal prompt deltas: change one variable per attempt and hold everything else constant.
- Fixed seeds: reuse a seed when a shot is close but not perfect.
- Custom style training: for series work, train a small style or character model on your own approved frames.
- Asset naming: adopt a strict convention like project_shot07_v3_ref01 so nothing gets lost.
- Continuity rules: respect screen direction and eyelines across cuts, and keep wardrobe details tracked in a simple table.
- A final grade: one unified color pass across the whole timeline ties mismatched shots together.
Series and episodic work justify the extra effort of custom training. One-off social posts usually do not.
Budgeting iteration without surprises
Start from the deliverable and work backwards. Assume four to eight attempts per shot at draft quality, and two to three at final quality. Multiply by your cost per attempt, then add twenty percent for the shots that surprise you.
Four habits keep spending predictable:
- Draft-first policy: no high-quality renders before picture lock, without exception.
- Batching: group similar shots into one session so settings, lighting, and style stay stable.
- Off-peak scheduling: queue long jobs overnight and review in the morning.
- A spend log: record attempts, cost, and outcome per shot so you can estimate the next project from real data.
Set a kill rule in advance. If a shot fails after ten attempts, stop tuning parameters and change the shot itself: shorten it, reframe it, replace the action, or shoot it practically.
Quality control checklist before export
Run this list on a full-screen pass, then again at normal viewing size on a phone.
- Hands and fingers: count them, check joints, look for merging.
- Eyes and teeth: watch for flicker, asymmetry, or glassy stares.
- Text and logos: any unreadable glyph or warped brand mark must be cut or replaced.
- Wardrobe and hair: continuous across every cut of the same scene.
- Screen direction and eyelines: consistent through dialogue and movement.
- Motion artifacts: check fast pans, water, hair, and fabric frame by frame.
- Jump cuts and stutter: watch for frames that repeat or drop.
- Audio: sync, ambience beds, transitions, loudness consistency.
- Aspect ratio and safe areas: titles not clipped in vertical crops.
- Color: banding in gradients, drifting white balance between shots.
- Rights: every asset traced to a source with usable terms.
- Export settings: bitrate, codec, and frame rate matched to the delivery platform.
Common mistakes to avoid
Writing shots like screenplay scenes rather than single actions produces unfilmable prompts. Generating at maximum quality first wastes budget on shots that get cut. Judging drafts at full screen instead of inside an edit hides pacing problems. Reusing a prompt that worked once without logging the seed turns a repeatable result into a one-off. Skipping sound design leaves the video feeling synthetic, even when the images are strong. And using AI video for shots that a still photograph, a stock clip, or a simple practical shot would serve better is the most expensive mistake of all.
The final trap is pride of authorship: keeping a shot because it took effort rather than because it works.
FAQ
Can text-to-video replace a full production?
For short-form content, product visuals, and stylized sequences, often yes. For dialogue-driven narrative with complex performance, it is usually a supplement rather than a replacement. The most reliable results come from hybrid pipelines where AI handles environments, inserts, and transitions while practical footage carries the performances.
How long should each generated shot be?
Three to six seconds. Shorter shots hide artifacts, keep pacing tight, and give you more editing flexibility. If a moment needs to breathe, generate two adjacent shots and cut between them rather than asking one model to hold a long take.
How do I stop a character from changing between shots?
Use an image reference for every shot, keep the prompt identical apart from the action, reuse the same seed where available, and track wardrobe details in a table. A final color pass across the whole timeline covers the remaining drift.
Should I use text-to-video or image-to-video?
Image-to-video whenever you already know what the frame should look like, which is most of the time in a planned project. Text-to-video is best for exploration, mood pieces, and shots where you genuinely do not know the composition you want yet.
What resolution and aspect ratio should I deliver?
Match the platform first and generate as close to native as possible. Generating wide and cropping to vertical loses composition and often reveals artifacts at the edges. If you need both, generate both versions from the same reference rather than reformatting one.
Do I need more than one model?
Usually, yes. No single tool is best at faces, landscapes, motion, and stylized looks simultaneously. Treat models as a toolkit and standardize on two or three you know well. Consistency comes from your pipeline and your final grade, not from using one tool for everything.
How much time should I budget?
Roughly forty percent planning and reference building, thirty percent draft generation and editing, twenty percent final renders, and ten percent finishing and quality control. Teams that invert those proportions spend most of their time regenerating shots that were never going to survive the edit.
