What actually changed in AI video generation
A year ago, most AI video output was judged on a single question: does the movement look real for two or three seconds? That bar is gone. Today the interesting questions are about directability, physical plausibility, shot length, character persistence, and whether a generated clip can survive being cut into a real timeline with real sound design.
That shift matters because it changes how you work. When models were unreliable, the workflow was a lottery: write a prompt, generate, hope, repeat. Now the strongest models respond to structured direction — camera language, blocking, lighting notes, timing — and they hold a look across multiple shots if you feed them consistent references. The craft has moved from prompting luck to shot planning.
At the same time, no single model wins everything. Some are extraordinary at photoreal human faces but mediocre at fast action. Some handle complex camera moves beautifully but drift on identity. Some are cheap, fast and predictable, which makes them perfect for animatics and previz even if they never appear in the final cut. The practical consequence is that serious creators run a multi-model pipeline, assigning each shot to the engine most likely to deliver it in the fewest attempts.
This guide walks through that pipeline: how the main model families differ, how to pick an engine per shot, how to keep characters consistent across engines, how to prompt so that your instructions survive translation between tools, and how to control iteration cost without strangling creativity.
The leading models and what each does best
It helps to stop thinking in terms of a ranked list and start thinking in terms of capability profiles. Almost every production decision comes down to four axes: realism, motion control, consistency, and cost per usable second.
Cinematic realism and physics
Sora-class models built their reputation on longer, more coherent scenes with plausible object interaction — liquid pouring, cloth folding, a hand pushing a door that actually swings. Their strength is narrative continuity: a single prompt can yield a shot with a beginning, middle and end rather than a loop of pleasant motion. If your project depends on believable cause and effect, this family is usually the first stop.
The trade-off is controllability. Very capable models sometimes interpret your prompt as a suggestion and add their own blocking, extras, or camera drift. You get beauty, but you may not get the exact shot. Budget extra attempts for precision work.
Motion control and camera language
Kling-style engines earned their following through fine-grained control: start and end frames, motion strength, camera trajectory, and a strong feel for dynamic subjects. They are excellent for product spins, choreography, sports, and any shot where the path of movement matters more than the environment.
Their weakness is often consistency over long sequences and, occasionally, over-sharpened texture in skin and fabric. Treat them as your action and movement specialists rather than your universal engine.
Stylized and edit-friendly output
Runway and Luma occupy a slightly different niche: fast, flexible, strong at stylization, and deeply integrated with editing tools. Their video-to-video and style-transfer features make them the natural choice when you already have footage and want to restyle it, or when you need a dozen variations of a look in an afternoon.
They also tend to be the most forgiving when you need to iterate quickly. Output may be less photoreal than the flagship models, but the speed of the loop often matters more than the peak quality of a single frame.
Open-weight and local models
Open-weight video models changed the economics of the field. Running a generator on your own hardware removes per-generation pressure, which changes behavior: you experiment more, you throw away more, and you end up with better material. You also get privacy for unreleased client work and full control over licensing.
The cost is setup. You need a capable GPU, comfort with command-line tooling, and patience with quantization and memory tuning. For teams doing hundreds of iterations per project, that investment usually pays for itself quickly.
Specialized models round out the picture: some excel at image-to-video with strong subject preservation, others at lip sync and talking heads, others at high-frame-rate sport or animation styles. Keep a short list of three to five engines you know deeply rather than a spreadsheet of twenty you have never tested.
How to choose a model for a specific shot
Most wasted effort comes from asking, "which model is best?" instead of "which model is best for this shot, in this budget, at this stage?" A shot-by-shot assignment solves the problem.
A simple scoring rubric
Score each candidate from 1 to 5 on five criteria, then match against the shot's priority:
- Subject fidelity — how well it preserves a face, logo, or product silhouette from a reference image.
- Motion accuracy — how closely the movement follows your described path and speed.
- Physical plausibility — contact, weight, shadows, and object permanence.
- Continuity — likelihood that the same character looks the same in the next shot.
- Iteration speed — realistic time and budget to reach an acceptable take.
A hero shot in a brand film might weight fidelity and plausibility at 5 and speed at 2. A previz animatic weights speed at 5 and everything else at 2. Same engine, different score, different choice — which is exactly the point.
Match the model to the stage, not just the shot
Early in a project, optimize for volume: fast engines, low resolution, loose prompts, many variations. You are searching for the idea, not polishing it. Mid-project, switch to engines with strong control surfaces so you can execute the shots you selected. Late in the project, use the highest-fidelity engine only for the few seconds the audience will actually study — a face in close-up, a product reveal, a title moment.
This staging alone can cut total render spend dramatically without visibly reducing final quality, because most of the timeline never needed the expensive engine in the first place.
A practical multi-model workflow, step by step
Here is a pipeline that works for commercials, short films, music videos, and social content alike. It is deliberately boring, because boring pipelines ship.
Step 1: Script and shot list before any generation
Write the script, then break it into numbered shots with columns for duration, subject, action, camera, lighting, and required continuity items. This document becomes your prompt source and your review checklist. Teams that skip it spend the same time re-deciding things inside their generator UI, usually with worse results.
Step 2: Look development with still images
Generate keyframes as stills first. Stills are cheap, fast, and easy to compare side by side. Approve the look, wardrobe, palette, and framing before animating anything. When you do move to video, feed these approved stills in as the first frame — image-to-video is dramatically more controllable than text-to-video for anything with a specific subject.
Step 3: Animate with restrained motion prompts
Describe motion in physical terms: what moves, in which direction, at what speed, relative to what. "Slow push in on the left hand as the lid lifts, steam rising, background static" outperforms "cinematic, beautiful, masterpiece." Keep one primary motion and at most one secondary motion per shot; models blend competing motions into mush.
Step 4: Generate coverage, not single clips
Ask for three to five variations per shot with small prompt changes — different camera distance, different pace, different lighting direction. Coverage gives your editor options and protects you when a cut does not land. Treat the first acceptable clip as the start of the shot, not the end of the work.
Step 5: Upscale, interpolate and stabilize
Most engines output at modest resolution and frame rate. Run a dedicated upscaler, then interpolate to your delivery frame rate, then stabilize if the camera move needs to be locked. Do the stabilization before any speed ramps, not after. Keep the original clip — interpolation artifacts sometimes appear only after color grading.
Step 6: Assemble, sound design, color
Cut in an editor, add sound effects and music, and grade after the picture is locked. AI-generated footage often has inconsistent white balance between shots; a single grade pass with matched references fixes more perceived quality problems than another round of generations ever would. Sound is the fastest way to make generated footage feel real: footsteps, room tone, and cloth movement carry more weight than extra detail in the image.
Prompt patterns that travel across models
Because you will use several engines, write prompts in a portable structure so you can move a shot between them without rewriting from scratch.
- Subject and wardrobe — one clause, specific nouns, no adjectives stacked three deep.
- Action and timing — what happens and when, expressed in seconds.
- Camera — distance, height, lens feel, movement speed.
- Light and time of day — direction and quality, not mood words.
- Environment and atmosphere — location, weather, particles, background activity.
- Continuity anchors — the exact name you use for a character or prop, repeated verbatim across shots.
- Negative constraints — a short list of the artifacts you keep seeing: extra fingers, warped text, floating feet, morphing logos.
Keep this block in a spreadsheet cell or a text file. When you switch engines, you trim one clause and add another rather than starting over. Over a hundred shots, that discipline is worth more than any single prompt trick.
Consistency across shots and models
Consistency is the hardest part of AI video, and it is a systems problem rather than a prompting problem.
First, create a character sheet: three to five reference stills at different angles, plus a locked description. Use those same references everywhere. Second, when a model supports start-frame conditioning, chain shots by using the last frame of one clip as the first frame of the next — this creates a visible continuity thread even when the underlying engines differ. Third, lock wardrobe, props, and locations in writing, then check each generated clip against that list before accepting it.
Be realistic about cross-model consistency. Skin texture, contrast, and lens character vary between engines, and a cut between two engines can read as a jump in production value. Two mitigations work well: keep a single engine for all shots of the same character within a scene, and use a unifying grade plus grain pass across the entire sequence. If a mismatch is unavoidable, hide the cut behind a motivated transition — a whip pan, a match cut on movement, or a brief insert.
Budget, render time and iteration discipline
Whatever pricing model you use — subscription tiers, per-second billing, or your own electricity — the real constraint is iteration count. A disciplined workflow reduces attempts per accepted shot from double digits to low single digits.
Practical rules that consistently help:
- Approve the still before animating. Most rejected clips were doomed at the keyframe stage.
- Change one variable per attempt. If you alter subject, camera, and lighting at once, you learn nothing from the failure.
- Keep a log. Shot number, engine, prompt version, verdict, reason. After twenty shots you will see patterns: one engine keeps failing on hands, another on text on screen.
- Set a stop rule. Three attempts without improvement means the shot design is wrong, not the prompt.
- Batch similar shots. Same character, same location, same lighting: generate them in one session while the prompt is fresh.
Also plan for render time as a scheduling constraint, not just a cost. Long queues shape decisions: use fast engines for exploration during the day and high-fidelity engines overnight for hero shots.
Common mistakes and how to fix them
Overloaded prompts. Ten adjectives do not improve output; they average out. Fix: subject, action, camera, light. Delete the rest.
Text-to-video for specific subjects. If a face or product must match, start from an image. Fix: generate or photograph a keyframe, then animate it.
Chasing realism in the wrong place. Viewers notice hands, eyes, and text. They rarely notice background foliage. Fix: spend your best engine on close-ups and let wide shots use faster models.
Ignoring frame one. The first frame is the most scrutinized frame in any edit. Fix: reject clips whose opening frame is weak, even if the middle is beautiful.
No sound planning. Silent generated footage feels synthetic regardless of image quality. Fix: rough in sound effects before you judge a cut.
One engine for everything. Fix: assign shots by capability, and accept that your pipeline has three or four tools.
Quality control checklist before delivery
Run this before you export, and you will catch most embarrassing errors:
- Character identity holds across every cut within a scene.
- Hands, teeth, eyes, and jewelry are anatomically stable in motion.
- Any on-screen text, logos, or signage is legible and correctly spelled.
- Shadows and contact points are consistent with the light direction you established.
- Camera moves are motivated and do not drift unintentionally.
- Frame rate, resolution, and audio loudness match the delivery spec.
- No interpolated warping on fast motion or around silhouettes.
- Color and grain are unified across shots from different engines.
FAQ
Do I need several AI video tools, or can one do the job?
One tool can finish a short project, especially if the style is consistent and the shots are simple. Multi-model pipelines become worthwhile when a project mixes precise movement, photoreal close-ups, and long scenes — no single engine leads on all three.
Is text-to-video or image-to-video better for narrative work?
Image-to-video is almost always more controllable. Text-to-video is best for exploring ideas, atmosphere, and abstract sequences where exact framing does not matter.
How do I keep a character consistent between different engines?
Lock a reference set, keep the same character description verbatim in every prompt, avoid mixing engines within a scene, and unify the final look with a single grade and grain pass.
How long should a generated shot be?
Shorter than feels natural. Two to four seconds is typical for a usable take; longer clips tend to accumulate artifacts. Generate long, cut short — the edit is where coherence is created.
What should I do about the resolution being too low?
Upscale as a separate step, after you have accepted the motion. Do not judge sharpness before upscaling, and always keep the original file in case interpolation introduces warping.
Are open-weight models worth the setup effort?
If you iterate heavily or handle confidential client material, yes. The removed per-generation pressure changes how freely you experiment, and experimentation is what produces good footage.
How do I stop wasting attempts?
Approve stills first, change one variable per attempt, log every result, and stop after three failed iterations to rethink the shot rather than the prompt.
Where to start this week
Choose one project, one scene, and three engines. Write the shot list, generate keyframes, animate the two hardest shots in your strongest engine, and process everything else in the fastest one. Log every attempt. By the end of the scene you will have a personal capability map — which engine handles your faces, your movement, and your deadlines — and that map is worth more than any generic ranking of tools.
From there the workflow compounds. Each project sharpens your prompts, your reference library grows, and your pipeline needs fewer attempts per shot. The models will keep changing, and new releases will keep resetting the leaderboard. A disciplined process for planning shots, testing engines against your own requirements, and unifying the result in post is the part that survives every model update.
