How to Keep AI Video Characters Consistent Across Scenes

Why the same face keeps changing between shots

Generative video models do not remember your character. Every clip is a fresh sample drawn from a probability distribution, conditioned on whatever text and images you hand the model at that moment. There is no persistent cast member inside the network, no continuity department, no wardrobe supervisor. The only memory that exists is the memory you build outside the model: reference images, written specs, locked seeds, reusable prompt blocks, and a review process that catches drift before it snowballs.

That is why a character can look perfect in shot one and like a distant cousin in shot four. The problem is not talent or a magic prompt. It is engineering.

Character drift shows up in three predictable forms, and naming them makes them easier to fix:

Identity drift — bone structure, eye spacing, jaw width, hairline, skin tone, and distinguishing marks slowly shift. This is the most damaging type because viewers read faces preattentively. A two-millimeter change in the distance between the eyes reads as "different person" even when nothing else changed.
Wardrobe and prop drift — the jacket gains a pocket, the shirt color warms, the glasses lose their frame shape, the necklace disappears. These errors are easy to miss in isolation and glaring in a sequence.
Performance drift — posture, gesture vocabulary, speech rhythm, and emotional register change. A nervous, hands-in-pockets character becomes a confident, arms-crossed one. This is the subtlest failure and often the one that makes an otherwise clean sequence feel wrong.

Everything below is a system for controlling those three layers at once, from pre-production planning through final grade.

Define the identity contract before you generate a single frame

Most creators start generating and only later notice they have no authoritative answer to the question "what does this character actually look like?" The fix is to write an identity contract: a short, specific document that separates what must never change from what is allowed to change per scene.

Sort your details into three buckets

Immutable anchors. Face shape, eye color and spacing, brow shape, nose profile, lip shape, hairline and natural hair color, skin tone family, height and build, any scar, mole, tattoo, or birthmark that makes the character recognizable. These get copied verbatim into every prompt and every reference set.

Scenario variables. Clothing, hairstyle (as opposed to hairline), makeup, accessories, injuries, weather effects on the character, emotional state. These should change scene to scene, but they need to be written down in advance so the change is intentional rather than accidental.

Style constants. Lens character, color grade, grain, contrast curve, and rendering style. If your series has a consistent look, treat the look as part of the character contract. A character who is photoreal in one shot and semi-stylized in the next will read as inconsistent even if the face is a perfect match.

Build a reference sheet that actually helps the model

A good reference sheet is not a collage of your favorite renders. It is a technical document. Aim for six to twelve images that cover:

A neutral, evenly lit front-facing portrait with a relaxed expression.
Two three-quarter angles, left and right, with soft directional light.
A full profile view.
A full-body shot in the baseline wardrobe.
Two or three expressive variants — smiling, speaking mid-sentence, looking off-camera — so the model learns that expressions do not change identity.
One image under the scene's characteristic lighting (night interior, harsh daylight, neon) so the model is not fighting an unfamiliar color cast.

Keep backgrounds plain and consistent. Contradictory reference images — one with warm rim light, one with cold flat light — force the model to average them, which produces a muddy, generic face that matches neither. Avoid occlusions: no hands over the jaw, no hair covering an eye, no sunglasses. Every detail you hide in the reference is a detail the model will invent later.

Write the voice and performance notes

If your sequence uses synthesized speech, keep the voice timbre, pacing, and accent stable in the same way. Note the character's micro-behaviors: do they tilt their head when listening, gesture with the left hand, keep their shoulders squared? These notes become the vocabulary you use in motion prompts, and they are the difference between a character and a puppet with the right face.

Choose the right generation path for each shot type

No single model is best at everything. Consistency comes from matching the tool to the demand of the shot and then bridging between tools with a still frame.

Locked-off close-ups and dialogue coverage. Identity-focused still generation is strongest here. Image models that accept reference images or identity adapters, plus face-conditioning techniques, give you tight control. Generate the still, refine it, then animate with restrained motion.
Medium shots with moderate motion. Video-native models that accept a character reference image handle walking, turning, and gesturing well. Keep motion prompts modest: the faster and wider the motion, the more the model has to invent, and invention is where identity leaks away.
Action and full-body movement. Expect more drift. Plan for a post-pass or for staging the shot so the face is small, partially turned, or in shadow — choices that hide drift instead of fighting it.
Stylized or animated looks. A custom-trained style model or a small fine-tune on a tight, consistent image set is often more reliable than stacking reference images, because training bakes identity into the weights instead of relying on attention at inference time.

Still-first versus video-native

As a rule, generate the still first. Even if your final pipeline is video-native, a still lets you iterate cheaply: you can fix the jaw, swap a shirt, adjust the light, and only then commit to animation. You also end up with a frame that becomes the reference for every subsequent shot in that scene, which gives you a visual anchor for review.

When training is worth the setup

If your character will appear in dozens of shots, across multiple episodes, with a distinctive style, training is usually worth the effort. If the character appears in three shots of a single clip, reference-based conditioning is faster and good enough. Decide based on total screen time, not on the prestige of the technique.

The end-to-end workflow, step by step

Step 1: Lock the script and the shot list

Write the scene as a list of shots with duration, camera angle, and action. Continuity problems often begin as script problems: a scene that jumps between locations and lighting conditions without any connective tissue is hard to keep visually coherent.

Step 2: Approve the character bible and reference sheet

Nothing downstream should start until the reference set and written spec are frozen. Freezing means versioning — call it v1 and never edit v1. If you change the character, create v2 and regenerate anything that used v1.

Step 3: Produce a canonical or hero frame

Generate until you get a still that captures the character at their most representative. This is your north star. Every later shot gets compared against it, and every prompt block is derived from the prompt that produced it. Save the exact prompt, seed, model, sampler, step count, and reference images alongside the image file.

Step 4: Fix the prompt skeleton and the seed

Build a reusable prompt template with a stable identity block. Keep the identity block word-for-word identical across shots — do not paraphrase it, do not reorder it, do not swap synonyms. Models are sensitive to token order and wording in ways that are not intuitive, and "green eyes" versus "emerald eyes" can shift the result subtly enough to break a cut.

Step 5: Generate shot by shot, verifying as you go

For each shot: generate the still, compare it against the hero frame, adjust, then animate. Do not batch twenty shots and review them all at the end. Drift compounds — a small error in shot three becomes a large error by shot twelve because you start treating the drifted version as the new reference.

Step 6: Animate with restrained motion settings

Keep camera moves simple and motion strength moderate. If a shot needs a big movement, split it into two shorter shots and cut between them. Cuts forgive identity drift; long unbroken takes expose it.

Step 7: Run a continuity review pass

Before editing, lay every shot out as a contact sheet and review it as a sequence. This is covered in detail below.

Step 8: Grade, mix, and archive

A single color grade applied across all shots is the cheapest consistency tool you have. Then archive the full project: prompts, seeds, references, model versions, and settings. When you return in three months to add a scene, the archive is what makes shot thirty match shot one.

Prompt architecture for repeatable faces

A consistent prompt is structured, not poetic. Build it in fixed blocks:

[identity block] + [wardrobe block] + [action] + [environment] + [camera] + [lighting] + [style/technical]

A filled example for scene one:

40-year-old woman, oval face, wide-set dark brown eyes, straight nose,
full lower lip, sharp jawline, small mole below right eye, black shoulder-length
hair with center part | charcoal wool coat over cream turtleneck |
standing still, hands in pockets, calm expression | rain-slick city street at night |
medium close-up, 50mm | cool key light from left, neon rim light from behind |
photoreal, shallow depth of field, subtle grain

And the same identity block reused in scene four, where only the variables change:

40-year-old woman, oval face, wide-set dark brown eyes, straight nose,
full lower lip, sharp jawline, small mole below right eye, black shoulder-length
hair with center part | grey knit sweater, hair tied back |
seated at a kitchen table, leaning forward, tired expression |
sunlit apartment interior, morning | medium shot, 35mm |
soft window light from right | photoreal, shallow depth of field, subtle grain

The identity block is untouched. Everything around it moves. That is the whole principle.

Three practical rules follow from this:

Copy-paste, never retype. Retyping introduces errors. Keep the identity block in a text file and paste it.
Keep the visual vocabulary short and specific. Ten precise descriptors beat thirty vague ones. Long adjective stacks dilute attention and increase variance.
Use negatives deliberately. Add a negative block for artifacts you keep seeing: extra fingers, warped ears, asymmetric eyes, waxy skin, hair blending into background. Keep it identical across shots too.

Settings that quietly reduce drift

Generation settings matter more than most creators expect.

Resolution and aspect ratio. Generate at the highest native resolution your pipeline supports. Low-resolution generation followed by upscaling invents detail that does not match your character.
Seed behavior. Locking the seed gives repeatability for the still pipeline. For video, a fixed seed helps but does not guarantee identity: motion sampling introduces new randomness. Treat the seed as one lever among several.
Guidance strength. Very high guidance can produce over-sharp, over-saturated faces that look subtly different from the rest of the sequence. Find a moderate value and keep it constant.
Reference strength. If you are using reference images, tune how strongly they condition the generation. Too weak and identity drifts; too strong and every shot inherits the pose, framing, and lighting of the reference, which makes cuts feel like duplicates.
Motion amount. Reduce it. This is the single highest-leverage setting for face stability in video generation.

Continuity review: the step almost nobody does

Lay out one frame per shot in a grid and look at it as a sequence, not as individual images. Check in this order:

Face geometry. Trace an imaginary line across the eyes and another across the mouth. Do they sit at the same angle relative to the head? Is the jaw width consistent?
Hair. Hairline, part position, volume, and length. Hair is the second most identity-defining feature after the face and the one models improvise most freely.
Distinguishing marks. Moles, scars, tattoos, freckles, ear shape. These vanish and reappear constantly.
Wardrobe details. Collar shape, button count, sleeve length, fabric texture, and accessory placement.
Hands. Finger count, proportions, and nail shape. If hands keep failing, restage the shot rather than regenerating forever.
Lighting direction and color temperature. Consistency of light across shots does more for the perceived continuity of a character than any prompt tweak.
Performance. Gesture vocabulary, posture, and energy level.

Keep a numbered naming convention — sc01_sh03_charV1_v03 — so you can always find the approved version. Keep a changelog noting what changed and why.

Salvaging drift in post-production

When reshoots are not practical, repair in post. In rough order of cost:

Shorten the shot. The moment drift becomes visible is often the moment the shot should cut. Trimming three frames can save a clip.
Restage with inserts. Over-the-shoulder framing, reaction shots, back-of-head, hands in close-up, or a silhouette against a bright window all preserve narrative continuity while hiding facial inconsistency.
Face refinement pass. Dedicated face restoration and face-swap steps applied across shots can pull all faces toward the hero frame. Apply carefully: over-processing produces the uncanny "same mask" effect.
Unify with grade and grain. A single LUT, matched contrast, and consistent film grain across the sequence makes small identity inconsistencies far less noticeable.
Regenerate only the broken segment. Split the clip, regenerate the drifting portion with the hero frame as reference, and blend the seam with a cut or a whip-pan.
Voice continuity. If you use synthesized speech, clone from one clean, consistent source recording. Voice drift is as distracting as face drift, and it is usually easier to fix.

Mistakes that cost the most time

Reference sheets with conflicting lighting. The model averages them and produces a face that matches none of your footage.
Starting generation before the spec is frozen. Every later decision inherits the ambiguity.
Paraphrasing the identity block. Small wording changes produce small face changes, and small face changes break cuts.
Generating at low resolution and upscaling. Invented detail does not match the character.
Using a different model per shot without a bridging still. Cross-model sequences need an approved frame passed through as reference.
Reviewing shots one at a time. Drift is a sequence phenomenon; it is invisible in isolation.
Ignoring the last five percent. A single grade and matched grain resolves more inconsistency than an hour of prompt tinkering.

Frequently asked questions

Why does my character's face change in every clip?

Because each clip is an independent sample. Without a shared reference image, a verbatim identity prompt block, and consistent settings, the model has no reason to land on the same face twice.

Can I use just one image as a character reference?

You can, and it works for short projects with forgiving framing. For anything with close-ups or recurring episodes, use six to twelve images covering multiple angles, expressions, and the baseline wardrobe.

Do I need to train a custom model?

Only if the character carries a lot of screen time or has a distinctive style that reference conditioning cannot hold. For a handful of shots, reference-based generation plus a strong identity prompt is faster.

How do I keep the voice consistent?

Clone from one clean source recording and reuse the same voice configuration across all lines. Keep pacing and accent settings documented alongside the visual spec.

What about costume changes?

Document them in the identity contract as scenario variables. The immutable anchors stay fixed; the wardrobe block changes deliberately and is recorded so you can reproduce it later.

How many shots can one reference set cover?

There is no hard limit, but quality degrades as scenes diverge from the reference's lighting and framing. Add one environment-specific reference per distinct lighting setup rather than trying to make a single set cover everything.

A reusable checklist for your next sequence

Identity contract written and versioned
Reference sheet with consistent lighting, plain backgrounds, no occlusions
Hero frame approved and archived with full settings
Identity prompt block frozen and copy-pasted, never retyped
Seed, resolution, guidance, and motion settings documented
Shots generated sequentially with verification at each step
Contact-sheet continuity review completed before editing
Single grade, matched grain, and unified audio applied
Project archive containing prompts, seeds, references, and model versions

Character consistency in AI video is not a prompt trick. It is production discipline: define the character, freeze the reference, keep the prompt skeleton stable, choose the right generation path per shot, review as a sequence, and repair in post when necessary. Do those things and viewers stop noticing the seams — which is exactly the point.