Why AI Video Characters Drift Between Shots
Anyone who has produced more than two AI-generated scenes has met the same frustration: the hero looks perfect in shot one, slightly off in shot four, and like a distant cousin by shot twelve. The performance is fine. The cinematography is fine. The face is wrong.
This happens because most text-to-video and image-to-video pipelines treat every generation as an independent rendering task. The model samples from a latent space conditioned on your prompt, seed, and reference images, but it has no persistent memory of the character you approved two minutes ago. Even when the prompt text is identical, small shifts in camera angle, lighting language, motion description, and frame composition pull the sampled identity in a different direction. The result is visual fragmentation: a sequence that reads as several different people wearing the same costume.
The fix is not a single magic setting. It is a workflow. Consistency comes from treating character identity as structured data that travels with your project, not as a phrase you retype from memory. The sections below lay out a repeatable system: build an identity sheet, lock a descriptor block, anchor each scene, control variables one at a time, then run quality checks before the sequence ever reaches an editor.
Step 1: Build a Character Identity Sheet Before You Generate Anything
The most common mistake in AI video production is starting with the script instead of the cast. Scripts are cheap to revise; a drifting character means regenerating every shot that features them.
A usable identity sheet contains six to twelve reference images of the same person, generated or photographed in controlled conditions:
- Front-facing neutral expression, even lighting, plain mid-grey background.
- Three-quarter view, both left and right, to give the model volume information.
- Profile view, which stabilizes jawline, nose bridge, and hairline.
- Two or three expression variants — smile, serious, speaking — since emotion changes facial geometry.
- Full-body reference with a height marker, so proportions stay believable when the character shares a frame with someone else.
- Wardrobe variant for each costume the character wears across the story.
- Detail close-ups of identifying marks: a scar, freckle pattern, earring, tattoo, or unusual eye colour.
Keep all references in the same aspect ratio and roughly the same lighting temperature. Mixed reference conditions teach the model that your character's skin tone changes with the weather, and it will oblige you later.
Name the folder with a convention you will still understand in three months, something like cast/mira/refs/v03/. Version numbers matter because you will inevitably decide the character needs a different hairstyle halfway through production, and you will want the old set back.
Step 2: Write a Locked Descriptor Block
The descriptor block is the textual half of your character identity. It is a short, fixed paragraph — usually forty to seventy words — that appears in every prompt featuring that character, in exactly the same order, with exactly the same wording.
Order matters more than most people expect. Language models and video models both weight early tokens more heavily, so the most identity-defining attributes go first:
MIRA — woman, late twenties, South Asian, oval face, high cheekbones,
straight black hair tied low, dark brown eyes, slim build, small scar
above left eyebrow, wearing a charcoal bomber jacket over a cream
knit top, silver hoop earrings, matte skin, no glasses.
Two rules keep the block stable. First, never reorder it. Second, never paraphrase it inside a prompt, even for variety. If you find yourself writing "the same woman but with warmer light," you have already introduced a new variable into the one part of the prompt that must never vary.
Store the block in a plain text file next to the reference images. Paste it into prompts rather than retyping. When multiple people share a scene, place each block in a consistent order — character A, then character B — and never swap them, because prompt position influences which identity the model prioritizes when faces overlap.
Step 3: Use Anchor Frames to Seed Every Scene
An anchor frame is a single approved still that defines how a character looks within one scene's lighting and wardrobe. You generate it first, evaluate it against the identity sheet, and only then produce the rest of the shots in that scene by referencing it.
A practical rhythm looks like this:
- Write the scene's shot list with a rough duration for each shot.
- Generate a wide or medium anchor shot with the locked descriptor block plus the scene's lighting and location language.
- Compare the face to the identity sheet. Regenerate until it matches within acceptable tolerance.
- Use that anchor as an image reference for the remaining shots in the scene.
- Re-anchor at any scene change involving lighting, wardrobe, or time of day.
Keep anchor chains short. Three to five downstream shots is usually safe before cumulative drift becomes visible; longer chains progressively lose identity fidelity because each generation inherits the previous generation's small errors. When a sequence needs twenty shots, use three or four anchors and distribute the shots between them.
The anchor frame also solves a second problem: continuity of props and set dressing. If the anchor shows a red kettle on the counter, downstream shots that reference it are far more likely to keep the kettle red and in the same place.
Step 4: Combine Identity, Style, and Pose References Deliberately
Modern video tools accept multiple reference images and let you weight them against each other. The temptation is to stack four or five references and hope the model sorts it out. In practice, each additional reference dilutes the others, and identity is usually the first thing to suffer because faces carry the most detail.
A workable weighting strategy for a three-reference setup:
- Identity reference: highest weight. This is the face and body you must keep.
- Style reference: medium weight. Colour grading, film grain, lens character, illustration style if you are working in a non-photoreal register.
- Pose or composition reference: lowest weight. Enough to guide framing, not enough to overwrite identity.
If the output looks like a stiff passport photo, identity weight is too high. If the character looks related to your hero rather than identical, identity weight is too low or a competing reference is winning. Adjust one weight at a time between generations so you can actually attribute the change.
For stylized projects — animation, painterly, or graphic-novel looks — generate the identity sheet in the target style rather than in photoreal. A photoreal reference and a heavily stylized style reference fight each other, and the model resolves the conflict by averaging, which is exactly the drift you are trying to prevent.
Step 5: Change One Variable at a Time
Consistency is a controlled experiment. When a shot fails, the fastest path to a fix is knowing which change caused it. Change one thing per generation round and keep a short log:
| Variable | Example change | Watch for |
|---|---|---|
| Camera | medium shot → close-up | facial proportion shifts |
| Lighting | soft key → hard side light | skin tone and shadow depth |
| Wardrobe | jacket on → jacket off | hairline and shoulder width |
| Motion | static → walking | body proportions |
| Duration | 5s → 10s | late-clip identity melt |
Longer clips are a special case. Many models maintain identity well for the first few seconds and then slowly morph as the motion compounds. If a ten-second shot drifts at second seven, either shorten the shot and cut around it, or split it into two generations that share the same anchor frame.
Also standardize your lighting language. "Soft key at 45 degrees, warm practicals in background, shallow depth of field" is reusable across every scene. "Beautiful cinematic lighting" is not — it means something different in every generation, and that variability leaks into the face.
Step 6: Keep Your Character Portable Between Tools
No single generation tool will be best at every shot forever. Wide establishing shots, dialogue close-ups, and action beats often call for different strengths. Plan for portability from the start by maintaining a plain-text character bible that contains:
- The locked descriptor block and any permitted variants.
- A list of reference images with file names and version.
- Known-good seeds and settings per tool, if the tool supports seeds.
- Notes on which tools have produced the best results for which shot types.
- A short style paragraph describing the overall look of the project.
When you move a character into a new tool, expect to rewrite the descriptor block slightly. Vocabulary is model-specific: some respond to "high cheekbones," others to "angular face," and the difference matters. What should not change is the underlying identity data. The references and the attribute list stay fixed; only the phrasing adapts.
Run a single test shot in the new tool before committing a scene to it. Twenty minutes of calibration beats regenerating forty shots later.
Step 7: Run a Frame-Level Quality Check Before Editing
Do not hand a sequence to an editor until you have checked it against the identity sheet. A contact sheet — a grid of one frame from every shot — makes drift obvious in seconds. Things to verify:
- Hairline and parting on the character's left side.
- Eye colour and spacing under different lighting.
- Jaw and chin shape in profile shots.
- Identifying marks present and in the correct location.
- Wardrobe details — collar type, button count, sleeve length.
- Height relationships when two characters share the frame.
- Hand and finger geometry, which fails more often than faces and is more distracting when it does.
Mark each shot as pass, minor fix, or regenerate. Fix minor issues with targeted inpainting or a corrective close-up rather than regenerating an otherwise good shot; regeneration is a lottery ticket for identity drift.
Troubleshooting the Failures You Will Actually Hit
The face changes when the camera angle changes. This usually means your reference set lacks the angle you are asking for. Add a profile and a three-quarter reference, then regenerate.
Identity holds but the character ages. Age drift typically comes from lighting descriptors. Harsh overhead light with deep shadows reads as older; soft frontal light reads as younger. Lock your lighting language and the effect stabilizes.
Wardrobe swaps mid-scene. Describe clothing in the same position in the prompt every time, and include it in the reference image rather than only in text.
Two characters merge visually. Separate them clearly in the shot list — different heights, different palettes, different screen sides — and generate them at different prompt positions consistently.
Background elements bleed into the character. Reduce style reference weight, or generate the background separately and composite. Heavy style references sometimes transfer texture onto skin.
Everything looks right but the sequence feels wrong. Check eyelines and screen direction. Consistency of face without consistency of blocking still reads as a broken scene.
Scaling the Workflow to Episodic Content
Once the system works for one short video, the question becomes whether it survives a series. It does, if you treat assets like production data rather than throwaway files.
Use a folder structure that mirrors the story: project/cast/, project/locations/, project/scenes/s01/, project/scenes/s02/. Inside each scene folder, keep the anchor frame, the shot list, the prompts used, and a short note about which generations were approved and why. When a character appears in episode nine, you want the episode two anchor frame in front of you, not a vague memory of it.
A lightweight database or even a structured spreadsheet helps once you pass twenty shots: columns for shot ID, scene, character, anchor used, tool, settings, status. It sounds bureaucratic for a creative project, but it converts consistency from luck into something you can reproduce on demand — and it makes handoffs to collaborators possible without a two-hour briefing.
Finally, budget time for re-anchoring. In episodic work, roughly one generation in five becomes an anchor. That ratio is normal. It is not wasted effort; it is the cost of a character the audience recognizes.
FAQ
How many reference images does a character actually need?
Six is the practical minimum: front, both three-quarters, profile, full body, and one expression variant. More helps only if the additional images add new angles or wardrobe states. Ten near-identical front shots add nothing.
Can I reuse a character across different projects?
Yes, if you keep the descriptor block and reference set intact. Treat the character as a reusable asset and version it. Just be aware that a change in overall project style may require regenerating the identity sheet in that new style.
Do I need a seed to keep a face stable?
Seeds help reproducibility within a single tool, but they are not portable between tools and they do not guarantee identity across different prompts. Use seeds as a convenience, not as your consistency strategy.
Why does the character look right in stills but wrong in motion?
Motion adds temporal compounding. Identities that hold in a static frame can drift over several seconds of movement. Shorten shots, split long takes, and verify identity in the middle and end of each clip — not just the first frame.
Is it better to generate in the final style from the start?
Almost always yes. Generating photoreal and stylizing afterwards costs you facial detail, and stylizing mid-pipeline tends to homogenize distinguishing features.
How do I handle a character who must appear at different ages?
Build two identity sheets with a shared base — same bone structure, same marks — and adjust age cues through hair, posture, wardrobe, and lighting language. Lock each age block separately and never mix them within one scene.
What is the single biggest time saver?
The locked descriptor block. Pasting an identical forty-word identity paragraph into every prompt removes the most common source of drift and takes ten minutes to set up.
