Why identity drift is the real bottleneck in AI video
Anyone who has shipped an AI-generated video knows the pattern. The opening shot looks fantastic: the face is right, the lighting is right, the wardrobe matches the script. Twenty shots later, the same character has a slightly different nose, a different jawline, and a hairline that has migrated half an inch. By the final scene, the protagonist looks like a close relative rather than the same person.
This is identity drift, and it is the single biggest reason AI video projects get abandoned halfway through. It is not a rendering quality problem. Modern video models produce beautiful frames. It is a continuity problem, and continuity is what separates a demo from something an audience will actually watch to the end.
The causes are structural, not accidental. Text-to-video models interpret a prompt probabilistically, so every generation samples a slightly different point in the model's learned distribution of "a person who looks like this." Camera angle changes shift the geometry. Lighting changes shift skin tone. Motion blur and compression soften the micro-features that make a face recognizable. And stylization, which is often the whole point of an AI video, pushes the model away from photorealism and toward its own aesthetic priors.
Multi-image fusion is the practical answer to this. Instead of handing the model a single reference image and hoping for the best, you build a small, deliberate reference set and let the model fuse identity signals from multiple angles, expressions, and lighting conditions into one stable character concept. This guide walks through how that works, how to build the reference set, and how to run a scene-to-scene workflow that holds up over dozens of shots.
How multi-image fusion actually works
From a single reference to a reference stack
A single-image reference gives the model one data point. It can see the front of a face and infer almost nothing reliable about the profile, the back of the head, the way the fabric of a jacket falls, or how the character looks under warm versus cool light. When the next shot requires a three-quarter turn, the model invents what it cannot see, and invention is where drift begins.
Multi-image fusion changes the input. You supply several images of the same character, and the pipeline extracts identity information from all of them, then conditions generation on the combined signal. Depending on the tool, this happens in one of a few ways:
- Cross-attention conditioning. Reference images are encoded into embeddings that are attended to during sampling, similar in spirit to adapter techniques used in image models. Multiple references average out noise in any single image.
- Identity embeddings. A face or subject encoder produces a numeric identity vector. Several references produce a tighter, more stable cluster than one.
- Lightweight fine-tuning. Some workflows train a small adapter on a handful of reference images, which bakes identity into the model rather than into the prompt.
- Latent concatenation. References are stitched into the generation canvas at the latent level, guiding structure as well as appearance.
You do not need to know which mechanism your tool uses. What matters is a practical consequence: more consistent references in, more consistent character out.
The three signals worth stacking
Not every image in a reference set does the same job. It helps to think in three stacks:
- Identity stack — face and body features. Front, three-quarter left, three-quarter right, profile, back, and at least one close-up.
- Style stack — frames that show the target look, whether that is photoreal, painterly, cel-shaded, or something stranger. Style references stop the identity from warping when the aesthetic shifts.
- State stack — the character in different emotional and physical states: neutral, smiling, angry, wet hair, running, injured. These keep a face stable when the performance changes.
Most drift complaints come from an identity-only reference set. The model holds the face but loses the wardrobe, or holds the wardrobe but forgets what the character looks like when they are not smiling.
Building a character bible before you generate a frame
The six-angle reference set
Before you open a video model, assemble the reference library. Six angles is the practical minimum for a character who will be seen in more than a couple of shots:
- Straight-on front, neutral expression, even lighting
- Three-quarter left
- Three-quarter right
- Full profile
- Back view, including hair from behind
- Tight close-up of the face
If you cannot generate these from scratch, generate them with an image model first, approve them, and treat them as canon. Consistency starts upstream of video. A sloppy character sheet guarantees a sloppy sequence.
Locking wardrobe, palette, and props
Write down the non-negotiables as literal specs rather than adjectives. "Dark navy wool coat" is weak. "Navy wool coat, #1B2A44, double-breasted, brass buttons, calf-length" is strong. Note hair color, hair length, eye color, skin tone descriptors, distinguishing marks, accessories, and any prop the character carries.
Color is where drift is most visible to audiences, even when they cannot articulate why something feels off. Keep a small palette per character, ideally three to five hex codes, and reuse it across every scene description.
Writing the identity card
The identity card is a short, stable block of text you paste into every prompt. It should never change between scenes of the same character. A typical card reads like this:
Female protagonist, late twenties, East Asian, oval face, straight black hair to mid-back with blunt fringe, dark brown eyes, small scar on left eyebrow, medium build. Wearing charcoal turtleneck, olive field jacket, dark denim.
Notice that nothing in the card describes mood, location, or camera. Those belong to the scene block. Mixing them is one of the fastest ways to introduce drift, because a mood word like "furious" in the identity card will get re-rolled differently in every shot.
A practical scene-to-scene workflow
Step 1: generate and approve a hero frame
Generate a single, well-lit, front-facing shot of the character. Iterate until it is unmistakably right. This hero frame becomes the anchor reference for everything downstream and the visual standard you compare every later shot against.
Do not skip approval. If the hero frame is 80 percent right, every subsequent shot inherits the missing 20 percent and amplifies it.
Step 2: keyframe the scene beats, not every shot
You do not need to generate every frame with full reference conditioning. Identify the beats where the character's identity is most exposed: close-ups, direct address to camera, profile turns, dramatic lighting changes, and entrances. Generate those as keyframes with the full reference stack.
For transitional or wide shots where the character is small in frame, you can relax to a lighter reference set and save generation time. The audience reads continuity from the close-ups, not from a figure twenty meters away.
Step 3: batch-generate with a consistent seed family
When a tool supports seeds, reuse a small set of seeds across a scene rather than randomizing per shot. Seeds are not magic, but a consistent seed family reduces the compositional churn that pushes identity around. Combined with an unchanged identity card, this turns generation from a lottery into a controlled process.
Work in batches of five to eight shots. Reviewing a smaller batch keeps your eye calibrated; if you generate fifty shots before reviewing, your standard will drift along with the character.
Step 4: review with a continuity checklist
Before accepting a shot, check it against the hero frame side by side, not from memory. Compare face shape, hairline, eye spacing, wardrobe details, and color temperature. Reject anything that reads as "a different person" or "the same person in a different show."
Regenerating one shot is cheap. Repairing an entire sequence after the fact is not.
Prompt patterns that change the scene without changing the face
A consistent prompt structure does more for continuity than any single setting. Use a fixed order and only edit the scene block:
Identity card (never changes) → wardrobe lock (rarely changes) → scene and action (changes every shot) → camera and lens (changes per shot) → lighting (changes per scene) → negative constraints (stable).
A few rules that pay off immediately:
- Do not re-describe the face in new words each time. Every new adjective is a new instruction. "Sharp cheekbones" in shot three and "sculpted features" in shot nine push the model in two directions.
- Vary one axis at a time. Change the location first, keep the camera identical, then change the camera. Two changes at once make it impossible to diagnose what caused drift.
- Describe light in technical terms. "Low-key side light from camera left, 4300K" is more reproducible than "moody lighting."
- Keep negatives consistent. If you ban glasses in one shot, ban them in all of them.
- State what stays the same. Some models respond well to explicit instructions such as "same character as the reference, identical facial structure."
A compact example, with the scene block swapped between shots, looks like this:
[identity card] + [navy coat spec] + "standing on a rain-slick platform at night, looking down at a phone" + "medium close-up, 50mm, shallow depth of field" + "cool practical light from signage, 4300K" + "no hats, no glasses, no text"
Then, for the next shot, keep everything before and after the scene block untouched and change only the action, location, camera, and light.
Hard cases: profiles, motion, stylized looks, and crowds
Some shots break consistency more than others, and they deserve special handling.
Profile and back-of-head shots. These fail when the reference set lacks the right angles. This is exactly why the six-angle set matters. If you cannot produce a back view, generate one from your hero frame using an image model and add it to the stack.
Fast motion. Motion blur and limb occlusion hide identity cues. For action sequences, generate a clean keyframe of the pose and use it as a structural reference, so the model has an identity anchor plus a pose anchor rather than only a prompt.
Heavy stylization. When the art direction is painterly or anime-influenced, identity lives in shape language rather than pores. Add at least two stylized references of the same character to the style stack; otherwise the model applies its generic style and the face normalizes toward the average.
Crowds and groups. With more than one character in frame, reference conflict is common. Either condition on the primary character and let secondaries be generic, or generate each character separately and composite. Trying to fully condition three characters in one generation usually produces three half-characters.
Aging, injury, and wardrobe changes. These are intentional changes, not drift. Document them in the character sheet as scene-specific overrides so you do not accidentally regenerate the "normal" version in the middle of an arc.
Choosing and evaluating a multi-image workflow
Not every tool handles reference stacking equally well. When you evaluate options, test with the same character sheet and the same three shots so the comparison is fair. The criteria that actually matter:
- How many references are accepted, and whether weights are adjustable. Four to six usable references with per-image weighting covers most production needs.
- Pose and structure control. Keyframe or pose conditioning is what keeps body language consistent, not just the face.
- Seed reproducibility. Without it, you cannot re-run a good result or debug a bad one.
- Batch behavior. Consistent output across a batch of ten matters more than one spectacular frame.
- Output resolution and artifact handling. Faces degrade fast at low resolution, and degraded faces drift.
- Export and editing fit. You want clean frames and clips that drop into an editor without conversion gymnastics.
- Rights and ownership terms. Confirm what you can publish commercially and how the platform treats your inputs.
A practical test: generate the same character in a bright interior, a night exterior, and a tight close-up. If all three read as the same person without retouching, the workflow is production-ready. If two of three drift, you have a reference-set problem more often than a model problem.
Common mistakes and fast fixes
- Too few references. Fix: build the six-angle set before anything else.
- Mixing resolution and lighting wildly across references. Fix: normalize references to similar framing and light before stacking them.
- Rewriting the identity card between shots. Fix: freeze it in a text file and paste, never retype.
- Chasing a perfect frame instead of a consistent set. Fix: judge shots against the hero frame, not against your imagination.
- Generating too many shots before review. Fix: batch in fives and eights.
- Ignoring color drift. Fix: compare hue values, not just faces; a skin tone shift of a few hundred Kelvin reads as a different person.
- Over-conditioning. Fix: if the output looks stiff or copied, reduce reference weights rather than adding more references.
Quality control: the shot approval checklist
Run every shot through the same five questions:
- Is the face recognizable as the hero-frame character at a glance?
- Do hairline, eye spacing, and jaw silhouette match?
- Is the wardrobe spec intact, including color and details?
- Does the lighting sit in the same world as the neighboring shots?
- Would an audience member notice a change if this shot were cut next to the previous one?
If any answer is no, regenerate before moving on. Keep a simple continuity log with shot number, references used, seed, and any accepted deviations. It takes a minute per shot and saves hours of rework.
FAQ
How many reference images do I actually need?
Four is workable, six is comfortable, and eight to ten only helps if the extra images add new angles or states. Duplicates do not improve fusion.
Can I fix drift after generation?
Partially. Face restoration and color matching can rescue a near-miss shot, but they cannot recreate a character who was never conditioned properly. Prevention is faster.
Does multi-image fusion work for non-human characters?
Yes, and it often works better. Creatures, robots, and stylized designs rely on silhouette and surface detail rather than facial micro-features, so a clean set of orthographic-style references is highly effective.
Why does the character change most when the lighting changes?
Lighting alters shadow structure, and shadow structure is a large part of how we recognize a face. Add at least one reference of your character in warm light and one in cool light so the model has seen both.
Should I train a custom model or use reference conditioning?
Reference conditioning is faster to iterate and easier to adjust. Training pays off when you have a long-running series, a large approved image set, and a need for speed across many shots.
How do I keep continuity across separate sessions?
Store the character sheet, the exact reference images, the seed family, and the prompt template together as a project asset. Continuity across days is a documentation problem more than a technical one.
The takeaway
Character consistency is not a single setting you toggle. It is a pipeline: a locked character sheet, a multi-angle reference set, a stable prompt structure, disciplined batching, and a review checklist you actually use. Multi-image fusion is the engine at the center of that pipeline, because it gives the model enough evidence to keep the same person in every frame instead of guessing.
Start small. Build one character bible, generate one hero frame, then push that character through three deliberately difficult shots. Once those hold, scale the same process across the sequence. The teams that ship watchable AI video are rarely using secret tools; they are simply refusing to let the identity drift.
