Why Character Consistency Breaks First in AI Video
Anyone who has generated more than a handful of clips has met the same disappointment: the first shot is gorgeous, the second is almost the same person, and by the fifth you are looking at a stranger wearing the same haircut. Faces are the most information-dense part of any frame, and viewers are biologically tuned to notice when one changes. A slightly different nose bridge, a jawline that drifts two millimeters, eyes that shift from hazel to green — the audience may not be able to name the problem, but they will feel that two different people are cutting back and forth.
Multi-image fusion is the family of techniques that fixes this. Instead of describing a character in words and hoping the model lands in the same place every time, you supply several images of the same person and let the system blend their identity cues into a single reusable reference. Words still matter, but they stop carrying the entire burden of identity.
This guide covers how fusion works, how to build reference sets that actually help, how to structure a shoot so continuity holds across dozens of shots, and how to repair drift when it appears.
How Multi-Image Fusion Works Under the Hood
Identity versus structure
Generative video models respond to two broad categories of conditioning. Identity conditioning answers "who is this?" — facial geometry, skin tone, hairline, age, the small asymmetries that make a person recognizable. Structure conditioning answers "how is this framed?" — pose, camera angle, body position, depth of field.
A single reference image is weak on identity because it tangles identity and structure together. Give the model one portrait of someone looking left in a red sweater and it may treat "looking left" and "red sweater" as part of who the person is. Multi-image fusion disentangles the two by averaging across references: supply the same face front-on, in profile, in three-quarter view, under different light, and the identity signal survives while incidental details cancel out.
Reference slots and weighting
Most fusion interfaces expose two or three reference slots plus an influence slider. Practical rules:
- Slot one should be the cleanest, most neutral image: even lighting, no extreme expression, eyes visible, face filling roughly a third of the frame.
- Slot two should add dimensionality — a three-quarter or profile view.
- Slot three, if available, adds an expression or costume state you need repeatedly.
- Keep total identity weight moderate. Pushing it to maximum often freezes the face into a mask that resists acting.
Where drift actually begins
Drift rarely starts at the face. It starts at scale, angle, and lighting. When the camera moves from a wide to a close-up, the model must invent facial detail it never rendered, and it invents that detail from the prompt. When lighting shifts from soft daylight to hard practicals, skin tones move. When a character turns away and turns back, the face is re-rolled. Fusion references shrink each re-roll, but they do not eliminate it — which is why workflow discipline matters as much as the tool.
Building a Reference Set That Fusion Can Read
A reference set is not a photo dump. Six to twelve carefully chosen images will beat forty random ones.
The minimum viable set
- A neutral front-on portrait, mouth closed, no strong expression.
- A three-quarter view.
- A profile view, both sides if possible.
- A half-body shot to establish proportions and shoulder width.
- A full-body shot to lock height and build.
- One expression image if the story needs it — laughing, angry, exhausted.
- Two or three costume variations if wardrobe changes across the film.
Technical quality rules
Resolution should be high enough that the eyes contain real detail — roughly 1024 pixels on the short side at minimum. Avoid motion blur, heavy filters, beauty retouching, and strong color grading; the model will bake those artifacts into the identity. Keep backgrounds plain and consistent. Avoid sunglasses, hats, or hands covering the face in your primary slots.
Mistakes that quietly wreck a set
- Using the same photo at different crops and calling it a reference set. Duplicates add no information and over-weight a single angle.
- Mixing ages. Include a photo from ten years ago and expect an averaged face that looks like neither version.
- Including wildly different lighting temperatures. The model may read the color shift as skin variation.
- Forgetting body shots, then wondering why the character's height changes between scenes.
A Shot-First Workflow for Continuity
Consistency is cheaper to design than to repair. Before generating anything, write a shot list with three columns: shot number, framing, and character state.
Step 1: Lock the story beats
Divide the film into scenes, and each scene into shots of eight to twelve seconds. Short shots are easier to keep consistent because each one contains fewer opportunities for the face to change.
Step 2: Batch shots by setup
Reorganize the list so all shots of the same character, in the same location and light, are generated together. If you render a close-up, then a wide, then return to a close-up an hour later, you will forget which reference weights and settings worked. Group by setup.
Step 3: Generate hero frames first
Before any motion, generate a still for each shot with fusion enabled. Compare them side by side on a contact sheet. Fix identity mismatches at the still stage — it costs a fraction of what it costs to regenerate a moving clip.
Step 4: Animate from approved keyframes
Most reliable pipelines animate from an approved frame rather than generating text-to-video from scratch. The still carries identity into the video model, and the fusion reference keeps it from drifting as the character moves.
Step 5: Keep a continuity ledger
Record the seed, reference set, fusion weight, prompt version, and any style adapter used for every approved shot. When shot 34 drifts, you can diff its settings against shot 12 and find the culprit in minutes instead of hours.
Prompting for Identity Stability
Describe behavior, not appearance
If fusion handles identity, spend your prompt words on action and camera. "She turns from the window, picks up the keys, and looks back over her shoulder" gives the model motion. Repeating "a woman with brown hair and green eyes" in every prompt adds noise and invites the text encoder to fight the image reference.
Anchor the environment once
Write a compact scene block — location, time of day, light quality, lens — and reuse it verbatim. Rewording the environment between shots is one of the most common causes of unintended visual jumps.
Keep motion verbs modest
Extreme motion such as running, spinning, or falling forces the model to synthesize large areas of new facial geometry. If a shot must include violent movement, cut it shorter and keep the face small in frame. Negative prompts are best reserved for artifacts — extra fingers, warped teeth, plastic skin, mismatched eye color — rather than for identity, since what you exclude does not tell the model who to render.
Mixing Models Without Losing the Face
Different models have different strengths: some handle faces better in close-up, others handle physics and camera movement better, and stylized work sits somewhere else entirely. Mixing them is normal, but each encodes identity differently.
The reference-image bridge
The most portable approach is to treat the approved keyframe as the source of truth and feed it into every model as an image-to-video input. Identity travels with the pixels rather than with a model-specific embedding.
Style adapters and personalization layers
If a model supports a trained personalization layer, build it from fifteen to thirty curated images of the same character and use it alongside fusion rather than instead of it. The layer handles identity; fusion handles per-shot framing and expression.
Matching output between models
Two models will not produce identical skin texture. Bridge the gap in post: apply shared film grain, a shared color grade, and the same sharpen-and-denoise pass to every shot. A unified grade hides small inconsistencies that would be glaring in raw output.
Wardrobe, Voice, and Performance Continuity
A face is only part of a person. Costume and voice continuity are usually neglected, and both show.
Wardrobe as a reference slot
Silhouette reads at a distance, so keep one slot for the costume — jacket shape, bag, specific boots. Generate a small wardrobe sheet with front, back, and detail views of each outfit, and attach the relevant one to shots in that scene. Props matter too: if a character always carries a mug or wears a locket, that object belongs in the reference images, or the model will treat it as optional detail and drop it in wide shots.
Voice reference clips
If your pipeline supports voice cloning, collect twenty to sixty seconds of clean speech for the character: consistent microphone distance, no background music, natural pacing. Generate all dialogue for that character from the same voice profile, then vary only emotional delivery. When animating dialogue, favor shots where the mouth is visible enough to sync; extreme profile shots and heavy occlusion make lip sync unreliable, so consider cutting away to a listener instead.
Performance continuity across shots
Performance includes posture, gesture speed, and eye-line. Note these in your continuity ledger next to each shot. "Talks with hands, leans forward" is easy to keep; "sad" is not.
A Continuity Audit Checklist
Run this after every generation batch, before moving on:
- Face check. Place the new still beside a known-good still at 100% zoom. Compare eye spacing, nose width, jaw shape, hairline.
- Scale check. Is the character the same height relative to fixed objects — doors, chairs, other characters?
- Wardrobe check. Same layers, fastenings, and colors within the scene.
- Light check. Does the light direction on the face match the establishment shot?
- Color check. Skin tone consistent across the scene when viewed in sequence.
- Motion check. Does the animation preserve the face during the first and last frames? Ends are where drift usually appears.
- Ledger check. Settings recorded for every approved shot.
Auditing at batch level rather than shot level is what keeps momentum. Fixing a drifting face inside a fifteen-shot scene is far easier than discovering the problem after sixty shots have been rendered.
Troubleshooting Common Drift Problems
The face changes between the first and last frame of a clip. Use the approved still as the first frame and, if the tool allows, as the last-frame reference too. Otherwise shorten the clip.
The character looks like a blend of two people. Your reference set likely includes two visually similar people, or one image is far higher resolution than the others and is dominating. Trim to one person and normalize resolution.
The face is frozen and emotionless. Identity weight is too high, or the reference set is entirely neutral expressions. Add an expression reference and lower the weight.
Skin looks plastic. Reference images are over-retouched. Add a natural, unedited photo and reduce denoise strength in post.
Eyes change color between shots. Encode eye color in the prompt block for close-ups and add mismatched eye color to negatives. Better still, lock it in the reference set.
Everything matches except the hairline. This is often a lighting artifact. Compare the reference set's lighting against the scene lighting and add a reference shot captured under similar light.
FAQ
How many reference images do I need? Six to twelve curated images cover most needs. More only helps if each image adds a genuinely new angle, expression, or costume.
Can I get consistency without multi-image fusion? Partially. A trained personalization layer or a locked seed with a fixed prompt can hold a character across a few shots. Across a full film, fusion or a keyframe-first pipeline is what keeps results stable.
Do I need the same seed for every shot? Not necessarily. Once image references are in play, seeds affect composition more than identity. Consistency of references matters more than consistency of seed.
Should I fix a drifting face in generation or in post? Generate first. Post-production face replacement works but softens skin detail and rarely matches lighting. Treat it as a rescue tool, not a plan.
Does fusion work for stylized or animated characters? Yes, and often better than for photoreal faces, because there are fewer subtle details available to drift.
How do I keep a crowd scene consistent? Treat only the two or three characters the camera follows as fusion subjects. Background figures can be generic; viewers track the ones with dialogue.
Can I reuse a reference set across projects? Yes, and you should. A library of well-built character sets is the single biggest time saver in long-form AI video work.
Start small. Pick one character, build a nine-image reference set, generate a five-shot scene, and audit it. The workflow you build on that short test will scale to a full film far more reliably than any single setting you tweak on a fifty-shot timeline.
