Why Characters Drift Between Shots
Anyone who has produced more than a handful of AI video shots has run into the same wall. The first clip looks extraordinary: a believable face, natural lighting, a costume that reads exactly as you imagined. Then you generate the next shot, and the person is subtly — or catastrophically — someone else. The jawline softens, the eye colour shifts, the jacket changes cut, and the hairline migrates. By shot six you are no longer telling a story about a character; you are telling a story about a family of near-identical strangers.
This is not a bug in a single tool. It is a structural property of how most generative video pipelines work. A text-to-video model has no persistent memory of a person. Each generation starts from noise and is steered by a prompt plus, at best, a seed. Nothing in that process guarantees that the entity described as "a woman in her thirties with auburn hair and a green canvas jacket" will be the same entity next time. The model is not remembering; it is re-imagining from scratch, and small sampling differences compound into visible identity drift.
The practical costs are real. Reshoots burn time. Editors end up cropping faces out of frame to hide inconsistencies. Voice-over and dialogue scenes become impossible because the mouth shapes no longer match the same person. Long-form narrative — the format where AI video would be most valuable — becomes the hardest format to produce.
Multi-image fusion is the technique that addresses this directly. Instead of describing a character in words or handing the model a single portrait, you supply a set of reference images and let the system build a fused identity representation from them. The rest of this guide covers what that means in practice, how to prepare references, where the approach still fails, and how to fold it into a repeatable production workflow.
What Multi-Image Fusion Actually Does
The core idea is simple to state and fiddly to execute: identity is better represented by a distribution of appearances than by one photograph. A single reference image forces the model to copy that image's specific lighting, angle, expression, and colour cast along with the person. Give it twelve images of the same person across different angles, lighting conditions, and moods, and the model can begin to separate what is constant about them — bone structure, proportions, distinguishing features — from what is incidental — the shadow under their nose, the blue tint of a window-lit room.
Mechanically, most modern systems implement this through some combination of reference conditioning and attention injection. The reference images are encoded into feature vectors. Those vectors are then inserted into the generation process at specific layers so that attention heads can consult them while denoising. When several references are supplied, the encoder either averages them in a learned latent space or maintains separate slots the model can attend to selectively, depending on which one is most relevant to the current view.
The practical consequence is a spectrum. At one end, weak fusion behaves almost like a single reference: the model locks onto whichever image is most similar to the target pose. At the other end, strong fusion produces a stable identity that survives changes in framing, wardrobe, and lighting — which is what narrative video requires.
It is worth distinguishing fusion from two neighbouring techniques. Face swapping replaces a face after generation and often produces a rubbery seam between head and body. Fine-tuning on an individual bakes identity into model weights but is expensive, slow to iterate, and hard to combine with other characters in one scene. Multi-image fusion sits in the middle: faster than training, more coherent than swapping, and flexible enough to handle several characters in the same frame.
Building a Reference Set That Actually Works
Quality of references determines quality of output more than any parameter you will tune later. Most disappointing results trace back to a weak reference set, not to the model.
Angle coverage
Aim for a full rotational sweep of the head: front, three-quarter left, three-quarter right, profile left, profile right, and at least one slightly elevated and one slightly below-eye-level view. Models learn three-dimensional structure from parallax-like variation. If every reference is a frontal headshot, the identity will collapse the moment you rotate the camera.
Lighting and colour variety
Include references shot in warm indoor light, cool daylight, and flat overcast conditions. This teaches the encoder to ignore colour temperature. If all references share the same golden-hour glow, your character will look amber in every scene, including a sterile laboratory.
Expression range
Neutral, smiling, speaking mid-sentence, and one with eyes partly closed. Expression variety prevents the model from welding a permanent half-smile onto the face, which is a common artefact of single-reference workflows.
What to exclude
Remove heavy filters, beauty retouching, strong Instagram-style colour grading, sunglasses, masks, and extreme motion blur. Also remove any image where the subject is the smallest element in the frame — a reference where the face occupies 40 pixels contributes noise, not information. Aim for at least 512 pixels across the face region, ideally more.
A practical target is eight to fifteen references for a primary character and six to ten for a supporting character who appears in fewer shots. More is not automatically better; twenty near-duplicate frames from the same photoshoot can bias the fusion toward one lighting condition and actually reduce flexibility.
Style Locking: Keeping More Than Faces Stable
Character consistency is only half the problem. Audiences also notice when the film's look changes between cuts. A scene that is desaturated and grainy followed by one that is glossy and high-contrast reads as amateur, even if the protagonist's face is flawless.
The same fusion logic extends to style. Supply a small set of style references — frames from a previous shot in the same project, or a mood board with consistent colour, contrast, and grain — alongside your character references. Keep the style set separate from the identity set so you can weight them independently. When you need to move from a warm interior to a cold exterior, adjust the style weight rather than rewriting your entire prompt, which risks disturbing the identity conditioning.
In practice, three to five style references are enough. Beyond that, you are usually repeating yourself, and repeated style frames can overpower scene-specific lighting you actually want.
A Step-by-Step Multi-Image Fusion Workflow
The following sequence works for narrative shorts, explainer videos, and product storytelling alike. It assumes you have access to a generation tool that accepts multiple reference images with adjustable influence.
Step 1: Lock the shot list before generating anything
Write every shot as a one-line description: framing, action, location, light direction, and emotional beat. This matters because fusion pipelines encourage experimentation, and experimentation without a shot list produces a pile of beautiful but unusable clips. A shot list also tells you which angles you will need references for.
Step 2: Assemble and tag references
Create a folder per character with references named by angle and lighting: ava_front_neutral_day.jpg, ava_threequarter_left_warm.jpg. Tagging sounds tedious and saves hours later, because you will want to swap out a single problematic reference without breaking everything else.
Step 3: Write prompts that describe change, not identity
This is the single most common mistake. Once identity comes from references, your prompt should describe what is different about this shot — camera movement, action, environment, mood — and should avoid re-describing the character's physical appearance in detail. Long descriptions of hair and eye colour compete with the reference conditioning and can pull the result toward the generic prompt interpretation.
A workable structure is: subject reference slot, action, environment, camera, light, style slot. Keep it to one action per shot. Layered actions such as "turns, then walks, then sits" fragment the identity across a single clip.
Step 4: Lock seeds and reusable style blocks
For a shot that works, record the seed, the reference set, the reference weights, and the prompt. Reuse that exact combination for reverse angles of the same moment. If your tool supports saved style presets or character presets, build them now rather than re-entering values shot by shot.
Step 5: Generate in passes and review at thumbnail scale
Generate three to five variations per shot, then review them as a contact sheet at small size before looking at any clip full-screen. Identity drift is usually visible at thumbnail scale — an eye spacing that is subtly off, a nose that is too narrow — and this pass saves you from falling in love with a clip that does not actually match.
Step 6: Assemble early, fix late
Cut the sequence together as soon as you have one usable clip per shot. Problems invisible in isolation — colour jumps, inconsistent pace, a character who reads younger in shot nine — appear immediately in sequence. Then regenerate only the failing shots, keeping everything else untouched.
Prompt Patterns and Parameter Decisions
How many references to supply
Start with four to six strong references for a first test. If identity destabilises across angles, add more angle coverage. If the character looks over-fitted to one photo, you likely have too many near-duplicates and should diversify instead of adding volume.
Balancing identity against scene
Most tools expose some form of reference strength. High strength gives faithful identity but resists the scene lighting and can produce a pasted-in look. Lower strength integrates better but drifts. A useful habit is to start at a moderate value, generate a comparison grid at low, medium, and high, and choose from evidence rather than from a default.
Negative prompts
Negative prompts are useful for structural problems: extra limbs, warped hands, text artefacts, watermarks, blur. They are much less useful for identity. Trying to fix a drifting face with a negative prompt such as "different person" rarely helps, because the model has no mechanism for reasoning about that instruction. Fix identity with references and strength, not with negation.
Motion prompts
Keep camera language specific and modest. "Slow push in" beats "dynamic cinematic movement" because the latter invites the model to improvise, and improvisation is where identity degrades. For dialogue, prefer locked-off or gently drifting shots; rapid motion reduces the effective resolution available to render facial detail.
Resolution and aspect ratio
Match reference aspect ratio to output aspect ratio where possible. Cropping a 16:9 reference into a 9:16 vertical output forces the encoder to guess at the sides of the head, which is a frequent source of distorted hairlines in vertical formats.
Troubleshooting Common Failures
| Symptom | Likely cause | Fix |
|---|---|---|
| Face changes shape between cuts | Insufficient angle coverage in references | Add profile and three-quarter views |
| Character looks identical in every shot | Over-fitted references or excessive strength | Diversify references, lower strength, vary expression |
| Wardrobe changes unexpectedly | Costume described in prompt rather than shown | Show the costume in a reference image instead |
| Character looks pasted onto the background | Reference strength too high, scene light ignored | Reduce identity weight, add scene lighting detail |
| Colour shifts between shots | No style references | Add three to five consistent style frames |
| Skin looks plastic | Reference images over-retouched | Replace with unretouched photography |
| Identity holds but expression is frozen | References all share one expression | Add smiling, speaking, and neutral variants |
| Details dissolve in fast motion | Too much movement per second | Shorten action, slow camera, split into two shots |
A second class of failure is subtler: the character is consistent but wrong. This usually means the fused identity has averaged in a feature from a stray reference — a sibling who appears in one photo, a different hairstyle from an old shoot. Audit the reference folder whenever the output feels slightly off-model.
Choosing Tools and Building Your Stack
Not all video generators handle multi-image reference conditioning equally, and the difference matters more than raw visual quality. When evaluating options, weigh these criteria:
- Maximum reference count. Can you supply eight or more images, or are you limited to one or two?
- Adjustable influence. Is there a strength control, or is conditioning all-or-nothing?
- Per-subject slots. Can several characters each carry their own reference set in one shot?
- Shot length and coherence. Longer single generations reduce the number of joins where drift accumulates.
- Motion realism. Identity is worthless if the walk cycle looks like a puppet.
- Aspect ratio flexibility. Vertical, square, and widescreen without quality loss.
- Iteration speed. Fast turnarounds let you test reference combinations rather than guessing.
- Export and resolution. Enough pixels for your final delivery format, with clean alpha or clean backgrounds if you plan to composite.
A realistic stack combines more than one tool: a strong identity-conditioned generator for character shots, a capable image model for building reference sheets in the first place, and a standard editor for stitching, colour matching, and sound. Many creators also keep an upscaler in the loop, since regeneration at higher resolution after a shot is approved is cheaper and more predictable than trying to get everything right in one pass.
Scaling Consistency Across a Series
Once a single scene works, the challenge shifts from technique to process. Consistency across twenty scenes is a logistics problem.
Build a character bible: a folder containing the approved reference set, a written description for internal use, key wardrobe variants, and a record of the seed and strength values that produced approved shots. Treat it as a living document; when you discover a better reference, version it.
Standardise naming. Every asset should encode project, character, shot, and version, so that a file named out of context is still identifiable.
Create shot templates for recurring setups — the same desk, the same street corner, the same doorway — with prompt blocks you can paste and adjust. Templates reduce the temptation to freehand prompts, which is where inconsistencies creep in.
Introduce review gates. Before approving a batch, check identity, wardrobe, colour, and motion against the previous approved batch, not just against the prompt. A short checklist beats a long memory.
Finally, budget for regeneration. Even mature pipelines produce shots that need a second attempt. Plan for a regeneration rate of roughly one in four shots rather than assuming a clean run.
Frequently Asked Questions
Does multi-image fusion require training a custom model?
Usually not. Reference-conditioned generation inserts identity information at inference time, so you can switch characters between projects without retraining. Training an individual model can still improve fidelity for a flagship character, but it is rarely the starting point.
How many reference images is ideal?
Around eight to fifteen for a lead character with heavy screen time, and four to eight for a supporting role. Diversity across angles, lighting, and expression matters more than raw count.
Can I keep two characters consistent in the same shot?
Yes, if the tool supports per-subject reference slots. Keep the two reference sets clearly separated and describe their positions and interaction in the prompt rather than re-describing their faces.
Why does the character look right but the scene look wrong?
You are likely over-weighting identity and under-weighting style. Add style references and reduce identity strength slightly so scene lighting can influence the render.
Does this work for non-human characters?
It works well for stylised creatures, animated characters, and product models, provided the references are visually consistent with each other. Inconsistency in the references becomes inconsistency in the output.
What about hands and props?
Hands remain the weakest area. Reduce hand prominence in framing, keep hands still where possible, and expect to regenerate. Props that matter to the story should appear in reference images, not only in the prompt.
Is a seed enough on its own?
No. Seeds help reproducibility within one tool and prompt combination, but they do not encode identity. They are a supplement to references, not a substitute.
A Practical Starting Checklist
Before generating your next sequence, confirm the following. You have a shot list. You have eight or more diverse reference images per lead character, with no heavy filters. You have three to five style references. Your prompts describe action, environment, and camera rather than physical appearance. You are generating in small batches and reviewing at thumbnail scale. You are assembling as you go rather than at the end. And you are keeping a record of the seeds and weights behind every approved shot.
Do those things consistently and the technical problem of character consistency largely disappears. What remains is the harder, more interesting work: deciding what your character actually does, shot to shot, in a story worth watching.
