Why Character Consistency Still Breaks in AI Video
Anyone who has generated a short clip with a photoreal character knows the letdown of the second shot. The opening render looks convincing: believable skin, plausible lighting, a face that reads as a real person. Then shot two arrives with a slightly wider jaw, different eye spacing, hair falling on the wrong side, and a jacket that has quietly changed colour. Nothing in the prompt changed. The character simply stopped being the same person.
That is not a bug you can prompt your way out of. It is structural. Most text-to-video and image-to-video models do not carry a persistent representation of "this specific person" between generations. They carry a representation of "a person who roughly matches this description." Every new clip resamples identity from the prompt and whatever conditioning images you supplied, which turns facial geometry into a random variable. Small sampling differences compound across frames, and by the third or fourth shot the character has become a cousin of the original.
Three failure modes show up again and again. Identity drift changes bone structure, apparent age, and facial proportions. Wardrobe drift changes garments, textures, and accessories, which breaks continuity harder than most creators expect because viewers track clothing as a timeline marker. Photometric drift changes skin tone, white balance, and shadow direction, so even a technically identical face looks like it was filmed on a different day. Multi-image fusion is the technique that attacks all three at once, and understanding why it works makes it far easier to apply deliberately instead of hoping a lucky seed saves the edit.
What Multi-Image Fusion Actually Does
Fusion means conditioning a generation on several reference images of the same subject rather than one. The model extracts features from each reference and blends them into a shared identity signal that guides every frame of the output. Architectures differ, but the general pipeline has four moving parts.
First, feature extraction. Encoders convert each reference into an embedding that captures geometry, texture, and colour statistics. Second, identity aggregation. The embeddings are merged, often with attention weights so that a sharp front-facing portrait contributes more than a blurry three-quarter shot. Third, cross-attention injection. The merged identity signal is fed into the denoising network alongside your text prompt, so it constrains facial structure without overriding pose, camera, or lighting. Fourth, temporal attention. Across frames, the model enforces smoothness so the fused identity does not flicker from frame to frame.
The practical consequences are easy to remember. Variety in references beats repetition, because a model cannot learn cheekbone depth from six near-identical portraits. Clean references beat stylised ones, because compression artefacts and heavy filters leak into the output. And a consistent reference set beats a large one, because contradictory inputs produce an averaged face that matches none of your source images.
Building a Reference Set the Model Can Read
The six shots every character set needs
A workable set covers the angles the story actually uses. Start with a neutral front-facing portrait in even light. Add a three-quarter view from each side, because most dialogue scenes live in that range. Include one profile to pin down nose and jaw silhouette. Add a full-body or waist-up frame to lock wardrobe proportions. Finish with one expressive frame, such as a smile or a concerned look, so the model does not collapse into a neutral mask whenever emotion is requested.
Lighting and background hygiene
Light your references the way you intend to light the scene. If the finished video is a warm interior, unflattering blue studio shots will fight the prompt. Keep backgrounds plain and uncluttered so the encoder does not absorb furniture into the identity signal. Remove glasses, hats, and heavy jewellery from at least the core portraits, then add separate accessory references if the character wears them throughout. Crop tight enough that the face occupies a good share of the frame, but not so tight that hair and jaw edges are cut off.
Signs your reference set is weak
If every output has the same soft, generic face, your references are too similar. If the character ages between shots, your references span different age ranges or lighting intensities. If skin tone wanders, your references have conflicting white balance. If the model invents a hairstyle, your set never showed the hair clearly from the side or back. All four symptoms are fixable with replacements rather than more prompting.
A Repeatable Fusion Workflow, Step by Step
Step 1: write the character bible before you generate anything
Write one short paragraph describing fixed traits and a second paragraph describing variable traits. Fixed traits are the ones that must survive every shot: face shape, eye colour, hairline, skin tone, build, and wardrobe. Variable traits are pose, expression, camera angle, and setting. This separation stops you from accidentally encoding a mood into the identity description, which is the most common cause of a character that only looks right when they are calm.
Step 2: normalise your references
Resize everything to the model's preferred resolution, convert to a consistent colour profile, and level the exposure so no single image is dramatically darker or lighter than the rest. Where a tool supports per-image weighting, give your sharpest and most neutral portrait the highest weight and use the others as supporting views. Where it supports masking, mask out background and hands so they cannot contaminate the identity embedding.
Step 3: lock a prompt scaffold
Write a reusable prompt template with three slots: identity, performance, and camera. The identity slot repeats verbatim in every shot. The performance slot changes with the beat of the scene. The camera slot describes framing and motion. Treating the identity slot as immutable is the single highest-leverage habit in this workflow, because it keeps the conditioning signal stable while everything else varies.
Step 4: generate a calibration clip
Before committing to a scene, render one short, simple shot: the character standing still, neutral expression, medium framing, no camera movement. This calibration clip becomes your reference standard. Compare every subsequent generation against it for jaw width, interpupillary distance, hair volume, and skin tone. If the calibration clip itself is wrong, fix the reference set before producing anything else.
Step 5: run a drift audit
Render your shots in batches and review them side by side at thumbnail size. Thumbnails strip away detail and expose structural differences, which is exactly what audience members notice first. Score each shot from one to five on identity, wardrobe, and lighting. Anything scoring three or lower gets regenerated with adjusted references rather than patched with more adjectives.
Step 6: repair rather than restart
Full restarts waste hours. If identity holds for the first two seconds and drifts afterwards, shorten the clip and extend with a second generation seeded from the last good frame. If the face is right but the lighting is wrong, fix it in post with a colour match instead of regenerating. If the pose is wrong but identity is perfect, keep the identity conditioning and rewrite only the performance slot.
Choosing Tools: What to Compare
Every serious video platform now advertises reference-based character control, so compare on capability rather than marketing language. Test each candidate against the same reference set and the same six-shot script.
| Criterion | What to look for |
|---|---|
| Reference capacity | How many images can be conditioned at once, and can you weight them? |
| Identity mechanism | Does it embed identity, swap faces, or rely on keyframes? |
| Temporal stability | Does the face flicker, warp, or re-render between frames? |
| Clip length | Useful duration before drift becomes visible |
| Control surface | Camera, pose, motion strength, and negative prompts |
| Editability | Can you extend, inpaint, or repair a clip without regenerating it? |
Broadly, three approaches exist. Reference-conditioned generators fuse multiple images directly. Keyframe-and-interpolation pipelines generate strong stills first and animate between them, which produces excellent identity stability at the cost of fluid motion. Post-production identity tools repair footage after the fact, which is the most controllable but the slowest. Many professional workflows combine all three: generate stills, animate between them, then repair the roughest frames.
Prompt Patterns That Hold a Face Together
Describe identity before describing action. Models weight early tokens more heavily, so leading with facial structure and hair rather than scene setting produces noticeably more stable results.
Use concrete, physical language. "Narrow jaw, high cheekbones, dark brown eyes, straight brows, close-set features" gives the model something to latch onto, while "beautiful face" gives it a stereotype. Avoid contradictory adjectives in the same prompt, because "youthful" and "weathered" in one sentence average into an unpredictable middle.
Keep the style clause identical across every shot in a scene. Changing from "cinematic realism" to "documentary realism" between shots changes the renderer's texture model and, indirectly, the face. Same for colour grading language.
Use negative prompts conservatively. A short list of specific exclusions works better than a long list of generic ones. Reusing the same seed where your tool allows it reduces variance, but never rely on a seed alone to preserve identity across very different camera angles.
Finally, describe motion amplitude. "Slow head turn, minimal movement" produces fewer warping artefacts than an unconstrained prompt, and less motion means less opportunity for the temporal model to smear facial features.
Fixing Temporal Distortion and Lighting Mismatch
Flicker usually comes from under-constrained temporal attention. Shortening the clip, reducing motion speed, and adding more keyframes all help. Warping around the mouth and eyes appears at high motion amplitudes or fast camera moves; lower the motion strength and let editing create the sense of speed instead.
Hair and teeth are the most common local artefacts. Thin strands break into noise because they are high-frequency detail the encoder cannot track frame to frame. Simplify hairstyles in the reference set, or keep the character's head relatively still in shots where the hair is prominent. Teeth blur when the model has no clear reference for the mouth in an open position, so add one smiling reference frame.
Lighting mismatch is best solved in post. Generation-time fixes such as adding colour temperature language to the prompt help, but a colour match node, a consistent lookup table, and a fixed white point across all clips will do more in ten minutes than an hour of prompt tinkering. If your character appears in both interior and exterior scenes, generate a calibration clip in each lighting environment so you know how the identity behaves before committing to a full sequence.
Quality Control: A Shot-Level Checklist
Run every approved shot through the same list before it enters the timeline.
- Face geometry matches the calibration clip at thumbnail size
- Eye colour and shape are stable across the whole clip
- Hairline and hair volume are unchanged
- Wardrobe colours, patterns, and layers match the continuity sheet
- Skin tone and white balance match adjacent shots
- Shadow direction is consistent with the scene's key light
- Hands and accessories do not morph mid-clip
- Motion does not exceed the amplitude the model handles cleanly
- No frame contains an identity collapse or a doubled face
Recording scores in a simple spreadsheet turns quality control from a feeling into a decision, and it gives you data when a particular reference image turns out to be the culprit.
Scaling a Multi-Scene Production
Once a single character works, production discipline matters more than any model setting. Keep a per-character folder with the approved reference set, the calibration clip, the identity prompt block, and the latest continuity sheet. Version the reference set rather than overwriting it, because replacing one image can shift the whole look and you may want to revert.
Batch similar shots together. Generating all medium shots of one character in a single session with identical settings reduces variance compared with scattering them across days of experimentation. Set review gates: no shot enters compositing until it passes the checklist, and no scene is locked until all its shots pass side by side.
For multiple characters, generate a shared reference frame containing all of them together. Even if you never use the frame directly, it helps you spot proportion inconsistencies before they become expensive, and it gives you a single image that establishes relative height and styling.
FAQ
How many reference images are enough? Four to eight well-chosen images is the practical sweet spot. Below four, identity is under-specified. Above ten, contradictory details start averaging into a face that resembles none of your inputs.
Can I get away with a single reference image? Yes for short, static, front-facing shots. The moment the camera moves to a profile or the character turns, a single view leaves too many unknowns and the model invents the rest.
Why does the face change when the camera moves? Because most models treat a new camera angle as a new generation problem. Adding profile and three-quarter references, and keeping motion amplitude low, is the most reliable fix.
Do I need a different reference set for a stylised or animated look? Usually yes. If the final output is illustrated, build the reference set in that illustration style. Mixing photographic references with an illustrated prompt creates a hybrid look that rarely satisfies either goal.
Should I generate stills first? For dialogue-heavy or identity-critical sequences, yes. A keyframe-and-interpolation workflow gives you exact control over the moments that matter and lets the model handle only the motion between them.
What is the fastest way to recover a drifting scene? Find the last frame where identity is correct, extend from that frame, and tighten the identity prompt block. Restarting from scratch should be the last resort, not the first reflex.
How do I handle costume changes without losing the face? Keep the core identity references unchanged and add costume-specific references as a secondary set. Describe the Costume in the performance slot, never the identity slot.
Get these habits in place and character consistency stops being a lottery. The reference set carries the identity, the workflow protects it, and quality control catches the handful of shots where the model still guesses wrong.
