Why Character Consistency Is the Hardest Problem in AI Video
Ask any generative video model for a five-second shot of a person walking through rain, and you will get something that looks like a film still. Ask for the same person in the next shot, and the illusion collapses. The jawline softens. The eye spacing changes by a few millimetres. The hairline migrates. The wardrobe quietly swaps a collar for a hoodie. Nothing is catastrophically wrong in any single frame, yet the moment you cut the two shots together, the audience feels that these are two different people wearing the same coat.
This is not a rendering problem. It is an identity problem. Video models are trained to optimise for per-frame plausibility, not for continuity along a timeline. Every frame is generated from a probability distribution over what a plausible frame should look like, and a face is just one more texture pattern inside that distribution. Text prompts are a remarkably lossy way to describe a face, which is why a prompt-based pipeline will always produce a cousin, not a clone.
Multi-image fusion is the family of techniques that closes this gap. Instead of describing a character in words, you supply several reference images and let the model derive a shared identity signal from them. The result is not perfect, but it is predictable enough to build narrative series work on: recurring hosts, episodic shorts, explainer series with a fixed presenter, brand characters, and fiction that depends on the audience recognising a face.
How Multi-Image Fusion Actually Works
Multi-image fusion is an umbrella term rather than a single feature. Depending on the tool, it may be called reference conditioning, subject locking, character memory, identity preservation, or multi-reference guidance. Under the hood, the same idea recurs: several images of one subject are passed through an encoder, the resulting feature vectors are pooled into an identity representation, and that representation steers every subsequent generation.
From single-image conditioning to multi-reference input
Early pipelines used one still image plus a text prompt. This works for a single output but breaks immediately across a sequence. The model tends to copy the pose, the lighting, and the framing of the reference image along with the face, because it has no way to separate identity from everything else in the picture. You end up with a character who is always photographed from the same angle under the same lamp.
Multi-reference input solves this by giving the model variance. When the references show the same person front-on, in three-quarter view, in profile, in different light, and with different expressions, the model is forced to find what stays constant across all of them. That constant is identity. Everything that changes is treated as a variable you can control with the prompt.
Identity anchoring versus style anchoring
The single most useful mental model is to split your references into two buckets.
Identity anchors carry facial geometry, skin tone, hairline, eyebrow shape, distinguishing marks, and overall proportions. These should be as neutral and as technically clean as possible: consistent focal length, even lighting, minimal makeup variance, no dramatic colour grading.
Style anchors carry the look of the production: colour grade, lens character, grain, wardrobe palette, environment mood. These can be cinematic and stylised, because you are not asking them to define a face.
When you mix the two buckets in one reference set, the model averages them. A heavily graded reference pulls the identity toward that grade, and you lose control of both. Keep the buckets separate, and you can restyle a character for a new episode without regenerating the face.
What the model actually learns
It is tempting to imagine the model storing your character in a database. It does not. It builds a weighted embedding that biases sampling toward a region of latent space. That embedding is only as coherent as the references that produced it. Feed it five images of the same person taken over ten years, and the embedding averages the ages into a face that matches none of them. Feed it two images that disagree about hair colour, and you get a character whose hair flickers between shots. Consistency begins with your reference pack, not with the tool settings.
Building a Reference Pack That Survives Every Scene
A good reference pack is small, boring, and technically clean. It is the least glamorous part of the workflow and the single biggest lever on output quality.
The minimum viable set
For most models, six to ten images is the sweet spot. Fewer than four and the identity signal is too weak; more than twelve and contradictory details start to average out. A reliable baseline:
- Two neutral frontal portraits, slight variation in expression
- Two three-quarter views, one left and one right
- One profile view, ideally mid-turn
- One full-body shot to capture proportions and posture
- One image in the intended production lighting, used as a style anchor
- One image showing a signature detail, such as a scar, a specific hairstyle, or a piece of jewellery
Coverage: angles, light, and expression
The goal is coverage of the identity, not coverage of the wardrobe. Angles matter most, because most identity drift happens when the character turns their head. If every reference is frontal, the model has no information about the shape of the jaw in profile and will invent one. Lighting matters second, because harsh shadows can be mistaken for facial structure. Expressions matter least, but a slight smile in one reference prevents the model from freezing the character into a permanent neutral stare.
What to leave out
Exclude anything the model might mistake for identity. Sunglasses, masks, heavy filters, motion blur, low-resolution crops, group photos, watermarks, and extreme wide-angle distortion all contaminate the embedding. Also exclude images that show a deliberately different look, such as a character with a beard in one reference and clean-shaven in another, unless you are building two separate characters.
Prompting Alongside Reference Images
Once your references are doing the heavy lifting on identity, prompts should handle everything the references cannot express.
Describe what the references cannot show
References encode appearance. They do not encode action, emotion trajectory, camera language, or environment. Prompt for motion, mood, lighting direction, and shot type. Do not re-describe the face in detail. Long descriptions of cheekbones and eye colour fight the reference embedding and drag the output toward a generic average face. Short, structural prompts win.
Locking wardrobe, age, and props
Write wardrobe as short, stable noun phrases and reuse them verbatim across every shot: "charcoal wool coat, cream scarf, leather satchel." Change one word and you introduce a variable the model may resolve by changing something else. Keep a prompt template file per character so that every shot inherits identical phrasing. If your character appears at two ages, treat those as two characters with separate reference packs and separate templates.
Drift-control phrasing
Some models respond to phrasing like "the same person as the reference," "unchanged facial structure," or "consistent identity." Others expose a numeric identity strength or reference weight. Start moderately strong, inspect the first three outputs, and adjust. If the face looks like a mask pasted onto a body, you have pushed identity weight too high. If the character looks like a relative rather than the same person, raise it.
Keeping a Face Stable Through Motion
Static consistency is only half the battle. Video adds a second axis of drift: identity that degrades as the clip progresses.
Temporal coherence and frame-to-frame drift
Most generative video drifts gradually. Frame one is accurate, frame thirty is a stranger. The practical fix is short generations with deliberate extension. Generate three to five seconds, then extend from the last frame rather than asking the model for a single long take. Overlap extensions by a fraction of a second so you have trimming room, and check the seam frame by frame.
Fast motion, occlusion, and profile turns
Turning heads are the single most common trigger for identity breaks. Slow the turn, cut around it, or place a reaction shot between the start and end of the rotation. Occlusion, such as a hand passing over the face, is a second trigger, because the model has to reconstruct the face from memory. If a hand must cross the face, keep the crossing fast and insert a clean frame immediately afterwards.
Camera moves that help and hurt
Slow dollies, gentle orbits, and slight push-ins preserve identity well, because they keep the face at a similar scale and angle. Whip pans, extreme close-ups, heavy handheld shake, and rapid focus pulls all invite drift. If you need a dramatic move, build it from shorter stable segments and cut on motion rather than generating the whole arc.
One Character, Many Models
Most serious workflows end up using more than one model: one for keyframes, one for animation, one for stylised inserts, another for upscaling. Each engine has its own facial priors, so the same references can produce subtly different people.
Building a model-agnostic character bible
Write down everything that defines the character in a single document: the reference pack, the prompt template, the wardrobe list, proportions, vocal tone if relevant, and a no-go list of features the character must never have. Treat this as production documentation rather than a note to yourself. It is the artefact that lets a collaborator, or a newer model, reproduce the look months later.
Normalising a look across engines
The most reliable technique is keyframe locking. Generate a portrait or mid-shot in the engine that handles the character best, then feed that still as the identity reference into the animation engine. This converts a soft identity embedding into a hard visual anchor and dramatically reduces cross-engine variance. Where a model supports training a small personal adapter, a lightweight character LoRA can also help, provided the training set follows the same rules as your reference pack.
Regenerate versus fix: decision criteria
Use drift rate as your guide. If fewer than roughly one in five frames shows identity problems, regenerate the shot or repair it with a masked retake. If problems appear in close to half of frames, stop iterating on the prompt; your reference pack is the bottleneck. Rebuild the references and try again. Prompt tinkering on a broken identity embedding wastes time.
A Repeatable Workflow for Series Work
The order of operations matters as much as the settings. A pipeline that produces consistent characters across dozens of shots looks like this.
Step one: define the blueprint
Collect references, choose identity and style anchors, write the prompt template, and decide on wardrobe versions. Do this before generating anything.
Step two: generate keyframes first
Generate still images for every shot in the episode, or at least for every distinct setup. Stills are cheap to iterate and easy to compare side by side. Only move forward when the whole keyframe set reads as one person.
Step three: animate in short segments
Animate each keyframe into a three-to-five-second clip, extending from the last frame where continuity is required. Keep a naming convention that ties each segment to its shot number and character version.
Step four: quality control and repair
Review at reduced speed, watching for the exact frames where identity slides. Repair with masked retakes or by regenerating a single segment rather than the whole shot.
Step five: lock and archive
Once a character version is approved, freeze it. Archive the reference pack, the exact prompt template, and the model versions used. Future episodes start from the frozen version, not from memory.
Common Failure Modes and How to Fix Them
Most consistency problems fall into a handful of recognisable patterns.
- Face melting mid-clip. Usually caused by long single generations. Split into shorter segments and extend.
- Wardrobe swapping between shots. Caused by paraphrased prompts. Copy wardrobe phrases verbatim.
- Age drift across a series. Caused by references from different periods. Curate a single-period pack.
- Style bleeding into identity. Caused by mixing graded style references with neutral identity references. Separate the buckets.
- Cousin-face outputs. Caused by too many contradictory references or an over-described face prompt. Reduce references, shorten the prompt.
- Background identity bleed. Caused by distinctive background elements in the references. Crop references tightly around the subject.
- Over-sharpened, mask-like results. Caused by identity strength pushed too high. Reduce and let the prompt handle performance.
Quality Control Checklist
Before you approve any shot, run this list:
- Does the face read as the same person as the approved keyframe at 100% zoom?
- Does identity hold in the first frame, the middle frame, and the last frame?
- Is the wardrobe wording identical to the template?
- Are hair length, hairline, and colour stable?
- Do skin tone and undertone match across shots?
- Are proportions consistent at full-body scale?
- Does the shot cut cleanly with the shot before and after it?
- Have you archived the exact settings used?
FAQ
How many reference images do I actually need? Four is the practical minimum and ten is plenty for most characters. Add references only when you are missing an angle, not to increase strength.
Do I need to train a custom model? Not necessarily. Multi-image fusion handles most recurring-character work. A small personal adapter helps when a character appears in dozens of shots with heavy stylisation, but it is an optimisation, not a prerequisite.
Can I use one character across different video tools? Yes, with a character bible and keyframe locking. Generate an approved still in your strongest engine and use that still as the identity reference everywhere else.
Why does my character still change between shots when I use references? Usually because the references contradict each other, or because the prompt spends too many words describing the face. Fix the pack first, then shorten the prompt.
What resolution should my references be? At least one megapixel, square or near-square crops, sharp focus on the face, and no compression artefacts. Higher is better up to the point where the tool downsamples anyway.
Does multi-image fusion work for animated or stylised characters? Often better than for photoreal people, because stylised characters have fewer subtle identity cues and are more tolerant of drift.
How do I handle two characters in one shot? Give each character its own reference set and avoid overlapping bodies where possible. If the model keeps blending faces, generate the characters separately and composite.
What is the fastest way to improve consistency right now? Rebuild the reference pack with clean, varied angles and split your identity and style anchors. That single change fixes more drift than any prompt rewrite.
