Why character consistency is the hardest problem in AI video
Anyone who has produced more than two AI-generated shots of the same person knows the feeling. Shot one gives you a confident, sharp-jawed lead with a slightly crooked smile. Shot two returns a cousin of that person. Shot three returns a stranger wearing the same jacket. The lighting is beautiful, the camera move is cinematic, and the whole sequence is unusable because the audience cannot tell who they are watching.
Classical animation and live-action production solved this with reference bibles, continuity photography, and a continuity supervisor whose entire job is to catch drift between takes. Generative video has the same requirement but almost none of the infrastructure. The model does not remember your character between generations unless you give it something concrete to hold onto. Text alone is a weak anchor: words like "short dark hair, brown eyes, denim jacket" describe millions of people, and each generation will happily pick a different one.
Multi-image fusion is the practical fix. Instead of describing a character, you show the model several images of that character and let the generation blend them into a single identity that then persists across shots. Done well, the result looks like a cast member rather than a lottery draw. Done badly, it produces a wax figure that only looks correct in the exact pose of the reference.
This guide is about the second outcome. It covers what fusion actually does internally, how to build a reference set that carries the information a model needs, how to prompt around the limits of the technique, and how to run a repeatable workflow from concept to a locked character that survives an entire scene list.
What multi-image fusion actually does
Fusion is not a face swap and it is not a simple average of your inputs. It is contextual inference: the model examines the shared features across your reference images, discards what is pose-specific or background-specific, and constructs an internal representation of the identity. That representation is then injected into the generation as conditioning, so the diffusion process is biased toward it at every sampling step.
The practical consequence is important. The model is not copying pixels from your references. It is inferring a concept. If your references disagree — different people, different ages, different hair colors — the inference splits the difference and you get a face that resembles nobody in your set. If your references are consistent but narrow, the inference is sharp but fragile: it will reproduce the identity only in conditions similar to the references.
The single most common mistake is treating fusion as a magic identity lock. It is better understood as a strong prior. It heavily weights the probability of certain facial structures, skin tones, and proportions, but it still loses to conflicting signals from the prompt, the pose, or the lighting.
Reference sets versus single-image conditioning
A single reference image constrains the model to one observation. It tells you what the character looks like from one angle, under one light, with one expression. When the next shot asks for a profile view in low light, the model has no information and improvises.
A reference set of four to eight images gives the model enough viewpoints to triangulate. The improvement is not linear; it is a step change. Going from one reference to four typically fixes eye spacing, nose length, and jaw width. Going from four to eight mainly improves how the character holds up under unusual angles and lighting.
The three anchors: geometry, wardrobe, lighting
Think of your reference set as carrying three independent signals:
- Geometry — bone structure, face shape, eye spacing, nose profile, body proportions, height relative to a known object.
- Wardrobe and grooming — the specific jacket, the haircut, the beard length, the accessories that make the character recognizable in a wide shot where the face is tiny.
- Lighting response — how the skin behaves in daylight versus shadow, whether there is a strong key light, how much specularity the skin has.
Most weak reference sets over-index on geometry and ignore the other two. That is why characters look right in close-ups and wrong everywhere else.
Building a character reference sheet that works
The reference sheet is the highest-leverage artifact in the entire workflow. Two hours spent here saves twenty hours of regeneration.
Aim for six to eight images, all of the same character, all cleanly lit, all at a similar level of sharpness. Cover these angles:
- Full frontal, neutral expression
- Three-quarter left and three-quarter right
- Profile left and profile right
- A slight upward angle and a slight downward angle
- One full-body or three-quarter-body shot for proportions and wardrobe
- One shot with a genuine, non-forced expression
Shot angles to include and skip
Include: anything that shows the face without distortion. Skip: extreme close-ups where a phone lens has already stretched the nose, and dramatic low-angle hero shots, because the model will read the perspective distortion as actual anatomy.
Also skip group photos unless you crop tightly. Even then, be careful: if two faces appear in one reference and the model cannot tell which one you mean, you have poisoned the set.
Resolution, background, and lighting rules
- Resolution — each reference should be at least 1024 pixels on the short side. Below that, facial micro-detail is mush and the model invents features.
- Background — plain, uncluttered, mid-gray or a single soft color. Busy backgrounds leak into generations as unwanted texture.
- Lighting — soft, even, front-facing. Harsh single-source light makes half the face unreadable and the model will interpolate that shadow as a permanent feature.
- Consistency — do not mix a studio-lit headshot with a beach selfie. The model treats lighting as an identity trait.
- Color grade — keep references in the same color space. If half are warm and half are cool, the identity inference has to fight the difference.
One more rule that is easy to miss: no heavy makeup, no sunglasses, no hats that cover the hairline, and no filters. Every one of those is a variable the model will try to preserve as part of the person.
Prompting alongside fusion
Fusion handles identity. Prompts handle everything else. The two must be written to complement rather than compete.
Describing what the reference cannot carry
The reference shows the character standing still. The prompt has to specify action, camera, and environment. Write the prompt as if the character is already cast and you are directing the scene:
Medium shot, character walking through a rain-slicked alley at night, handheld camera drifting left, shallow depth of field, practical neon light from the left, light rain, subtle breath vapor.
Notice what is absent: any description of the face. That is deliberate. Every adjective you spend on the character's appearance is a chance to contradict the reference set.
Stability language that helps, and language that hurts
Helpful phrases tend to be about camera and continuity: "consistent lighting across the shot," "locked camera," "stable framing," "natural motion of hair and fabric." These nudge the model toward temporal coherence.
Harmful phrases are usually about beauty and intensity: "stunning," "perfect skin," "flawless," "hyper-detailed face." These push the generation toward an idealized average face that drifts away from your reference. If your character has a scar, a mole, or an asymmetric eyebrow, words like "flawless" actively erase them.
A repeatable workflow: from concept to a locked character
This sequence is deliberately boring. Boring is what makes it repeatable across twenty shots.
Step 1 — Lock the design on paper first
Before generating anything, write a one-page character sheet: age range, build, hair, distinguishing features, wardrobe palette, three adjectives for personality, and two reference images from real life or stock that capture the vibe. This is the document you will check every future shot against.
Step 2 — Generate a candidate face, then approve one
Generate a wide spread of candidate images with a text-to-image tool. Pick exactly one face. Do not pick two and hope fusion merges them into something better. Approval should be a single, unambiguous decision.
Step 3 — Build the reference set around the approved face
Use an image editor or an image-to-image workflow to produce the six to eight angles described above from the approved face. Keep wardrobe identical across all of them. Save the set as a named folder with the character name and a version number.
Step 4 — Test the identity across three stress shots
Before committing to a full scene, run three cheap test generations: a profile close-up, a wide shot at night, and a shot with strong emotion. If the identity holds in all three, the set is good. If it holds in two, fix the weak angle by adding a reference rather than by rewriting the prompt.
Step 5 — Generate shots in a fixed order
Generate establishing shots first, then medium shots, then close-ups. Working from wide to tight makes continuity errors obvious early, when they are cheap to fix. It also gives you a library of approved frames you can feed back as additional references for difficult shots.
Step 6 — Run a continuity QA pass before editing
Watch the shots back-to-back at speed, then frame by frame at the cut points. Track five things: hairline shape, eye color, jaw width, wardrobe details, and any distinguishing mark. Anything that drifts more than a little gets regenerated, not color-corrected.
Step 7 — Archive the approved set
When the project ends, keep the reference set, the prompt templates that worked, and a short note about what failed. The next project with the same character starts at step four.
Adapting the workflow to different visual styles
Photoreal and cinematic
Photoreal is the most demanding style because audiences have the most practice reading real faces. Use the highest-resolution references you can, keep the skin's natural texture, and avoid any prompt language about retouching. Consider generating at a slightly lower resolution and upscaling afterward, because some pipelines sharpen faces aggressively at high resolution and smooth away the very features that make your character identifiable.
Animation and stylized looks
Stylized characters are easier in one sense — the model is not fighting photorealism — but harder in another, because stylization magnifies small inconsistencies. A five percent jaw difference is invisible on a real face and glaring on a cartoon one. Keep line weight and shading style identical across all references, and expect to need more of them.
Mixed-media and hybrid shots
When a character appears in both live-action-style and animated-style shots, build two separate reference sets rather than one merged set. The identity traits stay the same; the rendering style varies. Trying to fuse across styles usually produces a muddy compromise.
Common mistakes and how to fix them
The identity drifts between shots. Usually caused by inconsistency in the reference set or by prompt language describing the face. Remove appearance adjectives from prompts and audit references for lighting and wardrobe mismatches.
The character looks like a wax figure. A sign that the model is over-weighting identity conditioning and under-weighting the scene. Reduce fusion strength slightly and add more scene detail to the prompt.
The character looks correct but the expressions are dead. References with neutral expressions teach the model that neutral is the only valid state. Add two references with genuine, distinct expressions.
Age drifts upward or downward across a sequence. Age reads through skin texture and hair. If your references are all heavily retouched, the model has no age signal and will guess.
Backgrounds bleed in. Always caused by cluttered reference images. Rebuild the set on a plain background.
Everything looks slightly orange or slightly green. Color space mismatch in the reference set. Normalize before fusing.
Hands and bodies are inconsistent while the face is fine. Fusion conditions the identity, not the anatomy. Use consistent framing and pose descriptions, and avoid extreme body angles the references do not support.
The fifth shot breaks after four good ones. This is often a prompt drift issue rather than a fusion issue. Reuse the exact phrasing that worked instead of rewording for variety.
Choosing the right tooling for the job
Not every project needs the same level of investment. Use these criteria to decide how much machinery to build:
- Single shot, no recurring character — skip fusion entirely. Text-to-video with a good prompt is enough.
- Two to five shots, same character, similar framing — three references and careful prompting will carry it.
- Six to thirty shots with varied angles — build the full reference set and run the stress test. This is the sweet spot where fusion pays for itself many times over.
- Thirty-plus shots or multiple characters — treat it as a production. Maintain a shared reference library, version every set, and assign one person to run continuity QA.
- Multiple characters in the same frame — keep references strictly separate and describe each character's position and action explicitly. Do not attempt to fuse two identities into one generation call.
When evaluating any tool, check four things: how many references it accepts at once, whether it lets you weight them, whether it preserves identity across camera motion, and how quickly you can iterate on a failed shot. Iteration speed matters more than peak quality, because consistency is achieved through repeated small corrections.
Frequently asked questions
How many reference images is enough?
Four is the practical minimum for a recognizable identity. Six to eight is the reliable range. Beyond roughly twelve, returns flatten and you risk introducing contradictory signals.
Can I use images of two different people to create a new character?
You can, but treat the output as a brand-new design, not a blend of two known faces. Once you approve the result, generate a fresh, single-person reference set from it.
Do I need to describe the character in the prompt at all?
Only for traits the references cannot show — an accent in the performance, a specific emotion, a costume change. Otherwise, let the images do the work.
Why does the character change when the camera angle changes?
The reference set does not cover that angle. Add a matching reference rather than rewriting the prompt, because the prompt cannot invent geometry it was never given.
Should I upscale before or after fusion?
Fuse first, upscale afterward. Heavy upscaling can smooth away the fine skin and hair detail that the identity inference depends on.
Can I reuse a reference set across projects?
Yes, and you should. Keep it versioned and add new approved frames over time. A well-maintained set becomes more robust the longer you use it.
What if the model ignores my references completely?
Check that the images are actually being passed as conditioning and not just added to the prompt as text. Then check for conflicting appearance descriptions in the prompt, which can override conditioning.
A short pre-flight checklist
Before you generate the first real shot of a sequence, confirm all of the following:
- One character design is approved and documented.
- Six to eight references exist, plain background, even light, consistent wardrobe.
- All references are at least 1024 pixels on the short side and share a color space.
- The prompt contains no facial appearance adjectives.
- Three stress shots have passed: profile, wide, emotional.
- A naming and versioning convention is in place.
- One person owns continuity QA.
Character consistency is not a single setting you switch on. It is a discipline of narrowing variables until the only thing left that can change between shots is what you intended to change. Multi-image fusion gives you the strongest single lever available, but the lever only works when the reference set, the prompt, and the QA pass all point in the same direction. Build the set carefully, keep the prompts clean, test before you commit, and your characters will hold their identity from the first frame to the last.
