AI Face Synthesis Video: A Character Consistency Guide

Why Face Consistency Is the Hardest Problem in AI Video

Almost everyone who starts generating video with AI hits the same wall. The first clip looks astonishing. The second clip features what appears to be the same person — except the jaw is wider, the eyes sit a few pixels lower, and the hairline has migrated. By the third clip you have a stranger wearing your character's wardrobe.

Backgrounds forgive almost anything. A tree can warp, a wall can breathe, a crowd can melt into texture soup, and most viewers will never notice. Faces do not get that grace. Humans dedicate an enormous amount of neural real estate to reading faces, and it runs constantly in the background. We track eye spacing, brow angle, mouth width, skin texture, and micro-asymmetry without conscious effort. A drift of three or four pixels in the distance between the pupils reads as "wrong" long before anyone can explain why.

That is why face consistency is not a polish problem you fix at the end. It is an architectural problem you solve before the first frame is generated. There are three separate layers to keep stable, and they fail for different reasons:

Identity — who the person is. Eye color, bone structure, age, skin tone, hair. Failure looks like a different actor stepping in.
Performance — what the person is doing. Blink rate, gaze direction, mouth shapes, head sway, breathing. Failure looks like a puppet or a mannequin.
Continuity — the state carried across cuts. Wardrobe, hair position, lighting direction, wounds, sweat, props in hand. Failure looks like two scenes shot on different planets.

Most beginners fix one layer and assume the others will follow. In practice, a project that nails identity but ignores continuity still feels broken to an audience, because the brain reads the inconsistency even when the eye cannot name it.

How Modern Video Models Handle Faces — and Where They Break

Identity conditioning versus text-only prompting

Modern video generators are built on diffusion transformers operating in a compressed latent space. They are trained to produce plausible motion and plausible detail, not the same person twice. When you prompt with words alone, the model samples a new face every time. Identity becomes a random variable.

To get repeatable faces you need to inject identity through a second channel. In practice there are three approaches:

Reference conditioning. You supply one or more still images of the character, and the model binds its output to those features. This is fast, requires no training, and works well for short clips. It is the default choice for most projects.
Adapter-style identity injection. A lightweight network trained to map a face embedding into the generation process. Stronger identity lock than plain reference images, but it can fight against the prompt when the character needs to do something unusual.
Fine-tuning a character-specific model. You train a small adapter or low-rank layer on a curated image set. This gives the tightest identity control and is the right answer for a series, a recurring brand mascot, or anything with more than a dozen shots.

The trade-off is straightforward: reference conditioning is fast and flexible but drifts over long sequences; fine-tuning is slow to set up and less flexible stylistically but holds a face across an entire episode.

The anatomy of a face failure

Knowing the specific failure mode tells you which lever to pull. The recurring ones:

Melting. Features dissolve during fast motion, occlusion, or when a hand passes across the face. Fix with shorter shot lengths, fewer occlusions, and higher reference weight.
Identity bleed. In two-shots, two characters swap features. Fix by generating each character in a separate pass and compositing, or by using distinct, clearly separated training sets.
Progressive drift. The face slowly ages, slims, or shifts over a long clip. Fix by generating in shorter segments anchored to the same reference, then joining them.
Texture boiling. Skin detail crawls and shimmers frame to frame. Fix with temporal denoising, deflicker passes, or higher resolution generation followed by downscale.
Teeth and gaze artifacts. Mouth interiors turn to mush, pupils wander. Fix by avoiding open-mouth extremes and by specifying gaze direction explicitly in the prompt.

What "good enough" actually means

Set a measurable bar before you start, otherwise you will keep regenerating forever. A reasonable target for narrative work: a stable identity held across ten to fifteen seconds per shot, eyes that track consistently, no visible identity change across a cut, and less than five percent of frames requiring manual repair. If more than a tenth of your frames need fixing, the problem is upstream in your references or your prompt structure, not in your luck.

Choosing the Right Model for Face-Heavy Shots

Marketing pages all claim cinematic realism. What matters for face work is narrower and much more concrete. Compare candidates on these criteria:

Identity conditioning support. Can you feed reference images, and how heavily does the model respect them? A model with weak reference handling will cost you hours in retries.
Usable clip length. Not the maximum advertised length, but the longest clip before identity drift becomes visible.
Native resolution and face-scale detail. Faces occupy a small percentage of most frames. If the native resolution is low, faces turn to mush and no upscaler fully rescues them.
Motion fidelity. Aggressive camera moves and fast action stress identity. Test the moves you actually plan to use.
Controllability. Support for start frames, end frames, and motion direction is the difference between directing and gambling.
Style range. Photoreal models resist stylization; stylized models resist photorealism. Pick the one that matches your project.
Cost predictability. Per-second and per-retry economics matter more than the headline rate, because face work involves retries.
Downstream integration. Can you export clean frames, alpha, or depth passes for compositing?

Matching model strengths to shot types

Different shots have different failure tolerances. A talking head in a locked-off medium close-up has nowhere to hide. A walk-and-talk with camera movement hides small drifts. A wide ensemble shot hides a lot but exposes identity bleed between characters. A stylized animated look tolerates deformation that would destroy photorealism.

Match accordingly: use your most controllable model for dialogue close-ups, and your most cinematic model for wides and establishing shots where faces are small. Do not spend your best tool on frames where the face is twenty pixels tall.

Run a twenty-minute test before you commit

Pick three representative prompts — a static close-up, a moving mid-shot, and a two-character scene. Generate all three with at least three candidate models using the same character references and the same prompt text. Watch them side by side at full resolution, then at thumbnail size. The thumbnail test reveals identity drift faster than any technical inspection.

Building a Character Bible: Reference Sets and Locked Attributes

An actor shows up with the same face every day. AI models do not, unless you hand them the data. A character bible is a folder plus a one-page specification, and it is the highest-leverage document in your entire pipeline.

The reference set

Aim for eight to twelve images at minimum. More is not always better — a large set of inconsistent images teaches the model that your character is a shape-shifter.

One frontal, neutral expression, eyes open, mouth closed
Three-quarter views from both left and right
A true profile from each side
Slight upward and slight downward angles
Three or four distinct expressions: neutral, smiling, speaking mid-vowel, concerned
Two lighting conditions: soft frontal and harder side light
Identical wardrobe and hairstyling across every image

Keep resolutions consistent, use clean backgrounds, and avoid harsh filters. If a reference image has motion blur, exclude it — the model will learn the blur as a feature.

The locked attributes list

Write down, in plain language, every attribute that must not change: hair color and styling, hairline shape, brow thickness, eye color, nose bridge width, mouth width, lip fullness, jaw angle, chin shape, skin tone and texture, visible marks, approximate age, build, and every item of clothing including layering. Then write a second list of attributes that are allowed to change under named conditions — for example, hair may be damp in rain scenes, or a jacket may be removed indoors.

This sounds bureaucratic. It saves entire evenings. When a generated shot looks wrong and you cannot name why, the locked-attribute list usually names it for you.

Fine-tuning versus reference conditioning

If your project has fewer than ten shots, reference conditioning is almost always sufficient. If you are building a recurring series, a channel mascot, or a character who appears in dozens of scenes, invest in training a dedicated character model. The upfront cost in setup time is repaid by cutting retries down by an order of magnitude.

Prompt Engineering for Faces: A Structure That Holds Up

Free-form prompting produces free-form results. A repeatable structure keeps faces stable across an entire shoot.

The five-slot formula

Write every prompt in the same five slots, in the same order:

Identity anchor — a verbatim descriptive phrase, character name, and reference tag. Copy this word for word across every shot. Never paraphrase it. "Marcus, 40s, close-cropped dark hair with grey at the temples, deep-set brown eyes, square jaw, faint scar above the left eyebrow" is a reusable token. "A middle-aged man with dark hair" is a new person every time.
Performance — the action and emotional register. Includes gaze direction, pace of movement, and whether the character is speaking.
Camera — lens, height, distance, and movement. "Locked-off medium close-up at eye level, 50mm equivalent" is far more useful than "cinematic shot."
Lighting and environment — the anchor that ties shots together.
Style and grade — film stock feel, contrast, grain, color bias.

Lighting and environment anchors

Lighting is the most underrated consistency tool. If character A is lit from the left in shot one and from the right in shot two with no motivated reason, the audience reads discontinuity even if the face is identical. Decide a lighting plan for each location and repeat its phrasing verbatim in every prompt for that location. Change it only when the story motivates the change, and when you do, change it in one deliberate step.

Negative constraints and drift control

Most generators accept negative prompts or weight modifiers. Useful constraints for face work include: no identity change, no face warping, no extra fingers near the face, no teeth distortion, no sudden aging, no facial hair changes, no costume changes. Keep the negative list short and specific — a sprawling negative prompt starts suppressing the things you want.

Keyframes, Motion, and Nonverbal Performance

Blocking with start and end frames

If your model supports start and end frame conditioning, this is the single most powerful control you have. Generate a still of the character in the opening pose, another in the closing pose, then let the model interpolate the motion between them. Identity is anchored at both ends, which massively reduces mid-clip drift.

Micro-expressions, blink rate, and breath

Faces that do not blink look dead. Faces that blink on a metronome look robotic. Real blink rates run roughly every three to five seconds, faster under stress, slower when reading or thinking. If your tool exposes blink or micro-expression parameters, tune them. If not, break long dialogue into shorter clips and vary the delivery.

Add small involuntary motion: a slight head sway, a breath that lifts the shoulders, a momentary glance away and back. These cost nothing and they are the difference between a generated clip and a performance.

Lip sync and dialogue

Where lip sync is generated separately, drive it from the actual audio waveform rather than from phoneme guesses, and check the result at half speed. Plosives and rounded vowels are where sync breaks down most visibly.

Multi-Character Scenes and Long Sequences

Stable IDs and separated training sets

Every character needs a unique, stable identifier used consistently across prompts, file names, and project folders. When two characters share a training set or a reference folder, features bleed. Keep sets physically separate, and if a model supports multiple reference slots, make sure each slot maps to exactly one character.

Coverage strategy that sidesteps the hard problem

Two-shots are where identity bleed happens. The professional workaround is not better generation, it is better coverage: shoot each character in a solo pass and assemble the scene in the edit with over-the-shoulder framing, singles, and cutaways. Audiences read a well-cut sequence of singles as a conversation. This approach costs a little more in shot count and saves enormous amounts of regeneration.

Managing long sequences

Number every shot, log the character state for each one, and keep a continuity sheet listing wardrobe, hair, props, and lighting direction per scene. When you assemble a sequence, generate shots in narrative order so drift accumulates in a controlled direction rather than jumping around.

Post-Production Repair and Continuity

Generation is roughly seventy percent of the work. The remaining thirty percent happens in post:

Face restoration and upscaling. Targeted face restoration recovers detail on small, soft faces, but apply it gently. Over-restoration produces the plastic, airbrushed look that signals AI video instantly.
Deflicker and temporal smoothing. Removes texture boiling without smearing motion.
Frame interpolation. Raises frame rate smoothly, which helps fast head turns.
Color grading. A unified grade across all shots makes identity differences far less noticeable — the audience attributes variation to lighting rather than to a different person.
Film grain. A light, consistent grain layer unifies shots generated at different times with slightly different texture.

Finally, watch the whole sequence at normal speed on a phone screen. Small devices hide detail problems and expose rhythm and continuity problems, which is exactly the ratio you care about at this stage.

An End-to-End Production Workflow

Write the shot list first. Decide which shots need faces in close-up and which can be staged so faces are small or partially obscured. Design your coverage to hide the hard problems.
Build the character bible. Assemble references, lock the attribute list, and store the identity anchor phrase where everyone can copy it verbatim.
Test three models in parallel. Twenty minutes of comparison beats a week of guessing.
Generate stills before video. Approve the character's look as a still image first. Regenerating a still is cheap; regenerating twenty seconds of video is not.
Shoot in short segments. Ten to fifteen seconds per generation, anchored to the same references and keyframes.
Log continuity as you go. Update the continuity sheet after every accepted shot, not at the end.
Assemble and repair. Edit for rhythm first, then run restoration, deflicker, and grade.
Review at three scales. Full resolution, thumbnail grid, and phone screen. Each reveals a different class of defect.

Common Mistakes and FAQ

Mistakes that quietly ruin face consistency

Paraphrasing the identity anchor. Consistency lives in literal repetition. Rewriting your character description "to keep it fresh" resets the identity.
Using a large, inconsistent reference set. Ten mismatched photos teach the model that the face is variable.
Changing lighting without motivation. The most common cause of "why does this feel like a different scene?"
Occluding the face with hands or props. Occlusion is where features melt.
Over-restoring in post. Aggressive face enhancement reads as fake more than mild softness does.
Generating everything before editing. Edit early; the cut hides problems that generation cannot fix.

How many reference images do I actually need?

Eight to twelve consistent images covering frontal, three-quarter, profile, and a couple of expressions. Consistency of the set matters more than its size.

When should I train a character model instead of using references?

When the character appears in more than roughly a dozen shots, or when the project is episodic and the character will return in future sessions. Below that threshold, reference conditioning is faster overall.

Why does my character change age across a clip?

Age drift usually comes from a long clip length combined with reference weight that decays over time. Generate shorter segments and anchor each one to a keyframe.

Can I fix a drifting face in post?

Partially. Face restoration can pull features back toward a reference, and grading can mask small differences. It cannot repair a fundamentally different person. When drift is severe, regenerate.

Do stylized projects have the same problems?

Identity drift is identical; the visual tolerance is higher. Stylized faces can deform substantially before an audience objects, which is why animation-style characters are more forgiving than photoreal ones.

What is the fastest way to improve results today?

Stop paraphrasing your character description, add matched start and end frames to every shot, and switch to singles instead of two-shots. Those three changes fix the majority of consistency complaints before you touch a single setting.