Why Character Consistency Is Still the Hardest Problem in AI Video
Ask anyone who has finished a short narrative film built with generative video what nearly defeated them, and the answer is rarely render time or prompt writing. It is watching a hero's face change shape between shot four and shot five. One convincing frame is easy. A sequence in which the same invented person walks through a door, sits, argues, and turns to camera without their jawline migrating is a different class of problem entirely.
The reasons are structural rather than a matter of effort.
Every generation is a fresh sample. Diffusion and video models do not remember a person the way a film camera does. They reconstruct an image from noise, conditioned on your prompt and any reference inputs. When conditioning is weak, small changes in wording, framing, or motion resolve into a visibly different human.
Language describes categories, not individuals. A description like a woman in her early thirties with auburn hair and a scar over one eyebrow fits thousands of people. The model will cheerfully paint a new one on every run.
Identity lives in geometry. The ratio of eye spacing to nose length, the angle of the jaw, the way the hair parts, the thickness of the brows. These are spatial facts that text is poor at specifying.
Motion amplifies error. The moment a character turns their head or speaks, the model re-interprets the face from an unseen angle. Small deviations compound frame over frame, which is why a clip can look fine at second one and wrong at second six.
So consistency is not won by writing better adjectives. It is won by giving the model a durable visual reference and then holding everything around that reference stable. That is a pipeline problem with a pipeline solution, and the rest of this guide walks through it stage by stage.
Start With a Character Bible, Not a Prompt
Before you generate anything, write down who this person is. Not in a paragraph of backstory, but in a compact specification you will copy and paste into every tool for the life of the project.
Every usable character bible covers four pillars.
| Pillar | What you decide | Why it matters on screen |
|---|---|---|
| Silhouette | Height, build, posture, hair volume, signature clothing shape | Lets viewers recognise the character even in a wide or backlit shot |
| Face | Age, bone structure, eye colour and shape, brows, nose, mouth, distinctive marks | The core of identity, and the part that drifts first |
| Wardrobe | One primary outfit plus one variation, exact colours and fabrics | Colour is the cheapest continuity anchor you have |
| Behaviour | How they stand, gesture, walk, react under stress | Translates directly into motion and performance prompts |
The visual DNA sheet
Write a fixed descriptor block, a sentence or two of roughly thirty to fifty words, that you paste into every prompt, verbatim, in every tool. Repetition is the point. The block might read: Mira, 34, Spanish-Portuguese, olive skin, sharp jaw, straight dark brows, hazel eyes set wide, small scar through left eyebrow, black hair in a low knot, charcoal wool coat over slate turtleneck, calm, still hands.
Note what is and is not in there. No emotion words, no camera words, no lighting words. Those change per shot, and mixing them into the identity block is how people accidentally corrupt their own anchor. Keep the block pure identity; keep everything else in a separate, per-shot prompt layer.
Behaviour belongs in the bible too
Amateur character work stops at appearance. Professional work specifies performance. Does this person lean forward when listening? Do they fidget? Blink slowly, hold eye contact too long, avoid it entirely? Write four or five behavioural rules and turn them into motion prompt fragments such as slow deliberate blink or shoulders squared, minimal hand movement. This is the difference between a good-looking avatar and a character an audience reads as a person.
Building the Reference Set: From Concept to Identity Anchor
The reference set is the raw material that teaches a model who your character is. Ten to twenty images is the sweet spot for most approaches: enough to cover variation, few enough that every image is genuinely on-model.
Step by step
- Establish one canonical portrait. Front-facing, neutral expression, even lighting, plain background. This is the master reference, and everything else is judged against it.
- Shoot the rotation. Add three-quarter left, three-quarter right, profile, plus a slight low and high angle. Faces are three-dimensional, and video will ask for every angle sooner or later.
- Cover expression range. Neutral, small smile, talking, frowning, surprised. Identity should survive emotion.
- Cover lighting conditions. Soft daylight, hard side light, warm interior light, low key. If your references are all lit identically, the model will fight you in any scene with different light.
- Add two full-body frames. One static standing, one mid-motion, to lock proportions, height, and wardrobe silhouette.
- Curate ruthlessly. Delete anything with an off-model jaw, inconsistent hairline, wrong eye colour, or artefacts. A single bad reference teaches the model exactly the wrong lesson.
What makes a reference image usable
Good references share a handful of properties: the face occupies a consistent proportion of the frame, the eyes are clearly visible and in focus, there is no heavy shadow across the nose or brow, resolution is high enough that skin texture is legible, and the character is the only subject. Background clutter, strong motion blur, and heavy stylisation all reduce a reference's usefulness.
The mistakes that ruin a reference set
- One lighting setup everywhere. Produces characters that only work in that light.
- Generated references treated as ground truth. If a reference image disagrees with the others, you are training on noise. Regenerate or retouch it.
- Age and hairstyle drift across the set. Your output will average the two.
- All beauty shots, no utility shots. No neutral angle means no clean identity anchor.
- Too many images, too little curation. A tight set of twelve strong images outperforms a loose set of forty.
Choosing the Right Model for Each Stage
No single model excels at every stage of a character build, so treat your stack as a pipeline rather than a favourite tool. Four stages matter.
Concept and key art
Use the strongest image model you have access to for ideation. This is where you explore, not where you commit. Expect to discard most outputs. Push style prompts sideways, across different eras, lenses, and art directions, to find the silhouette that reads best at thumbnail size.
Identity locking
This is the decisive stage. Three broad approaches exist, and they trade effort against precision:
- Reference conditioning. You supply one or more images at generation time and instruct the model to preserve the subject. Fast, no training, works well for short sequences. Weakest at extreme angles.
- Fine-tuning a small personal model. You train a lightweight adapter on your curated set. Highest fidelity and best angle coverage, but it requires a clean dataset, compute time, and patience.
- Face-region compositing. You generate the shot, then transplant a face from a reference. Useful for salvage work, dangerous as a default because lighting and angle mismatches are visible.
Most projects should start with reference conditioning and graduate to fine-tuning only when a character will appear in many scenes.
Video generation
Video models vary enormously in how well they hold a reference across motion. Test each candidate on the same three shots before committing: a slow head turn, a walking shot at medium distance, and a talking close-up. The talking close-up is the honest test; everything looks fine in a slow pan.
Finishing
Upscaling, face restoration, and colour matching are where a sequence starts to feel like one film. Apply the same restoration settings across all shots. Inconsistent restoration is a subtle but real source of continuity failure.
Locking Identity Across Shots: Techniques That Actually Hold
With a reference set and a chosen stack, consistency becomes a discipline of repetition.
Reference conditioning in a stable order
Always supply the reference images in the same order, with the same weights, and with the canonical portrait first. Ordering changes output more than most people expect.
Seed discipline
Fix the random seed for every shot within a scene. When you need variation, change one variable at a time, whether that is framing, expression, or light, and then re-lock. Random seeding across an entire scene is the fastest way to produce a slideshow of near-strangers.
The fixed descriptor block
Paste your identity block unchanged into every prompt and place per-shot instructions after it, clearly separated. Never paraphrase the block to make a prompt read more naturally. The model does not care about prose quality, only about consistency of tokens.
Wardrobe, props, and light continuity
Colour-grade your whole scene to the same palette before you generate, not after. Keep props identical: same mug, same coat, same bag. Continuity of objects does as much work as continuity of face, because audiences use context to confirm identity.
A Shot-by-Shot Workflow for a Character-Led Scene
Here is a workflow you can run end to end on a two-minute character piece.
1. Pre-production
Lock the character bible, the reference set, the colour palette, and the shot list. Decide how many shots the scene needs and resist adding more later; every new shot is a new consistency risk.
2. Generate an anchor frame per scene
Before any video, produce one still that is unquestionably on-model for each scene, with the correct light, wardrobe, and angle family. Approve it, then treat it as the reference for every video generation in that scene.
3. Work in coverage order
Generate close-ups first while your reference set is freshest and your prompts are cleanest, then mediums, then wides. Wides hide identity, close-ups expose it, so doing the hard shots first means the easy ones cannot embarrass you.
4. Run a side-by-side review gate
Put every approved shot next to the canonical portrait at the same scale. If the jaw, eye spacing, or hairline reads differently, regenerate before you move on. Fixing it at the review gate costs one generation; fixing it after assembly costs a day.
5. Assemble and cut on motion
Cut on movement, sound, and reaction rather than on static frames. Editing rhythm disguises tiny inconsistencies and amplifies genuine performance.
Fixing the Most Common Character Failures
| Symptom | Likely cause | Fix |
|---|---|---|
| Face drifts across shots | Weak or inconsistent reference conditioning | Re-order references, add angles, consider fine-tuning |
| Character looks older or younger | Age adjectives varying in the prompt | Freeze the identity block, remove stray age words |
| Wardrobe mutates | Clothing described differently per shot | Define the outfit once, reuse exact wording |
| Hair changes length or parting | No profile or rear reference | Add profile and back-of-head shots to the set |
| Expression looks pasted on | Reference set has no expression range | Add talking, smiling, and frowning references |
| Flicker or melting in motion | Model struggling with motion complexity | Simplify the action, shorten the clip, add motion blur |
| Style mismatch between shots | Different model or settings per shot | Standardise the stack for a whole scene |
Most of these failures are cheap to prevent and expensive to repair. That asymmetry is the whole argument for building the reference set properly the first time.
Organizing Assets So You Never Rebuild a Character Twice
A character is an asset, and assets need a home. A simple structure works:
/characters/mira/
bible.md
/refs
/training
/scenes
/exports
Name files with character, scene, shot, and version, for example mira_s03_sh07_v04.png. Six weeks later, versioning is the only thing standing between you and a full rebuild. Keep a short note next to every approved shot recording the seed, the model, and the prompt used. This is the least glamorous and most valuable habit in the entire workflow.
Turning One Character Into a Series
Once a character is built, the marginal cost of the second appearance is a fraction of the first. That is where character work pays off: a recurring hero in a channel, a mascot across a campaign, a cast that returns episode after episode.
To scale without losing the thread:
- Freeze the identity block permanently. Treat it as a contract, not a draft.
- Add variations deliberately. Give the character a seasonal outfit or a new prop as a controlled variant, documented as such.
- Build a character family. Secondary characters with their own bibles make crossover scenes straightforward.
- Keep a continuity log. When did the scar appear? Which coat belongs to which period? Write it down once.
- Audit every ten shots. Compare back to the canonical portrait at the same scale. Drift is gradual and easy to miss.
FAQ
How many reference images do I need?
Ten to twenty curated images covers most projects. Fewer than eight makes angle coverage unreliable, and more than twenty-five rarely improves results unless you are fine-tuning.
Can I use AI-generated images as references?
Yes, with care. They must be mutually consistent. Regenerate or retouch any reference that disagrees with the canonical portrait.
Do I need to train a model for one short video?
No. Reference conditioning is usually enough for a single scene. Training becomes worthwhile when the character appears in several scenes or across episodes.
Why does my character look fine in stills but wrong in video?
Motion exposes angles and expressions your reference set never covered. Add profile, expression, and mid-motion references to close the gap.
How do I stop the wardrobe changing between shots?
Write the outfit once as an exact string and paste it into every prompt unchanged. Never paraphrase it for flow.
What is the fastest way to fix one bad shot in an otherwise good sequence?
Regenerate with the scene anchor frame as the reference and the original seed locked. If that fails twice, simplify the shot's action rather than fighting the identity.
Should the whole team use the same model stack?
For a single project, yes. Mixed stacks inside one scene are the most common cause of style drift that no amount of prompt tuning repairs.
