Why character consistency is the real bottleneck in AI video
Almost everyone who starts making narrative AI video hits the same wall within a week. A single shot looks astonishing. The lighting is right, the face is expressive, the camera move feels intentional. Then you generate the next shot, cut them together, and the illusion collapses. Your lead character's jaw is a little wider. Her eyes have drifted from green to hazel. The leather jacket that was matte is suddenly glossy, the hairline has moved two centimetres north, and she appears to have lost three years of age between cuts.
Audiences are remarkably forgiving about physics. They will accept a car that morphs slightly on impact, a crowd that behaves strangely, or a background that dissolves into soup. They will not forgive a face that changes identity between two shots, because faces are the one thing the human brain is hardwired to track. A viewer may not be able to explain why your short film feels off, but they will stop caring about the story the moment they stop believing the person in it.
Consistency problems come from a handful of predictable sources:
- Stateless generation. Most video models do not store a persistent identity. Every render is a fresh sample from a probability distribution, and small prompt or seed changes move you to a different point in that distribution.
- Competing instructions. A style prompt that says "gritty neo-noir" can easily override a character prompt, quietly changing skin tone, contrast, and facial structure.
- Reference starvation. One blurry headshot is not enough information to reconstruct a face across profiles, extreme close-ups, and full-body action.
- Pipeline drift. If each shot is upscaled with a different tool, or colour graded differently, or interpolated at a different frame rate, the audience reads that as a change in the character.
The fix is not a single magic setting. It is a production discipline: build a reference ground truth, lock your language, choose models per shot type, and run continuity checks like a script supervisor would. What follows is a complete working method you can reuse across a short film, an episodic series, or a character-driven brand campaign.
Building a reference pack: the visual ground truth
Everything downstream depends on this step. A reference pack is a small, tightly controlled set of images that defines who your character is. Treat it as a casting photograph session, not a mood board.
What belongs in a reference pack
Aim for eight to twelve images per principal character, all generated or shot in the same session with the same lighting logic and the same approximate focal length. Include:
- Straight-on front view, neutral expression
- Three-quarter view, neutral expression
- Full profile, left and right
- Close-up of the face, filling the frame
- Full-body standing pose, neutral stance
- Two or three signature expressions (warm smile, fear, determination)
- Costume detail shots — collar, cuffs, boots, jewellery, insignia
- One action pose that shows silhouette in motion
Hairline, ears, and jaw shape are the strongest identity anchors, so front and profile views are non-negotiable. Silhouette matters more than you think: if your character reads clearly as a shape, you can get away with a softer face render in wide shots.
Turning raw images into usable references
Raw outputs need cleaning before they become references. Crop every image to the same aspect ratio, remove busy backgrounds in favour of a flat neutral backdrop, and upscale only once with a single tool so you are not stacking artefacts. Then name files systematically:
kira/front_neutral_v3.png
kira/profile_left_neutral_v3.png
kira/costume_jacket_detail_v3.png
Version numbers matter because you will iterate. Create a locked folder and freeze a version once you are happy with it. Every downstream shot should point at the locked set, never at a working folder you are still editing. When a character starts drifting across a production, the first diagnostic question is always the same: are we still using the locked references, or did someone quietly swap in a newer experiment?
Reference packs for non-human characters
Creatures, robots, and stylised characters need extra attention to mechanical details. Panel lines, joint shapes, eye lights, and asymmetric features are the equivalents of a human face. If your robot has a specific arrangement of vents on the left cheek, every reference should show it clearly, and your prompt should describe it in the same words every time.
Choosing the right model for each shot type
No single model is best at everything, and trying to force one to be is the fastest route to inconsistency. Instead, assign models to shot types and accept that your pipeline will be a chain rather than a single button.
| Shot type | What the model needs to do well | Decision criteria |
|---|---|---|
| Locked keyframe | Follow a detailed character prompt exactly | Prompt adherence, identity strength, still quality |
| Dialogue close-up | Preserve face over 3–6 seconds | Identity retention, micro-expression quality, lip sync |
| Action sequence | Handle motion without melting anatomy | Temporal coherence, limb stability, motion blur realism |
| Wide establishing shot | Maintain world style and scale | Style consistency, camera control, background stability |
| Insert / detail shot | Match grade and texture | Texture fidelity, colour response |
When you evaluate a model, score it on six things: identity lock, motion realism, prompt adherence, controllability (pose, depth, camera move), maximum clip length, and how clean the output is at the edges. Write those scores down. A model with brilliant motion but weak identity lock should be used for crowd and background work, never for your protagonist's close-ups.
Two more practical notes. First, prefer image-to-video over text-to-video for any shot containing a principal character, because it inherits identity from your locked reference instead of inventing it. Second, keep a fixed set of supporting tools: one upscaler, one interpolation tool, one colour pipeline, one face-detail pass. Mixing tools per shot is one of the most common hidden causes of drift.
Writing a character bible that AI models can actually follow
A character bible is not prose for humans. It is a structured block of language you paste, unchanged, into every prompt that features that character.
The identity block
Build it in fixed order so the model sees the most important tokens first:
SUBJECT: adult woman, late 30s, athletic build, 172 cm
FACE: oval face, strong jaw, high cheekbones, slightly asymmetric left brow,
small scar above right lip
HAIR: dark auburn, shoulder length, centre part, subtle waves, tied back in action scenes
EYES: deep green, hooded lids, thick dark lashes
WARDROBE: charcoal wool overcoat, black turtleneck, worn leather gloves
SILHOUETTE: clean vertical line, narrow shoulders, long coat hem
PALETTE: #2B2B2E, #101014, #7A3B2E accent
LENS & GRADE: 50 mm, shallow depth of field, cool shadows, warm skin tones
Drift guards and negative prompts
Write a short negative list and keep it identical everywhere: no facial hair, no glasses, no bright saturated colours, no plastic skin, no wide-angle distortion on close-ups, no change of hair length. Consistency comes from repetition, not variety. The moment you start paraphrasing your own character description, you have introduced a variable.
Locking style separately from identity
Keep two blocks in your prompt: an identity block that never changes, and a scene block that changes every shot. Style words belong in a third, global style block shared by every shot in the production — the grade, film stock feel, grain, contrast curve. Never let scene-specific style words leak into the identity block, and never let identity words drift into the style block.
A repeatable shot-by-shot workflow from script to locked frames
Here is the production loop that keeps drift under control. It takes longer per shot than one-click generation, but it saves entire scenes from being discarded.
Step 1: Beat-board the script
Break your script into beats, then into shots. For each shot record: shot ID, characters present, wardrobe state, time of day, camera move, and emotional beat. This ledger becomes your continuity source of truth.
Step 2: Generate the style frame
Create one hero frame that establishes the look of the whole production — grade, grain, contrast, lens character. Approve it before generating anything else. Everything is compared against this frame.
Step 3: Generate keyframes per shot
Using locked references plus the identity block, produce a still keyframe for each shot. Reject anything with even mild identity drift at this stage; it will get worse once motion is added, not better.
Step 4: Animate from keyframes
Feed approved keyframes into an image-to-video model. Keep the motion prompt short and physical: "slow push in, subtle head turn to camera, coat hem moving in wind." Long poetic motion prompts introduce randomness.
Step 5: Motion and detail passes
Add secondary motion, fix hands and eyes, and stabilise frames that flicker. Do detail restoration at low strength — strong face enhancement is the number one cause of "same costume, different person" syndrome.
Step 6: Assemble and check continuity
Cut the scene together before any final grade. Watch it three times at normal speed, then once frame by frame around every cut. Note every identity wobble in the ledger.
Step 7: Grade and deliver
Apply one consistent grade across the entire scene. If a shot cannot be graded to match, regenerate it rather than patching it. Patched shots always read as patched.
Fusion techniques: blending references and models for fidelity
"Fusion" in practice means combining multiple signals rather than trusting a single generation. Four techniques are worth building into your routine.
Multi-image reference blending. Instead of feeding one portrait, feed three to five complementary references: front, three-quarter, and a costume detail. Most reference-capable models weight these images to construct a more stable identity than any single photo can provide.
Character-specific training. If a character appears in dozens of shots, train a small identity adapter on your locked reference pack. This gives you a reusable identity token that behaves consistently across scenes and even across different base models.
Layered compositing. For complex shots, generate a clean plate (background and lighting only), generate the character on a neutral background, then composite. This keeps the character's render settings identical between shots and makes the world the only variable.
Final-pass face consistency. When two shots still do not match, a targeted face pass can harmonise them — but apply it to the whole scene, not just the offending shot, or you create a new mismatch with everything around it.
Continuity across scenes, outfits, aging, and episodes
A cinematic universe is not a single video. It is a library of videos that must agree with each other. That requires a continuity ledger that is more like a costume department's paperwork than a prompt file.
- Wardrobe variants. Give each costume a code (A1 winter coat, A2 winter coat damaged, B1 field gear). Reference images exist per variant, not per character.
- Damage and progression states. If a jacket is torn in episode four, it stays torn until it is repaired on screen. Generate a new locked reference when a state change occurs rather than describing it ad hoc.
- Time jumps and aging. Produce a new reference pack for each era of a character's life. Do not try to age a character through prompt words alone; you will get a different person each time.
- Lighting continuity. Keep a shot list of time-of-day and practical light sources. Faces rendered under wildly different colour temperatures read as different characters even when the geometry matches.
- Cross-episode checks. At the start of each new episode, place a frame of the character beside the previous episode's hero frame. If you cannot tell they are the same person at a glance, fix it before you shoot anything.
Quality control checklist and common mistakes
The checklist below takes five minutes per scene and prevents most rework:
- Identity block unchanged from the master document
- Locked reference pack used, not a working folder
- Keyframe approved before animation
- One upscaler, one interpolator, one face pass across the whole scene
- Aspect ratio and frame rate identical across shots
- Colour temperature consistent between adjacent cuts
- Hands, eyes, and hairline checked at full resolution, not thumbnail
- Audio recorded or generated only after picture lock
Common mistakes that break consistency, roughly in order of how often they appear:
- Over-prompting. Twenty lines of style language drown out the four lines that define the face.
- Chasing a better single shot. Regenerating one shot with new settings to make it prettier almost always costs you the match with its neighbours.
- Heavy face restoration. Full-strength enhancement standardises faces into a generic template and erases your character's distinctive features.
- Mixing aspect ratios. A shot rendered in a different framing changes apparent facial proportions and reads as a different person.
- No locked reference version. If the reference set keeps moving, the character keeps moving with it.
- Grading at the end with no reference frame. Without a hero frame to match against, each shot drifts in a different direction.
FAQ
How many reference images do I really need? For a minor character who appears in three or four shots, three images (front, three-quarter, costume detail) are enough. For a lead, eight to twelve across angles and expressions. More is not automatically better — inconsistent references are worse than few.
Can I keep the same character across different models? Yes, but only with a strong identity block and a consistent reference pack. Expect some reinterpretation, so keep your protagonist in one primary model per production and use others for background or inserts.
Why does my character look right in stills but wrong in motion? Motion models sample across time and can accumulate drift. Always animate from an approved keyframe, keep motion prompts short and physical, and check the first and last frames of every clip against your references.
How do I handle two characters in the same shot? Generate each character separately on a neutral background with identical lighting and lens settings, then composite. Models handle multi-character prompts poorly and tend to blend features between subjects.
Do I need to rebuild references for a sequel? You should re-approve them. Even without deliberate change, a new production introduces new lighting, wardrobe, and grade. Re-lock the reference pack against your new style frame before generating anything.
What is the single highest-impact fix? Lock your reference pack and stop editing it mid-production. Most consistency failures are pipelines, not models.
Turning the method into a habit
The teams that ship coherent AI video series are not using secret models. They are running a disciplined pipeline: a locked reference pack, a fixed identity block, a model assigned per shot type, and a continuity ledger that a script supervisor would recognise. The first episode you make this way will feel slower. The third will feel faster than your old one-click approach, because you will stop throwing away scenes.
Start small. Pick one character, build a twelve-image reference pack, write one identity block, and produce a single sixty-second scene with three shots. Check the cut frame by frame. Once that scene holds together, scale the process to an episode, then to a season. Consistency is not a feature you switch on — it is a habit you build shot by shot, and it is the difference between a collection of impressive clips and a universe an audience believes in.
