There are two serious architectures for keeping an AI character consistent: fine-tuning (training a LoRA on the character's face) and reference conditioning (attaching verified photos of the character to every generation). Both work. They fail differently, cost differently, and suit different people — here's the engineer's-eye comparison.
How each one works
LoRA: collect 20–50 images of the face, train a small adapter on top of a base image model until it learns the identity as weights. Generation then needs only a trigger word — the face is baked in.
Reference conditioning: maintain a curated library of verified character images; each generation request attaches the most relevant references, and the model copies identity from pixels at inference time. Nothing is trained; the library is the identity.
Where LoRA wins
- Identity strength at the extremes. A well-trained LoRA holds the face through stylization, unusual compositions, and crowded scenes where reference conditioning can weaken.
- Marginal cost. Once trained, generation is just inference — no reference plumbing per request. If you run your own GPU, marginal images are nearly free.
- Portability. The LoRA file works in any compatible local pipeline; you own the artifact outright.
Where LoRA hurts
- The dataset is destiny — and it's frozen. Train on a subtly inconsistent dataset and the drift is baked into the weights. You can't curate it out afterward; you re-collect and re-train. This is the most common DIY failure: the reference-count question answered sloppily, permanently.
- The bootstrap paradox. A LoRA needs 20–50 consistent images of a person who doesn't exist yet — which means you need a consistency pipeline before you can train. (Solution in practice: build a golden set first, then optionally train on it later — the two architectures compose.)
- Time and iteration. Realistically 10–30 hours across dataset prep, captioning, training runs, and evaluation before the first production-grade results. The skills are learnable and genuinely fun for tinkerers — and a tax for everyone else.
- Frozen identity, frozen base. New base models arrive constantly; your LoRA is married to the one it was trained on. Migrating means retraining.
Where reference conditioning wins
- Zero training, minutes to operational. Seed → generated coverage matrix → automated face filtering → curate → locked. The identity exists the same afternoon.
- Correctable forever. A bad reference gets deleted from the library; a bad training image lives in the weights forever. Curation beats retraining as a maintenance model.
- Model-agnostic identity. The library is just photos — when a better generation model ships, the same references condition it. The identity outlives any particular model.
Where reference conditioning hurts
- Per-request reference selection adds plumbing (handled for you on a platform, real work if self-built).
- At the artistic extremes — heavy stylization, complex multi-character scenes — conditioning loosens before a strong LoRA does.
The economics, honestly
DIY LoRA: $5–20/training run in GPU rental plus your hours; near-zero marginal images if self-hosted. Platform: ~$25 per character, $0.25 per image, zero hours of infrastructure. The crossover favors DIY only when your time is free and volume is enormous — for an operator running accounts rather than a hobbyist running experiments, the platform math wins long before the first thousand images.
The pragmatic answer
They're not actually rivals: the highest-end pipelines build a verified golden set first (it's the quality-controlled dataset a LoRA wants anyway) and graduate to fine-tuning only if they hit conditioning's limits. Start with the path that takes minutes — a curated, ArcFace-verified character for $19 — and keep the LoRA option in your back pocket; your locked reference library is the training set, pre-cleaned, if that day comes.