The instinct, when you decide a product needs a custom model, is to fine-tune the thing the user touches. The user uploads a photo, so fine-tune something that takes photos.
I did the opposite. The adapter I trained for VIOFY has never seen a user photo and never will.
It is a 3B text model that receives a structured emotional brief and returns one or two sentences. Every pixel is handled upstream by a general-purpose vision model I rent by the token and could swap tomorrow.
That split is the decision that made shipping a custom model to production defensible at all.
The two jobs inside one feature
The feature looks like a single capability: look at this photo and react to it like a friend would.
It is two jobs with almost nothing in common.
Perception. What is in the frame, what the mood reads as, is this a gym photo or a travel photo, is anything here unsafe. Everyone needs this. Everyone's version is roughly the same. Frontier labs are competing to sell it to me at a falling price.
Voice. Sounding like a specific friend, in a specific register, that says This feels earned, like the work happened when nobody was watching instead of Great photo! The lighting really works here. Nobody sells that. It is the entire product.
A single model doing both is a model where the part I cannot buy is welded to the part I can. Improving perception means retraining voice. Changing voice means re-validating perception. The two get better at completely different rates, and one of them gets better for free while I sleep.
So the pipeline separates them:
image
→ general vision model (rented, swappable, sees the photo)
→ structured vision JSON (validated + normalized at the seam)
→ LoRA writer on serverless GPU (mine, never sees the photo)
→ one short reaction
Fine-tune the layer you cannot rent. Rent the layer that is becoming a commodity. If you cannot say which layer is which, you are not ready to fine-tune anything.
What the writer never receives
The vision stage returns a fairly rich object. Category and confidence, secondary categories, reasoning, perceived mood, emotional signals, and also lighting_and_colors, subjects_and_composition, key_contextual_objects, plus a set of risk flags.
The writer receives this:
{
"category": "fitness",
"emotional_focus": ["discipline", "consistency", "progress", "self-respect", "effort", "growth"],
"avoid_focus": ["body rating", "sexualized comments", "unsafe fitness advice", "only describing muscles"],
"vision_summary": "focused. discipline, effort, quiet determination. Training effort is the emotional center.",
"mood": "focused",
"language": "en",
"max_new_tokens": 80,
"temperature": 0.75
}
No image. No URL. No composition. No object list.
The omissions are deliberate, and the code says so at the point where it happens:
/**
* Build the writer-facing summary from emotional context only.
*
* Deliberately skips `subjects_and_composition` and `key_contextual_objects`:
* feeding literal object/framing descriptions to the writer is what pushes it
* toward inventory-style output, which the product rules forbid.
*/
I learned this the expensive way. Give a language model a list of what is visible and it will tell you what is visible. Love the dumbbells and the natural window light is a correct sentence about the photo and a complete failure of the product, which exists to answer what does this feel like, not what is in frame.
The fix is not a stronger instruction telling the model to avoid describing objects. The fix is not sending it the objects. An instruction is a request. A missing field is a guarantee.
The same logic runs the other direction through avoid_focus, which is not model output at all. It comes from a hand-written per-category guideline map. For fitness it forbids body rating and unsafe advice. That is a product rule, and product rules do not belong in weights where they can only be changed by retraining. They belong in a JSON file that a human can edit and a test can assert against.
The seam is a validation boundary, not a mapping
The easy version of this architecture pipes one model's output into another model's input. That is not what the middle arrow is.
Between the two stages sits a function whose job is to refuse:
// Without any emotional context the writer has nothing to react to, and the
// LoRA will fall back to generic filler. Treat that as a pipeline error.
if (!visionSummary && !perceivedMood && emotionalSignals.length === 0) {
return { ok: false, reason: 'invalid_vision_json', detail: 'no_emotional_context' }
}
It also collapses uncertainty rather than passing it along:
export function resolveEffectiveCategory(primaryCategory, confidence, unclearCategory) {
if (!isGuidelineCategory(primaryCategory)) return FALLBACK_CATEGORY
if (unclearCategory === true) return FALLBACK_CATEGORY
if (typeof confidence === 'number' && confidence < 0.5) return FALLBACK_CATEGORY
return primaryCategory
}
A vision model that is 40% sure this is a couples photo is a vision model that does not know. Forwarding couples at 0.4 hands the writer a confident-looking category built on a coin flip, and the writer has no way to know it should hedge, because it was fine-tuned to commit. Better to say general_lifestyle out loud.
Every string that crosses is trimmed, collapsed, deduplicated and length-capped. Not because the vision model is adversarial, but because the field between two models is exactly where an unbounded string turns into a prompt-injection surface, and "the upstream model probably won't do that" is not a security property.
The boundary is symmetric. Output gets validated on the way back with the same suspicion:
const LEAKED_PROMPT_MARKERS = ['System:', 'User:', 'Assistant:', 'You need to', 'You have to']
If any of those appear in a reaction, the run is rejected as prompt_leakage and the request is served by the commodity writer instead. That check exists because it caught a real failure: a missing chat template put the adapter out of distribution and it started echoing its own scaffolding into user-facing text.
Fifteen typed failure reasons, from missing_api_key to reaction_too_long, and the client never sees any of them. They go to a metrics column so I can tell slow from broken without guessing.
What the split actually bought
A swappable perception layer. A better vision model ships, I change one config value, and the adapter is unaffected because its input contract did not move. That is the whole point of the JSON in the middle.
Privacy that is structural rather than promised. A photo cannot leak out of a component that never received it. My custom GPU worker, the least battle-tested part of the stack, has no access to user images. That is an absence, not a policy.
Tests that run in milliseconds. The client module reads no globals (config and fetch are injected), so the entire writer path is unit-testable on a laptop with no GPU, no Deno runtime, and no API key. The parts of an ML feature most likely to break are the parts that need no accelerator.
A place to put the guardrails. Safety rules, category direction, length limits and language routing all live in ordinary code around the model. When a rule changes, I ship a deploy, not a training run.
The part that is easy to leave out
Two honest costs.
The first: the brief is a ceiling. The adapter can only be as good as the summary it receives. If perception misreads a graduation as a nightlife photo, the writer will produce a fluent, well-toned reaction to the wrong moment, and it will do it confidently, because it was trained to. My worst outputs are not writing failures. They are category failures wearing good prose. So the stage worth monitoring is the vision call, not the adapter I trained.
The second, and the one I would skip if I were selling this: the custom model serves a minority of traffic. The adapter is viofy-qwen2.5-3b-lora-en, English only. VIOFY supports eleven languages. So the locale router sends ten of them to the rented writer, and that path is recorded as a plain success rather than a fallback, because being skipped by design is not a failure and should not pollute the failure metric.
I spent weeks on a model that currently writes for one language out of eleven.
I would do it again, and the reason is the architecture rather than the model: because voice is routed rather than welded, widening it is a config change (RUNPOD_WRITER_LOCALES=en,es) the day a Spanish adapter finishes training. No code change. No redeploy of the pipeline. And on any request where the custom writer is off, unsupported, slow, or wrong, a boring commodity model finishes the job and the user never learns which one they got.
The rule I would give someone starting this
Before fine-tuning anything, split the feature into the part that is becoming a commodity and the part that is your product. Rent the first. Train the second. Put a validated, typed, deliberately narrow contract between them, and treat every field you leave out of that contract as a feature rather than an oversight. The photo never crosses because the design says it never crosses.
A tiny vote of confidence
Was this note worth your time?
Notes from the build
More from the AI engineering trenches
Evals, observability, prompt design, and real lessons from shipping AI products. No fluff.
By subscribing, you agree to receive Orlando's emails. No spam. Unsubscribe anytime.