Every AI Feature Needs a Second Writer

The fine-tuned model is the interesting part of my pipeline and the least trustworthy thing in it.

It runs on a serverless GPU I do not operate, behind a queue I cannot see into, in a container I built myself, serving weights I trained on 400 examples. Every other component in the stack has more production hours behind it than my adapter has minutes.

So it does not get to be load-bearing. A commodity model sits behind it and can take the next request without a deploy, without a code change, and without the user learning which one answered.

That is not hedging. It is the condition under which shipping the custom model was a reasonable thing to do at all.

I have written about the general shape of reliable AI systems before. This is the concrete implementation, including the part I got wrong first.

The naive version fails at the metric, not the code

Writing a fallback is easy. try the good model, catch, call the other one. Twenty minutes.

Then you add a counter, because an unmonitored fallback is just a slower way to be broken, and the counter immediately becomes useless.

Here is why. There are four ways a request can end up served by the commodity writer, and only one of them is a problem:

Rows two and three are the system working as designed. The adapter is English-only, so every Spanish request routes to the commodity writer on purpose — that is a correct outcome, not a degraded one. If I count it as a fallback, my fallback rate sits permanently near 90%, and the number that is supposed to tell me the custom model is sick tells me nothing, because it is already at 90% on a healthy day.

So the distinction is drawn in code rather than in a dashboard filter someone has to remember:

// Being switched off or skipped by design is not a failure, so those stay
// plain 'openrouter'. Only a genuine RunPod problem counts as a fallback.
const isRealFailure =
  failureReason !== null &&
  failureReason !== 'disabled' &&
  failureReason !== 'locale_unsupported'

Three provider values, not two. runpod, openrouter, openrouter_fallback. The middle one is the by-design route and it is logged as a success.

An alert you have learned to ignore is worse than no alert, because it costs attention every time it fires and buys nothing. Most of the work in monitoring is deciding what does not count.

The second writer has to be held to the same bar

The failure mode I did not anticipate: a fallback path that quietly lowers quality.

If the custom writer is checked for safety and length and the commodity writer is not, then every failure of the custom model becomes an unchecked response. The fallback would be doing the opposite of its job — turning a loud failure into a silent quality regression, which is the exact trade the whole design is meant to prevent.

So the LoRA's output goes through the same quality gate the commodity writer already passed, and tripping any of these routes the request to the other model:

const DISQUALIFYING_QUALITY_FLAGS = [
  'unsafe_language',
  'too_technical',
  'mentions_literal_objects',
  'too_long',
]

mentions_literal_objects is a product rule, not a safety one — the feature exists to say what a photo feels like, and a reaction that inventories what is in the frame has failed even though it is a perfectly good sentence. A model that produces one is not having a bad day. It is off-spec, and off-spec output should be replaced rather than shipped.

One flag is deliberately excluded, and the comment explains why so nobody adds it back:

/**
 * `too_generic` is excluded: it is a soft style signal that the OpenRouter path
 * retries rather than hard-fails on, and treating it as fatal here would inflate
 * the fallback rate without a safety benefit.
 */

That is the same judgment as the provider taxonomy, one layer down. A signal worth logging is not automatically a signal worth failing on.

Fifteen ways to fail, none of them the user's problem

The client for the custom writer never throws. Every path resolves to a typed result:

export type RunpodFailureReason =
  | 'disabled' | 'missing_api_key' | 'missing_endpoint_id'
  | 'invalid_vision_json' | 'timeout' | 'network_error'
  | 'unauthorized' | 'rate_limited' | 'server_error'
  | 'service_unavailable' | 'http_error' | 'malformed_json'
  | 'job_failed' | 'job_not_completed' | 'missing_output'
  | 'empty_reaction' | 'prompt_leakage' | 'reaction_too_long'

Two things about that list.

It distinguishes my failures from the platform's. missing_api_key is a deploy mistake. rate_limited is capacity. prompt_leakage is the model itself misbehaving. Collapsing them into error would leave me reading raw logs at exactly the moment I need an answer fast.

None of it reaches the client. The reason goes to a database column and a structured log line. The mobile app receives the same response shape it always receives, from whichever writer produced it. A user is not owed my infrastructure taxonomy, and a response format that changes based on which model answered would leak the fallback into every client that has to parse it.

The typed-result choice also means the fallback cannot be skipped by accident. An exception thrown three frames deep can escape a try block someone refactored. A returned union type cannot — the compiler makes the caller handle it.

The kill switch is a config value

The rollback for the entire custom-model path:

supabase secrets set RUNPOD_WRITER_ENABLED=false

No redeploy. No build. The next invocation reads the new value and every request goes to the commodity writer.

This matters more than it looks. A rollback that requires a deploy is a rollback with a build step, a queue, and a cold start between me and the fix — and it will be needed at the worst possible time, because that is when rollbacks are needed. Any switch I would want to flip during an incident has to be reachable without shipping code.

The same principle covers the language routing. Supported locales are parsed from config rather than hardcoded:

export function runpodSupportsLocale(locale: string, supportedLocales = ['en']): boolean {
  return supportedLocales.includes(writerLanguage(locale))
}

When a Spanish adapter finishes training, widening the custom model to a second language is RUNPOD_WRITER_LOCALES=en,es. If that check had been if (locale === 'en') — which is what I wrote first, and it was shorter and clearer — the retrain would have needed a code change, a review, and a deploy to reach production. The config version is worse code and a better system.

The honest cost

Two writers means two prompt formats, two output validators, two sets of tests, and a permanent obligation to keep the commodity path working even during the long stretches when it serves almost nothing. It is dead weight most of the time.

It is also the only reason I sleep after deploying a model I trained myself.

And there is a failure mode the fallback cannot catch, which is worth naming since it is the one that will eventually get me: the fallback only fires on outputs that are detectably wrong. Empty, leaked, unsafe, too long, off-spec. A reaction that is fluent, well-formed, on-topic and simply worse than the commodity model would have produced sails straight through every check and gets recorded as runpod success.

My instrumentation can prove the custom writer is alive. It cannot prove it is better. That needs an eval set and side-by-side human scoring, and until I run that, the honest statement is that I have a reliable way to serve a model whose superiority I have not measured.

Which is a strange thing to publish about your own work, and also the actual state of it.