What Actually Breaks When You Put a LoRA Adapter in Production

Every write-up about fine-tuning a small model ends at the same place: the loss curve flattens, a sample generation looks good, and the post concludes.

That is where the work starts.

I fine-tuned a LoRA adapter on Qwen/Qwen2.5-3B-Instruct to write short, specific reactions to photos for VIOFY. Training was the short part. The three failures that cost the most time came after, and they shared one property that made them expensive:

None of them raised an exception. Each one produced output that looked like output. A wrong prompt format still generates text. A broken cache path still resolves to a directory. A container that cannot start still reports a status.

The shape of the deployment

Worth establishing, because it explains why the failures were hard to see.

image
  → general vision model        (rented, general-purpose)
  → structured vision JSON      (validated + normalized)
  → LoRA writer on serverless GPU
  → one short reaction

The adapter never sees a photo. It receives a structured brief (category, emotional focus, things to avoid, a vision summary, a mood) and returns one or two sentences. The base model plus adapter run in a container on serverless GPU; a server-side function holds the credentials and calls it.

The adapter is trained on one narrow job. That narrowness is the point, and it is also what made the first bug invisible.

Failure 1: the adapter shipped a tokenizer without a chat template

The adapter repository contains its own tokenizer_config.json. It has no chat_template key.

The worker loaded the tokenizer from the adapter repo, found no template, and fell back to a plaintext layout:

System:
...

User:
...

Assistant:

Qwen2.5 was trained on ChatML:

<|im_start|>system
...<|im_end|>
<|im_start|>user
...<|im_end|>
<|im_start|>assistant

So every request went to the model in a format it had never seen during either pretraining or fine-tuning. The adapter was out of distribution on every single call.

It still produced English sentences. That is why it took a while to find.

The symptoms, once I knew what to look for:

• reactions that ran to the full max_new_tokens budget instead of stopping, because the model never emitted a stop token it recognized;
• fragments of the system prompt appearing in user-facing output, like You need to write in natural American English showing up inside a reaction;
• role markers like Assistant: leaking into text meant for a product surface.

I had been reading those as prompt-quality problems. They were a formatting problem.

The fix, and the principle behind it

Two changes.

First, always load the tokenizer from the base model, never from the adapter repo:

tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL_ID,          # Qwen/Qwen2.5-3B-Instruct
    token=hf_token(),
    cache_dir=MODEL_CACHE_DIR,
    trust_remote_code=True,
)
ensure_chat_template(tokenizer)

The adapter changes weights. It does not change how turns are delimited. The base model is the authority on that, so it should be the source.

Second, and more important, make the fallback impossible:

def ensure_chat_template(tokenizer) -> None:
    """Fail fast if the tokenizer cannot produce Qwen ChatML."""
    if not getattr(tokenizer, "chat_template", None):
        raise RuntimeError(
            f"Tokenizer for {BASE_MODEL_ID} has no chat_template. "
            "Refusing to start: the LoRA requires Qwen ChatML formatting."
        )

A worker that cannot format prompts correctly should not accept jobs. Refusing to start is a visible failure that costs one deploy. Degrading quietly is an invisible failure that costs weeks and reaches users.

That is the general rule this bug taught me:

A fallback is only safe when the degraded mode is still correct. If the fallback produces wrong-but-plausible behavior, it is not a fallback. It is a bug with a retry policy.

The same reasoning applies one layer down, to stop tokens. Generation terminates on a set of EOS ids, so the worker resolves them explicitly and refuses to boot if the set is empty:

if not EOS_IDS:
    raise RuntimeError(
        "No usable EOS token ids resolved. Generation would never terminate."
    )

For this tokenizer the resolved ids are [151645, 151643]: <|im_end|> plus the tokenizer's own EOS. Empty would mean every request runs to the token limit. Better to never start.

Failure 2: an unpinned dependency removed CUDA support

The base image is pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime. It already provides torch 2.5.1 built against CUDA 12.4.

requirements.txt listed torch, unpinned.

So pip install -r requirements.txt did exactly what it was asked to do: it saw a torch requirement, resolved the newest thing on PyPI, and replaced the CUDA-linked build with a generic wheel. Meanwhile transformers was also unpinned and resolved to a new major, and huggingface_hub moved past the version range that transformers accepts.

Nothing in the build failed. The image built successfully. Then:

• workers crashed on startup;
• the endpoint reported unhealthy workers;
• requests sat in the queue with no error attributable to any request.

No single request was wrong. There was nothing alive to serve them.

The fix is short enough to be embarrassing. Pin everything, and deliberately omit torch:

# torch is intentionally ABSENT: the base image already provides torch 2.5.1
# built against CUDA 12.4. Listing it here makes pip replace that with a
# generic PyPI wheel and breaks GPU support.

runpod>=1.7,<2
transformers==4.57.1
peft==0.17.1
accelerate==1.10.1
bitsandbytes==0.48.1
# transformers 4.57 requires huggingface-hub >=0.34,<1.0
huggingface_hub>=0.34,<1.0
sentencepiece==0.2.2
protobuf>=4,<6

The comment matters as much as the pin. A version pin with no reason attached gets "cleaned up" by the next person to touch the file — including a future version of me.

And since the failure only appears at container start on a GPU, resolution is worth checking before a rebuild rather than after:

echo "torch==2.5.1" > /tmp/constraints.txt
uv pip compile requirements.txt --constraint /tmp/constraints.txt \
  --python-version 3.11 --python-platform x86_64-unknown-linux-gnu \
  -o /tmp/resolved.txt
grep -E '^(torch|transformers|huggingface-hub)==' /tmp/resolved.txt

The general point: in a GPU container, the base image is part of your dependency graph. Treating requirements.txt as the whole picture is how you silently unlink CUDA.

Failure 3: two sources of truth for one cache directory

The Dockerfile set HF_HOME, TRANSFORMERS_CACHE, and HF_HUB_CACHE to paths under /runpod-volume/....

The handler had its own hardcoded default.

Both were confident. They disagreed. The handler's value won, so the image's environment variables were quietly ignored, and no network volume was attached to the endpoint anyway, which means /runpod-volume was never a real volume. Three configuration values pointing at a path that did not exist, overridden by a fourth that nobody had written down.

The failure mode here is not a crash. It is downloading several gigabytes of model weights into a directory you did not intend, on every cold worker, until the disk fills.

The fix was to delete the configuration rather than correct it. One resolver, one precedence order, owned by the code:

def resolve_cache_dir() -> str:
    """Single source of truth for the Hugging Face cache location."""
    return os.getenv("MODEL_CACHE_DIR", os.getenv("HF_HOME", DEFAULT_CACHE_DIR))

MODEL_CACHE_DIR  →  HF_HOME  →  /tmp/huggingface

The Dockerfile now sets none of them, and says why:

# Cache location is resolved at runtime by prompting.resolve_cache_dir():
#   MODEL_CACHE_DIR -> HF_HOME -> /tmp/huggingface
# Not hardcoded here so the image works with or without a network volume.

When two layers can both configure a value, the fix is usually to remove one layer's ability to have an opinion rather than to make the two agree.

The assumption that cost the most: quantization is not automatically faster

I shipped 4-bit NF4 first, because a 3B model in 4-bit is obviously the efficient choice.

On a 24 GB GPU, bf16 was faster.

The reasoning is not exotic once you say it out loud. Quantization trades compute for memory. If the model already fits comfortably in VRAM, you have bought nothing and paid for the dequantization overhead on every forward pass. NF4 earns its keep when memory is the binding constraint. At 3B on 24 GB, it is not.

So bf16 became the default and 4-bit became opt-in behind USE_4BIT, off unless something forces it:

def use_4bit() -> bool:
    return env_flag("USE_4BIT", False)

Combined with the chat-template fix, this moved generation meaningfully: before, execution time sat around 8.4 seconds per request, largely because generation ran to the token limit instead of stopping at <|im_end|>. Cold start was about 47 seconds before enabling the platform's fast-boot option.

The habit worth keeping: measure the default before optimizing away from it. I inherited "quantize small models" as a belief rather than a measurement, and it cost me both latency and a day.

Make the worker say what it resolved

All three failures share a second property, alongside not raising an exception: in each case the worker knew the answer and never said it out loud. It knew which tokenizer it loaded. It knew which cache directory it picked. It knew which precision it was running.

So the worker now logs its resolved configuration before it accepts a single job:

HF token present: True
model cache dir: /tmp/huggingface
precision: bfloat16
emojis allowed: True
loading tokenizer from Qwen/Qwen2.5-3B-Instruct
stop token ids: [151645, 151643]

Six lines. Every one of them would have caught one of the bugs above in seconds instead of days.

Two details:

• Log resolved values, not configured ones. model cache dir is printed by the resolver, after precedence is applied, not read back from an environment variable. A log line that echoes your config tells you what you asked for. You need to know what you got.
• Log the presence of a secret, never the secret. HF token present: True answers the only question worth asking at startup. The token itself never appears in a log line, not even truncated.

Downstream of the worker, the calling function records which provider, which model, queue delay, and execution time for every run. That is what makes "slow" separable from "broken": a rising queue delay means scale up, a rising execution time means the generation itself changed.

Testing any of this without a GPU

The reason these bugs survived so long is that I had no way to check them cheaply. Everything lived in one file that imported torch, so the only test environment was a live GPU endpoint.

Splitting the module fixed that. All prompt construction, input normalization, and output sanitizing live in a file that imports no torch, peft, transformers, or serverless SDK. The heavy file only wires it up.

Now the regression that caused the P0 has a test:

def test_build_chat_prompt_refuses_plaintext_fallback(self):
    """The old code silently degraded here. It must now fail loudly."""
    with self.assertRaises(RuntimeError):
        build_chat_prompt(FakeTokenizer(chat_template=""), "anything")

def test_rendered_prompt_uses_qwen_chatml_markers(self):
    prompt = build_chat_prompt(_BASE_TOKENIZER, "Task: react.")
    self.assertIn("<|im_start|>system", prompt)
    self.assertIn("<|im_end|>", prompt)
    self.assertTrue(prompt.rstrip().endswith("<|im_start|>assistant"))

A handful of tests load the real public base tokenizer to assert the template exists and that <|im_end|> resolves to a real token id. They skip automatically when transformers is missing or the machine is offline. The rest need nothing but Python.

The lesson generalizes past this project: the parts of an ML service that break are mostly not the parts that need a GPU. Prompt formatting, input validation, bounds clamping, output cleanup, config resolution: all of it is ordinary software, and all of it is testable in milliseconds if you refuse to let heavy imports leak into those modules.

The checklist I use now

Before an adapter serves a real request:

1. Load the tokenizer from the base model, not the adapter repo.
2. Assert the chat template exists at startup, and refuse to boot without it.
3. Assert the stop-token set is non-empty at startup.
4. Render one prompt and read it. Confirm the special tokens are the ones the model was trained on.
5. Pin every dependency, and write down which ones the base image already provides.
6. Resolve dependencies locally before rebuilding the image.
7. Give every configuration value exactly one owner.
8. Benchmark full precision before enabling quantization.
9. Keep prompt and output logic in modules with no heavy imports, and test them.
10. Log which provider, model, and precision served each request — otherwise none of the above is observable in production.

None of that is about fine-tuning.

The model was the easy part. It behaved exactly as trained on every request, including all the ones where I was handing it a format it had never seen. It had no way to tell me. Everything that made those requests wrong lived in the container, the dependency resolver, and the environment variables, which is the boring layer ML tutorials skip. That layer is where the failures are.