The Training Set Was a Text File a Client Wrote by Hand

Fine-tuning content has a standard opening: load a dataset from the hub, format it, start training. The dataset is a given. It arrives clean, labeled, and split.

Mine arrived as client_feedback_examples.txt. 43 KB. A category name on a line by itself, then blocks like this:

Selfie:
Ejemplo 1:
"Ok, esta sí la subiría 👀. Tiene una vibra súper natural y se siente
mucho más auténtica que esas fotos donde uno intenta demasiado."

Ejemplo 2:
"No sé qué tiene esta exactamente, pero me gusta más que muchas selfies
perfectamente planeadas. Se siente más tú."

Four hundred of those, written by hand by the client, in Spanish, with paragraphs of guidance in between the sections. That file was the entire specification for what the product's voice was supposed to sound like. There was no other source of truth for it.

The script that turns it into trainable JSONL is 852 lines. The training run that consumed the output took an afternoon.

That ratio is the note.

Why a line-splitter does not work

The obvious parse is: split on Ejemplo N:, take what follows. It shreds the file, because examples wrap across lines and the client wrote naturally rather than in records.

So the parser tracks an open quote instead of a line:

def is_quote_complete(lines: list[str]) -> bool:
    text = "\n".join(lines).strip()
    if len(text) < 2:
        return False

    expected_closer = OPENING_QUOTES.get(text[0])
    return bool(expected_closer and text.endswith(expected_closer))

OPENING_QUOTES maps five opening marks to their partners — straight quotes, curly quotes, and «» — because a human writing across a week does not use one quote style consistently, and a typographic quote is a different codepoint from an ASCII one no matter how identical it looks in the terminal.

An example accumulates lines until its own opening mark is closed. That is a small state machine, and it is the difference between 400 clean records and 400 truncated ones.

The line I am least proud of

Categories arrive as bare headers — Selfie:, Gym:, Viajes: — so the parser carries a current_category forward and assigns it to everything underneath.

Except the client also wrote prose between sections. Guidance, context, requests. And that prose sits at the same indentation as everything else, so a naive parser keeps the last category active and misfiles every example that follows.

The fix:

# Long prose between sections is client guidance, not feedback. It ends
# the current category so unlabeled future examples are not misfiled.
if len(line) > 40:
    current_category = None

Forty characters. A magic number with no theory behind it, chosen because it worked on this file and would need re-tuning on any other.

I left it in and wrote the comment, and I would defend both. The alternative — a classifier, or a more general grammar — is more engineering for a one-time input that has exactly one producer. What makes the hack acceptable is not its elegance. It is that unresolved records do not get quietly guessed. They fail:

if base_record["category"] is None:
    parse_errors.append({
        "line_number": base_record["line_number"],
        "reason": "Missing category header",
        "raw": base_record["feedback_text"],
    })
    continue

Parse errors are written to parse_errors.txt rather than swallowed. The file is currently zero bytes, which is the only reason I trust the 40.

A heuristic with a visible failure channel is engineering. The same heuristic failing silently is data corruption that shows up months later as a model that is bad at one category and nobody knows why.

There is also a repair pass: if a later line says Categoría: fitness, every example still holding None gets backfilled. Recovering the label from context beats discarding real client writing.

Adaptation, not translation

The product ships in English. The data is Spanish. The obvious move is to run it through a translator.

I refused, and this is the decision I would defend hardest.

Literal translation preserves meaning and destroys register. "Esta transmite una confianza muy tranquila" becomes "This transmits a very calm confidence" — grammatical, meaningless as social speech, and something no English speaker has ever said to a friend. Train on 400 of those and you get a model that is fluent and foreign. The uncanny valley of voice is worse than a plain sentence, because plain reads as terse and translated reads as wrong.

So the pipeline produces two datasets and labels the difference in the data itself:

{
  "id": "selfie_001",
  "language": "en",
  "source": "client_adapted",
  "translation_status": "adapted"
}

The Spanish set keeps source: "client_original". The English set is written to sound like American social speech expressing the same emotional beat, per category:

'fitness': [
    'This feels earned. Like the kind of progress that only shows up after a lot of quiet work.',
    "Lowkey, this has 'I kept showing up even when nobody noticed' energy.",
    'This is not just about how it looks. It feels like discipline finally becoming visible.',
    ...
]

Emoji get carried across separately, because they are the one part of the register that survives the language change intact. 27% of the client's examples contain one, and an English set with no emoji would have taught the model to write in a flatter voice than the client actually uses.

The project README states the rule so that a future me cannot quietly break it:

Do not use the Spanish examples alone to judge or train English VIOFY. The English dataset is meant to sound like natural American social feedback, not literal translation.

Three numbers that say what the dataset cannot do

This is the part a portfolio piece would leave out, so it is the part worth writing.

400 Spanish examples collapse to 162 unique English ones. The adaptation cycles through eight templates per category, selected by example number. Fifty client examples per category map onto eight English strings, repeated. One line appears seven times. The Spanish set has 398 distinct texts out of 400; the English set has 162 out of 400.

That is not a diverse dataset. It is a style demonstration with the volume turned up, and a model trained on it learns eight shapes very well and the space between them not at all. Every generic-sounding output I have ever gotten traces back to this number.

sentiment_type is positive for all 400 rows. It is derived from a keyword table, and I called it a weak label when I built it. It is not a weak label. It is a constant with a column name — a field that never varies teaches nothing and costs a column, and its only real function was to make the CSV look more like a dataset than it was.

Eight of twelve categories have zero client examples. The guideline map covers twelve; the client wrote for eight. celebration, nightlife, food, and general_lifestyle have none.

The last one matters more than the other three combined, because general_lifestyle is the fallback category. When the upstream vision model is unsure — low confidence, unclear scene — the pipeline deliberately collapses to general_lifestyle rather than forwarding a coin-flip guess. Which means the exact path taken when the system is least certain leads to the category the writer was never trained on.

I did not design that. I found it while counting rows for this note. The upstream fallback logic and the training data were built weeks apart, each one locally correct, and nobody was holding both at once.

What I would do differently

Not "collect more data." That is the answer everyone gives and it costs money I do not have.

Write the adaptation examples first, then the parser. The parser is deterministic and testable and I enjoyed writing it. The 162 English strings are the actual product and I generated them from a template table because it was the fast part. I put the craft in the wrong place.

Audit the derived columns before training, not before writing about it. One value_counts() on sentiment_type would have shown 400 identical values in a second. I never ran it, because I built the field and therefore assumed I knew what was in it.

Cross-check the data against the runtime. The training set and the inference pipeline are one system. Nothing in my repo forced the category list in the guideline map to agree with the categories that actually have examples, and so it does not.

Fine-tuning is a small amount of well-documented work sitting on top of a large amount of undocumented judgment about what the data means. The tutorials cover the first part because it is the part that generalizes.

The second part is the job.