Fine-tuning a 3B open model to replace a frontier API: a worked case study

We previously wrote three checks to run before you fine-tune. This post is the sequel: what it looks like when a workload passes all three, and you actually do it.

The scenario below is hypothetical — a composite of patterns we see in real engagements, with the company and numbers invented so we can show everything end to end. The techniques, the failure modes, and the shape of the results are the real part.

The setup

Meet Meridian Freight, a fictional mid-size freight forwarder. Every day, roughly 40,000 emails and PDF attachments arrive: booking requests, customs paperwork, rate inquiries, delay notifications, invoices. A team used to triage these by hand. Eighteen months ago, Meridian wired the queue to a frontier LLM API with a long prompt that does two jobs per document:

1. Classify it into one of 12 operational intents (booking, customs query, claim, invoice dispute, and so on)
2. Extract a fixed set of structured fields: shipment reference, ports, container numbers, dates, requested action

The output is a JSON object that routes the document to the right team with the fields pre-filled. It works, and the business is genuinely built around it now. But three problems have been compounding:

• Cost. At ~1,200 input tokens and ~300 output tokens per document, 40,000 documents a day works out to roughly $7,600 a month in API spend — and volume is growing 8% a quarter.
• Latency. The API round-trip has a p95 near 6 seconds. Documents queue behind rate limits during morning peaks, which is exactly when the operations team needs them triaged.
• Data residency. Two new EU customers have contract language that prohibits shipment data from being processed by third-party model providers. Losing them is not an option; neither is redacting shipping documents into uselessness.

Notice what is not on that list: quality. The frontier model is good at this task. The problem is that Meridian is renting a 2-trillion-parameter generalist to do the same narrow transformation 40,000 times a day.

Why fine-tuning, and why a small model

Run the standard checks before reaching for training:

Can retrieval solve it? No — this is not a knowledge problem. The task is a fixed transformation from messy text to a strict schema. There is nothing to retrieve; the model needs a behavior, not facts.

Can better prompting solve it? Prompting is how the current system works, and it hits the quality bar. But no prompt fixes the cost structure, the latency, or the residency constraint. Those are properties of where the model runs and how big it is, not of the instructions.

Is there labeled data? This is the quiet advantage of having run the API pipeline for 18 months: Meridian is sitting on hundreds of thousands of input-output pairs, and — because low-confidence outputs went through a human review queue — a meaningful slice of them carry human corrections. The expensive part of fine-tuning, the dataset, has been accumulating as a side effect of production.

That combination — narrow task, stable schema, abundant labels, and pressure on cost, latency, and residency — is the textbook profile for a small fine-tuned model. Not a 70B model. A 12-intent classifier with slot extraction does not need encyclopedic world knowledge; it needs to reproduce one distribution extremely well. Models in the 1B–3B range are interesting precisely because they cross a deployment threshold: they fit comfortably on a single 24 GB GPU with room for batching, which means one boring, cheap machine inside Meridian's own VPC.

We chose Llama 3.2 3B Instruct as the primary candidate and its 1B sibling as the cheap challenger. Qwen's small instruct models are an equally reasonable pick; we had more prior calibration on Llama for European-language business text, and the open license terms fit a commercial deployment. The honest answer is that at this size, dataset quality matters far more than which of the top open 3B models you start from.

Building the dataset

This is where the project is won or lost, so it got the majority of the effort.

The raw material was 18 months of production logs. The refinement steps:

1. Prefer corrected outputs. Wherever a human reviewer had fixed the API's output, the corrected version became the label. These examples are gold: they concentrate exactly on the cases the teacher got wrong.
2. Dedupe aggressively. Freight documents are full of near-duplicates (the same booking template from the same shipper, hundreds of times). Near-duplicate clusters were collapsed to a handful of representatives each, so the model would not spend its capacity memorizing one customer's letterhead.
3. Stratify by intent. The natural distribution was brutally imbalanced — two intents covered 60% of traffic, and the rarest ("claim escalation") was under 1%. We capped the dominant classes and oversampled the rare ones toward a floor, so every intent had at least a few hundred examples.
4. Freeze the eval set first. Before any training, 1,000 examples were set aside, stratified across intents and document formats, with labels human-verified. This set never changes and never leaks into training. Every number in this post comes from it.

The final training set: ~14,000 examples, each formatted as a chat exchange — a short system prompt stating the schema, the document as the user turn, and the JSON object as the assistant turn. Notably, this is a distillation setup: the teacher (the frontier API) already did most of the labeling work as a paid byproduct of running production. The fine-tune's job is to compress that behavior into a model Meridian owns.

Training

The whole run fits on one rented 24 GB GPU. We used QLoRA — the base model quantized to 4-bit, with low-rank adapters trained in bf16 on top — via Hugging Face TRL:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="train.jsonl", split="train")

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

config = SFTConfig(
    model_name_or_path="meta-llama/Llama-3.2-3B-Instruct",
    max_length=2048,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    num_train_epochs=3,
    bf16=True,
    load_in_4bit=True,
    completion_only_loss=True,
    output_dir="./meridian-triage-3b",
)

trainer = SFTTrainer(model=config.model_name_or_path, args=config,
                     train_dataset=dataset, peft_config=peft_config)
trainer.train()

A few choices worth calling out:

• Completion-only loss. The model is graded only on the JSON it produces, not on re-predicting the input document. For structured-output tasks this noticeably sharpens format adherence.
• LoRA rank 16 across all linear layers. For a task this narrow, we saw no benefit from rank 64 in early sweeps — just slower training. Small task, small adapter.
• Three epochs at 2e-4, cosine decay. ~1,300 optimizer steps total. The full run took a bit over three hours; at spot prices for a 24 GB card, the training cost less than a day of the API bill.

What went wrong first

The first run looked great on aggregate accuracy and was quietly broken. Per-class metrics showed the two rare intents ("claim escalation" and "dangerous goods declaration") had F1 scores in the 60s — the stratification floor had been set too low, and three epochs was not enough exposure for the model to learn their boundaries against neighboring classes. The fix was unglamorous: raise the oversampling floor for rare classes, add ~400 targeted examples mined from the logs by searching for the confusion pairs, and rerun. This is the normal texture of fine-tuning work — the second dataset iteration matters more than any hyperparameter.

The 1B model was worth testing and worth rejecting. On pure intent classification it landed within a point of the 3B. But on multi-field extraction — especially documents with several container numbers and dates competing for the same slots — it dropped about four points of exact-match, and no amount of extra data closed the gap in our budget. The lesson generalizes: classification compresses further than extraction. If Meridian's task had been routing alone, the 1B would have shipped.

Results

All numbers on the frozen 1,000-example eval set. "Field accuracy" is exact-match across all extracted fields; routing is macro-F1 over the 12 intents.

System                      Field acc.   Routing F1   p95 latency   Cost / month
Frontier API (incumbent)    96.1%        0.95         5.8 s         ~$7,600
Llama 3.2 3B, zero-shot     61.3%        0.71         0.7 s         ~$620
Llama 3.2 1B, fine-tuned    91.2%        0.93         0.4 s         ~$620
Llama 3.2 3B, fine-tuned    95.4%        0.94         0.7 s         ~$620

The zero-shot row is the one people skip and should not: it proves the win came from the fine-tune, not from the base model. Out of the box, the 3B model is nowhere near usable for this task. After fine-tuning, it sits within 0.7 points of a frontier model — on this distribution, and only on this distribution.

Shipping it

Training is half the project. The deployment shape:

• Merge and serve. The LoRA adapter is merged into the base weights and served with vLLM on a single L4 inside Meridian's VPC. At 40,000 documents a day (about half a request per second on average, a few per second at peak), one card has headroom to spare.
• Constrain the output. vLLM's structured-output mode enforces the JSON schema at decode time. Format errors go from "rare but real" to structurally impossible — a bigger reliability gain than most people expect, because downstream code no longer needs a repair path.
• Keep a fallback. Documents where the model's self-reported confidence is low, or that fail business-rule validation, still go to the frontier API — now about 3% of traffic instead of 100%. The residency-constrained EU customers are pinned to the local path only.
• Keep the flywheel. The human review queue stays. Its corrections accumulate into the next dataset iteration, and the frozen eval set gets re-run on every retrain. The eval set, not the checkpoint, is the durable asset: models will be swapped again, and the eval set is what makes swapping safe.

Net effect for the hypothetical Meridian: roughly 12x lower cost, 8x lower p95 latency, shipment data that never leaves their network, and a quality gap small enough that the review queue absorbs it.

What to take from this

1. Small fine-tuned models are specialists, not small generalists. The zero-shot row is terrible and the fine-tuned row is near-frontier. Both facts matter.
2. Your API logs are a dataset subsidy. If a frontier model runs your task today, you are already paying for labels. Human corrections on its mistakes are the most valuable examples you own.
3. Freeze the eval set before training anything. Every decision in this project — 1B vs 3B, dataset iterations, ship or not — was arbitrated by the same 1,000 examples.
4. Expect the second dataset iteration. The first training run's job is to reveal what the dataset is missing, usually in the rare classes.
5. Size to the task. Classification alone might have shipped on the 1B. Extraction needed the 3B. Nothing here needed a 70B.

Fine-tuning a small open model is not a way to beat frontier models. It is a way to stop paying frontier prices, frontier latency, and frontier data-governance costs for a task that stopped needing frontier intelligence a long time ago.