Practical LLM Fine-Tuning Without a GPU Farm

Full fine-tuning a 7B-parameter model requires ~112GB of GPU VRAM. Most of us don't have that. But with QLoRA (Quantized Low-Rank Adaptation), you can fine-tune a Llama 3 8B model on a single 24GB consumer GPU — or on AWS spot instances for a few dollars per run.

Understanding LoRA and Why It Works

LoRA works by decomposing weight update matrices into two low-rank matrices. Instead of training all ~7 billion parameters, you train a tiny fraction — often less than 1% — and freeze the rest. QLoRA extends this by quantizing the frozen base model to 4-bit precision, slashing memory requirements dramatically.

from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

Dataset Quality Over Quantity

I've consistently found that 1,000 high-quality, diverse examples outperform 50,000 mediocre ones. Fine-tuning is teaching the model a style or domain — not teaching it new knowledge from scratch. Curate ruthlessly.

• Format matters: use the same prompt template during training and inference
• Diversity matters: cover edge cases, not just the happy path
• Quality filtering: use GPT-4 or Claude to score your training examples before training
• Instruction following: mix instruction-response pairs with raw completion examples

Training on Spot Instances

AWS g5.2xlarge (A10G GPU, 24GB VRAM) costs ~$1.00/hr on-demand, ~$0.30/hr spot. A typical 3-epoch fine-tuning run on 2K examples takes under 2 hours. That's under $1 for a domain-adapted model. Use Hugging Face Accelerate + checkpoint saving every 100 steps to survive spot interruptions.

Evaluation: Perplexity Isn't Enough

Perplexity tells you if the model is learning — not if it's learning the right thing. Use task-specific evaluation: ROUGE scores for summarization, exact-match for structured extraction, and human preference scoring (side-by-side A/B) for instruction following. Automate this into your CI pipeline with LLM-as-judge patterns.