LoRA Large Model Fine-Tuning

Thu, 11 Dec 2025 00:00:00 +0800

1. What is LoRA?

LoRA (Low-Rank Adaptation) is a large model fine-tuning technique.

The Problem

Large models (like Qwen, Llama) have 7B, 70B parameters or more. Retraining all parameters every time you want to teach them something new requires dozens of A100 GPUs running for days — far beyond what most people can afford.

The LoRA Approach

Don’t touch the original model — just attach a “small plugin” next to it.

Think of it this way:

Approach	Analogy
Full fine-tuning	Rewriting the entire textbook into your desired version
LoRA	Sticking a few Post-it notes in the margins with your additions

The original model weights are completely unchanged. LoRA inserts a tiny matrix (low-rank matrix) next to certain key layers, and only updates this matrix during training.

Why “Low-Rank Adaptation”?

Mathematically, a large matrix can be approximated by the product of two smaller matrices. LoRA leverages this: the original weight matrix $W$ stays frozen, while two small matrices $A$ and $B$ are trained. The final output becomes $W + A \cdot B$. Together, $A$ and $B$ may be only one-thousandth the size of the original weights.

2. What is LoRA Used For?

Its core ability is simple: change a large model’s “behavior” at very low cost.

📌 LoRA changes style/format/behavior, not knowledge

Want the model to learn new knowledge → Use RAG (Retrieval-Augmented Generation)

Want the model to speak or act differently → Use LoRA

Typical Use Cases

Scenario	Example
Style Transfer	Make a general model write in Lu Xun’s literary style
Format Constraint	Force the model to output strict JSON, nothing more
Role-playing	Turn Qwen into Zhongli from Genshin Impact
Domain Tone	Medical consultation tone, legal document precision
Instruction Following	Make the model better follow system prompts without wandering

LoRA Advantages

Extremely low VRAM: A 7B model can be trained on a single RTX 3090 (24G), or even barely on 8G cards
Tiny file size: Training output is just a few MB. One model can host dozens of different LoRAs, plug-and-play
Fast training: A few hundred samples, results in 30 minutes
No base damage: Unload the LoRA and the model reverts to original, zero risk

3. How to Use LoRA?

Four main steps:

Step 1: Prepare Data

Write example dialogues of the “behavior” you want the model to learn.

Suppose you want the model to become a sarcastic customer service agent. Write 50-200 dialogues like this:

User: Where's my package?
Agent: Well, unless there's a surprise, it's probably sunbathing in the delivery truck. I suggest you sunbathe too while you wait.

User: Can I get a refund?
Agent: Sure, the process is about as long as a pilgrimage. Get your order number ready and I'll guide you on this journey westward.

You don’t need much data — 50-200 high-quality examples will show noticeable results. Quality matters far more than quantity.

Step 2: Choose a Tool

There are many tools out there, but they all do essentially the same thing. Two recommendations:

Tool	Features	Best For
LLaMA-Factory	Web UI, click-based operation	Complete beginners who don’t want to code
Unsloth	2-5x faster, half the VRAM	Those who know some Python and want efficiency

Unsloth’s core code is under 30 lines. The essence: load model → attach LoRA → feed data → train → save.

Step 3: Train

Set a few key parameters, then wait:

Parameter	Purpose	Typical Value
LoRA rank	Determines the “plugin” capacity. Higher = more refined but more VRAM	8 or 16 is enough
Learning rate	How much to learn each step	5e-5, don’t overthink it
Epochs	How many times to go through the data	3-5

Step 4: Use

After training, you get a LoRA file of a few MB. To use it, load both the original model and this LoRA file — the model now behaves as you trained it.

Want to switch back? Just unload the LoRA, one second to restore.

One model can host multiple LoRAs simultaneously — for example, attach a “sarcastic style” LoRA and a “JSON output” LoRA, and the model becomes a sarcastic assistant that outputs JSON.

Concept	What It Does	Relationship with LoRA
Full Fine-tuning	Modifies all parameters	Alternative to LoRA, better results but hundreds of times more expensive
RAG	External knowledge retrieval	Complementary: RAG handles knowledge, LoRA handles style
Prompt Engineering	Writing prompts	Lightest approach, but limited. Use LoRA when prompts aren’t enough
QLoRA	LoRA + 4-bit quantization	VRAM-saving version of LoRA, can run 7B on 8G cards
PEFT	Parameter-Efficient Fine-tuning category	LoRA is the most popular member of the PEFT family

5. One-Sentence Summary

LoRA = a small plugin for large models that changes behavior without changing knowledge, with extremely low training cost and plug-and-play MB-sized files.

Large Language Model on Yanmu