7B 模型微调为何需要 112 GB 显存：LoRA 与 QLoRA 的内存账

Ask how much memory it takes to fine-tune a 7B model and the instinct is "the model's 14 GB in fp16, so a bit more than that". The real figure is about 112 GB, before you've stored a single activation. The model is 14 GB of it.

Once you see where the other 98 GB goes, LoRA and QLoRA stop looking like clever tricks and start looking obvious.

Where the memory actually goes

The accounting comes from the ZeRO paper, and it's worth reading in their words:

In total, this results in 2Ψ + 2Ψ + KΨ = 16Ψ bytes of memory requirement. For a model such as GPT-2 with 1.5 Billion parameters, this leads to a memory requirement of at least 24 GB, which is significantly higher than the meager 3 GB of memory required to hold the fp16 parameters alone.

Ψ is the parameter count. Per parameter, mixed-precision training with Adam holds:

• 2 bytes — fp16 weights
• 2 bytes — fp16 gradients
• 12 bytes — what ZeRO calls K: an fp32 master copy of the weights plus Adam's two moment buffers, 4 bytes each

Sixteen bytes per parameter. For 7 billion parameters that's 112 GB, and only 14 GB of it is the model you're actually trying to change. The optimizer state alone is 84 GB, six times the weights.

That's the number I wish I'd seen earlier. Fine-tuning memory isn't dominated by the model. It's dominated by the bookkeeping needed to update it.

LoRA: stop paying for the bookkeeping

If most of the cost is gradients and optimizer state, the obvious move is to have far fewer things that need them.

That's LoRA. Freeze every pretrained weight, and train a pair of small matrices injected into each layer instead. The frozen base still sits in memory at fp16 — 14 GB — but it carries no gradients and no optimizer state. Only the small matrices do.

How small? Take a Llama-style 7B: 32 layers, hidden size 4096, with adapters on the query and value projections. At rank 8, each adapter is 8 × (4096 + 4096) = 65,536 values. Across 32 layers and two projections that's 4,194,304 trainable parameters.

About 0.06% of the model. You're training four million numbers, not seven billion.

Hu et al. put the GPT-3 result plainly: compared to GPT-3 175B fine-tuned with Adam, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times", while it "performs on-par or better than finetuning in model quality" on the models they tested.

And because the low-rank update merges back into the base weight afterwards, there's "no additional inference latency". That's what separated it from earlier adapter methods, which left an extra layer sitting in the forward pass permanently.

QLoRA: shrink the part you can't avoid

LoRA leaves one big cost standing: the frozen base still needs 14 GB at fp16.

QLoRA goes after exactly that. Store the frozen base in 4-bit and it drops to about 3.5 GB — slightly more in practice, because quantization needs constants of its own. Dettmers et al. squeezed those too: their double quantization cuts the overhead "from 32/64 = 0.5 bits, to 8/64 + 32/(64 · 256) = 0.127 bits" per parameter.

Gradients still flow through the 4-bit base into LoRA adapters kept at higher precision. The headline result is the one that made it famous — enough memory saved "to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance".

The cost is time. The 4-bit weights have to be dequantized to do any arithmetic with them, on every pass, so it runs slower than plain LoRA. You're trading speed for memory.

What I'd take from it

The three methods are really one question: which of the 16 bytes per parameter are you willing to stop paying for?

• Full fine-tuning pays all 16. Worth it when hardware isn't the constraint, or the task is far from what the model was trained on.
• LoRA stops paying the 14 bytes of bookkeeping on the base. The 2-byte fp16 copy stays.
• QLoRA shrinks that remaining 2 bytes to roughly half a byte.

If someone says their 7B model "needs 14 GB", they're quoting inference. Ask about the other 98.

Longer version with the full memory table and a rank-by-rank worked example at diffstudy.com. Sources: ZeRO, arXiv 1910.02054, LoRA, arXiv 2106.09685, QLoRA, arXiv 2305.14314.