The message that eats your afternoon

Nothing derails a fine-tuning run faster than a wall of red text ending in torch.cuda.OutOfMemoryError. It usually shows up minutes into training, after you've already downloaded weights, tokenized a dataset, and warmed the GPU — which is exactly why it stings. The good news is that CUDA out-of-memory (OOM) during Hugging Face fine-tuning is one of the most predictable failures in the stack. Once you understand the memory budget and apply fixes in order of impact, you can almost always land a model that "shouldn't fit" on the hardware you have.

This guide walks through reading the error, decomposing GPU memory into its four consumers, and then a prioritized list of fixes — from QLoRA down to activation offloading — with realistic TrainingArguments and TRL SFTConfig snippets you can paste into a run. If you get stuck on a specific stack trace, our Hugging Face proxy job support team debugs exactly these situations live.

Reading the OOM error correctly

A typical message looks like this:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 23.68 GiB of which 512.00 MiB is free.
Process has 23.18 GiB memory in use. Of the allocated memory
21.90 GiB is allocated by PyTorch, and 640.00 MiB is reserved by
PyTorch but unallocated.

Three numbers matter. Total capacity is your card (here a 24 GB GPU). Allocated by PyTorch is memory holding live tensors — weights, gradients, optimizer state, activations. Reserved but unallocated is the caching allocator's fragmentation headroom; if this number is large while allocation still fails, you have fragmentation, not a true capacity problem. The "Tried to allocate" figure tells you the size of the single allocation that tipped you over — often an activation buffer during the forward pass or an optimizer state during the first step.

A practical first move is to set the allocator to expandable segments, which dramatically reduces fragmentation-driven OOMs:

import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

The memory budget: where your VRAM actually goes

During training, GPU memory is consumed by four things. Understanding their relative sizes tells you which fix will help most.

  • Model weights. In fp16/bf16, a parameter costs 2 bytes. A 7B model is ~14 GB just to hold the weights. In 4-bit (NF4) it drops to roughly 3.5–4 GB.
  • Gradients. One gradient per trainable parameter, same dtype as the parameter. Full fine-tuning a 7B model adds another ~14 GB. This is the number LoRA slashes: you only have gradients for the tiny adapter.
  • Optimizer states. Adam keeps two moments per parameter. In fp32 that's 8 bytes per parameter — for a 7B model, ~56 GB. This is usually the single biggest consumer in full fine-tuning and the reason naive full-FT of a 7B on a 24 GB card is impossible.
  • Activations. Intermediate tensors kept for the backward pass. These scale with batch size × sequence length × hidden size × layers, and they are the part that grows when you increase either batch size or max_seq_length.

The takeaway: weights are fixed, but gradients and optimizer states are what you attack with parameter-efficient methods, and activations are what you attack with checkpointing and shorter sequences. Now let's fix things in order of impact.

Fix 1 (biggest win): QLoRA — 4-bit base + LoRA adapters

If you are OOMing on a single consumer GPU, the highest-leverage change by far is QLoRA. It loads the base model in 4-bit NF4 (via bitsandbytes), freezes it, and trains only small low-rank LoRA adapters on top. Because the frozen 4-bit base has no gradients and no optimizer state, you eliminate the two largest consumers in one move. Combined with a paged optimizer, QLoRA routinely fits 7B–13B fine-tuning on a 24 GB card.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,      # extra ~0.4 bits/param saved
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

peft_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules="all-linear",   # covers q,k,v,o + MLP projections
    task_type="CAUSAL_LM",
)

The bnb_4bit_compute_dtype is the dtype used for the matmuls (weights are dequantized on the fly), not the storage dtype — keep it bf16 on Ampere or newer. For a deeper treatment of adapter configuration and the training loop, see our QLoRA fine-tuning job support and the broader PEFT fine-tuning job support pages.

Fix 2: Gradient checkpointing

Gradient (activation) checkpointing trades compute for memory: instead of keeping every intermediate activation for the backward pass, it keeps a few checkpoints and recomputes the rest on demand. This cuts activation memory dramatically — often 60–70% — at the cost of roughly a 20–30% slowdown. It is almost always worth enabling when you are memory-bound.

from trl import SFTConfig

args = SFTConfig(
    output_dir="out",
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},  # required with PEFT
    bf16=True,
)

The use_reentrant: False flag matters: the non-reentrant implementation is the supported path with PEFT and avoids the "does not require grad" warnings that otherwise silently disable checkpointing. If you also see a warning about inputs not requiring gradients, call model.enable_input_require_grads() before training.

Fix 3: Gradient accumulation instead of a big batch

A common misconception is that you need a large physical batch. You don't — you need a large effective batch. Gradient accumulation runs several small micro-batches, sums their gradients, and steps once. The activation memory is set by the micro-batch (per_device_train_batch_size), while the effective batch is that times gradient_accumulation_steps.

args = SFTConfig(
    output_dir="out",
    per_device_train_batch_size=1,       # tiny micro-batch = low activation memory
    gradient_accumulation_steps=16,      # effective batch = 16
    gradient_checkpointing=True,
    bf16=True,
)

Effective batch = per_device_train_batch_size × gradient_accumulation_steps × num_gpus. When you hit OOM, drop the micro-batch to 1 and raise accumulation to keep the effective batch (and therefore your learning dynamics) unchanged. This is the cheapest knob that preserves training quality.

Fix 4: Paged 8-bit optimizer

Even with LoRA, Adam's optimizer state can spike memory during the first optimizer step. The paged_adamw_8bit optimizer stores state in 8-bit and uses NVIDIA unified memory to page state to CPU RAM when the GPU is tight, preventing the classic "OOM on step 1" that catches people who thought their run was safe.

args = SFTConfig(
    output_dir="out",
    optim="paged_adamw_8bit",
    learning_rate=2e-4,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    bf16=True,
)

This pairing — QLoRA base, LoRA adapters, gradient checkpointing, accumulation, and a paged 8-bit optimizer — is the canonical single-GPU recipe. If it still won't fit, the next levers are sequence length and sharding.

Fix 5: Reduce max_seq_length

Activation memory grows roughly linearly with sequence length (and quadratically for non-flash attention). If your data is mostly short, a 4096-token context is wasted VRAM. Cap it, and use packing so short samples are concatenated to fill the window efficiently rather than padded.

args = SFTConfig(
    output_dir="out",
    max_length=1024,     # was 4096 — big activation savings
    packing=True,        # concatenate short samples, avoid pad waste
)

Halving sequence length can free several gigabytes. Profile your token-length distribution before committing to a large context — many datasets have a p95 well under 1024. The TRL configuration surface is covered further in our Hugging Face SFT job support and TRL job support resources.

Fix 6: device_map="auto" for inference-time offload

For loading and evaluation (not the training step itself), device_map="auto" uses Accelerate to lay the model across available GPUs, then CPU RAM, then disk. It is how you load a model that is larger than one card. Note the caveat: naive device_map offloading is for inference and single-adapter QLoRA, not a substitute for proper sharded training — for multi-GPU training you want FSDP or DeepSpeed below.

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
    max_memory={0: "22GiB", "cpu": "64GiB"},   # cap per device to leave headroom
)

Fix 7: Shard with FSDP or DeepSpeed ZeRO via Accelerate

When you have multiple GPUs, sharding is the real answer to full fine-tuning. Fully Sharded Data Parallel (FSDP) and DeepSpeed ZeRO both partition parameters, gradients, and optimizer states across ranks so no single GPU holds the whole thing. ZeRO-3 shards all three; ZeRO-2 shards gradients and optimizer states only. Both are driven through Accelerate — start with accelerate config, then launch with accelerate launch.

# fsdp_config.yaml (generated by `accelerate config`)
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
mixed_precision: bf16
fsdp_config:
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_offload_params: true        # offload sharded params to CPU when idle
  fsdp_state_dict_type: SHARDED_STATE_DICT
accelerate launch --config_file fsdp_config.yaml train.py

With fsdp_offload_params: true you can even fine-tune models that exceed aggregate GPU memory, at the cost of PCIe traffic. The mechanics of multi-GPU launches and the trade-offs between FSDP and ZeRO are exactly what our Accelerate distributed training job support covers.

Fix 8: Flash Attention, cache clearing, and activation offload

Flash attention. FlashAttention-2/3 computes attention without materializing the full N×N score matrix, turning attention's memory from quadratic to linear in sequence length. Pass attn_implementation="flash_attention_2" at load time (shown in Fix 1). This is a strict win for long contexts and is now the default path many Transformers v5 models expect.

Clearing the cache. Between evaluation and training, or after loading, fragmentation can leave "reserved but unallocated" memory. Reclaim it:

import gc, torch
gc.collect()
torch.cuda.empty_cache()
torch.cuda.reset_peak_memory_stats()
print(torch.cuda.memory_summary())   # inspect what actually holds memory

Activation / CPU offload. If you are still a few hundred megabytes short, offloading activations or optimizer state to CPU (via DeepSpeed ZeRO-Offload or FSDP CPU offload) buys the final headroom. It is slower per step but often the difference between running and not running.

Related tuning — quantization choices (bitsandbytes vs GPTQ vs AWQ) and broader throughput work — is covered in our Hugging Face quantization job support and GPU optimization job support pages.

A decision checklist

  • Single 24 GB GPU, 7B–13B: QLoRA + gradient checkpointing + accumulation + paged_adamw_8bit + flash attention. Cap max_length.
  • OOM on step 1 specifically: switch to a paged/8-bit optimizer — that spike is optimizer state.
  • OOM mid-epoch on long samples: lower max_length, enable packing, confirm flash attention is active.
  • "Reserved but unallocated" is large: set expandable_segments:True and clear the cache — it's fragmentation.
  • Multi-GPU full fine-tuning: FSDP FULL_SHARD or DeepSpeed ZeRO-3 through Accelerate, with CPU offload if needed.

Most OOMs fall to the first recipe. When they don't, it is usually a subtle interaction — checkpointing silently disabled, a stray fp32 optimizer, or a device_map that put a layer where it shouldn't be. The general training and inference mechanics behind all of this live in our Transformers training job support and Transformers inference job support guides.

Get unblocked fast

If you're staring at a stack trace on a deadline and none of the standard fixes are landing, don't burn the afternoon guessing. Our engineers pair with you on your actual repo, GPU, and dataset to get the run green. Start with Hugging Face proxy job support and we'll help you turn that red OOM wall into a training curve.