PEFT (Parameter-Efficient Fine-Tuning)

Parameter-Efficient Fine-Tuning (PEFT) is a collection of methods for adapting large pre-trained models, such as large language models (LLMs), to specific tasks with minimal computational and resource costs. Unlike traditional full fine-tuning, which requires updating all of the model's parameters, PEFT methods focus on modifying only a small fraction of the weights (less than 1-5% of the total), leaving the majority of the model unchanged ("frozen").

This approach significantly reduces memory, storage, and training time requirements, making the adaptation of powerful foundation models more accessible and robust against the problem of catastrophic forgetting.

Challenges of Full Fine-Tuning

Traditional full fine-tuning, where all model parameters are updated, faces several critical challenges that have catalyzed the development of PEFT:

• High computational costs: Updating hundreds of billions of parameters requires vast computational resources (high-performance GPUs/TPUs) and large amounts of video memory (VRAM), making the process expensive and inaccessible for many researchers.
• Storage inefficiency: For each new task, a full, multi-gigabyte copy of the model must be stored, leading to an exponential increase in disk space requirements.
• Catastrophic Forgetting: As a model adapts to new data, it "forgets" the general knowledge acquired during pre-training, which degrades its performance on other tasks.
• Risk of Overfitting: On small datasets, models with billions of parameters are prone to "memorizing" training examples instead of learning generalizable patterns.

Taxonomy of PEFT Methods

PEFT methods can be classified by how they modify the model's parameters. There are three main categories: additive, selective, and reparameterization-based.

Additive Methods

These methods freeze all original model weights and add new, small trainable modules.

• Adapters: The earliest additive method. Small neural network modules with a "bottleneck" architecture are inserted between transformer layers. Only the weights of these adapters are trained.
• Soft Prompt-based Methods: Instead of changing the model's weights, these methods add trainable vectors ("virtual tokens") to the input data, which guide the model's behavior. Key variants include:
  • Prompt Tuning: Adds trainable vectors only to the input embeddings.
  • Prefix-Tuning: Adds trainable vector prefixes to the hidden states at each layer of the attention mechanism, allowing for finer control.
  • P-Tuning v2: A generalization of Prefix-Tuning that applies trainable prompts to all layers of the model, achieving performance comparable to full fine-tuning.

Selective Methods

These methods do not add new parameters but select and fine-tune a small subset of existing ones.

• BitFit: An extremely economical method that fine-tunes only the bias terms and the parameters of the normalization layers, updating less than 0.1% of the total parameters.
• Differential Pruning (Diff Pruning): Uses a trainable mask to dynamically determine which weights should be updated during the training process.

Reparameterization Methods

This category is based on the hypothesis that the weight updates for model adaptation have a low "intrinsic rank." Instead of updating full-sized weight matrices, these methods update their low-rank representation.

• LoRA (Low-Rank Adaptation): The most popular PEFT method to date. It assumes that the weight update matrix `ΔW` can be approximated by the product of two low-rank matrices: `ΔW = BA`. During fine-tuning, the original matrix `W` is frozen, and only `A` and `B` are trained.
• QLoRA: Combines LoRA with quantization techniques to further reduce memory requirements, allowing models with 65 billion parameters to be fine-tuned on a single consumer GPU.

Performance and Resource Comparison

PEFT methods can achieve performance comparable to full fine-tuning while radically reducing costs.

Comparative performance and efficiency of PEFT methods

Model	Method	Trainable Parameters (%)	Benchmark Result (Avg. Accuracy)	Source
BERT-Large	Full Fine-Tuning	100%	80.4 (GLUE)
BERT-Large	Adapters	3.6%	80.0 (GLUE)
LLaMA-7B	LoRA	0.83%	74.7%
LLaMA-7B	DoRA (a LoRA variant)	0.84%	78.1%

Key advantages of PEFT in resource savings:

• GPU Memory (VRAM): For the LLaMA 65B model, full fine-tuning theoretically requires >780 GB of VRAM, whereas QLoRA allows it to be fine-tuned on a GPU with <48 GB of VRAM.
• Storage Space: Checkpoints saved after PEFT take up megabytes instead of gigabytes. This allows for storing hundreds of "adapters" for different tasks in the space that a single fully fine-tuned model would occupy.

Application Areas

Initially developed for NLP, PEFT methods have been successfully adapted for a wide range of tasks:

• Computer Vision (CV) and Vision-Language Models (VLM): Adapting models like Vision Transformer (ViT) and Segment Anything Model (SAM) for image segmentation tasks, including in biomedicine.
• Code Generation and Analysis: Customizing LLMs for the specifics of particular software projects, internal APIs, or codebases.
• Generative Models: "Styling" image generation models, such as Stable Diffusion, using LoRA adapters (the Dreambooth technique).
• Speech Synthesis: Adapting models to generate speech with a specific voice, intonation, or emotional tone.

External links

• PEFT library repository on GitHub
• Official documentation for the PEFT library by Hugging Face
• Fine-tuning (deep learning) — Wikipedia

See also

• LLM cost optimization

Literature

• Hu, E.J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
• Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
• Houlsby, N. et al. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019.
• Li, X.L.; Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190.
• Liu, X. et al. (2022). P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. arXiv:2110.07602.
• Ben Zaken, E.; Ravfogel, S.; Goldberg, Y. (2021). BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language Models. arXiv:2106.10199.
• Guo, D.; Rush, A.M.; Kim, Y. (2020). Parameter-Efficient Transfer Learning with Diff Pruning. arXiv:2012.07463.
• Jiang, Z. et al. (2024). MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning. arXiv:2405.12130.
• Mao, K. et al. (2024). A Survey on LoRA of Large Language Models. arXiv:2407.11046.
• Chen, S. et al. (2024). Parameter-Efficient Fine Tuning: A Comprehensive Analysis Across Applications. arXiv:2404.13506.
• Zhang, J. et al. (2025). Parameter-Efficient Fine-Tuning for Foundation Models. arXiv:2501.13787.