The article from Hugging Face questions whether LoRA should be the default choice for parameter-efficient fine-tuning (PEFT) by analyzing its popularity and comparing it to other techniques via systematic benchmarks. LoRA dominates usage: 98.4% of PEFT model cards on the Hub mention LoRA, 95% of PEFT checkpoints on an external site are LoRA, and 71.3% of GitHub code snippets for PEFT configs use LoRA. The authors argue this dominance may be self-reinforcing due to LoRA‘s early visibility and broad support, not necessarily superior performance.

The PEFT library, developed by Hugging Face, implements over 40 distinct PEFT techniques behind a unified API. To enable objective comparison, the team added benchmarks for LLM mathematical reasoning (MetaMathQA on Llama-3.2-3B, evaluated on GSM8K) and image generation (fine-tuning FLUX.2-klein-base-4B on a new concept). All techniques are evaluated under identical conditions (same base model, dataset, code, hardware), tracking test performance, VRAM usage, runtime, and checkpoint size.

Results reveal that LoRA is not universally optimal. On the LLM benchmark, LoRA with rank-stabilized initialization (rs-LoRA) achieves 53.2% test accuracy at 22.6 GB VRAM, placing it on the Pareto frontier alongside techniques like BEFT (32.9%, 20.2 GB) and Lily (54.9%, 25.6 GB). Vanilla LoRA only reaches 48.1% at 22.5 GB, so using a variant matters. On the image generation benchmark, LoRA scores 0.697 dino similarity at 9.97 GB, whereas OFT achieves 0.708 at 9.01 GB, strictly dominating LoRA.

The article acknowledges limitations: hyperparameter choices can favor certain techniques; the benchmarks don’t capture all capabilities (e.g., long-prompt compression with Cartridges); and downstream packages like vLLM often only support LoRA. However, PEFT now supports converting non-LoRA adapters into LoRA format with minimal score loss (e.g., GraLoRA: 0.702 → 0.694 similarity).

The key takeaway: LoRA should not be the automatic default. Developers can easily switch between techniques via PEFT‘s unified API (e.g., replacing LoraConfig with OFTConfig). Even if sticking with LoRA, using variants like rs-LoRA or LoRA-FA can yield better trade-offs. The team invites community contributions to expand benchmarks and improve hyperparameter sweeps.