Optimal per-adapter LoRA Scale selection

Determine, for each Stable Diffusion XL (SDXL) Low-Rank Adapter (LoRA), the optimal usage of the LoRA Scale hyperparameter by systematically characterizing how the Scale affects CARLoS’s CLIP-diff-based Strength metric across prompts and seeds, and specify the per-adapter scale value(s) that best capture the adapter’s intended effect.

Background

CARLoS quantifies each LoRA’s generative effect using CLIP-diff vectors, summarizing them with three metrics: Semantic Direction, Strength (magnitude of effect), and Consistency (stability). The LoRA Scale hyperparameter in SDXL influences Strength, but the authors fixed it to 1.0 for all adapters in their study.

The paper notes that the relationship between LoRA Scale and Strength is non-trivial and varies across adapters, making a one-size-fits-all setting inadequate. Identifying an adapter-specific optimal scale is important for practical retrieval and generation, as overly strong LoRAs can override prompts and reduce usability, while too-weak settings may fail to manifest the intended effect.

References

Although we find that the “LoRA Scale” hyperparameter influences Strength, we fix it (to 1) across all adapters. We find the connection between Strength and Scale is non-trivial (see supplementary materials) across different LoRAs, and leave the computationally intensive investigation of optimal scale usage for each adapter as future work.

— CARLoS: Retrieval via Concise Assessment Representation of LoRAs at Scale  (2512.08826 - Sarfaty et al., 9 Dec 2025) in Section 3.2 (Adapter Indexing), Strength paragraph