Recent Trends in Parallel Computing Review Article
A Review Study on CPU-Optimized Parameter-Efficient Fine-Tuning for Large Language Models to Increase Accuracy Using LoRA
Abstract
The fast proliferation of large language models (LLMs) has increased the need to optimize the process of fine-tuning, but the existing workflows that require a graphics processing unit (GPU) are still expensive, intensive, and unavailable to most researchers. This paper is driven by the desire to have a more cost-efficient and democratized version by examining a CPU-efficient implementation of parameter-efficient fine-tuning (PEFT) based on low-rank adaptation (LoRA). The major purpose of the study is to find out whether CPUs with low-rank matrix updates, mixed-precision quantization, SIMD vectorization, operator fusion, and NUMA-sensitive scheduling can be as precise as GPU-based LoRA at a lower computational cost. The suggested methodology presents a full CPU-LoRA pipeline that consists of selective transformation to the layer of transformers, quantizing from 4-bit to 8-bit, implementing matrix operations using high-performance basic linear algebra subprogram (BLAS) libraries, and using dynamic memory efficient batching. Open-source LLMs, including large language model meta AI (LLaMA), Falcon, and Mistral, were experimented on with standard natural language processing (NLP) datasets. The measures of evaluation were accuracy, perplexity, F1/ Recall-oriented understudy for gisting evaluation (ROUGE) measures, throughput, memory consumption, and power. Empirical evidence shows that CPU-optimized LoRA can be trained with close to 92–98% accuracy levels of GPU-LoRA and has considerable practical advantages: up to 70–90% memory reductions and massive operational and energy savings. According to the study, critical CPU bottlenecks are also discovered, and dedicated optimizations are provided to enable CPU-based fine-tuning as a practical alternative to small institutions and researchers who lack access to GPUs. All in all, the results support the idea that CPU-optimized LoRA is a viable, economically effective, and scalable option in terms of customization of LLM, which can further the larger objective of making high-quality model adaptation more accessible and sustainable.
Keywords
References (13)
- Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners [preprint]. 2020. arXiv:2005.14165. doi:10.48550/arXiv.2005.14165.
- Narayanan D, Shoeybi M, Casper J, LeGresley P, Patwary MM, Korthikanti VA, et al. Efficient large-scale language model training on GPU clusters using Megatron-LM [preprint]. 2021. arXiv:2104.04473. doi:10.48550/arXiv.2104.04473.
- Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models [preprint]. 2021. arXiv:2106.09685. doi:10.48550/arXiv.2106.09685.
- Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Attention is all you need [preprint]. 2017. arXiv:1706.03762. doi:10.48550/arXiv.1706.03762.
- Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). Online; Punta Cana, Dominican Republic; 2021:3045–3059. doi:10.18653/v1/ emnlp-main.243.
- Han Z, Gao C, Liu J, Zhang J, Zhang SQ. Parameter-efficient fine-tuning for large models: a comprehensive survey [preprint]. 2024. arXiv:2403.14608. doi:10.48550/arXiv.2403.14608.
- Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: efficient finetuning of quantized large language models. Advances in Neural Information Processing Systems (NeurIPS 36). New Orleans, USA; 2023:10088–10115. doi:10.52202/075280-0441.
- Houlsby N, Giurgiu A, Jastrzebski S, Morrone B, de Laroussilhe Q, Gesmundo A, et al. Parameter-efficient transfer learning for NLP [preprint]. 2019. arXiv:1902.00751. doi:10.48550/arXiv.1902.00751.
- Carneiro AR, Serpa MS, Navaux POA. Lightweight deep learning applications on AVX-512. 2021 IEEE Symposium on Computers and Communications (ISCC), Athens, Greece. 2021:1–6. doi:10.1109/ISCC53001.2021.9631464.
- Hasan MM, Islam MM. High-performance computing architectures for training large-scale transformer models in cyber-resilient applications. ASRC Procedia Glob Perspect Sci Scholarsh. 2022;2:193–226. doi:10.63125/6zt59y89.
- Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: open and efficient foundation language models [preprint]. 2023. arXiv:2302.13971. doi:10.48550/arXiv.2302.13971.
- Gao T, Fisch A, Chen D. Making pre-trained language models better few-shot learners. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021). Online; 2021:3816–3830. doi:10.18653/v1/2021.acl-long.295.
- Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: state-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP 2020). Online; 2020:38–45. doi:10.18653/v1/2020.emnlp-demos.6.