STM Journals Peer-reviewed research from Consortium e-Learning Network

Journal of Web Engineering & Technology Review Article

LLM Evolution: Secrets and Disadvantages

  1. Saharsh Harshad Wakhale Department of Computer Science, Narayana E-Techno School Uttrahalli, Bengaluru

Abstract

The advancement of large language models (LLMs) has initiated a significant transformation in artificial intelligence, with substantial effects on fields including natural language processing, machine learning, and human-computer interaction. This research examines the diverse improvements in LLMs, emphasizing significant milestones from early models such as GPT-2 to contemporary state-of-the-art designs. The investigation highlights the novel training methodologies, such as unsupervised learning and transfer learning, which have markedly improved the capabilities of LLMs in language understanding and production. Notwithstanding their exceptional powers, LLMs possess considerable drawbacks. Ethical issues regarding biases inherent in training data result in the perpetuation of detrimental stereotypes, prompting essential enquiries about accountability and equity in AI systems. The resource-intensive aspect of training and deploying LLMs presents sustainability concerns, as their environmental impact can be significant because of the energy demand involved. These issues necessitate a sophisticated comprehension of the trade-offs inherent in LLM development. The article advocates responsible AI methods, emphasizing the application of bias mitigation measures and the promotion of transparency in model development. Proposed future research directions aim to further examine the ethical implications and improve the robustness of LLMs, seeking a balance between technical progress and public welfare. By promoting interdisciplinary conversation, we may more effectively traverse the complexity of LLM evolution and leverage its potential while avoiding inherent risks.

Keywords

References (37)

  1. Vaswani A. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998–6008.
  2. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. [Preprint]. arXiv Prepr. 2018. doi:10.48550/arXiv.1810.04805.
  3. Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–901.
  4. Howard J, Ruder S. Universal language model fine-tuning for text classification. [Preprint]. arXiv:1801.06146. 2018. doi:10.48550/arXiv.1801.06146. 2018.
  5. Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. [Preprint]. arXiv:2001.08361. 2020. doi:10.48550/arXiv.2001.08361.
  6. Reynolds L, McDonell K. Prompt programming for large language models: Beyond the few-shot paradigm. Ext Abstr CHI Conf Hum Factors Comput Syst. 2021;1–7. doi:10.1145/3411763.
  7. Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys. 2021;54(6):1-35. doi:10.1145/3457607
  8. Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y. Large language models are zero-shot reasoners. Adv Neural Inf Process Syst. 2022;35:22199–213.
  9. Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst. 2022;35:24824–37.
  10. Mialon G, Dessì R, Lomeli M, Nalmpantis C, Pasunuru R, Raileanu R, Rozière B, Schick T, Dwivedi-Yu J, Celikyilmaz A, Grave E, LeCun Y, Scialom T. Augmented language models: a survey. [Preprint]. arXiv:2302.07842. doi:10.48550/arXiv.2302.07842. 2023.
  11. Barocas S, Hardt M, Narayanan A. Fairness and Machine Learning: Limitations and Opportunities. Cambridge, Massachusetts, United States: MIT Press; 2023.
  12. Gebru T, Morgenstern J, Vecchione B, Vaughan JW, Wallach H, III HD, et al. Datasheets for datasets. Communications of the ACM. 2021;64(12):86-92. doi:10.1145/3458723
  13. Brundage M, Avin S, Wang J, Belfield H, Krueger G, Hadfield G, et al. Toward trustworthy AI development: mechanisms for supporting verifiable claims. [Preprint]. arXiv:2004.07213. 2020. doi:10.48550/arXiv.2004.07213. 2020.
  14. Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, et al. Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019:220-229. doi:10.1145/3287560.3287596
  15. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. HuggingFace's Transformers: state-of-the-art natural language processing. arXiv [Preprint]. arXiv:1910.03771. 2020. DOI: https://doi.org/10.48550/arXiv.1910.03771.
  16. Holmes W. Artificial intelligence in education: Promises and implications for teaching and learning. Boston: Center for Curriculum Redesign; 2019.
  17. Seldon A, Abidoye O. The Fourth Education Revolution. London: Legend Press Ltd; 2018.
  18. Woolf BP. Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing E-Learning. Burlington, Massachusetts, United States: Morgan Kaufmann Publishers; 2010.
  19. Floridi L, Cowls J. A Unified Framework of Five Principles for AI in Society. Harvard Data Science Review. 2019. doi:10.1162/99608f92.8cd550d1
  20. Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. [Preprint]. arXiv:1412.6572. 2014. doi:10.48550/arXiv.1412.6572.
  21. Christiano PF, Leike J, Brown T, Martic M, Legg S, Amodei D. Deep reinforcement learning from human preferences. Adv Neural Inf Process Syst. 2017;30:4299–310.
  22. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7
  23. Patel BN, Rosenberg L, Willcox G, Baltaxe D, Lyons M, Irvin J, et al. Human–machine partnership with artificial intelligence for chest radiograph diagnosis. npj Digital Medicine. 2019;2(1). doi:10.1038/s41746-019-0189-7
  24. Schneider G. Automating drug discovery. Nat Rev Drug Discov. 2018;17:97–113. DOI: 10.1038/ 2017.232. PubMed: 29242609.
  25. Whittlestone J, Nyrup R, Alexandrova A, Cave S. The Role and Limits of Principles in AI Ethics. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 2019:195-200. doi:10.1145/3306618.3314289
  26. Holstein K, McLaren BM, Aleven V. Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity. Journal of Learning Analytics. 2019;6(2). doi:10.18608/jla.2019.62.3
  27. McKenzie L. (2024). AI in Education: Emerging Trends and Critical Challenges - ISC Research. [online] ISC Research. Available from: https://iscresearch.com/ai-in-education-emerging-trends-and-critical-challenges/.
  28. Luckin R, Holmes W, Griffiths M, Forcier LB. Intelligence Unleashed: An Argument for AI in Education. London: Pearson Education; 2016.
  29. Williamson B, Eynon R. Historical threads, missing links, and future directions in AI in education. Learning, Media and Technology. 2020;45(3):223-235. doi:10.1080/17439884.2020.1798995
  30. Cho M, Yun H, Ko E. Contactless marketing management of fashion brands in the digital age. European Management Journal. 2023;41(4):512-520. doi:10.1016/j.emj.2022.12.005
  31. Chaturvedi R, Verma S. Opportunities and challenges of AI-driven customer service. Artif Intell Customer Serv. 2023;33–71.
  32. Pang B, Lee L. Opinion Mining and Sentiment Analysis. Foundations and Trends® in Information Retrieval. 2008;2(1-2):1-135. doi:10.1561/1500000011
  33. Følstad A, Skjuve M. Chatbots for customer service. Proceedings of the 1st International Conference on Conversational User Interfaces. 2019:1-9. doi:10.1145/3342775.3342784
  34. Chen M, Tworek J, Jun H, Yuan Q, Pinto HPO, Kaplan J, et al. Evaluating large language models trained on code. [Preprint]. arXiv:2107.03374. 2021. doi:10.48550/arXiv.2107.03374.
  35. Das R, Sandhane R. Artificial Intelligence in Cyber Security. Journal of Physics: Conference Series. 2021;1964(4):042072. doi:10.1088/1742-6596/1964/4/042072
  36. Ivanov D, Dolgui A. A digital supply chain twin for managing the disruption risks and resilience in the era of Industry 4.0. Prod Plan Control. 2021;32:775–788. doi:10.1080/09537287. 1768450.
  37. Putha S. AI-driven predictive maintenance for smart manufacturing: Enhancing equipment reliability and reducing downtime. J Deep Learn Genomic Data Anal. 2022;2:160–203.
Support