STM Journals Peer-reviewed research from Consortium e-Learning Network

Recent Trends in Programming languages Review Article

Comparison of Models of Machine Learning and Hyperparameter optimization methods on various datasets

  1. Naviya Shetty Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development & Research (TIMSCDR)
  2. Anuja Shinde Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development & Research (TIMSCDR)
  3. Shiksha Dubey Department of Computer Science and Engineering, Thakur Institute of Management Studies, Career Development & Research (TIMSCDR)

Abstract

The most likely phase in achieving powerful and robust machine learning models is probably the hyperparameters tuning step. The traditional exhaustive methods of search (Grid Search and others) ensure that the search space is covered, but are computationally very inexpensive; random search is less expensive and can still miss good regions; and lastly, the modern model-based and population-based methods (Bayesian Optimization, Tree-structured Parzen Estimator (TPE), Genetic Algorithms) are thought to provide better trade-offs between performance and resource utilization. This study performs a comparative study of five tuning algorithms (Grid Search, Random Search, Bayesian Optimization, TPE/Optuna, Genetic Algorithm) applied to a wide range of supervised models (Random Forest, XGBoost, SVM, MLP) on a number of classification and regression data sets. To record the results, we configured the experiments to measure accuracy (or RMSE), running time, memory consumption, and interpret and visualize the results in one of the analytical dashboards of Streamlit. Based on the results, model-based optimizers (TPE and Bayesian) achieved near-optimal solutions at a fraction of the time and memory that I needed theGrid Search method, thus is a suitable fit in low-resource environments. The paper outlines replication procedure and stages and technical configurations of the dashboard and export/reporting pipeline (refer to project files in the code modules). The choice of hyperparameters, including learning rates, regularization coefficients, the depth of trees, estimators number, etc., affect the model generalization and efficiency greatly. It is not always possible to run an exhaustive tuning of a large parameter grid when using complex models and multiple datasets, thus the importance of using more intelligent searching strategies increases again, particularly when they are limited to running the experiments on small CPUs, memory, or edge hardware. Here, we comparatively tune a range of tuning methods on smaller data and track model and wrap experiments, visualization, and statistical testing on a reusable Streamlit dashboard. We will arrive at solutions to both tuning strategies that provided a good trade-off between accuracy and computational cost and provide a well-documented, reproducible workflow to practitioners.

Keywords

References (34)

  1. Yang L, Shami A. On hyperparameter optimization of machine learning
  2. algorithms: Theory and practice. Neurocomputing. 2020 Nov 20;415:295-
  3. Bischl B, Binder M, Lang M, Pielok T, Richter J, Coors S, Thomas J,
  4. Ullmann T, Becker M, Boulesteix AL, Deng D. Hyperparameter
  5. optimization: Foundations, algorithms, best practices, and open challenges.
  6. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery.
  7. 2023 Mar;13(2):e1484.
  8. Turner R, Eriksson D, McCourt M, Kiili J, Laaksonen E, Xu Z, Guyon I.
  9. Bayesian optimization is superior to random search for machine learning
  10. hyperparameter tuning: Analysis of the black-box optimization challenge
  11. 2020. InNeurIPS 2020 competition and demonstration track 2021 Aug 7 (pp.
  12. 3-26). PMLR.
  13. Dasgupta S, Sen J. A comparative study of hyperparameter tuning methods.
  14. Data Science in Theory and Practice. 2024 Sep 13:258.
  15. Franceschi L, Donini M, Perrone V, Klein A, Archambeau C, Seeger M,
  16. Pontil M, Frasconi P. Hyperparameter optimization in machine learning.
  17. Foundations and Trends® in Machine Learning. 2025 Oct 2;18(6):975-1109.
  18. Snoek J, Larochelle H, Adams RP. Practical bayesian optimization of
  19. machine learning algorithms. Advances in neural information processing
  20. systems. 2012;25.
  21. Bergstra J, Yamins D, Cox D. Making a science of model search:
  22. Hyperparameter optimization in hundreds of dimensions for vision
  23. architectures. InInternational conference on machine learning 2013 Feb 13
  24. (pp. 115-123). PMLR.
  25. Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A. Hyperband:
  26. Bandit-based configuration evaluation for hyperparameter optimization.
  27. InInternational Conference on Learning Representations 2017 Feb 6.
  28. Feurer M, Hutter F. Hyperparameter optimization. InAutomated machine
  29. learning: Methods, systems, challenges 2019 May 18 (pp. 3-33). Cham:
  30. Springer International Publishing.
  31. . Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: A next-generation
  32. hyperparameter optimization framework. InProceedings of the 25th ACM
  33. SIGKDD international conference on knowledge discovery & data mining
  34. 2019 Jul 25 (pp. 2623-2631).
Support