STM Journals Peer-reviewed research from Consortium e-Learning Network

International Journal of Data Structure Studies Review Article

Improving Dataset Integrity Through Automated Data Cleaning Techniques

  1. Rajani Kanta Sahu Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar
  2. Achinta Kumar Palit Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar
  3. Ushashree Nayak Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar

Abstract

High-quality data is a fundamental requirement in data science for producing trustworthy analytical insights and effective machine learning models. Problems, including incomplete records, inconsistent entries, duplicate observations, and anomalous values, can severely reduce the accuracy and robustness of predictive systems. As modern datasets continue to expand in both volume and structural complexity, relying on manual data cleaning methods become time-consuming and error-prone, highlighting the growing importance of automated data preprocessing solutions. This paper explores the automation of data cleaning within machine learning workflows, reviewing tools such as Open Refine, Trifacta, Data Cleaner, and Python libraries like Pandas and Dedupe. It examines key algorithmic approaches, including imputation techniques, to find unusual or abnormal data points (e.g., Z-score, interquartile range (IQR), Isolation Forest) and duplicate detection via fuzzy matching. The role of data profiling and validation frameworks in early-stage diagnosis is also discussed. Furthermore, the paper investigates how machine learning can be applied to data cleaning itself, enabling models to learn patterns for anomaly detection and correction. Challenges such as scalability, interpretability, and domain-specific customization are addressed, alongside future directions for autonomous data quality management. Interactive platforms like Data Lens, which combine profiling, error detection, and human-in-the-loop repair mechanisms, are highlighted for their role in building reproducible, ML-aligned workflows. The findings emphasize the transformative potential of AI-powered data cleaning in improving data integrity while reducing manual effort and operational cost.

Keywords

References (23)

  1. Dasu T, Johnson T. Exploratory Data Mining and Data Cleaning. Wiley Series in Probability and Statistics. 2003. doi:10.1002/0471448354
  2. Aggarwal CC. Outlier Analysis. 2017. doi:10.1007/978-3-319-47578-3
  3. Rahm E, Do HH. Data cleaning: problems and current approaches. IEEE Data Eng Bull. 2000;23(4):3–13.
  4. Redman TC. The impact of poor data quality on the typical enterprise. Communications of the ACM. 1998;41(2):79-82. doi:10.1145/269012.269025
  5. io. (2023). Dedupe.io - De-duplicate and find matches in your Excel spreadsheet or database. [online] Dedupe.io. Available at: https://dedupe.io/
  6. Pipino LL, Lee YW, Wang RY. Data quality assessment. Communications of the ACM. 2002;45(4):211-218. doi:10.1145/505248.506010
  7. (2026). OpenRefine: a powerful free, open source tool. [online] OpenRefine. Available from: https://openrefine.org/
  8. PyData Sphinx Theme. (2026). Pandas Documentation — Pandas 3.0.2 documentation. Version: 3.0.2. [online] Pandas: Open Source, BSD-Licensed Library. Available from: https://pandas.pydata.org/docs/
  9. Scikit-learn. (2025). Scikit-Learn: machine learning in Python — scikit-learn 1.8.0 documentation. [online] Available from: https://scikit-learn.org/stable/
  10. (2023). Trifacta is Now Alteryx Designer Cloud. [online] Alteryx. Available from: https://www.alteryx.com/about-us/trifacta-is-now-alteryx-designer-cloud
  11. van der Walt S, Colbert SC, Varoquaux G. The NumPy Array: A Structure for Efficient Numerical Computation. Computing in Science & Engineering. 2011;13(2):22-30. doi:10.1109/mcse.2011.37
  12. McKinney W. Data Structures for Statistical Computing in Python. Proceedings of the Python in Science Conference. 2010:56-61. doi:10.25080/majora-92bf1922-00a
  13. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.
  14. Wickham H. Tidy Data. Journal of Statistical Software. 2014;59(10). doi:10.18637/jss.v059.i10
  15. Chu X, Ilyas IF, Krishnan S, Wang J. Data Cleaning. Proceedings of the 2016 International Conference on Management of Data. 2016:2201-2206. doi:10.1145/2882903.2912574
  16. Bertrand Cariou (2020). New Dataprep AI features for data wrangling. [online] Google Cloud Blog. Available from: https://cloud.google.com/blog/products/data-analytics/new-dataprep-ai-features-for-data-wrangling
  17. Kandel S, Paepcke A, Hellerstein J, Heer J. Wrangler. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011:3363-3372. doi:10.1145/1978942.1979444
  18. Bantilan N. pandera: Statistical Data Validation of Pandas Dataframes. Proceedings of the Python in Science Conference. 2020:116-124. doi:10.25080/majora-342d178e-010
  19. Python Cerberus. (2023). Validation Schemas. [online] Cerberus — Data validation for Python. Available from: https://docs.python-cerberus.org/schemas.html
  20. Hershberg T, Burstein A, Dockhorn R. Record Linkage. Historical Methods Newsletter. 1976;9(2-3):137-163. doi:10.1080/00182494.1976.10112639
  21. (2020). FuzzyWuzzy: fuzzy string matching in Python [Online]. GitHub. Available from: https://github.com/seatgeek/fuzzywuzzy
  22. Rubin DB. Multiple Imputation for Nonresponse in Surveys. Wiley Series in Probability and Statistics. 1987. doi:10.1002/9780470316696
  23. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research. 2002;16:321-357. doi:10.1613/jair.953