International Journal of Data Structure Studies Review Article

Improving Dataset Integrity Through Automated Data Cleaning Techniques

  1. Rajani Kanta Sahu Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar
  2. Achinta Kumar Palit Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar
  3. Ushashree Nayak Department of Computer science and Engineering, Gandhi Institute of Excellent Technocrats, Bhubaneswar

Abstract

High-quality data is a fundamental requirement in data science for producing trustworthy analytical insights and effective machine learning models. Problems, including incomplete records, inconsistent entries, duplicate observations, and anomalous values, can severely reduce the accuracy and robustness of predictive systems. As modern datasets continue to expand in both volume and structural complexity, relying on manual data cleaning methods become time-consuming and error-prone, highlighting the growing importance of automated data preprocessing solutions. This paper explores the automation of data cleaning within machine learning workflows, reviewing tools such as Open Refine, Trifacta, Data Cleaner, and Python libraries like Pandas and Dedupe. It examines key algorithmic approaches, including imputation techniques, to find unusual or abnormal data points (e.g., Z-score, interquartile range (IQR), Isolation Forest) and duplicate detection via fuzzy matching. The role of data profiling and validation frameworks in early-stage diagnosis is also discussed. Furthermore, the paper investigates how machine learning can be applied to data cleaning itself, enabling models to learn patterns for anomaly detection and correction. Challenges such as scalability, interpretability, and domain-specific customization are addressed, alongside future directions for autonomous data quality management. Interactive platforms like Data Lens, which combine profiling, error detection, and human-in-the-loop repair mechanisms, are highlighted for their role in building reproducible, ML-aligned workflows. The findings emphasize the transformative potential of AI-powered data cleaning in improving data integrity while reducing manual effort and operational cost.

Keywords

References (23)

  1. Dasu T, Johnson T. Exploratory Data Mining and Data Cleaning. Hoboken (NJ, US): Wiley Interscience; 2003. doi:10.1002/0471448354.
  2. Aggarwal CC. Outlier Analysis. 2nd ed. Cham: Springer; 2017. doi:10.1007/978-3-319-47578-3.
  3. Rahm E, Do HH. Data cleaning: problems and current approaches. IEEE Data Eng Bull. 2000;23(4):3–13.
  4. Redman TC. The impact of poor data quality on the typical enterprise. Commun ACM. 1998;41(2):79–82. doi:10.1145/269012.269025.
  5. io. (2023). Dedupe.io - De-duplicate and find matches in your Excel spreadsheet or database. [online] Dedupe.io. Available at: https://dedupe.io/
  6. Pipino LL, Lee YW, Wang RY. Data quality assessment. Commun ACM. 2002;45:211–218. doi:10.1145/505248.506010.
  7. (2026). OpenRefine: a powerful free, open source tool. [online] OpenRefine. Available from: https://openrefine.org/
  8. PyData Sphinx Theme. (2026). Pandas Documentation — Pandas 3.0.2 documentation. Version: 3.0.2. [online] Pandas: Open Source, BSD-Licensed Library. Available from: https://pandas.pydata.org/docs/
  9. Scikit-learn. (2025). Scikit-Learn: machine learning in Python — scikit-learn 1.8.0 documentation. [online] Available from: https://scikit-learn.org/stable/
  10. (2023). Trifacta is Now Alteryx Designer Cloud. [online] Alteryx. Available from: https://www.alteryx.com/about-us/trifacta-is-now-alteryx-designer-cloud
  11. Van der Walt S, Colbert SC, Varoquaux G. The NumPy array: a structure for efficient numerical computation. Comput Sci Eng. 2011;13(2):22–30. doi:10.1109/MCSE.2011.37.
  12. McKinney W. Data structures for statistical computing in Python. Python in Science Conference, Austin, Texas, USA. 2010. p. 56–61. doi:10.25080/Majora-92bf1922-00a.
  13. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–2830.
  14. Wickham H. Tidy data. J Stat Softw. 2014;59(10):1–23. doi:10.18637/jss.v059.i10.
  15. Chu X, Ilyas IF, Krishnan S, Wang J. Data cleaning: overview and emerging challenges. International Conference on Management of Data (SIGMOD/PODS’16), San Francisco, California, USA. 2016 Jun 26–Jul 1. p. 2201–2206. doi:10.1145/2882903.2912574.
  16. Bertrand Cariou (2020). New Dataprep AI features for data wrangling. [online] Google Cloud Blog. Available from: https://cloud.google.com/blog/products/data-analytics/new-dataprep-ai-features-for-data-wrangling
  17. Kandel S, Paepcke A, Hellerstein J, Heer J. Wrangler: interactive visual specification of data transformation scripts. CHI Conference on Human Factors in Computing Systems (CHI ‘11), Vancouver, BC, Canada. 2011 May 7–12. p. 3363–3372. doi:10.1145/1978942.1979444.
  18. Bantilan N. Pandera: statistical data validation of pandas dataframes. Proc Python Sci Conf. 2020:116–124. doi:10.25080/Majora-342d178e-010.
  19. Python Cerberus. (2023). Validation Schemas. [online] Cerberus — Data validation for Python. Available from: https://docs.python-cerberus.org/schemas.html
  20. Hershberg T, Burstein A, Dockhorn R. Record linkage. Hist Methods Newsl. 1976;9(2-3):137–163. doi:10.1080/00182494.1976.10112639
  21. (2020). FuzzyWuzzy: fuzzy string matching in Python [Online]. GitHub. Available from: https://github.com/seatgeek/fuzzywuzzy
  22. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York (NY, US): John Wiley & Sons; 1987. doi:10.1002/9780470316696.
  23. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321–357. doi:10.1613/jair.953.

Read more of it

Subscribe to one journal or build your own list of several. One invoice, one payment, access for a full year.

Subscribe to journals →

Publish your article

Peer-reviewed journals across every subject, with open access options. Send your manuscript to the right editor today.

Publish with us → Calculate your APC
Support