STM Journals Peer-reviewed research from Consortium e-Learning Network

Recent Trends in Parallel Computing Review Article

A Hierarchical Orchestration Framework for Parallelized Multi-Entity Web Ingestion and NoSQL Synchronization

  1. Taufique Khan Asadulla Khan Department. of Computer Applications, GH Raisoni University
  2. Ankush D. Bambal Department. of Computer Applications, GH Raisoni University
  3. Kartik S. Kharabe Department. of Computer Applications, GH Raisoni University
  4. Sonali Nimbhorkar Department. of Computer Applications, GH Raisoni University

Abstract

In the current EdTech sector, gathering structured data from various fragmented web portals is a major challenge for building reliable, real-time data systems. Standard sequential ETL (Extract, Transform, Load) pipelines often encounter significant bottlenecks, as a single failure in one module can halt the entire ingestion process. Furthermore, linear execution is often too slow to meet the demands for large, multi-category datasets. This paper presents a Modular Hierarchical Orchestration framework designed to solve these issues through high-speed, parallel data collection. The system utilizes a centralized “Master–Orchestrator” to manage multiple independent sub-scripts simultaneously each specializing in different domains such as college profile, academic course details, competitive exams, and career trajectories. By running these tasks in parallel, the framework optimizes server CPU utilization and drastically reduces the total ingestion window. The collected data is synchronized and stored in MongoDB NoSQL database, which provides the necessary schema flexibility to handle heterogenous data formats without sacrificing relational integrity. This system was rigorously tested in a real-world industrial environment, where it demonstrated a robust reduction in processing time and superior fault tolerance compared to traditional, single-track scraping methods.

Keywords

References (12)

  1. Rucco C, Longo A, Saad M. Enhancing data ingestion efficiency in cloud-based systems: a design pattern approach. Data Sci Eng. 2026 Jun;11(2):287–302.
  2. Talakola S. Automated end to end testing with Playwright for React applications. Int J Emerg Res Eng Technol. 2024 Mar 30;5(1):38–47.
  3. MongoDB Documentation Team. What is MongoDB? [Online]. MongoDB; 2026. Available from: https://www.mongodb.com/docs/manual/
  4. de Lima Melo FF, Ferreira F. Performance analysis of NoSQL databases with large volumes of open educational data. Int J Comput Appl. 2024;174(29):9–17.
  5. Ma J. A high performance computing web search engine based on big data and parallel distributed models. Informatica. 2024;48(20):27–38.
  6. Bezawada N, Rajagopal AR, Swaminathan A, Smrithy GS, Elakkiya R. Optimizing data extraction techniques in the automation industry. In: International Conference on Hybrid Intelligent Systems; 2023 Dec 11. p. 89–98. Cham: Springer Nature Switzerland.
  7. Eyzenakh DS, Rameykov AS, Nikiforov IV. High performance distributed web-scraper. Tr Inst Sist Program RAN. 2021;33(3):87–100.
  8. Chaganti S. Automated schema enforcement and stateless deduplication in distributed data pipelines: a contract-driven approach. In: SoutheastCon 2026; 2026 Feb 20. p. 1–5.
  9. Richardson L, Ruby S. RESTful web services. Sebastopol (CA): O'Reilly Media; 2008 Dec 17.
  10. Chodorow K. MongoDB: the definitive guide. Sebastopol (CA): O'Reilly Media; 2013.
  11. Lewis J, Fowler M. Microservices: a definition of this new architectural term [Online]. MartinFowler.com. 2014 Mar 25. Available from: https://martinfowler.com/articles/microservices.html
  12. Roth J, Duan Y, Mahner FP, Kaniuth P, Wallis TS, Hebart MN. Ten principles for reliable, efficient, and adaptable coding in psychology and cognitive neuroscience. Commun Psychol. 2025 Apr 15;3(1):62.
Support