data engineering
2 articles · search the full text for this term
-
A Hierarchical Orchestration Framework for Parallelized Multi-Entity Web Ingestion and NoSQL Synchronization
Abstract: In the current EdTech sector, gathering structured data from various fragmented web portals is a major challenge for building reliable, real-time data systems. Standard sequential ETL (Extract, Transform, Load) pipelines often encounter significant bottlenecks, as a single failure in one module can halt the entire ingestion process. Furthermore, linear execution is often too slow to meet the demands for large, multi-category datasets. This paper presents a Modular Hierarchical Orchestration framework …
Published in Recent Trends in Parallel Computing · Vol. 13, Issue 2, 2026 · pp. 1–6 Read article
-
The Future of Multi-cloud Infrastructures in Data Engineering for Challenge of Handling Large and Dynamic Datasets
Abstract: This paper explores the future of multi-cloud infrastructures in data engineering for the challenge of handling large and dynamic datasets. It delves into the opportunities and challenges presented by multi-clouds, focusing on their role in managing large volumes of data and complex computational tasks. The paper discusses the advantages of multi-clouds, such as manageability, flexibility, and cost-effectiveness, and their potential to enhance the efficiency and accuracy of recommender systems. It …
Published in Recent Trends in Parallel Computing · Vol. 10, Issue 3, 2023 · pp. 18–23 Read article