Methods for prioritizing the processes of transferring data to central storage

Objectives. The efficient management of parallel ETL (Extract, Transform, Load) process execution in central data warehouses critically impacts overall processing time. Existing orchestration tools such as Apache Airflow, NiFi, Luigi employ simplified prioritization algorithms which ignore dependency graph topology and resource dynamics, leading to suboptimal scheduling. The objective of this work is to develop and validate a novel task prioritization method for ETL pipelines, aimed at minimizing their total duration through deep analysis of structural features of Directed Acyclic Graphs (DAGs), as well as the use of simulation modeling to evaluate various scheduling strategies under conditions of competition for limited concurrency slots.

Methods. The study proposed a Python simulation model, replicating ETL process execution in an environment with limited concurrency slots. The model generates a DAG which reflects the dependency structure of processes for building a central data warehouse and compares 9 prioritization algorithms. These include basic algorithms (prioritization by minimum/maximum average execution time), topological algorithms (prioritization by minimum/maximum layer level, maximization of dependency count), and hybrid algorithms (splitting slots into queues for minimum and maximum execution time). Experiments were conducted on graphs of a variety of topologies using the developed simulation model.

Results. The hybrid algorithm (slot allocation: 50% for tasks with maximum execution time, 50% for tasks with minimum execution time) demonstrated the highest level of efficiency. It reduced total execution time by 15–17%, when compared to basic algorithms, minimized task idle time by 20–25%, and showed resilience to graph topology variations. A linear combination of optimized coefficients (execution time being the most significant factor) ranked second in terms of efficiency.

Conclusions. Prioritization based on DAG topology analysis and hybrid strategies significantly reduces ETL pipeline execution time. The hybrid algorithm is recommended for implementation in orchestrators, since it balances minimizing pipeline duration and task idle time. A promising area for further study is the development of adaptive algorithms that account for real-time dynamic resource load.

1. Kimball R., Ross M. The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling. 3nd ed. Kimball group. Wiley; 2013, 608 p.

2. Simitsis A., Skiadopoulos S., Vassiliadis P. The History, Present, and Future of ETL Technology. In: DOLAP, CEUR Workshop Proceedings. 2023;3369:3–12.

3. Tian W. Enhancing Financial Decision-Making Through Automated Business Intelligence Systems. Int. J. e-Collaboration (IJeC). 2025;21(1):1–20. Available from URL: https://www.igi-global.com/article/enhancing-financial-decision-making-through-automated-business-intelligence-systems/367575. Accessed January 20, 2025.

4. El-Sappagh S.H.A., Hendawi A.M.A., El Bastawissy A.H. A proposed model for data warehouse ETL processes. Journal of King Saud University – Computer and Information Sciences (J. King Saud Univ.). 2011;23(2):91–104. https://doi.org/10.1016/j.jksuci.2011.05.005

5. Wijaya R., Pudjoatmodjo B. An overview and implementation of extraction-transformation-loading (ETL) process in data warehouse. In: 2015 3rd International Conference on Information and Communication Technology (ICoICT). 2015. P. 70–74. https://doi.org/10.1109/ICoICT.2015.7231399

6. Kuzmina Yu.V., Kubanskikh O.V. Brief description of the ETL process. Uchenye Zapiski Bryanskogo Gosudarstvennogo Universiteta = Scientific Notes of the Bryansk State University. 2017;1(5):33–36 (in Russ.). https://www.elibrary.ru/zmwlez

7. Dhaouadi A., Bousselmi K., Gammoudi M.M., Monnet S., Hammoudi S. Data Warehousing Process Modeling from Classical Approaches to New Trends: Main Features and Comparisons. Data. 2022;7(8):113. https://doi.org/10.3390/data7080113

8. Vassiliadis P., Simitsis A., Skiadopoulos S. Graph-Based Modeling of ETL Activities with Multi-level Transformations and Updates. In: Tjoa A.M., Trujillo J. (Eds.). Data Warehousing and Knowledge Discovery. Part of the book series: DaWaK 2005. Lecture Notes in Computer Science. 2005. V. 3589. P. 43–52. https://doi.org/10.1007/11546849_5

9. Yasmin J., Wang J.A., Tian Y., Adams B. An empirical study of developers’ challenges in implementing Workflows as Code: A case study on Apache Airflow. J. Syst. Software. 2024;219(5):112248. https://doi.org/10.1016/j.jss.2024.112248

10. Mikhailov A.N. Using Apache Airflow for Data Processing Orchestration. Vestnik Nauki. 2024;10(79):783–787 (in Russ.). https://www.elibrary.ru/ijihms

11. Gromov N.D., Platoshin A.I., Panov A.V. Comparative analysis of tools and platforms for automation of ETL processes in modern data warehouses. Mezhdunarodnyi zhurnal gumanitarnykh i estestvennykh nauk = International Journal of Humanities and Natural Sciences. 2023;11-4(86):46–48 (in Russ.). https://doi.org/10.24412/2500-1000-2023-11-4-46-48

12. Zhdanov D.E. Building ETL Processes Based on Cron and Luigi Task Orchestrator. Aktual’nye issledovaniya = Current Research. 2023;46-1(176):63–68 (in Russ.). https://www.elibrary.ru/nenikj

13. Ueter N., Günzel M., Brüggen G., Chen J. Parallel Path Progression DAG Scheduling. IEEE Transactions on Computers. 2023;72(10):3002–3016. https://doi.org/10.1109/TC.2023.3280137

14. Bogatyrev V.A., Bogatyrev S.V., Bogatyrev A.V. Assessment of the readiness of a computer system for timely servicing of requests when combined with informational recovery of memory after failures. Nauchno-tekhnicheskii vestnik informatsionnykh tekhnologii, mekhaniki i optiki = Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 2023;23(3):608–617 (in Russ.). https://doi.org/10.17586/2226-1494-2023-23-3-608-617

15. Bogatyrev V.A., Bogatyrev S.V., Bogatyrev A.V. Recovery of Real-Time Clusters with the Division of Computing Resources into the Execution of Functional Queries and the Restoration of Data Generated Since the Last Backup. In: Vishnevskiy V.M., Samouylov K.E., Kozyrev D.V. (Eds.). Distributed Computer and Communication Networks: Control, Computation, Communications. Book series: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2024. V. 14123. P. 236–250. https://doi.org/10.1007/978-3-031-50482-2_19

16. Karagiannis A., Vassiliadis P., Simitsis A. Scheduling strategies for efficient ETL execution. Inform. Syst. 2013;38(6): 927–945. https://doi.org/10.1016/j.is.2012.12.001

17. Topcuoglu H., Hariri S., Min-You Wu. Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing. IEEE Trans. Parallel Distrib. Syst. 2007;13(3):260–274. https://doi.org/10.1109/71.993206

18. Strengholt P. Building Medallion Architectures. 1st ed. Sebastopol (CA): O’Reilly Media; 2024, 209 р.

19. Serra J. Deciphering Data Architectures. 1st ed. Sebastopol (CA): O’Reilly Media; 2024, 146 р.

20. Blažić G., Poščić P., Jakšić D. Data warehouse architecture classification. In: 2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). 2017. P. 1491–1495. https://doi.org/10.23919/MIPRO.2017.7973657