Processing streams in a monitoring cloud cluster

The creation of monitoring clusters based on cloud computing technologies is a promising direction for the development of systems for continuous monitoring of objects for various purposes in the web space. Hadoop web-programming environment is the technological basis for the development of algorithmic and software solutions for the synthesis of monitoring clusters, including information security and information counteraction systems. The International Telecommunication Union’ (ITU) recommendations Y. 3510 present the requirements for cloud infrastructure that require monitoring the performance of deployed applications based on the collection of real-world statistics. Often, computing resources of monitoring clusters of cloud data centers are allocated for continuous parallel processing of high-speed streaming data, which imposes new requirements to monitoring technologies, necessitating the creation and research of new models of parallel computing. The need to use service monitoring plays an important role in the cloud computing industry, especially for SLA/QoS assessment, as the application or service may experience problems even if the virtual machines on which the work is taking place appear to be operational. This requires to study the methodological possibilities of organization to study of parallel processing high-speed streaming services with the processing of huge amounts of bit data, and, simultaneously, to estimate the necessary computational resource. In the conditions of high dynamics of changes in the bit rate of information generation from the source, a model of the bit rate of Discretized Stream (DStream) formation is proposed, which has a common application. Based on the poly-burst nature of the bit rate model, a model of group content traffic of any sources of different services processed in the cloud cluster was created. The obtained results made it possible to develop mathematical models of parallel DStreams from sources processed in a cloud cluster via Hadoop technology using the micro-batch architecture of the Spark Streaming module. These models take into account the flow of requests for maintenance from sources of different services, on the one hand, and, on the other hand, the needs of services in bit rate, taking into account the multichannel traffic of sources of various services. At the same time, analytical relations are obtained to calculate the required performance of the Hadoop cluster at a given value of the probability of batch loss.

monitoring, Hadoop, Spark, batch, bit rate, micro-batch, architecture, parallel flow, cloud computing, expectation, variance, probability, probability distribution function, probability distribution density, random process, Delta function

1. Gudkova I., Maslovskaya N. Probability model for analysing impact of delays due to monitoring on mean service time in cloud computing. T-comm. 2014;8(6):13-15 (in Russ.).

2. Basharin G., Gaidamaka Y., Samoylov K. Mathematical theory of teletraffic and its application to the analysis of multiservice communication of next generation networks. Automatic Control and Computer Sciences. 2013;47(2):62-69. https://doi.org/10.3103/S0146411613020028

3. Buyya R., Broberg J., Goscinski A. Cloud Computing. Principles and Paradigms. New Jersey: John Wiley & Sons, Inc., 2011. 637 p. https://doi-org/10.1002/9780470940105

4. Focus Group on Cloud Computing. Technical Report. Part 1: Introduction to the cloud ecosystem: definitions, taxonomies, use cases and higher-level requirements. ver. 1.0 (02/2012). International Telecommunication Union, 2012. 62 p. http://handle.itu.int/11.1002/pub/808604ae-en

5. Khaled S., Boutaba R. Estimating service response time for elastic cloud applications. In: Proceed. of the 1th International Conference on Cloud Networking CLOUDNET, IEEE, 2012; pp. 12-16. https://doi.org/10.1109/CloudNet.2012.6483647

6. Recommendation ITU-T, 2013, Cloud Computing framework and high-level requirements, Y.3501, p. 27.

7. Recommendation ITU-T, 2013, Cloud Computing infrastructure requirements, Y.3510, p. 22.

8. Recommendation ITU-T, 2015, Cloud Computing framework for end-to-end resource management, Y.3520, (09/15).

9. Recommendation ITU-T, 2016, Overview of end-to-end cloud computing management, Y.3521/M.3070 (03/16).

10. Recommendation ITU-T, 2016, End-to-end cloud service lifecycle management requirements, Y.3522 (09/16).

11. Recommendation ITU-T, 2018, Cloud computing – Overview of inter-cloud trust management, Y.3517 (12/18).

12. Recommendation ITU-T, 2018, Cloud computing – functional requirements of inter-cloud data management, Y.3518 (12/18).

13. Karau H., Konwinski A., Wendell P., Zaharia М. Learning Spark: Lighting-fast data analysis. US: O’Reilly, 2015. 257 p.

14. Erokhin S.D. A review of scientific research on artificial intelligence. In: Proceed. 2019 Systems of signals generating and processing in the field of on board communications. IEEE, 2019. 4 p. INSPEC Accession Number: 18638425. https://doi.org/10.1109/SOSG.2019.8706723

15. Chesnokov A.S., Gorodnichev M.G., Gavrish K.A., Zhidkova M.A. Intelligent vehicle condition monitoring system. In: Proceed. 2019 Systems of signals generating and processing in the field of on board communications. IEEE, 2019. 4 p. INSPEC Accession Number: 18638469. https://doi.org/10.1109/SOSG.2019.8706727

16. Lam C.P. Hadoop in Action. Publisher: Manning Publications Company, 2011. 334 p.

17. Nazarov A., Nazarov M., Pantiuhin D, Sychev A., Pokrova S. Automation of monitoring processes in webbased neuro-fuzzy formalism. T-comm. 2015;9(8):26-33 (in Russ.).

18. Munerman V.I. The implementation of parallel data processing in cloud systems. Sovremennye informacionnye tehnologii i IT-obrazovanie = Modern Information Technology and IT-education. 2017;13(2):57-63 (in Russ.). https://doi.org/10.25559/SITITO.2017.2.223

19. Nazarov A.N. Model of parallel processing of tasks in the cloud cluster Hadoop. In: Proceedings of the XIII International Industry Scientific and Technical Conference “Technologies of the information society” (March 20-21, 2019, Moscow, MTUCI). In 2 v. V. 2. M.: Publishing House Media Publisher, 2019; pp. 69-71 (in Russ.).

20. Grigoriev V.R., Nazarov A.N. Methodological aspects of parallel problem solving in the cloud cluster of cyberattacks monitoring. In: Proceedings of the XVIII Scientific and Practical Conference “Information technologies in public administration. Digital transformation into human capital” (April 25, 2019). M.: Research Institute “Voskhod”, 2019. 4 p. (in Russ.).

21. Sokolov G.A., Gladkih I.M. Mathematical statistics. М.: Ekzamen Publ., 2004. 432 p. (in Russ.).