mireabulletinRussian Technological JournalRussian Technological Journal2782-32102500-316XRTU MIREA10.32362/2500-316X-2022-10-2-59-74mireabulletin-485Research ArticleМАТЕМАТИЧЕСКОЕ МОДЕЛИРОВАНИЕMATHEMATICAL MODELINGПрименение биоинспирированных алгоритмов глобальной оптимизации для повышения точности прогнозов компактных машин экстремального обученияApplication of bioinspired global optimization algorithms to the improvement of the prediction accuracy of compact extreme learning machineshttps://orcid.org/0000-0003-4516-3746ДемидоваЛ. А.DemidovaL. A.

Демидова Лилия Анатольевна, д.т.н., профессор, профессор кафедры корпоративных информационных систем Института информационных технологий

119454, Москва, пр-т Вернадского, д. 78

Scopus Author ID 56406258800

ResearcherID R-6077-2016

Liliya A. Demidova, Dr. Sci. (Eng.), Professor, Professor, ERP Systems Department, Institute of Information Technologies

78, Vernadskogo pr., Moscow, 119454

Scopus Author ID 56406258800

ResearcherID R-6077-2016 

demidova.liliya@gmail.comhttps://orcid.org/0000-0003-1977-8165ГорчаковА. В.GorchakovA. V.

Горчаков Артём Владимирович, аспирант кафедры корпоративных информационных систем Института информационных технологий

119454, Москва, пр-т Вернадского, д. 78

Scopus Author ID 57215001290

ResearcherID ABC-8911-2021

Artyom V. Gorchakov, Postgraduate Student, ERP Systems Department, Institute of Information Technologies

78, Vernadskogo pr., Moscow, 119454

Scopus Author ID 57215001290

ResearcherID ABC-8911-2021 

worldbeater-dev@yandex.ruМИРЭА – Российский технологический университетРоссияMIREA – Russian Technological UniversityRussian Federation2022030420221025974Copyright © Демидова Л.А., Горчаков А.В., 20222022Демидова Л.А., Горчаков А.В.Demidova L.A., Gorchakov A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.rtj-mirea.ru/jour/article/view/485

Цели. В результате современных исследований в машинном обучении, направленных на повышение точности и снижение вычислительной сложности алгоритмов анализа данных, была предложена новая архитектура искусственной нейронной сети – машина экстремального обучения. Это нейронная сеть прямого распространения с единственным скрытым слоем. В этой сети веса соединений между входными нейронами и нейронами скрытого слоя инициализируются случайно, а веса соединений между нейронами скрытого слоя и выходными нейронами вычисляются с использованием операции псевдообращения Мура – Пенроуза. Замена итерационного процесса обучения, присущего многим архитектурам нейронных сетей, на случайную инициализацию одной части весов и вычисление другой части делает рассматриваемый инструмент существенно более производительным, с сохранением хорошей обобщающей способности. Однако случайная инициализация входных весов не гарантирует оптимальной точности прогнозов. Цель работы – разработка и исследование подходов к интеллектуальной настройке входных весов в машинах экстремального обучения биоинспирированными алгоритмами для повышения точности прогнозов этого инструмента анализа данных в задачах восстановления регрессии.Методы. Использованы методы теории оптимизации, теории эволюционных вычислений и роевого интеллекта, теории вероятностей и математической статистики, системного анализа.Результаты. Разработаны и исследованы подходы к интеллектуальной настройке входных весов в машинах экстремального обучения, основанные на применении генетического алгоритма, алгоритма роя частиц, алгоритма поиска косяком рыб, алгоритма хаотического поиска косяком рыб с экспоненциальным убыванием шага, предложенного авторами настоящего исследования. Выявлено, что применение биоинспирированных алгоритмов способно улучшить точность прогнозов машин экстремального обучения в задачах восстановления регрессии, причем машине экстремального обучения с уточненными биоинспирированными алгоритмами весами требуется меньшее число нейронов на скрытом слое для достижения высокой точности прогнозов на тренировочных и тестовых наборах данных. С помощью хаотического алгоритма поиска косяком рыб с экспоненциальным убыванием шага могут быть получены наилучшие конфигурации машин экстремального обучения в рассмотренных задачах.Выводы. Полученные результаты показывают, что точность прогнозов машин экстремального обучения может быть улучшена посредством применения биоинспирированных алгоритмов интеллектуальной настройки входных весов. Для выполнения настройки весов требуются дополнительные вычисления, поэтому использование машин экстремального обучения в сочетании с биоинспирированными алгоритмами может быть целесообразно в тех областях, где необходимо получение наиболее точной и компактной конфигурации машины экстремального обучения.

Objectives. Recent research in machine learning and artificial intelligence aimed at improving prediction accuracy and reducing computational complexity resulted in a novel neural network architecture referred to as an extreme learning machine (ELM). An ELM comprises a single-hidden-layer feedforward neural network in which the weights of connections among input-layer neurons and hidden-layer neurons are initialized randomly, while the weights of connections among hidden-layer neurons and output-layer neurons are computed using a generalized Moore– Penrose pseudoinverse operation. The replacement of the iterative learning process currently used in many neural network architectures with the random initialization of input weights and the explicit computation of output weights significantly increases the performance of this novel machine learning algorithm while preserving good generalization performance. However, since the random initialization of input weights does not necessarily guarantee optimal prediction accuracy, the purpose of the present work was to develop and study approaches to intelligent adjustment of input weights in ELMs using bioinspired algorithms in order to improve the prediction accuracy of this data analysis tool in regression problems.Methods. Methods of optimization theory, theory of evolutionary computation and swarm intelligence, probability theory, mathematical statistics and systems analysis were used.Results. Approaches to the intelligent adjustment of input weights in ELMs were developed and studied. These approaches are based on the genetic algorithm, the particle swarm algorithm, the fish school search algorithm, as well as the chaotic fish school search algorithm with exponential step decay proposed by the authors. By adjusting input weights with bioinspired optimization algorithms, it was shown that the prediction accuracy of ELMs in regression problems can be improved to reduce the number of hidden-layer neurons to reach a high prediction accuracy on learning and test datasets. In the considered problems, the best ELM configurations can be obtained using the chaotic fish school search algorithm with exponential step decay.Conclusions. The obtained results showed that the prediction accuracy of ELMs can be improved by using bioinspired algorithms for the intelligent adjustment of input weights. Additional calculations are required to adjust the weights; therefore, the use of ELMs in combination with bioinspired algorithms may be advisable where it is necessary to obtain the most accurate and most compact ELM configuration.

нейронные сетимашины экстремального обучениябиоинспирированные алгоритмыгенетический алгоритмалгоритм роя частицалгоритм поиска косяком рыбмашинное обучениерегрессионный анализneural networksextreme learning machinebioinspired algorithmsgenetic algorithmparticle swarm optimization algorithmfish school search algorithmmachine learningregression analysisReferencesWu Y., Ianakiev K., Govindaraju V. Improved k-nearest neighbor classification. Pattern Recognition. 2002;35(10):2311–2318. https://doi.org/10.1016/S0031-3203(01)00132-7Wu Y., Ianakiev K., Govindaraju V. Improved k-nearest neighbor classification. Pattern Recognition. 2002;35(10):2311–2318. https://doi.org/10.1016/S0031-3203(01)00132-7Noble W.S. What is a support vector machine? Nat. Biotechnol. 2006;24(12):1565–1567. https://doi.org/10.1038/nbt1206-1565Noble W.S. What is a support vector machine? Nat. Biotechnol. 2006;24(12):1565–1567. https://doi.org/10.1038/nbt1206-1565Demidova L.A. Two-stage hybrid data classifiers based on SVM and kNN algorithms. Symmetry. 2021;13(4):615. https://doi.org/10.3390/sym13040615Demidova L.A. Two-stage hybrid data classifiers based on SVM and kNN algorithms. Symmetry. 2021;13(4):615. https://doi.org/10.3390/sym13040615Lin W., Wu Z., Lin L., Wen A., Li J. An ensemble random forest algorithm for insurance Big Data analysis. IEEE Access. 2017;5:16568–16575. https://doi.org/10.1109/ACCESS.2017.2738069Lin W., Wu Z., Lin L., Wen A., Li J. An ensemble random forest algorithm for insurance Big Data analysis. IEEE Access. 2017;5:16568–16575. https://doi.org/10.1109/ACCESS.2017.2738069Deng L., Hinton G., Kingsbury B. New types of deep neural network learning for speech recognition and related applications: An overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013:8599–8603. https://doi.org/10.1109/ICASSP.2013.6639344Deng L., Hinton G., Kingsbury B. New types of deep neural network learning for speech recognition and related applications: An overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013:8599–8603. https://doi.org/10.1109/ICASSP.2013.6639344Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review. 1958;65(6):386–408. https://doi.org/10.1037/h0042519Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review. 1958;65(6):386–408. https://doi.org/10.1037/h0042519Affonso C., Debiaso Rossi A.L., Antunes Vieira F.H., Ponce de Leon Ferreira de Carvalho A.C. Deep learning for biological image classification. Expert Systems with Applications. 2017;85:114–122. https://doi.org/10.1016/j.eswa.2017.05.039Affonso C., Debiaso Rossi A.L., Antunes Vieira F.H., Ponce de Leon Ferreira de Carvalho A.C. Deep learning for biological image classification. Expert Systems with Applications. 2017;85:114–122. https://doi.org/10.1016/j.eswa.2017.05.039Chen N., Xiong C., Du W., Wang C., Lin X., Chen Z. An improved genetic algorithm coupling a back-propagation neural network model (IGA-BPNN) for water-level predictions. Water. 2019;11(9):1795. https://doi.org/10.3390/w11091795Chen N., Xiong C., Du W., Wang C., Lin X., Chen Z. An improved genetic algorithm coupling a back-propagation neural network model (IGA-BPNN) for water-level predictions. Water. 2019;11(9):1795. https://doi.org/10.3390/w11091795Such F.P., Madhavan V., Conti E., Lehman J., Stanley K.O., Clune J. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567. 2017. https://arxiv.org/abs/1712.06567Such F.P., Madhavan V., Conti E., Lehman J., Stanley K.O., Clune J. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567. 2017. https://arxiv.org/abs/1712.06567Shao B., Li M., Zhao Y., Bian G. Nickel price forecast based on the LSTM neural network optimized by the improved PSO algorithm. Mathematical Problems in Engineering. 2019;2019(2):1934796. https://doi.org/10.1155/2019/1934796Shao B., Li M., Zhao Y., Bian G. Nickel price forecast based on the LSTM neural network optimized by the improved PSO algorithm. Mathematical Problems in Engineering. 2019;2019(2):1934796. https://doi.org/10.1155/2019/1934796Ruder S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. 2016. https://arxiv.org/abs/1609.04747Ruder S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. 2016. https://arxiv.org/abs/1609.04747Куликов А.А. Структура локального детектора модели репринта объекта на изображении. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13 [Kulikov A.A. The structure of the local detector of the reprint model of the object in the image. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13]Куликов А.А. Структура локального детектора модели репринта объекта на изображении. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13 [Kulikov A.A. The structure of the local detector of the reprint model of the object in the image. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13]Dean J., Corrado G.S., Monga R., Chen K., Devin M., Le Q.V., Mao M.Z., Ranzato M.A., Senior A., Tucker P., Yang K., Ng A.Y. Large scale distributed deep networks. Advances in Neural Information Processing Systems. 2012;25:1223–1231.Dean J., Corrado G.S., Monga R., Chen K., Devin M., Le Q.V., Mao M.Z., Ranzato M.A., Senior A., Tucker P., Yang K., Ng A.Y. Large scale distributed deep networks. Advances in Neural Information Processing Systems. 2012;25:1223–1231.Huang G.B., Zhu Q.Y., Siew C.K. Extreme learning machine: theory and applications. Neurocomputing. 2006;70(1–3):489–501. https://doi.org/10.1016/j.neucom.2005.12.126Huang G.B., Zhu Q.Y., Siew C.K. Extreme learning machine: theory and applications. Neurocomputing. 2006;70(1–3):489–501. https://doi.org/10.1016/j.neucom.2005.12.126Rao C.R. Generalized inverse of a matrix and its applications. In: Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability. 1972. V. 1. Theory of Statistics. 1972:601–620. https://doi.org/10.1525/9780520325883-032Rao C.R. Generalized inverse of a matrix and its applications. In: Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability. 1972. V. 1. Theory of Statistics. 1972:601–620. https://doi.org/10.1525/9780520325883-032Cai W., Yang J., Yu Y., Song Y., Zhou T., Qin J. PSO-ELM: A hybrid learning model for short-term traffic flow forecasting. IEEE Access. 2020;8:6505–6514. https://doi.org/10.1109/ACCESS.2019.2963784Cai W., Yang J., Yu Y., Song Y., Zhou T., Qin J. PSO-ELM: A hybrid learning model for short-term traffic flow forecasting. IEEE Access. 2020;8:6505–6514. https://doi.org/10.1109/ACCESS.2019.2963784Liu Y., Loh H.T., Tor S.B. Comparison of extreme learning machine with support vector machine for text classification. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Innovations in Applied Artificial Intelligence. 2005;3533:390–399. http://doi.org/10.1007/11504894_55Liu Y., Loh H.T., Tor S.B. Comparison of extreme learning machine with support vector machine for text classification. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Innovations in Applied Artificial Intelligence. 2005;3533:390–399. http://doi.org/10.1007/11504894_55Li G.X. Application of extreme learning machine algorithm in the regression fitting. In: 2016 International Conference on Information System and Artificial Intelligence (ISAI). 2016:419–422. https://doi.org/10.1109/ISAI.2016.0095Li G.X. Application of extreme learning machine algorithm in the regression fitting. In: 2016 International Conference on Information System and Artificial Intelligence (ISAI). 2016:419–422. https://doi.org/10.1109/ISAI.2016.0095Song S., Wang Y., Lin X., Huang Q. Study on GA-based training algorithm for extreme learning machine. In: 2015 7th International Conference on Intelligent HumanMachine Systems and Cybernetics. IEEE. 2015;2:132–135. https://doi.org/10.1109/IHMSC.2015.156Song S., Wang Y., Lin X., Huang Q. Study on GA-based training algorithm for extreme learning machine. In: 2015 7th International Conference on Intelligent HumanMachine Systems and Cybernetics. IEEE. 2015;2:132–135. https://doi.org/10.1109/IHMSC.2015.156Никонов В.В., Горчаков А.В. Тренировка моделей машинного обучения с использованием современных средств контейнеризации и облачной инфраструктуры. Промышленные АСУ и контроллеры. 2021;6:33–43. https://doi.org/10.25791/asu.6.2021.1288 [Nikonov V.V., Gorchakov A.V. Train machine learning models using modern containerization and cloud Infrastructure. Promyshlennye ASU i kontrollery = Industrial Automated Control Systems and Controllers. 2021;6:33–43 (in Russ.). https://doi.org/10.25791/asu.6.2021.1288]Никонов В.В., Горчаков А.В. Тренировка моделей машинного обучения с использованием современных средств контейнеризации и облачной инфраструктуры. Промышленные АСУ и контроллеры. 2021;6:33–43. https://doi.org/10.25791/asu.6.2021.1288 [Nikonov V.V., Gorchakov A.V. Train machine learning models using modern containerization and cloud Infrastructure. Promyshlennye ASU i kontrollery = Industrial Automated Control Systems and Controllers. 2021;6:33–43 (in Russ.). https://doi.org/10.25791/asu.6.2021.1288]Eremeev A.V. A genetic algorithm with tournament selection as a local search method. J. Appl. Ind. Math. 2012;6(3):286–294. https://doi.org/10.1134/S1990478912030039Eremeev A.V. A genetic algorithm with tournament selection as a local search method. J. Appl. Ind. Math. 2012;6(3):286–294. https://doi.org/10.1134/S1990478912030039Kennedy J., Eberhart R. Particle swarm optimization. In: Proceedings of ICNN’95-International Conference on Neural Networks. 1995;4:1942–1948. https://doi.org/10.1109/ICNN.1995.488968Kennedy J., Eberhart R. Particle swarm optimization. In: Proceedings of ICNN’95-International Conference on Neural Networks. 1995;4:1942–1948. https://doi.org/10.1109/ICNN.1995.488968Bastos Filho C.J.A., de Lima Neto F.B., Lins A.J.C.C., Nascimento A.I.S., Lima M.P. A novel search algorithm based on fish school behavior. In: 2008 IEEE Int. Conference on Systems, Man and Cybernetics. 2008:2646–2651. https://doi.org/10.1109/ICSMC.2008.4811695Bastos Filho C.J.A., de Lima Neto F.B., Lins A.J.C.C., Nascimento A.I.S., Lima M.P. A novel search algorithm based on fish school behavior. In: 2008 IEEE Int. Conference on Systems, Man and Cybernetics. 2008:2646–2651. https://doi.org/10.1109/ICSMC.2008.4811695Demidova L.A., Gorchakov A.V. A study of chaotic maps producing symmetric distributions in the fish school search optimization algorithm with exponential step decay. Symmetry. 2020;12(5):784. https://doi.org/10.3390/sym12050784Demidova L.A., Gorchakov A.V. A study of chaotic maps producing symmetric distributions in the fish school search optimization algorithm with exponential step decay. Symmetry. 2020;12(5):784. https://doi.org/10.3390/sym12050784Cao W., Gao J., Ming Zh., Cai Sh. Some tricks in parameter selection for extreme learning machine. IOP Conf. Ser.: Mater. Sci. Eng. 2017;261(1):012002. https://doi.org/10.1088/1757-899X/261/1/012002Cao W., Gao J., Ming Zh., Cai Sh. Some tricks in parameter selection for extreme learning machine. IOP Conf. Ser.: Mater. Sci. Eng. 2017;261(1):012002. https://doi.org/10.1088/1757-899X/261/1/012002Dos Santos W., Barbosa V., de Souza R., Ribeiro R., Feitosa A., Silva V., Ribeiro D., Covello de Freitas R., Lima M., Soares N. Image reconstruction of electrical impedance tomography using fish school search and differential evolution. In: Critical Developments and Applications of Swarm Intelligence. IGI Global; 2018. P. 301–338. https://doi.org/10.4018/978-1-5225-5134-8.ch012Dos Santos W., Barbosa V., de Souza R., Ribeiro R., Feitosa A., Silva V., Ribeiro D., Covello de Freitas R., Lima M., Soares N. Image reconstruction of electrical impedance tomography using fish school search and differential evolution. In: Critical Developments and Applications of Swarm Intelligence. IGI Global; 2018. P. 301–338. https://doi.org/10.4018/978-1-5225-5134-8.ch012Demidova L.A., Gorchakov A.V. Application of chaotic Fish School Search optimization algorithm with exponential step decay in neural network loss function optimization. Procedia Computer Science. 2021;186(6):352–359. https://doi.org/10.1016/j.procs.2021.04.156Demidova L.A., Gorchakov A.V. Application of chaotic Fish School Search optimization algorithm with exponential step decay in neural network loss function optimization. Procedia Computer Science. 2021;186(6):352–359. https://doi.org/10.1016/j.procs.2021.04.156Harrison D. Jr., Rubinfeld D.L. Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manag. 1978;5(1):81–102. https://doi.org/10.1016/0095-0696(78)90006-2Harrison D. Jr., Rubinfeld D.L. Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manag. 1978;5(1):81–102. https://doi.org/10.1016/0095-0696(78)90006-2Kibler D., Aha D.W., Albert M.K. Instance-based prediction of real-valued attributes. Comput. Intell. 1989;5(2):51–57. https://doi.org/10.1111/j.1467-8640.1989.tb00315.xKibler D., Aha D.W., Albert M.K. Instance-based prediction of real-valued attributes. Comput. Intell. 1989;5(2):51–57. https://doi.org/10.1111/j.1467-8640.1989.tb00315.xLi H., Xu Z., Taylor G., Studer C., Goldstein T. Visualizing the loss landscape of neural nets. In: NIPS’18: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018:6391–6401. https://arxiv.org/abs/1712.09913v3Li H., Xu Z., Taylor G., Studer C., Goldstein T. Visualizing the loss landscape of neural nets. In: NIPS’18: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018:6391–6401. https://arxiv.org/abs/1712.09913v3Dai B., Gu C., Zhao E., Zhu K., Cao W., Qin X. Improved online sequential extreme learning machine for identifying crack behavior in concrete dam. Adv. Struct. Eng. 2019;22(2):402–412. https://doi.org/10.1177/1369433218788635Dai B., Gu C., Zhao E., Zhu K., Cao W., Qin X. Improved online sequential extreme learning machine for identifying crack behavior in concrete dam. Adv. Struct. Eng. 2019;22(2):402–412. https://doi.org/10.1177/1369433218788635

The authors declare that there are no conflicts of interest present.