Application of bioinspired global optimization algorithms to the improvement of the prediction accuracy of compact extreme learning machines

Objectives. Recent research in machine learning and artificial intelligence aimed at improving prediction accuracy and reducing computational complexity resulted in a novel neural network architecture referred to as an extreme learning machine (ELM). An ELM comprises a single-hidden-layer feedforward neural network in which the weights of connections among input-layer neurons and hidden-layer neurons are initialized randomly, while the weights of connections among hidden-layer neurons and output-layer neurons are computed using a generalized Moore– Penrose pseudoinverse operation. The replacement of the iterative learning process currently used in many neural network architectures with the random initialization of input weights and the explicit computation of output weights significantly increases the performance of this novel machine learning algorithm while preserving good generalization performance. However, since the random initialization of input weights does not necessarily guarantee optimal prediction accuracy, the purpose of the present work was to develop and study approaches to intelligent adjustment of input weights in ELMs using bioinspired algorithms in order to improve the prediction accuracy of this data analysis tool in regression problems.
Methods. Methods of optimization theory, theory of evolutionary computation and swarm intelligence, probability theory, mathematical statistics and systems analysis were used.
Results. Approaches to the intelligent adjustment of input weights in ELMs were developed and studied. These approaches are based on the genetic algorithm, the particle swarm algorithm, the fish school search algorithm, as well as the chaotic fish school search algorithm with exponential step decay proposed by the authors. By adjusting input weights with bioinspired optimization algorithms, it was shown that the prediction accuracy of ELMs in regression problems can be improved to reduce the number of hidden-layer neurons to reach a high prediction accuracy on learning and test datasets. In the considered problems, the best ELM configurations can be obtained using the chaotic fish school search algorithm with exponential step decay.
Conclusions. The obtained results showed that the prediction accuracy of ELMs can be improved by using bioinspired algorithms for the intelligent adjustment of input weights. Additional calculations are required to adjust the weights; therefore, the use of ELMs in combination with bioinspired algorithms may be advisable where it is necessary to obtain the most accurate and most compact ELM configuration.

1. Wu Y., Ianakiev K., Govindaraju V. Improved k-nearest neighbor classification. Pattern Recognition. 2002;35(10):2311–2318. https://doi.org/10.1016/S0031-3203(01)00132-7

2. Noble W.S. What is a support vector machine? Nat. Biotechnol. 2006;24(12):1565–1567. https://doi.org/10.1038/nbt1206-1565

3. Demidova L.A. Two-stage hybrid data classifiers based on SVM and kNN algorithms. Symmetry. 2021;13(4):615. https://doi.org/10.3390/sym13040615

4. Lin W., Wu Z., Lin L., Wen A., Li J. An ensemble random forest algorithm for insurance Big Data analysis. IEEE Access. 2017;5:16568–16575. https://doi.org/10.1109/ACCESS.2017.2738069

5. Deng L., Hinton G., Kingsbury B. New types of deep neural network learning for speech recognition and related applications: An overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013:8599–8603. https://doi.org/10.1109/ICASSP.2013.6639344

6. Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review. 1958;65(6):386–408. https://doi.org/10.1037/h0042519

7. Affonso C., Debiaso Rossi A.L., Antunes Vieira F.H., Ponce de Leon Ferreira de Carvalho A.C. Deep learning for biological image classification. Expert Systems with Applications. 2017;85:114–122. https://doi.org/10.1016/j.eswa.2017.05.039

8. Chen N., Xiong C., Du W., Wang C., Lin X., Chen Z. An improved genetic algorithm coupling a back-propagation neural network model (IGA-BPNN) for water-level predictions. Water. 2019;11(9):1795. https://doi.org/10.3390/w11091795

9. Such F.P., Madhavan V., Conti E., Lehman J., Stanley K.O., Clune J. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567. 2017. https://arxiv.org/abs/1712.06567

10. Shao B., Li M., Zhao Y., Bian G. Nickel price forecast based on the LSTM neural network optimized by the improved PSO algorithm. Mathematical Problems in Engineering. 2019;2019(2):1934796. https://doi.org/10.1155/2019/1934796

11. Ruder S. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. 2016. https://arxiv.org/abs/1609.04747

12. Куликов А.А. Структура локального детектора модели репринта объекта на изображении. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13 [Kulikov A.A. The structure of the local detector of the reprint model of the object in the image. Russ. Technol. J. 2021;9(5):7–13. https://doi.org/10.32362/2500-316X-2021-9-5-7-13]

13. Dean J., Corrado G.S., Monga R., Chen K., Devin M., Le Q.V., Mao M.Z., Ranzato M.A., Senior A., Tucker P., Yang K., Ng A.Y. Large scale distributed deep networks. Advances in Neural Information Processing Systems. 2012;25:1223–1231.

14. Huang G.B., Zhu Q.Y., Siew C.K. Extreme learning machine: theory and applications. Neurocomputing. 2006;70(1–3):489–501. https://doi.org/10.1016/j.neucom.2005.12.126

15. Rao C.R. Generalized inverse of a matrix and its applications. In: Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability. 1972. V. 1. Theory of Statistics. 1972:601–620. https://doi.org/10.1525/9780520325883-032

16. Cai W., Yang J., Yu Y., Song Y., Zhou T., Qin J. PSO-ELM: A hybrid learning model for short-term traffic flow forecasting. IEEE Access. 2020;8:6505–6514. https://doi.org/10.1109/ACCESS.2019.2963784

17. Liu Y., Loh H.T., Tor S.B. Comparison of extreme learning machine with support vector machine for text classification. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Innovations in Applied Artificial Intelligence. 2005;3533:390–399. http://doi.org/10.1007/11504894_55

18. Li G.X. Application of extreme learning machine algorithm in the regression fitting. In: 2016 International Conference on Information System and Artificial Intelligence (ISAI). 2016:419–422. https://doi.org/10.1109/ISAI.2016.0095

19. Song S., Wang Y., Lin X., Huang Q. Study on GA-based training algorithm for extreme learning machine. In: 2015 7th International Conference on Intelligent HumanMachine Systems and Cybernetics. IEEE. 2015;2:132–135. https://doi.org/10.1109/IHMSC.2015.156

20. Никонов В.В., Горчаков А.В. Тренировка моделей машинного обучения с использованием современных средств контейнеризации и облачной инфраструктуры. Промышленные АСУ и контроллеры. 2021;6:33–43. https://doi.org/10.25791/asu.6.2021.1288 [Nikonov V.V., Gorchakov A.V. Train machine learning models using modern containerization and cloud Infrastructure. Promyshlennye ASU i kontrollery = Industrial Automated Control Systems and Controllers. 2021;6:33–43 (in Russ.). https://doi.org/10.25791/asu.6.2021.1288]

21. Eremeev A.V. A genetic algorithm with tournament selection as a local search method. J. Appl. Ind. Math. 2012;6(3):286–294. https://doi.org/10.1134/S1990478912030039

22. Kennedy J., Eberhart R. Particle swarm optimization. In: Proceedings of ICNN’95-International Conference on Neural Networks. 1995;4:1942–1948. https://doi.org/10.1109/ICNN.1995.488968

23. Bastos Filho C.J.A., de Lima Neto F.B., Lins A.J.C.C., Nascimento A.I.S., Lima M.P. A novel search algorithm based on fish school behavior. In: 2008 IEEE Int. Conference on Systems, Man and Cybernetics. 2008:2646–2651. https://doi.org/10.1109/ICSMC.2008.4811695

24. Demidova L.A., Gorchakov A.V. A study of chaotic maps producing symmetric distributions in the fish school search optimization algorithm with exponential step decay. Symmetry. 2020;12(5):784. https://doi.org/10.3390/sym12050784

25. Cao W., Gao J., Ming Zh., Cai Sh. Some tricks in parameter selection for extreme learning machine. IOP Conf. Ser.: Mater. Sci. Eng. 2017;261(1):012002. https://doi.org/10.1088/1757-899X/261/1/012002

26. Dos Santos W., Barbosa V., de Souza R., Ribeiro R., Feitosa A., Silva V., Ribeiro D., Covello de Freitas R., Lima M., Soares N. Image reconstruction of electrical impedance tomography using fish school search and differential evolution. In: Critical Developments and Applications of Swarm Intelligence. IGI Global; 2018. P. 301–338. https://doi.org/10.4018/978-1-5225-5134-8.ch012

27. Demidova L.A., Gorchakov A.V. Application of chaotic Fish School Search optimization algorithm with exponential step decay in neural network loss function optimization. Procedia Computer Science. 2021;186(6):352–359. https://doi.org/10.1016/j.procs.2021.04.156

28. Harrison D. Jr., Rubinfeld D.L. Hedonic housing prices and the demand for clean air. J. Environ. Econ. Manag. 1978;5(1):81–102. https://doi.org/10.1016/0095-0696(78)90006-2

29. Kibler D., Aha D.W., Albert M.K. Instance-based prediction of real-valued attributes. Comput. Intell. 1989;5(2):51–57. https://doi.org/10.1111/j.1467-8640.1989.tb00315.x

30. Li H., Xu Z., Taylor G., Studer C., Goldstein T. Visualizing the loss landscape of neural nets. In: NIPS’18: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018:6391–6401. https://arxiv.org/abs/1712.09913v3

31. Dai B., Gu C., Zhao E., Zhu K., Cao W., Qin X. Improved online sequential extreme learning machine for identifying crack behavior in concrete dam. Adv. Struct. Eng. 2019;22(2):402–412. https://doi.org/10.1177/1369433218788635