Feature space transformation in the support vector method

Objectives. This study focuses on the development and investigation of a generalized nonlinear Support Vector Machine (SVM) method incorporating an adaptive transformation of the feature space. Its aim is to improve computational efficiency while maintaining high classification accuracy. The binary classification problem is used as a case study. The main objective of the research is to quantitatively evaluate the performance of the proposed approach when compared to classical SVM models using fixed kernel functions, and to analyze how the transformation parameters affect classification quality.

Methods. The proposed approach involves a preliminary transformation of the input data using a learnable nonlinear mapping with a fixed structure. This mapping is implemented as a composition of elementary functions and is parameterized by a limited number of trainable weights which allows control over model complexity. A linear SVM with L2 regularization is applied after the transformation. The model is trained using conventional, unconstrained numerical optimization methods. The classification quality is evaluated using the Accuracy metric averaged over 10-fold cross-validation. The work also studies the behavior of the model with varying feature space dimensionality. In addition, computational complexity is analyzed in terms of the number of operations and inference time required on test datasets.

Results. Numerical experiments demonstrate that the proposed model significantly reduces classification time when compared to a polynomial-kernel SVM, while maintaining a comparable level of accuracy. The runtime analysis confirms that the proposed approach scales much better than traditional kernel methods. At the same time, the structure of the model remains interpretable and can be further adapted to the specifics of the application domain.

Conclusions. The method developed provides an efficient alternative to traditional kernel-based algorithms. Through the use of a parameterized transformation of the feature space, the method enables adaptability, interpretability, and scalability, making it promising for practical applications in machine learning tasks.

1. Vapnik V. Statistical Learning Theory. New York: Wiley; 1998, 736 p. ISBN 978-0-471-03003-4

2. Maggioni F., Spinelli A. A novel robust optimization model for nonlinear Support Vector Machine. Computers & Operations Research. 2024;157:105059.

3. Rubin N., Fischer K., Lindner J., Dahmen D., Seroussi I., Ringel Z., Krüger M., Helias M. From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning. arXiv preprint arXiv:2402.03210. 2024. https://doi.org/10.48550/arXiv.2502.03210

4. LeJeune D., Alemohammad S. An Adaptive Tangent Feature Perspective of Neural Networks. In: Proceedings of the 37th International Conference on Machine Learning (ICML). 2024. URL: https://proceedings.mlr.press/v234/lejeune24a/lejeune24a.pdf. Дата обращения 07.07.2025. / Accessed July 07, 2025.

5. Li Y., Lin Q. Diagonal Over-parameterization in Reproducing Kernel Hilbert Spaces as an Adaptive Feature Model: Generalization and Adaptivity. arXiv preprint arXiv:2501.08679. 2025. https://arxiv.org/abs/2501.08679

6. Kurbucz M.T., Benkő Z., Varga L., et al. Adaptive Law-Based Transformation (ALT): A Fast and Transparent Feature Transformation Method for Time Series Classification. arXiv preprint arXiv:2501.09217. 2025. https://arxiv.org/abs/2501.09217

7. Tikhonov A.N., Arsenin V.Y. Solutions of Ill-Posed Problems. Washington, D.C.: W.H. Freeman and Co.; 1977, 258 p.

8. Bishop C.M. Pattern Recognition and Machine Learning. New York: Springer; 2006, 738 p.

9. Fletcher R. Practical Methods of Optimization. New York: John Wiley & Sons; 1987, 464 p.

10. Baydin A.G., Pearlmutter B.A., Radul A.A., Siskind J.M. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research (JMLR). 2018;18(153):1–43.

11. Fisher J. Automatic differentiation with dual numbers. arXiv preprint arXiv:2201.00024.

12. Bischl B., Casalicchio G., Feurer M., Hutter F., Lang M., Mantovani R.G., van Rijn J.N., Vanschoren J. OpenML Benchmarking Suites. arXiv preprint arXiv:1708.03731v2 [stat.ML], 2019. https://arxiv.org/abs/1708.03731v2

13. Kohavi R. A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. In: Proceedings of Fourteenth International Joint Conference on Artificial Intelligence (IJCAI). 1995;14(2):1137–1143.

14. Labatut V., Cherifi H. Accuracy Measures for the Comparison of Classifiers. arXiv preprint arXiv:1207.3790. https://doi.org/10.48550/arXiv.1207.3790

15. Schölkopf B., Smola A.J. Learning with Kernels. Cambridge: MIT Press; 2002, 300 p.

16. Hofmann T., Schölkopf B., Smola A.J. Kernel Methods in Machine Learning. Ann. Statist. 2008;36(3):1171–1220. https://doi.org/10.1214/009053607000000677

17. Boser B.E., Guyon I.M., Vapnik V. A Training Algorithm for Optimal Margin Classifiers. In: Proceedings of the Fifth Annual Workshop on Computational Learning Theory (COLT). 1992. P. 144–152. https://doi.org/10.1145/130385.130401

18. Cortes C., Vapnik V. Support-Vector Networks. Mach. Learn. 1995;20(3):273–297. https://doi.org/10.1007/BF00994018