mireabulletinRussian Technological JournalRussian Technological Journal2782-32102500-316XRTU MIREA10.32362/2500-316X-2025-13-2-7-17OQUHWLmireabulletin-1122Research ArticleИНФОРМАЦИОННЫЕ СИСТЕМЫ. ИНФОРМАТИКА. ПРОБЛЕМЫ ИНФОРМАЦИОННОЙ БЕЗОПАСНОСТИINFORMATION SYSTEMS. COMPUTER SCIENCES. ISSUES OF INFORMATION SECURITYСбор и анализ датасета для задачи автоматической генерации сообщений коммитовDataset collection for automatic generation of commit messageshttps://orcid.org/0009-0009-1804-9412КосьяненкоИ. А.KosyanenkoIvan A.

Косьяненко Иван Александрович, аспирант, кафедра инструментального и прикладного программного обеспечения, Институт информационных технологий

119454, Москва, пр-т Вернадского, д. 78

Ivan A. Kosyanenko, Postgraduate Student, Department of Instrumental and Applied Software, Institute of Information Technologies

78, Vernadskogo pr., Moscow, 119454

kosyanenko.edu@gmail.comhttps://orcid.org/0000-0002-4922-7260БолбаковР. Г.BolbakovRoman G.

Болбаков Роман Геннадьевич, к.т.н., доцент, заведующий кафедрой инструментального и прикладного программного обеспечения, Институт информационных технологий

119454, Москва, пр-т Вернадского, д. 78

Scopus Author ID 57202836952

Roman G. Bolbakov, Cand. Sci. (Eng.), Associate Professor, Head of the Department of Instrumental and Applied Software, Institute of Information Technologies

78, Vernadskogo pr., Moscow, 119454

Scopus Author ID 57202836952

bolbakov@mirea.ruМИРЭА – Российский технологический университетРоссияMIREA – Russian Technological UniversityRussian Federation202506042025132717Copyright © Косьяненко И.А., Болбаков Р.Г., 20252025Косьяненко И.А., Болбаков Р.Г.Kosyanenko I.A., Bolbakov R.G.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://www.rtj-mirea.ru/jour/article/view/1122Цели

Цели. Для управления процессом разработки современного программного обеспечения нередко применяются системы контроля версий, которые позволяют фиксировать изменения в программном коде и передавать контекст этих изменений при помощи сообщений коммитов. Релевантное и качественное описание внесенных изменений при помощи таких сообщений требует от разработчика высокой компетенции и времени, но современные методы машинного обучения позволяют решать эту задачу автоматически. Целью работы является статистический и сравнительный анализ собранной выборки данных с наборами изменений в программном коде и их описаниями на естественном языке.

Методы

Методы. В исследовании использован комплексный подход, включающий сбор данных с популярных репозиториев на GitHub, предварительную обработку и фильтрацию данных, а также статистический анализ и метод обработки естественного языка (векторизация текста). Для оценки семантической близости между первым предложением и полным текстом сообщений коммитов было использовано косинусное сходство.

Результаты

Результаты. Проведено исследование структуры и качества сообщений коммитов, включающее сбор данных из репозиториев GitHub и их предварительную очистку. Осуществлена векторизация текста сообщений коммитов и оценка семантической близости между первыми предложениями и полными текстами сообщений с использованием косинусного сходства. Выполнен сравнительный анализ качества сообщений в собранном датасете и в нескольких аналогичных наборах данных с помощью классификации при помощи модели CodeBERT.

Выводы

Выводы. Проведенный анализ выявил низкий уровень косинусного сходства между первыми предложениями и полными текстами сообщений коммитов (0.0969), что свидетельствует о слабой семантической связи между ними и опровергает гипотезу о том, что первые предложения выступают в качестве обобщения содержания сообщений. Процентная доля пустых сообщений в собранном наборе данных составила лишь 0.0007%, что существенно ниже ожидаемого значения и указывает на высокое качество собранных данных. Классификационный анализ показал, что доля сообщений, отнесенных к категории «плохих», в собранном датасете составляет 16.82%, что значительно ниже аналогичных показателей в других сопоставимых наборах данных, где этот процент варьируется от 34.75% до 54.26%. Данный факт подчеркивает высокое качество собранного набора данных и его адекватность для дальнейшего применения в системах автоматической генерации сообщений коммитов

Objectives

Objectives. In contemporary software development practice, version control systems are often used to manage the development process. Such systems allow developers to track changes in the codebase and convey the context of these changes through commit messages. The use of such messages to provide relevant and high-quality descriptions of the changes generally requires a high level of competence and time commitment from the developer. However, modern machine learning methods can enable the automation of this task. Therefore, the work sets out to provide a statistical and comparative analysis of the collected data sample with sets of changes in the program code and their descriptions in natural language.

Methods

Methods. In this study, a comprehensive approach was used, including data collection from popular GitHub repositories, preliminary data processing and filtering, as well as statistical analysis and natural language processing method (text vectorization). Cosine similarity was used as a means of assessing the semantic proximity between the first sentence and the full text of commit messages.

Results

Results. A comprehensive study of the structure and quality of commit messages encompassed data collection from GitHub repositories and preliminary data cleansing. The research involved text vectorization of commit messages and evaluation of semantic similarity between the first sentences and full texts of messages using cosine similarity. The comparative analysis of message quality in the collected dataset and several analogous datasets used classification based on the CodeBERT model.

Conclusions

Conclusions. The analysis revealed a low level of cosine similarity (0.0969) between the first sentences and full texts of commit messages, indicating a weak semantic relationship between them and refuting the hypothesis that first sentences serve as summaries of message content. The low proportion of empty messages in the collected dataset at 0.0007% was significantly lower than expected, indicating high-quality data collection. The results of classification analysis showed that the proportion of messages categorized as “poor” in the collected dataset was 16.82%, substantially lower than comparable figures in other datasets, where this percentage ranged from 34.75% to 54.26%. This fact underscores the high quality of the collected dataset and its suitability for further application in automatic commit message generation systems.

генерация сообщений коммитовсистемы контроля версийописание изменений в программном кодекосинусное сходствофильтрация данныхвекторизация текстадатасетмашинное обучениеcommit message generationversion control systemsdescription of changes in software codecosine similaritydata filteringtext vectorizationdatasetmachine learningReferencesTian Y., Zhang Y., Stol K., Jiang L., Liu H. What makes a good commit message? Proceedings of the 44th International Conference on Software Engineering. 2022;44:2389–2401. https://doi.org/10.1145/3510003.3510205Tian Y., Zhang Y., Stol K., Jiang L., Liu H. What makes a good commit message? Proceedings of the 44th International Conference on Software Engineering. 2022;44:2389–2401. https://doi.org/10.1145/3510003.3510205Kosyanenko I.A., Bolbakov R.G. About automatic generation of commit messages in version control systems. International Journal of Open Information Technologies (INJOIT). 2022;10(4):55–60 (in Russ.).Kosyanenko I.A., Bolbakov R.G. About automatic generation of commit messages in version control systems. International Journal of Open Information Technologies (INJOIT). 2022;10(4):55–60 (in Russ.).Liu Z., Xia X., Hassan A., Lo D., Xing Z. Neural-machine-translation-based commit message generation: how far are we? In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018;33:373–384. https://doi.org/10.1145/3238147.3238190Liu Z., Xia X., Hassan A., Lo D., Xing Z. Neural-machine-translation-based commit message generation: how far are we? In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018;33:373–384. https://doi.org/10.1145/3238147.3238190Sun Z., Li L., Liu Y., Du X., Li L. On the importance of building high-quality training datasets for neural code search. In: Proceedings of the 44th International Conference on Software Engineering. 2022;44:1609–1620. https://doi.org/10.1145/3510003.3510160Sun Z., Li L., Liu Y., Du X., Li L. On the importance of building high-quality training datasets for neural code search. In: Proceedings of the 44th International Conference on Software Engineering. 2022;44:1609–1620. https://doi.org/10.1145/3510003.3510160Hawkins D.M. The problem of overfitting. J. Chem. Inf. Comput. Sci. 2004;44(1):1–12. https://doi.org/10.1021/ci0342472Hawkins D.M. The problem of overfitting. J. Chem. Inf. Comput. Sci. 2004;44(1):1–12. https://doi.org/10.1021/ci0342472Banko M., Brill E. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics. 2001;26–33. https://doi.org/10.3115/1073012.1073017Banko M., Brill E. Scaling to Very Very Large Corpora for Natural Language Disambiguation. In: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics. 2001;26–33. https://doi.org/10.3115/1073012.1073017Halevy A., Norvig P., Pereira F. The unreasonable effectiveness of data. IEEE Intell. Syst. 2009;24(2):8–12. https://doi.org/10.1109/MIS.2009.36Halevy A., Norvig P., Pereira F. The unreasonable effectiveness of data. IEEE Intell. Syst. 2009;24(2):8–12. https://doi.org/10.1109/MIS.2009.36Tao W., Wang Y., Shi E., Du L., Han S., Zhang H., Zhang D., Zhang W. On the evaluation of commit message generation models: An experimental study. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE. 2021;126–136. https://doi.org/10.48550/arXiv.2107.05373Tao W., Wang Y., Shi E., Du L., Han S., Zhang H., Zhang D., Zhang W. On the evaluation of commit message generation models: An experimental study. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE. 2021;126–136. https://doi.org/10.48550/arXiv.2107.05373Jiang S., McMillan C. Towards automatic generation of short summaries of commits. In: 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC). IEEE. 2017;320–323. https://doi.org/10.48550/arXiv.1703.09603Jiang S., McMillan C. Towards automatic generation of short summaries of commits. In: 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC). IEEE. 2017;320–323. https://doi.org/10.48550/arXiv.1703.09603Myagkova E.Yu. To the problem of “formal” and “inner” grammar. Vestnik Tverskogo gosudarstvennogo universiteta. Seriya: Filologiya = Herald of Tver State University. Series: Philology. 2012;24(4):96–102 (in Russ.).Myagkova E.Yu. To the problem of “formal” and “inner” grammar. Vestnik Tverskogo gosudarstvennogo universiteta. Seriya: Filologiya = Herald of Tver State University. Series: Philology. 2012;24(4):96–102 (in Russ.).Xu S., Yao Y., Xu F., Gu T., Tong H., Lu J. Commit message generation for source code changes. In: Proceedings of the TwentyEighth International Joint Conference on Artificial Intelligence (IJCAI-19). 2019;3975–3981. https://doi.org/10.24963/ijcai.2019/552Xu S., Yao Y., Xu F., Gu T., Tong H., Lu J. Commit message generation for source code changes. In: Proceedings of the TwentyEighth International Joint Conference on Artificial Intelligence (IJCAI-19). 2019;3975–3981. https://doi.org/10.24963/ijcai.2019/552Liu Q., Liu Z., Zhi H., Fan H., Du B., Qian Y. Generating commit messages from diffs using pointer-generator network. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE. 2019;299–309. http://doi.org/10.1109/MSR.2019.00056Liu Q., Liu Z., Zhi H., Fan H., Du B., Qian Y. Generating commit messages from diffs using pointer-generator network. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE. 2019;299–309. http://doi.org/10.1109/MSR.2019.00056Loyola P., Marrese-Taylor E., Matsuo Y. A neural architecture for generating natural language descriptions from source code changes. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017;287–292. https://doi.org/10.18653/v1/P17-2045Loyola P., Marrese-Taylor E., Matsuo Y. A neural architecture for generating natural language descriptions from source code changes. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017;287–292. https://doi.org/10.18653/v1/P17-2045Liu S., Gao C., Chen S., Yiu L., Liy Y. ATOM: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Trans. Software Eng. 2020;48(5):1800–1817. https://doi.org/10.48550/arXiv.1912.02972Liu S., Gao C., Chen S., Yiu L., Liy Y. ATOM: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Trans. Software Eng. 2020;48(5):1800–1817. https://doi.org/10.48550/arXiv.1912.02972Eliseeva A., Sokolov Y., Bogomolov E., Golubev Y., Dig D., Bryskin T. From Commit Message Generation to History-Aware Commit Message Completion. In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE. 2023;723–735. https://doi.org/10.48550/arXiv.2308.07655Eliseeva A., Sokolov Y., Bogomolov E., Golubev Y., Dig D., Bryskin T. From Commit Message Generation to History-Aware Commit Message Completion. In: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE. 2023;723–735. https://doi.org/10.48550/arXiv.2308.07655Dey T., Mousavi S., Ponce E. Detecting and characterizing bots that commit code. In: Proceedings of the 17th international conference on mining software repositories. 2020;209–219. https://doi.org/10.1145/3379597.3387478Dey T., Mousavi S., Ponce E. Detecting and characterizing bots that commit code. In: Proceedings of the 17th international conference on mining software repositories. 2020;209–219. https://doi.org/10.1145/3379597.3387478Kuchnik M., Smith V., Amvrosiadis G. Validating large language models with ReLM. Proceedings of Machine Learning and Systems. 2023;5:457–476. https://doi.org/10.48550/arXiv.2211.15458Kuchnik M., Smith V., Amvrosiadis G. Validating large language models with ReLM. Proceedings of Machine Learning and Systems. 2023;5:457–476. https://doi.org/10.48550/arXiv.2211.15458Haque S., Zachary E. Semantic similarity metrics for evaluating source code summarization. In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. 2022;36–47. https://doi.org/10.1145/3524610.3527909Haque S., Zachary E. Semantic similarity metrics for evaluating source code summarization. In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. 2022;36–47. https://doi.org/10.1145/3524610.3527909Rahutomo F., Kitasuka T., Aritsugi M. Semantic cosine similarity. In: The 7th International Student Conference on Advanced Science and Technology (ICAST). 2012;4(1):1–2.Rahutomo F., Kitasuka T., Aritsugi M. Semantic cosine similarity. In: The 7th International Student Conference on Advanced Science and Technology (ICAST). 2012;4(1):1–2.Roshan R., Bhacho I.A., Zai S. Comparative Analysis of TF–IDF and Hashing Vectorizer for Fake News Detection in Sindhi: A Machine Learning and Deep Learning Approach. Eng. Proc. 2023;46(1):5. https://doi.org/10.3390/engproc2023046005Roshan R., Bhacho I.A., Zai S. Comparative Analysis of TF–IDF and Hashing Vectorizer for Fake News Detection in Sindhi: A Machine Learning and Deep Learning Approach. Eng. Proc. 2023;46(1):5. https://doi.org/10.3390/engproc2023046005Aggarwal C.C., Yu P.S. Outlier Detection in High Dimensional Data. In: Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data. 2001;30(2):37–46. http://dx.doi.org/10.1145/376284.375668Aggarwal C.C., Yu P.S. Outlier Detection in High Dimensional Data. In: Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data. 2001;30(2):37–46. http://dx.doi.org/10.1145/376284.375668Feng Z., Guo D., Tang F., et al. CodeBERT: A pre-trained model for programming and natural languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020. P. 1536–1547. Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.139Feng Z., Guo D., Tang F., et al. CodeBERT: A pre-trained model for programming and natural languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020. P. 1536–1547. Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.139Qasim R., Bangyal W.H. A fine-tuned BERT-based transfer learning approach for text classification. J. Healthc. Eng. 2022;2022:3498123. https://doi.org/10.1155/2022/3498123Qasim R., Bangyal W.H. A fine-tuned BERT-based transfer learning approach for text classification. J. Healthc. Eng. 2022;2022:3498123. https://doi.org/10.1155/2022/3498123

The authors declare that there are no conflicts of interest present.