Метод конвертации акцента с клонированием голоса в реальном времени на основе неавторегрессионной нейросетевой модели

Цели. В настоящее время при разработке моделей для преобразования речи с акцентом в речь без акцента используются архитектуры глубоких нейросетей, а также ансамбли предобученных нейросетей для распознавания и генерации речи. При этом доступ к реализациям таких моделей является ограниченным, что затрудняет их применение, изучение и дальнейшее развитие. Также использование данных моделей ограничено особенностями архитектуры, которая не позволяет гибко менять тембр генерируемой речи и требует накопления контекста, что ведет к увеличению задержки при генерации и делает данные системы непригодными для использования в сценариях коммуникации двух и более людей в реальном времени. В связи с этим актуальной задачей и целью настоящей работы является разработка метода, позволяющего на основе входной речи с акцентом генерировать речь без акцента с минимальными задержками с возможностью сохранения, клонирования и модификации тембра говорящего, что позволит преодолеть ограничения текущих моделей.
Методы. Применены методы модификации, обучения и объединения глубоких нейросетей в единую сквозную архитектуру для прямого преобразования речи в речь. Для обучения использованы оригинальные и модифицированные наборы данных из открытых источников.
Результаты. Разработан метод конвертации акцента с клонированием голоса в реальном времени на основе неавторегрессионной нейросетевой модели, которая состоит из модулей определения акцента и пола, идентификации говорящего, преобразования речи в фонетическое представление, генерации спектрограммы и декодирования полученной спектрограммы в аудиосигнал. Метод демонстрирует высокое качество конвертации акцента с сохранением оригинального тембра, а также низкие задержки при генерации, приемлемые для использования в сценариях реального времени.
Выводы. Апробация разработанного метода подтвердила эффективность предложенной неавторегрессионной нейросетевой архитектуры. Разработанная прикладная нейросетевая модель продемонстрировала возможность работы в информационных системах на английском языке в режиме реального времени.

1. McMillin D.C. Outsourcing identities: Call centres and cultural transformation in India. Economic and Political Weekly. 2006;41(3):235–241.

2. Felps D., Bortfeld H., Gutierrez-Osuna R. Foreign accent conversion in computer assisted pronunciation training. Speech Communication. 2009;51(10):920–932. https://doi.org/10.1016/j.specom.2008.11.004

3. Probst K., Ke Y., Eskenazi M. Enhancing foreign language tutors–In search of the golden speaker. Speech Communication. 2002;37(3–4):161–173. https://doi.org/10.1016/S0167-6393(01)00009-7

4. Türk O., Arslan L. M. Subband based voice conversion. In: 7th International Conference on Spoken Language Processing, ICSLP2002 – INTERSPEECH 2002. Interspeech. 2002. P. 289–292.

5. Biadsy F., Weis, R.J., Moreno P.J., Kanevsky D., Jia Y. Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation. Interspeech. 2019. P. 4115–4119. http://doi.org/10.21437/Interspeech.2019-1789

6. Birner B. Why Do Some People Have an Accent? Linguistic Society of America. Washington, DC. 1999. 6 p.

7. Baese-Berk M.M., Morrill T.H. Speaking rate consistency in native and non-native speakers of English. J. Acoust. Soc. Am. 2015;138(3):EL223–EL228. https://doi.org/10.1121/1.4929622

8. Piske T., MacKay I.R.A., Flege J.E. Factors affecting degree of foreign accent in an L2: A review. J. Phonetics. 2001;29(2): 191–215. https://doi.org/10.1006/jpho.2001.0134

9. Munro M.J., Derwing T.M. Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning. 1995;45(1):73–97. https://doi.org/10.1111/j.1467-1770.1995.tb00963.x

10. Lev-Ari S., Keysar B. Why don’t we believe non-native speakers? The influence of accent on credibility. J. Exp. Soc. Psychol. 2010;46(6):1093–1096. https://doi.org/10.1016/j.jesp.2010.05.025

11. Rubin D.L., Smith K.A. Effects of accent, ethnicity, and lecture topic on undergraduates’ perceptions of nonnative Englishspeaking teaching assistants. Int. J. Intercult. Relat. 1990;14(3):337–353. https://doi.org/10.1016/0147-1767(90)90019-S

12. Nelson Jr. L.R., Signorella M.L., Botti K.G. Accent, gender, and perceived competence. Hispanic J. Behavior. Sci. 2016;38(2):166–185. https://doi.org/10.1177/0739986316632319

13. Zhao G., Gutierrez-Osuna R. Using phonetic posteriorgram based frame pairing for segmental accent conversion. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2019;27(10):1649–1660. https://doi.org/10.1109/TASLP.2019.2926754

14. Zhao G., Sonsaat S., Levis J., Chukharev-Hudilainen E., Gutierrez-Osuna R. Accent conversion using phonetic posteriorgrams. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2018. P. 5314–5318. https://doi.org/10.1109/ICASSP.2018.8462258

15. Aryal S., Gutierrez-Osuna R. Can voice conversion be used to reduce non-native accents? In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2014. P. 7879–7883. https://doi.org/10.1109/ICASSP.2014.6855134

16. Ding S., Zhao G., Gutierrez-Osuna R. Accentron: Foreign accent conversion to arbitrary non-native speakers using zero-shot learning. Computer Speech & Language. 2022;72:101302. https://doi.org/10.1016/j.csl.2021.101302

17. Quamer W., Das A., Levis J., Chukharev-Hudilainen E., Gutierrez-Osuna R. Zero-shot foreign accent conversion without a native reference. Proc. Interspeech. 2022. http://doi.org/10.21437/Interspeech.2022-10664

18. Jin M., Serai P., Wu J., Tjandra A., Manohar V., He Q. Voice-preserving zero-shot multiple accent conversion. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2023. P. 1–5. https://doi.org/10.1109/ICASSP49357.2023.10094737

19. Zhou Y., Wu Z., Zhang M., Tian X., Li H. TTS-guided training for accent conversion without parallel data. IEEE Signal Proc. Lett. 2023;30:533–537. https://doi.org/10.1109/lsp.2023.3270079

20. Zhao G., Ding S., Gutierrez-Osuna R. Converting foreign accent speech without a reference. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2021;29:2367–2381. https://doi.org/10.1109/TASLP.2021.3060813

21. Liu S., Wang D., Cao Y., Sun L., Wu X., Kang S., Wu Z., Liu X., Su D., Yu D., Meng H. End-to-end accent conversion without using native utterances. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2020. P. 6289–6293.

22. Zhou X., Zhang M., Zhou Y., Wu Z., Li H. Accented text-to-speech synthesis with limited data. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2024;32:1699–1711. https://doi.org/10.1109/TASLP.2024.3363414

23. Pinget A.F., Bosker H.R., Quené H., De Jong, N.H. Native speakers’ perceptions of fluency and accent in L2 speech. Language Testing. 2014;31(3):349–365. https://doi.org/10.1177/0265532214526177

24. Бархударова Е.Л. Методологические проблемы анализа иностранного акцента в русской речи. Вестник Московского университета. Серия 9. Филология. 2012;6:57–70. [Barkhudarova E.L. Methodological Problems in Analyzing Foreign Accents in Russian Speech. Vestnik Moskovskogo universiteta. Seriya 9. Filologiya = Lomonosov Philology J. 2012;6:57–70 (in Russ.).]

25. Arik S., Chen J., Peng K., Ping W., Zhou Y. Neural voice cloning with a few samples. Advances in Neural Information Processing Systems (NeurIPS 2018). 2018;31.

26. Cohen D. Issues in transnet packetized voice communication. In: Proceedings of the fifth Symposium on Data Communications (SIGCOMM’77). 1977. P. 6.10–6.13. https://doi.org/10.1145/800103.803349

27. Liang Y.J., Farber N., Girod B. Adaptive playout scheduling and loss concealment for voice communication over IP networks. IEEE Trans. Multimedia. 2003;5(4):532–543. https://doi.org/10.1109/TMM.2003.819095

28. Matzinger T., Pleyer M., Żywiczyński P. Pause Length and Differences in Cognitive State Attribution in Native and Non-Native Speakers. Languages. 2023;8(1):26. http://doi.org/10.3390/languages8010026

29. Medsker L.R., Jain L. (Eds.). Recurrent Neural Networks. Design and Applications. Boca Raton: CRC Press; 2001. 416 p.

30. Li J., Lavrukhin V., Ginsburg B., Leary R., Kuchaiev O., Cohen J.M., Nguyen H., Gadde R.T. Jasper: An End-to-End Convolutional Neural Acoustic Model. Interspeech 2019. 2019. https://doi.org/10.21437/interspeech.2019-1819

31. Dawalatabad N., Ravanelli M., Grondin F., Thienpondt J., Desplanques B., Na H. ECAPA-TDNN Embeddings for Speaker Diarization. arXiv preprint arXiv:2104.01466. 2021. https://doi.org/10.48550/arXiv.2104.01466

32. Ravanelli M., Bengio Y. Speaker recognition from raw waveform with SincNet. In: 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE; 2018. P. 1021–1028. https://doi.org/10.1109/SLT.2018.8639585

33. Snyder D., Garcia-Romero D., Sell G., Povey D., Khudanpur S. X-vectors: Robust DNN embeddings for speaker recognition. In: 2018 IEEE international Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2018. P. 5329–5333. http://doi.org/10.1109/ICASSP.2018.8461375

34. Deng J., Guo J., Xue N., Zafeiriou S. Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE; 2019. P. 4690–4699. https://doi.org/10.1109/CVPR.2019.00482

35. Gulati A., Qin J., Chiu C., Parmar N., Zhang Y., Yu J., Han W., Wang S., Zhang Z., Wu Y., Pang R. Conformer: Convolutionaugmented trans-former for speech recognition. Proc. Interspeech 2020. 2020. P. 5036–5040. https://doi.org/10.21437/interspeech.2020-3015

36. Glorot X., Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research. 2010. P. 249–256. URL: http://proceedings.mlr.press/v9/glorot10a.html

37. Gu J., Wang Z., Kuen J., Ma L., Shahroudy A., Shuai B., Liu T., Wang X., Wang G., Cai J., Chen T. Recent advances in convolutional neural networks. Pattern Recognition. 2018;77:354–377. https://doi.org/10.1016/j.patcog.2017.10.013

38. Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention is all you need. Advances in Neural Information Processing Systems. 2017;30:5999–6009. https://doi.org/10.48550/arXiv.1706.03762

39. Ren Y., Ruan Y., Tan X., Qin T., Zhao S., Zhao Z., Liu T.Y. FastSpeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems. 2019;32. https://doi.org/10.48550/arXiv.1905.09263

40. Graves A., Fernández S., Gomez F., Schmidhuber J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning. 2006. P. 369–376. https://doi.org/10.1145/1143844.1143891

41. Ghahremani P., BabaAli B., Povey D., Riedhammer K., Trmal J., Khudanpur S. A pitch extraction algorithm tuned for automatic speech recognition. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2014. P. 2494–2498. http://doi.org/10.1109/ICASSP.2014.6854049

42. Gerhard D. Pitch Extraction and Fundamental Frequency: History and Current Techniques. Masters Thesis. Regina, SK, Canada: Department of Computer Science, University of Regina; 2003. 23 p.

43. Łańcucki A. Fastpitch: Parallel text-to-speech with pitch prediction. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2021. P. 6588–6592. https://doi.org/10.1109/ICASSP39728.2021.9413889

44. Kong J., Kim J., Bae J. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems. 2020;33:17022–17033. http://doi.org/10.48550/arXiv.2010.05646

45. Kominek J., Black A.W. The CMU Arctic speech databases. In: Fifth ISCA Workshop on Speech Synthesis. 2004. P. 223–224.

46. Zhao G., Sonsaat S., Silpachai A., Lucic I., Chukharev-Hudilainen E., Levis J., Gutierrez-Osuna R. L2-ARCTIC: A Nonnative English Speech Corpus. Interspeech 2018. 2018. P. 2783–2787. http://doi.org/10.21437/Interspeech.2018-1110

47. Weinberger S.H., Kunath S.A. The Speech Accent Archive: towards a typology of English accents. In: Corpus-based Studies in Language Use, Language Learning, and Language Documentation. Brill; 2011. P. 265–281. https://doi.org/10.1163/9789401206884_014

48. Ardila R., Branson M., Davis K., Kohler M., Meyer J., Henretty M., Henretty M., Morais R., Saunders L., Tyers F., Weber G. Common Voice: A Massively-Multilingual Speech Corpus. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. 2020. P. 4218–4222. https://doi.org/10.48550/arXiv.1912.06670

49. Nagrani A., Chung J.S., Zisserman A. Voxceleb: a large-scale speaker identification dataset. Interspeech 2017. 2017. http://doi.org/10.21437/Interspeech.2017-950

50. Chung J., Nagrani A., Zisserman A. VoxCeleb2: Deep speaker recognition. Interspeech 2018. 2018. http://doi.org/10.21437/Interspeech.2018-1929

51. Panayotov V., Chen G., Povey D., Khudanpur S. Librispeech: an ASR corpus based on public domain audio books. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2015. P. 5206–5210. http://doi.org/10.1109/ICASSP.2015.7178964

52. Veaux C., Yamagishi J., MacDonald K. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit. University of Edinburgh. The Center for Speech Technology Research (CSTR). 2017. https://doi.org/10.7488/ds/2645

53. Chen G., Chai S., Wang G., Du J., Zhang W., Weng C., Su D., Povey D., TrmalJ., Zhang J., Jin M., Khudanpur S., Watanabe S., Zhao S., Zou W., Li X., Yao X., Wang Y., Wang Y., You Z., Yan Z. GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio. In: 22nd Annual Conference of the International Speech Communication Association, Interspeech 2021. International Speech Communication Association; 2021. P. 4376–4380. https://doi.org/10.21437/Interspeech.2021-1965

54. Bakhturina E., Zhang Y., Ginsburg B. Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization. Proc. Interspeech 2022. 2022. http://doi.org/10.48550/arXiv.2203.15917

55. Kudo T., Richardson J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2018. P. 66–71. https://doi.org/10.48550/arXiv.1808.06226

56. Koizumi Y., Zen H., Karita S., Ding Y., Yatabe K., Morioka N., Bacchiani M., Zhang Y., Han W., Bapna A. Libritts-r: A Restored Multi-Speaker Text-to-Speech Corpus. arXiv preprint arXiv:2305.18802. 2023. https://doi.org/10.48550/arXiv.2305.18802

57. Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G., Killeen T., Lin Z., Gimelshein N., Antiga L., Desmaison A., Kopf A., Yang E., DeVito Z., Raison M., Tejani A., Chilamkurthy S., SteinerB., Fang L., Bai J., Chintala S. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems. 2019;32:8024–8035.

58. Kuchaiev O., Li J., Nguyen H., Hrinchuk O., Leary R., Ginsburg B., Kriman S., Beliaev S., Lavrukhin V., Cook J., Castonguay P., Popova M., Huang J., Cohen J. Nemo: a toolkit for building ai applications using neural modules. arXiv preprint arXiv:1909.09577. 2019. https://doi.org/10.48550/arXiv.1909.09577

59. Bredin H., Yin R., Coria J.M., Gelly G., Korshunov P., Lavechin M., Fustes D., Titeux H., Bouaziz W., Gill M.P. Pyannote. Audio: neural building blocks for speaker diarization. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2020. P. 7124–7128. https://doi.org/10.1109/ICASSP40776.2020.9052974

60. Olatunji T., Afonja T., Yadavalli A., Emezue C.C., Singh S., Dossou B.F., Osuchukwu J., Osei S., Tonja A.L., Etori N., Mbataku C. AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR. Transactions of the Association for Computational Linguistics. 2023;11:1669–1685. https://doi.org/10.1162/tacl_a_00627

61. Majumdar S., Balam J., Hrinchuk O., Lavrukhin V., Noroozi V., Ginsburg B. Citrinet: Closing the gap between nonautoregressive and autoregressive end-to-end models for automatic speech recognition. arXiv preprint arXiv:2104.01721.2021. http://doi.org/10.48550/arXiv.2104.01721

62. Radford A., Kim J.W., Xu T., Brockman G., McLeavey C., Sutskever I. Robust speech recognition via large-scale weak supervision. In: Proceedings of the 40th International Conference on Machine Learning. PMLR 202. 2023. P. 28492–28518. http://doi.org/10.48550/arXiv.2212.04356