中圖分類號(hào)：TP181 文獻(xiàn)標(biāo)志碼：A DOI: 10.16157/j.issn.0258-7998.223117 中文引用格式： 孫靜含，任靜. 計(jì)算機(jī)文本分析算法發(fā)展綜述[J]. 電子技術(shù)應(yīng)用，2023，49(3)：42-47. 英文引用格式： Sun Jinghan，Ren Jing. A survey of the development of computer text analysis algorithms[J]. Application of Electronic Technique，2023，49(3)：42-47.
A survey of the development of computer text analysis algorithms
Sun Jinghan1，Ren Jing2
(1.Beijing University of Technology，Beijing100124， China； 2.The Sixth Research Institute of China Electronics Corporation， Beijing 100083， China)
Abstract： Abstract： Computer text analysis is an important branch in the field of natural language processing, and it is a computer technology that studies how to extract various types of information from a given corpus from text data. At present, computer text analysis has entered a new historical stage. On the one hand, the keyword extraction algorithm has gradually been completed. On the other hand, with the emergence of the BERT method, the word vector calculation problem has also made great progress. However, there are still some problems to be solved in both keyword extraction and word vector calculation. In addition, many existing studies suitable for using text analysis still use ancient text analysis methods. Therefore, in the future, how to better reduce the model size to promote the integration of disciplines and improve the comprehensive social benefits of text analysis will become an important issue in the development of text analysis algorithms.
Key words : text analysis；natural language processing；algorithm
0 引言
計(jì)算機(jī)文本分析是自然語(yǔ)言處理（Natural Language Processing, NLP）領(lǐng)域的一個(gè)重要分支，是指對(duì)文本數(shù)據(jù)或語(yǔ)料庫(kù)內(nèi)的語(yǔ)料進(jìn)行分析，最終提取出給定語(yǔ)料的各種信息，包括關(guān)鍵詞、詞向量等內(nèi)容的計(jì)算機(jī)技術(shù)，在一些文獻(xiàn)中也將這一領(lǐng)域的相關(guān)技術(shù)納入自然語(yǔ)言處理預(yù)訓(xùn)練技術(shù)之中。文本分析最早起源于20世紀(jì)50~60年代，這一階段的研究重點(diǎn)是如何對(duì)語(yǔ)言規(guī)則進(jìn)行設(shè)定。到20世紀(jì)70年代，隨著語(yǔ)料庫(kù)的豐富和硬件設(shè)備的進(jìn)步，文本分析技術(shù)開(kāi)始融合機(jī)器學(xué)習(xí)算法，并得到了快速發(fā)展。進(jìn)入21世紀(jì)之后，深度學(xué)習(xí)方法被運(yùn)用到文本分析之中，誕生了諸如Word2Vec、BERT等技術(shù)，進(jìn)一步提高了文本分析的應(yīng)用場(chǎng)景。在可以預(yù)見(jiàn)的未來(lái)，傳統(tǒng)的基于數(shù)字?jǐn)?shù)據(jù)的分析方法將逐漸無(wú)法滿足越發(fā)龐雜的應(yīng)用需求，文本分析和其所屬的自然語(yǔ)言處理領(lǐng)域?qū)?huì)是一個(gè)愈發(fā)重要的發(fā)展方向。
