中圖分類號(hào)：TP311 文獻(xiàn)標(biāo)志碼：A DOI: 10.16157/j.issn.0258-7998.223379 中文引用格式： 孫北寧，呂維新，曾俊，等. 一種結(jié)合TF-IDF和Simhash的科技項(xiàng)目文本相似性度量方法[J]. 電子技術(shù)應(yīng)用，2023，49(6)：89-93. 英文引用格式： Sun Beining，Lv Weixin，Zeng Jun，et al. An approach for text similarity measurement of science and technology projects combing TF-IDF and Simhash[J]. Application of Electronic Technique，2023，49(6)：89-93.
An approach for text similarity measurement of science and technology projects combing TF-IDF and Simhash
Sun Beining1，2，Lv Weixin3，Zeng Jun4，Xiao Heng4
(1.Department of Science Technology and Data， Yunnan Power Grid Co.， Ltd.， Kunming 650011， China； 2.School of Big Data and Intelligent Engineering， Southwest Forestry University， Kunming 650224， China； 3.Kunming Power Supply Bureau， Yunnan Power Grid Co.， Ltd.， Kunming 650011， China； 4.Yunnan Yundian Tongfang Technology Co.， Ltd.， Kunming 650214， China)
Abstract： To enhance the accuracy and performance of text similarity measurement of science and technology projects, this paper proposes a new approach for measuring text similarity of science and technology projects by combining TF-IDF and Simhash. Firstly, this method uses natural language processing technology to preprocess science and technology project texts to get a term set, then uses the TF-IDF method to calculate the TF-IDF value of each term in the term set, and selects the important term with higher TF-IDF value. Secondly, this method uses the Simhash algorithm to get the Simhash signature of the text through mapping the selected important terms into fixed binary strings. Finally, Hamming distance is used to calculate the similarity between two Simhash signatures. Experimental results show that compared to the traditional Simhash and TF-IDF, the proposed method can promote the evaluation metrics of precision, recall and F-measure.
Key words : science and technology project text；text similarity；TF-IDF；Simhash
