中圖分類號：TP399 文獻標志碼：A DOI: 10.16157/j.issn.0258-7998.233812
中文引用格式： 顧明珠，明瑞成，邱創(chuàng)一，等. 一種多教師模型知識蒸餾深度神經(jīng)網(wǎng)絡模型壓縮算法[J]. 電子技術(shù)應用，2023，49(8)：7-12.
英文引用格式： Gu Mingzhu，Ming Ruicheng，Qiu Chuangyi，et al. A multi-teacher knowledge distillation model compression algorithm for deep neural network[J]. Application of Electronic Technique，2023，49(8)：7-12.

A multi-teacher knowledge distillation model compression algorithm for deep neural network

Abstract： In order to minimize the accuracy loss when compressing huge deep learning models and deploying them to devices with limited computing power and storage capacity, a knowledge distillation model compression method is investigated and an improved multi-teacher model knowledge distillation compression algorithm with filtering is proposed. Taking advantage of the integration of multi-teacher models, the better-performing teacher models are screened for student instruction using the predicted cross-entropy of each teacher model as the quantitative criterion for screening, and the student models are allowed to extract information starting from the feature layer of the teacher models, while the better-performing teacher models are allowed to have more say in the instruction. The experimental results of classification models such as VGG13 on the CIFAR100 dataset show that the multi-teacher model compression method in this paper has better performance in terms of accuracy compared with other compression algorithms with the same size of the final obtained student models.

Key words : model compression；distillation of knowledge；multi-teacher model；cross entropy；feature layer