KD-AHOSVD: Neural Network Compression via Knowledge Distillation and Tensor Decomposition

In the field of Deep Learning, the high number of parameters in models has become a significant concern within the scientific community due to the increased computational resources and memory required for training and inference. Addressing this issue, we propose a novel tensorized technique to compress network architectures. Our approach aims to significantly reduce the network’s size and the number of parameters by integrating Averaged Higher Order Singular Value Decomposition with a novel Knowledge Distillation approach. Specifically, we replace certain layers of the original architecture with layers that perform linear projections onto a reduced space defined by our reduction technique. We conducted experiments on image classification tasks using multiple architectures and datasets. The evaluation focuses on final accuracy, model size, and parameter reduction, comparing our approach with both the original models and quantization, a widely used reduction method. The results underscore the effectiveness of our method in significantly reducing the number of parameters and the overall size of neural networks while maintaining high performance.