人工智能培训

搜索

人工智能论文:高效深度神经网络的训练秩修剪(Trained Rank Pruning for Efficient Deep Neural Networks)

[复制链接]
755144030 发表于 2018-12-7 11:33:37 | 显示全部楼层 |阅读模式
755144030 2018-12-7 11:33:37 35 0 显示全部楼层
人工智能论文:高效深度神经网络的训练秩修剪(Trained Rank Pruning for Efficient Deep Neural Networks)随着网络深度和宽度的增加,深度神经网络(DNN)的性能在近年来不断提升。为了在移动电话等边缘设备上启用DNN,研究人员提出了几种网络压缩方法,包括修剪,量化和分解。在基于因子化的方法中,低秩近似由于其坚实的理论基础和有效的实现而被广泛采用。以前的几个工作试图通过低秩分解直接逼近预训练模型;然而,参数中的小近似误差可以获得大的预测损失。因此,性能通常会显着下降,并且需要进行复杂的微调才能恢复准确性。我们认为将低秩近似与训练分开并不是最佳选择。与以前的作品不同,本文将低秩近似和正则化集成到训练中。我们提出训练等级修剪(TRP),它迭代低等级近似和训练。 TRP保持原始网络的容量,同时在训练期间施加低级别约束。利用随机亚梯度下降优化核正则化进一步促进TRP中的低秩。 TRP训练的网络本质上具有低秩结构,并且可以以可忽略的性能损失来近似,消除了低秩近似之后的微调。这些方法在CIFAR-10和ImageNet上进行了全面评估,优于使用低秩近似的先前压缩方法。
The performance of Deep Neural Networks (DNNs) keeps elevating in recentyears with increasing network depth and width.To enable DNNs on edge deviceslike mobile phones, researchers proposed several network compression methodsincluding pruning, quantization and factorization.Among thefactorization-based approaches, low-rank approximation has been widely adoptedbecause of its solid theoretical rationale and efficient implementations.Several previous works attempted to directly approximate a pre-trained model bylow-rank decomposition;however, small approximation errors in parameters canripple a large prediction loss.As a result, performance usually dropssignificantly and a sophisticated fine-tuning is required to recover accuracy.We argue that it is not optimal to separate low-rank approximation fromtraining.Unlike previous works, this paper integrates low rank approximationand regularization into the training.We propose Trained Rank Pruning (TRP),which iterates low rank approximation and training.TRP maintains the capacityof original network while imposes low-rank constraints during training.Astochastic sub-gradient descent optimized nuclear regularization is utilized tofurther encourage low rank in TRP.The TRP trained network has low-rankstructure in nature, and can be approximated with negligible performance loss,eliminating fine-tuning after low rank approximation.The methods arecomprehensively evaluated on CIFAR-10 and ImageNet, outperforming previouscompression methods using low rank approximation.人工智能论文:高效深度神经网络的训练秩修剪(Trained Rank Pruning for Efficient Deep Neural Networks) kZdo4Kr4wko73QoR.jpg
URL地址:https://arxiv.org/abs/1812.02402     ----pdf下载地址:https://arxiv.org/pdf/1812.02402    ----人工智能论文:高效深度神经网络的训练秩修剪(Trained Rank Pruning for Efficient Deep Neural Networks)
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则 返回列表 发新帖

755144030当前离线
新手上路

查看:35 | 回复:0

快速回复 返回顶部 返回列表