人工智能培训

搜索

深度学习论文:弹性八卦:使用类似八卦的协议分发神经网络训练(Elastic Gossip: Distributing Neural Network Traini

[复制链接]
zwb521 发表于 2018-12-7 11:42:22 | 显示全部楼层 |阅读模式
zwb521 2018-12-7 11:42:22 52 0 显示全部楼层
深度学习论文:弹性八卦:使用类似八卦的协议分发神经网络训练(Elastic Gossip: Distributing Neural Network Training Using Gossip-like  Protocols)分布式神经网络训练对于包括使用计算集群进行扩展,对诸如IOT设备和边缘服务器之类的数据进行训练,利用跨异构环境中的未充分利用资源等的若干原因特别感兴趣。大多数现代方法主要使用计算集群进行扩展,并且需要高网络带宽和频繁的通信。本文概述了标准分发培训,并提出了一种新技术,涉及使用类似Gossip的协议进行空中通信,称为弹性八卦。这种方法建立在称为弹性平均SGD(EASGD)的现有技术的基础上,类似于另一种称为Gossiping SGD的技术,它也是类似Gossip的协议。弹性八卦使用MNIST数字识别和CIFAR-10分类任务,使用常用的跨越多层感知器(MLP)和卷积神经网络(CNN)的神经网络架构,对Gossiping SGD进行经验评估。发现Elastic Gossip,Gossiping SGD和All-reduce SGD表现相当可观,即使后者需要相当高的通信成本。虽然弹性八卦在这些实验中表现优于Gossiping SGD,但可以通过对超参数空间进行更彻底的搜索(特定于agiven应用),可能会产生比弹性八卦更有效的Gossiping SGD配置。
Distributing Neural Network training is of particular interest for severalreasons including scaling using computing clusters, training at data sourcessuch as IOT devices and edge servers, utilizing underutilized resources acrossheterogeneous environments, and so on.Most contemporary approaches primarilyaddress scaling using computing clusters and require high network bandwidth andfrequent communication.This thesis presents an overview of standard approachesto distribute training and proposes a novel technique involvingpairwise-communication using Gossip-like protocols, called Elastic Gossip.Thisapproach builds upon an existing technique known as Elastic Averaging SGD(EASGD), and is similar to another technique called Gossiping SGD which alsouses Gossip-like protocols.Elastic Gossip is empirically evaluated againstGossiping SGD using the MNIST digit recognition and CIFAR-10 classificationtasks, using commonly used Neural Network architectures spanning Multi-LayerPerceptrons (MLPs) and Convolutional Neural Networks (CNNs).It is found thatElastic Gossip, Gossiping SGD, and All-reduce SGD perform quite comparably,even though the latter entails a substantially higher communication cost.WhileElastic Gossip performs better than Gossiping SGD in these experiments, it ispossible that a more thorough search over hyper-parameter space, specific to agiven application, may yield configurations of Gossiping SGD that work betterthan Elastic Gossip.深度学习论文:弹性八卦:使用类似八卦的协议分发神经网络训练(Elastic Gossip: Distributing Neural Network Training Using Gossip-like  Protocols) PCdMTEdU0kkTuu9e.jpg
URL地址:https://arxiv.org/abs/1812.02407     ----pdf下载地址:https://arxiv.org/pdf/1812.02407    ----深度学习论文:弹性八卦:使用类似八卦的协议分发神经网络训练(Elastic Gossip: Distributing Neural Network Training Using Gossip-like  Protocols)
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则 返回列表 发新帖

zwb521当前离线
新手上路

查看:52 | 回复:0

快速回复 返回顶部 返回列表