人工智能培训

搜索

深度学习论文:深度网络中线性区域的复杂性(Complexity of Linear Regions in Deep Networks)

[复制链接]
meidou. 发表于 2019-1-28 12:05:05 | 显示全部楼层 |阅读模式
meidou. 2019-1-28 12:05:05 465 0 显示全部楼层
深度学习论文:深度网络中线性区域的复杂性(Complexity of Linear Regions in Deep Networks)众所周知,神经网络的表现力取决于其体系结构,更深层的网络表达更复杂的功能。在计算分段线性函数的网络中,例如具有RELU激活的网络,不同线性区域的数量是表达性的自然度量。有可能构建线性区域的数量随深度呈指数增长的网络,或者仅仅是单个区域的网络;不清楚大多数网络在此范围内的哪个位置在训练之前或之后落入实践中。在本文中,我们提供了一个数学框架来计算分段线性网络的线性区域的数量,并测量这些区域之间的边界的体积。特别地,我们证明了对于初始化的网络,沿着任何维子空间的平均区域数量在神经元的总数中线性增长,而不是指数上限。我们还发现,初始化时最近区域边界的平均距离与神经元数量的倒数相似。我们的理论认为,即使在训练之后,线性区域的数量远低于指数,这是一种与经验观察相匹配的直觉。我们得出结论,神经网络的实际表现力可能远低于理论最大值,并且可以量化这个差距。
It is well-known that the expressivity of a neural network depends on itsarchitecture, with deeper networks expressing more complex functions.In thecase of networks that compute piecewise linear functions, such as those withReLU activation, the number of distinct linear regions is a natural measure ofexpressivity.It is possible to construct networks for which the number oflinear regions grows exponentially with depth, or with merely a single region;it is not clear where within this range most networks fall in practice, eitherbefore or after training.In this paper, we provide a mathematical framework tocount the number of linear regions of a piecewise linear network and measurethe volume of the boundaries between these regions.In particular, we provethat for networks at initialization, the average number of regions along anyone-dimensional subspace grows linearly in the total number of neurons, farbelow the exponential upper bound.We also find that the average distance tothe nearest region boundary at initialization scales like the inverse of thenumber of neurons.Our theory suggests that, even after training, the number oflinear regions is far below exponential, an intuition that matches ourempirical observations.We conclude that the practical expressivity of neuralnetworks is likely far below that of the theoretical maximum, and that this gapcan be quantified.深度学习论文:深度网络中线性区域的复杂性(Complexity of Linear Regions in Deep Networks) xLKd3eyK4Y4iSli3.jpg
URL地址:https://arxiv.org/abs/1901.09021     ----pdf下载地址:https://arxiv.org/pdf/1901.09021    ----深度学习论文:深度网络中线性区域的复杂性(Complexity of Linear Regions in Deep Networks)
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则 返回列表 发新帖

meidou.当前离线
新手上路

查看:465 | 回复:0

快速回复 返回顶部 返回列表