摘要
近年来深度学习在各个研究领域取得越来越多的成果,这些都离不开激活函数的发展.但现有的激活函数Tanh、Re LU和PRe LU在随着研究的深入暴露出越来越多的问题,比如存在"神经元死亡"和偏移现象,对噪声不具有鲁棒性.针对这些问题,结合Tanh和PRe LU的优点,提出了TReLU激活函数,通过参数α控制负半轴非饱和区间获得激活值,输出近似0均值,软饱和性对噪声鲁棒.实验结果表明,TReLU在四种不同的数据集上都取得了最好的效果,对不同优化方法具有鲁棒性,具有一定的实用价值.
In recent years,deep learning achieve more and more results in various research areas,and they can’t be separated from the development of the activation function. However,the existing activation functions Tanh,Re LU and PRe LU are exposed to more and more problems with the depth of the study,such as the existence of " neuronal death" and the bias shift,and they are not robust to noise. In viewof these problems,combined with the advantages of Tanh and PRe LU,the TReLU activation function is proposed to preserve the negative half-axis activation value,the mean shifts toward zero and the soft saturation is robust to noise. The experimental results showthat TReLU has the best effect on three different data sets,and is robust to different optimization methods,so TReLU has certain practical value.
作者
张涛
杨剑
宋文爱
宋超峰
ZHANG Tao;YANG Jian;SONG Wen-ai;SONG Chao-feng(School of Software,North University of China,Taiyuan 030051,China)
出处
《小型微型计算机系统》
CSCD
北大核心
2019年第1期58-63,共6页
Journal of Chinese Computer Systems
基金
山西省回国留学人员科研资助项目(2014-053)资助
关键词
深度学习
激活函数
软饱和性
鲁棒性
deep learning
activation function
soft saturation
robustness
