2025年 03期

A Recursive Back Propagation Algorithm for Data Stream Classification


摘要(Abstract):

针对传统深度神经网络因数据流中发生概念漂移而出现分类准确率较低的问题,为了增强深度神经网络模型的学习能力,提出一种用于数据流分类的递归反向传播算法。该算法融合在线梯度下降算法的强大数据流学习能力与递归最小二乘法的快速收敛特性,当数据流发生概念漂移时,首先利用递归最小二乘法逐步训练神经网络模型,达到一个相对稳定的状态后切换至在线梯度下降算法,进一步训练深度神经网络模型,实现更深层次的数据流学习,优化深度神经网络模型的分类性能,并在多个人工数据集和真实数据集中实验验证所提算法的有效性。结果表明:所提算法具有优异的概念漂移适应能力,数据流分类准确率超越仅使用在线梯度下降算法或递归最小二乘法训练神经网络模型的多种算法。

关键词(KeyWords):在线深度学习;在线梯度下降算法;递归最小二乘法;反向传播;深度神经网络;概念漂移

基金项目(Foundation):国家自然科学基金项目(62366011);; 广西重点研发计划项目(桂科AB21220023);; 广西图像图形与智能处理重点实验室资助项目(GIIP2306)

作者(Author): 刘展华,文益民,刘祥

DOI: 10.13349/j.cnki.jdxbn.20241227.001

参考文献(References):

[1] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84.

[2] BENGIO Y,COURVILLE A,VINCENT P.Representation learning:a review and new perspectives[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(8):1798.

[3] HOI S C H,WANG J L,ZHAO P L.Libol:a library for online learning algorithms[J].The Journal of Machine Learning Research,2014,15(1):495.

[4] CHEN K L,LEE C H,GARUDADRI H,et al.ResNEsts and DenseNEsts:block-based DNN models with improved representation guarantees[J].Advances in Neural Information Processing Systems,2021,34:3413.

[5] DAUPHIN Y N,PASCANU R,GULCEHRE C,et al.Identifying and attacking the saddle point problem in high-dimensional non-convex optimization[C]//NIPS’14:Proceedings of the 27th International Conference on Neural Information Processing Systems :Vol 2.New York:ACM,2014:2933.

[6] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[EB/OL].(2015-11-11) [2024-05-10].https://doi.org/10.48550/arXiv.1502.03167.

[7] NAIR V,HINTON G E.Rectified linear units improve restricted boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning (ICML-10),June 21-24,2010,Haifa,Israel.Madison:Omnipress,2010:807.

[8] SAHOO D,PHAM Q,LU J,et al.Online deep learning:learning deep neural networks on the fly[EB/OL].(2017-11-10)[2024-05-10].https://doi.org/10.48550/arXiv.1711.03705.

[9] ASHFAHANI A,PRATAMA M.Autonomous deep learning:continual learning approach for dynamic environments[C]//Proceedings of the 2019 SIAM international conference on data mining,May 2-4,2019,Calgary,Canada.Philadelphia:SIAM,2019:666.

[10] YANG Y,ZHOU D W,ZHAN D C,et al.Adaptive deep models for incremental learning:considering capacity scalability and sustainability[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,August 4-8,2019,Anchorage,USA.New York:ACM,2019:74.

[11] HUANG G B,ZHU Q Y,SIEW C K.Extreme learning machine:a new learning scheme of feedforward neural networks[C]//2004 IEEE International Joint Conference on Neural Networks,July 25-29,2004,Budapest,Hungary.Piscataway:IEEE,2004:2.

[12] HUANG G B,LIANG N Y,RONG H J,et al.On-line sequential extreme learning machine[J].Computational Intelligence,2005,2005:232.

[13] IGELNIK B,PAO Y H.Stochastic choice of basis functions in adaptive function approximation and the functional-link net[J].IEEE Transactions on Neural Networks,1995,6(6):1320.

[14] SHIVA S,HU M H,SUGANTHAN P N.Online learning using deep random vector functional link network[J].Pattern recognition,2022,129:108744.

[15] XUE H,REN Z.Sketch discriminatively regularized online gradient descent classification[J].Applied Intelligence,2020,50(5):1367.

[16] MARQUARDT D W.An algorithm for least-squares estimation of nonlinear parameters[J].Journal on the Society for Industrial and Applied Mathematics,1963,11(2):431.

[17] SONG Q,MI Y X,LAI W X.A novel variable forgetting factor recursive least square algorithm to improve the anti-interference ability of battery model parameters identification[J].IEEE Access,2019,7:61548.

[18] GOLUB G H,HANSEN P C,O’LEARY D P.Tikhonov regularization and total least squares[J].SIAM Journal on Matrix Analysis and Applications,1999,21(1):185.

[19] YING Y M,PONTIL M.Online gradient descent learning algorithms[J].Foundations of Computational Mathematics,2008,8:561.

[20] MASTERS D,LUSCHI C.Revisiting small batch training for deep neural networks[EB/OL].(2018-04-20)[2024-05-09].https://doi.org/10.48550/arXiv.1804.07612.

[21] CHEN W W,TAN D K,ZHAO L F.Vehicle sideslip angle and road friction estimation using online gradient descent algorithm[J].IEEE Transactions on Vehicular Technology,2018,67(12):11475.

[22] JANSSON P A.Neural networks:an overview[J].Analytical Chemistry,1991,63(6):357A.

[23] BISHOP C M.Neural networks and their applications[J].Review of Scientific Instruments,1994,65(6):1803.

[24] FRíAS-BLANCO I,CAMPO-áVILA J del,RAMOS-JIMéNEZ G,et al.Online and non-parametric drift detection methods based on Hoeffding’s bounds[J].IEEE Transactions on Knowledge and Data Engineering,2014,27(3):810.

[25] LU J,LIU A J,DONG F,et al.Learning under concept drift:a review[J].IEEE Transactions on Knowledge and Data Engineering,2018,31(12):2346.

[26] LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278.

[27] BAKHSHI S,GHAHRAMANIAN P,BONAB H,et al.A broad ensemble learning system for drifting stream classification[J].IEEE Access,2003,11:89315.