Magnified gradient function to improve first-order gradient-based learning algorithms

Sin Chun Ng; Chi Chung Cheung; Andrew Kwok Fai Lui; Shensheng Xu

doi:10.1007/978-3-642-31346-2_51

Magnified gradient function to improve first-order gradient-based learning algorithms

Sin Chun Ng, Chi Chung Cheung, Andrew Kwok Fai Lui, Shensheng Xu

The Hong Kong Polytechnic University

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

2 Citations (Scopus)

Abstract

In this paper, we propose a new approach to improve the performance of existing first-order gradient-based fast learning algorithms in terms of speed and global convergence capability. The idea is to magnify the gradient terms of the activation function so that fast learning speed and global convergence can be achieved. The approach can be applied to existing gradient-based algorithms. Simulation results show that this approach can significantly speed up the convergence rate and increase the global convergence capability of existing popular first-order gradient-based fast learning algorithms for multi-layer feed-forward neural networks.

Original language	English
Title of host publication	Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings
Pages	448-457
Number of pages	10
Edition	PART 1
DOIs	https://doi.org/10.1007/978-3-642-31346-2_51
Publication status	Published - 11 Jul 2012
Event	9th International Symposium on Neural Networks, ISNN 2012 - Shenyang, China Duration: 11 Jul 2012 → 14 Jul 2012

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 1
Volume	7367 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	9th International Symposium on Neural Networks, ISNN 2012
Country/Territory	China
City	Shenyang
Period	11/07/12 → 14/07/12

Keywords

backpropagation
gradient-based algorithms
magnified gradient function
Quickprop
Rprop

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

More information

10.1007/978-3-642-31346-2_51

Cite this

Ng, S. C., Cheung, C. C., Lui, A. K. F., & Xu, S. (2012). Magnified gradient function to improve first-order gradient-based learning algorithms. In Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings (PART 1 ed., pp. 448-457). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7367 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-31346-2_51

Ng, Sin Chun ; Cheung, Chi Chung ; Lui, Andrew Kwok Fai et al. / Magnified gradient function to improve first-order gradient-based learning algorithms. Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1. ed. 2012. pp. 448-457 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).

@inproceedings{0bea8956161645bbb73817c265e7aa05,

title = "Magnified gradient function to improve first-order gradient-based learning algorithms",

abstract = "In this paper, we propose a new approach to improve the performance of existing first-order gradient-based fast learning algorithms in terms of speed and global convergence capability. The idea is to magnify the gradient terms of the activation function so that fast learning speed and global convergence can be achieved. The approach can be applied to existing gradient-based algorithms. Simulation results show that this approach can significantly speed up the convergence rate and increase the global convergence capability of existing popular first-order gradient-based fast learning algorithms for multi-layer feed-forward neural networks.",

keywords = "backpropagation, gradient-based algorithms, magnified gradient function, Quickprop, Rprop",

author = "Ng, {Sin Chun} and Cheung, {Chi Chung} and Lui, {Andrew Kwok Fai} and Shensheng Xu",

year = "2012",

month = jul,

day = "11",

doi = "10.1007/978-3-642-31346-2_51",

language = "English",

isbn = "9783642313455",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 1",

pages = "448--457",

booktitle = "Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings",

edition = "PART 1",

note = "9th International Symposium on Neural Networks, ISNN 2012 ; Conference date: 11-07-2012 Through 14-07-2012",

}

Ng, SC, Cheung, CC, Lui, AKF & Xu, S 2012, Magnified gradient function to improve first-order gradient-based learning algorithms. in Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 7367 LNCS, pp. 448-457, 9th International Symposium on Neural Networks, ISNN 2012, Shenyang, China, 11/07/12. https://doi.org/10.1007/978-3-642-31346-2_51

Magnified gradient function to improve first-order gradient-based learning algorithms. / Ng, Sin Chun; Cheung, Chi Chung; Lui, Andrew Kwok Fai et al.
Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1. ed. 2012. p. 448-457 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7367 LNCS, No. PART 1).

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - Magnified gradient function to improve first-order gradient-based learning algorithms

AU - Ng, Sin Chun

AU - Cheung, Chi Chung

AU - Lui, Andrew Kwok Fai

AU - Xu, Shensheng

PY - 2012/7/11

Y1 - 2012/7/11

N2 - In this paper, we propose a new approach to improve the performance of existing first-order gradient-based fast learning algorithms in terms of speed and global convergence capability. The idea is to magnify the gradient terms of the activation function so that fast learning speed and global convergence can be achieved. The approach can be applied to existing gradient-based algorithms. Simulation results show that this approach can significantly speed up the convergence rate and increase the global convergence capability of existing popular first-order gradient-based fast learning algorithms for multi-layer feed-forward neural networks.

AB - In this paper, we propose a new approach to improve the performance of existing first-order gradient-based fast learning algorithms in terms of speed and global convergence capability. The idea is to magnify the gradient terms of the activation function so that fast learning speed and global convergence can be achieved. The approach can be applied to existing gradient-based algorithms. Simulation results show that this approach can significantly speed up the convergence rate and increase the global convergence capability of existing popular first-order gradient-based fast learning algorithms for multi-layer feed-forward neural networks.

KW - backpropagation

KW - gradient-based algorithms

KW - magnified gradient function

KW - Quickprop

KW - Rprop

UR - http://www.scopus.com/inward/record.url?scp=84865151498&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-31346-2_51

DO - 10.1007/978-3-642-31346-2_51

M3 - Conference article published in proceeding or book

AN - SCOPUS:84865151498

SN - 9783642313455

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 448

EP - 457

BT - Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings

T2 - 9th International Symposium on Neural Networks, ISNN 2012

Y2 - 11 July 2012 through 14 July 2012

ER -

Ng SC, Cheung CC, Lui AKF, Xu S. Magnified gradient function to improve first-order gradient-based learning algorithms. In Advances in Neural Networks, ISNN 2012 - 9th International Symposium on Neural Networks, Proceedings. PART 1 ed. 2012. p. 448-457. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). doi: 10.1007/978-3-642-31346-2_51

Magnified gradient function to improve first-order gradient-based learning algorithms

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Cite this