Pieter-Jan Hoedt, Sepp Hochreiter, Günter Klambauer,
"Characterising Activation Functions by Their Backward Dynamics Around Forward Fixed Points"
: Neural Information Processing Systems (NIPS 2018), 2018
Original Titel:
Characterising Activation Functions by Their Backward Dynamics Around Forward Fixed Points
Sprache des Titels:
Englisch
Original Buchtitel:
Neural Information Processing Systems (NIPS 2018)
Original Kurzfassung:
The forward dynamics in neural networks for various activation functions has been studied extensively in the context of initialisation and normalisation strategies, by mean field theory, edge of chaos theory, and fixed point analysis. However, the study of the backward dynamics appears to be largely disconnected to the insights obtained from the forward analysis. We argue that many of the ideas from the forward analysis could and should be applied to backward dynamics. We show that the ideas of mean field theory and fixed point analysis apply to the backward pass and allow to characterize activation functions.
Forschungs-