MyJournals Home  

RSS FeedsEntropy, Vol. 22, Pages 101: A Geometric Interpretation of Stochastic Gradient Descent Using Diffusion Metrics (Entropy)

 
 

15 january 2020 15:00:09

 
Entropy, Vol. 22, Pages 101: A Geometric Interpretation of Stochastic Gradient Descent Using Diffusion Metrics (Entropy)
 


This paper is a step towards developing a geometric understanding of a popular algorithm for training deep neural networks named stochastic gradient descent (SGD). We built upon a recent result which observed that the noise in SGD while training typical networks is highly non-isotropic. That motivated a deterministic model in which the trajectories of our dynamical systems are described via geodesics of a family of metrics arising from a certain diffusion matrix; namely, the covariance of the stochastic gradients in SGD. Our model is analogous to models in general relativity: the role of the electromagnetic field in the latter is played by the gradient of the loss function of a deep network in the former.


 
86 viewsCategory: Informatics, Physics
 
Entropy, Vol. 22, Pages 97: A Review of the Application of Information Theory to Clinical Diagnostic Testing (Entropy)
Entropy, Vol. 22, Pages 107: On the Composability of Statistically Secure Random Oblivious Transfer (Entropy)
 
 
blog comments powered by Disqus


MyJournals.org
The latest issues of all your favorite science journals on one page

Username:
Password:

Register | Retrieve

Search:

Physics


Copyright © 2008 - 2020 Indigonet Services B.V.. Contact: Tim Hulsen. Read here our privacy notice.
Other websites of Indigonet Services B.V.: Nieuws Vacatures News Tweets Nachrichten