MyJournals Home  

RSS FeedsEntropy, Vol. 23, Pages 1360: Information Bottleneck Theory Based Exploration of Cascade Learning (Entropy)

 
 

18 october 2021 17:09:49

 
Entropy, Vol. 23, Pages 1360: Information Bottleneck Theory Based Exploration of Cascade Learning (Entropy)
 


In solving challenging pattern recognition problems, deep neural networks have shown excellent performance by forming powerful mappings between inputs and targets, learning representations (features) and making subsequent predictions. A recent tool to help understand how representations are formed is based on observing the dynamics of learning on an information plane using mutual information, linking the input to the representation (I(X;T)) and the representation to the target (I(T;Y)). In this paper, we use an information theoretical approach to understand how Cascade Learning (CL), a method to train deep neural networks layer-by-layer, learns representations, as CL has shown comparable results while saving computation and memory costs. We observe that performance is not linked to information–compression, which differs from observation on End-to-End (E2E) learning. Additionally, CL can inherit information about targets, and gradually specialise extracted features layer-by-layer. We evaluate this effect by proposing an information transition ratio, I(T;Y)/I(X;T), and show that it can serve as a useful heuristic in setting the depth of a neural network that achieves satisfactory accuracy of classification.


 
142 viewsCategory: Informatics, Physics
 
Entropy, Vol. 23, Pages 1359: Beware the Black-Box: On the Robustness of Recent Defenses to Adversarial Examples (Entropy)
Entropy, Vol. 23, Pages 1361: Information Theory and Symbolic Analysis: Theory and Applications (Entropy)
 
 
blog comments powered by Disqus


MyJournals.org
The latest issues of all your favorite science journals on one page

Username:
Password:

Register | Retrieve

Search:

Physics


Copyright © 2008 - 2024 Indigonet Services B.V.. Contact: Tim Hulsen. Read here our privacy notice.
Other websites of Indigonet Services B.V.: Nieuws Vacatures News Tweets Nachrichten