Semanlink - [1503.02406] Deep Learning and the Information Bottleneck Principle

Tags:

Au sujet de ce document

sl:arxiv_author :
- Noga Zaslavsky
- Naftali Tishby
sl:arxiv_firstAuthor : Naftali Tishby
sl:arxiv_num : 1503.02406
sl:arxiv_published : 2015-03-09T09:39:41Z
sl:arxiv_summary : Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.@en
sl:arxiv_title : Deep Learning and the Information Bottleneck Principle@en
sl:arxiv_updated : 2015-03-09T09:39:41Z
sl:bookmarkOf : https://arxiv.org/abs/1503.02406
sl:creationDate : 2019-08-15
sl:creationTime : 2019-08-15T17:07:31Z

Infos sur le fichier

Bookmark of: https://arxiv.org/abs/1503.02406

Documents with similar tags (experimental)

[1908.01580] The HSIC Bottleneck: Deep Learning without Back-Propagation

Tags:

2019-08-15 A propos

[physics/0004057] The information bottleneck method

Tags:

> We define the relevant information in a signal x ∈ X as being the information that this signal provides about another signal y ∈ Y. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. **Understanding the signal x requires more than just predicting y, it also requires specifying which features of X play a role in the prediction. We formalize this problem as that of finding a short code for X that preserves the maximum information about Y.** That is, we squeeze the information that X provides about Y through a ‘bottleneck’ formed by a limited set of codewords X ̃... This approach yields an exact set of self consistent equations for the coding rules X → X ̃ and X ̃ → Y .

(from the intro) : how to define "meaningful / relevant" information? An issue left out of information theory by Shannon (focus on the problem of transmitting information rather than judging its value to the recipient) ->leads to
consider statistical and information theoretic principles as almost irrelevant
for the question of meaning.

> In contrast, **we argue here that information theory,
in particular lossy source compression, provides a natural quantitative
approach to the question of “relevant information.”** Specifically, we formulate
a **variational principle** for the extraction or efficient representation of
relevant information.

2019-08-15 A propos