Off the convex pathAlgorithms off the convex path.
http://offconvex.github.io/
Contrastive Unsupervised Learning of Semantic Representations: A Theoretical Framework<p><em>Semantic representations</em> (aka <em>semantic embeddings</em>) of complicated data types (e.g. images, text, video) have become central in machine learning, and also crop up in machine translation, language models, GANs, domain transfer, etc.
These involve learning a <em>representation function</em> $f$ such that for any data point $x$ its representation $f(x)$ is “high level” (retains semantic information while discarding low level details, such as color of individual pixels in an image) and “compact” (low dimensional).
The test of a good representation is that it should greatly simplify solving <em>new</em> classification tasks, by allowing them to be solved via linear classifiers (or other low-complexity classifiers) using small amounts of labeled data.</p>
<p>Researchers are most interested in <em>unsupervised</em> representation learning using unlabeled data. A popular approach is to use objectives similar to the <strong>word2vec</strong> algorithm for word embeddings, which work well for diverse data types such as molecules, social networks, images, text etc.
See the <a href="https://en.wikipedia.org/wiki/Word2vec">wikipage of word2vec</a> for references.
Why do such objectives succeed in such diverse settings?
This post is about an explanation for these methods by using our <a href="https://arxiv.org/abs/1902.09229">new theoretical framework</a> with coauthor Misha Khodak.
The framework makes minimalistic assumptions, which is a good thing, since word2vec-like algorithms apply to vastly different data types and it is unlikely that they can share a common Bayesian generative model for the data.
(An example of generative models in this space is described in an earlier <a href="http://www.offconvex.org/2016/02/14/word-embeddings-2/">blog post on the RAND-WALK model</a>.)
As a bonus this framework also yields principled ways to design new variants of the training objectives.</p>
<h2 id="semantic-representations-learning">Semantic representations learning</h2>
<p>Do good, broadly useful representations even <em>exist</em> in the first place?
In domains such as computer vision, we know the answer is “yes” because deep convolutional neural networks (CNNs), when trained to high accuracy on large multiclass labeled datasets such as <a href="http://www.image-net.org/">ImageNet</a>, end up learning very powerful and succinct representations along the way. The penultimate layer of the net — the input into the final <a href="https://en.wikipedia.org/wiki/Softmax_function">softmax layer</a> — serves as a good semantic embedding of the image in new unrelated visual tasks.
(Other layers from the trained net can also serve as good embeddings.)
In fact the availability of such embeddings from pre-trained (on large multiclass datasets) nets has led to a revolution in computer vision, allowing a host of new classification tasks to be solved with low-complexity classifiers (e.g. linear classifiers) using very little labeled data.
Thus they are the gold standard to compare to if we try to learn embeddings via unlabeled data.</p>
<p style="text-align:center;">
<img src="/assets/CURLheadless1.svg" width="50%" />
</p>
<h3 id="word2vec-like-methods-curl">word2vec-like methods: CURL</h3>
<p>Since the success of <a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">word2vec</a>, similar approaches were used to learn embeddings for <a href="https://arxiv.org/pdf/1803.02893.pdf">sentences and paragraphs</a>, <a href="https://arxiv.org/abs/1505.00687">images</a> and <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4640716/">biological sequences</a>.
All of these methods share a key idea: they leverage access to pairs of similar data points $x, x^+$, and learn an embedding function $f$ such that the inner product of $f(x)$ and $f(x^+)$ is on average higher than the inner product of $f(x)$ and $f(x^-)$, where $x^{-}$ is a random data point (and thus presumably dissimilar to $x$).
In practice similar data points are usually found heuristically, often using co-occurrences, e.g. consecutive sentences in a large text corpus, nearby frames in a video clip, different patches in the same image, etc.</p>
<p>A good example of such methods is <strong>Quick Thoughts</strong> (QT) from <a href="https://arxiv.org/pdf/1803.02893.pdf">Logeswaran and Lee</a>, which is the state-of-the-art unsupervised text embedding on many tasks. To learn a representation function $f$, QT minimizes the following loss function on a large text corpus</p>
<script type="math/tex; mode=display">L_{un}(f):=\mathop{\mathbb{E}}\limits_{x,x^+,x^-}\left[-\log\left(\frac{e^{f(x)^Tf(x^+)}}{e^{f(x)^Tf(x^+)}+e^{f(x)^Tf(x^-)}}\right)\right] (1),</script>
<p>where $(x, x^+)$ are consecutive sentences and presumably “semantically similar” and $x^-$ is a random negative sample.
For images $x$ and $x^+$ could be <a href="https://arxiv.org/abs/1505.00687">nearby frames</a> from a video.
For text, two successive sentences serve as good candidates for a similar pair: for example, the following are two successive sentences in the Wikipedia page on word2vec: <em>“High frequency words often provide little information.”</em> and <em>“Words with frequency above a certain threshold may be subsampled to increase training speed.”</em>
Clearly they are much more similar than a random pair of sentences, and the learner exploits this.
From now on we use <em>Contrastive Unsupervised Representation Learning (CURL)</em> to refer to methods that leverage similar pairs of data points and our goal is to analyze these methods.</p>
<h3 id="need-for-a-new-framework">Need for a new framework</h3>
<p>The standard framework for machine learning involves minimizing some loss function, and learning is said to succeed (or <em>generalize</em>) if the loss is roughly the same on the average training data point and the average test data point.
In contrastive learning, however, the objective used at test time is very different from the training objective: generalization error is not the right way to think about this.</p>
<blockquote>
<p><strong>Main Hurdle for Theory:</strong> We have to show that doing well on task A (minimizing the word2vec-like objective) allows the representation to do well on task B (i.e., classification tasks revealed later).</p>
</blockquote>
<p>Earlier methods along such lines include <em>kernel learning</em> and <em>semi-supervised learning</em>, but there training typically requires at least a few labeled examples from the classification tasks of future interest. Bayesian approaches using generative models are also well-established in simpler settings, but have proved difficult for complicated data such as images and text.
Furthermore, the simple word2vec-like learners described above do not appear to operate like Bayesian optimizers in any obvious way, and also work for very different data types.</p>
<p>We tackle this problem by proposing a framework that formalizes the notion of semantic similarity that is implicitly used by these algorithms and use the framework to show why contrastive learning gives good representations, while defining what <em>good representations</em> mean in this context.</p>
<h2 id="our-framework">Our framework</h2>
<p>Clearly, the implicit/heuristic notion of similarity used in contrastive learning is connected to the downstream tasks in some way — e.g., similarity carries a strong <em>hint</em> that on average the “similar pairs” tend to be assigned the same labels in many downstream tasks (though there is no hard guarantee per se). We present a simple and minimalistic framework to formalize such a notion of similarity.
For purposes of exposition we’ll refer to data points as “images”.</p>
<h3 id="semantic-similarity">Semantic similarity</h3>
<p>We assume nature has many <em>classes</em> of images, and has a measure $\rho$ on a set of classes $\mathcal{C}$, so that if asked to pick a class it selects $c$ with probability $\rho(c)$.
Each class $c$ also has an associated distribution $D_c$ on images i.e. if nature is asked to furnish examples of class $c$ (e.g., the class “dogs”) then it picks image $x$ with probability $D_c(x)$.
Note that classes can have arbitrary overlap, including no overlap.
To formalize a notion of <em>semantic similarity</em> we assume that when asked to provide “similar” images, nature picks a class $c^+$ from $\mathcal{C}$ using measure $\rho$ and then picks two i.i.d. samples $x, x^{+}$ from the distribution $D_{c^+}$.
The dissimilar example $x^{-}$ is picked by selecting another class $c^-$ from measure $\rho$ and picking a random sample $x^{-}$ from $D_{c^-}$.</p>
<p style="text-align:center;">
<img src="/assets/CURLframework.svg" width="80%" />
</p>
<p>The training objective for learning the representation is exactly the QT objective from earlier, but now inherits the following interpretation from the framework
<script type="math/tex">\min_{f\in\mathcal{F}}\ L_{un}(f)=\mathop{\mathbb{E}}\limits_{c^+,c^-\sim\rho}\ \mathop{\mathbb{E}}\limits_{x,x^+\sim D_{c^+}}\ \mathop{\mathbb{E}}\limits_{x^-\sim D_{c^-}}\left[\log\left(1+e^{f(x)^Tf(x^-)-f(x)^Tf(x^+)}\right)\right]</script></p>
<p>Note that the function class $\mathcal{F}$ is an arbitrary deep net architecture mapping images to embeddings (neural net sans the final layer), and one would learn $f$ via gradient descent/back-propagation as usual.
Of course, no theory currently exists for explaining when optimization succeeds for complicated deep nets, so our framework will simply assume that gradient descent has already resulted in some representation $f$ that achieves low loss, and studies how well this does in downstream classification tasks.</p>
<h3 id="testing-representations">Testing representations</h3>
<p>What defines a good representation?
We assume that the quality of the representation is tested by using it to solve a binary (i.e., two-way) classification task using a linear classifier.
(The paper also studies extensions to $k$-way classification in the downstream task.)
How is this binary classification task selected?
Nature picks two classes $c_1, c_2$ randomly according to measure $\rho$ and picks data points for each class according to the associated probability distributions $D_{c_1}$ and $D_{c_2}$.
The representation is then used to solve this binary task via logistic regression: namely, find two vectors $w_1, w_2$ so as to minimize the following loss
<script type="math/tex">L_{sup}(f, (c_1,c_2))=\inf\limits_{w_1,w_2}\left[\frac{1}{2}\mathop{\mathbb{E}}\limits_{x\sim D_{c_1}}\log(1+e^{f(x)^Tw_2-f(x)^Tw_1})+\frac{1}{2}\mathop{\mathbb{E}}\limits_{x\sim D_{c_2}}\log(1+e^{f(x)^Tw_1-f(x)^Tw_2})\right]</script></p>
<p>The quality of the representation is estimated as the <em>average</em> loss over nature’s choices of binary classification tasks.
<script type="math/tex">L_{sup}(f)=\mathop{\mathbb{E}}\limits_{c_1,c_2\sim\rho}\ \left[L_{sup}(f, (c_1,c_2))|c_1\neq c_2\right]</script></p>
<p>It is important to note that the latent classes present in the unlabeled data are the <em>same</em> classes present in the classification tasks.
This allows us to formalize a sense of ‘semantic similarity’ as alluded to above: the classes from which data points appear together more frequently are the classes that make up <em>relevant</em> classification tasks.
Note that if the number of classes is large, then typically the data used in unsupervised training may involve <em>no samples</em> from the classes used at test time.
Indeed, we are hoping to show that the learned representations are useful for classification on potentially unseen classes.</p>
<h2 id="provable-guarantees-for-unsupervised-learning">Provable guarantees for unsupervised learning</h2>
<p>What would be a dream result for theory? Suppose we fix a class of representation functions ${\mathcal F}$, say those computable by a ResNet 50 architecture with some choices of layer sizes etc.</p>
<blockquote>
<p><strong>Dream Theorem:</strong> Minimizing the unsupervised loss (using modest amount of unlabeled data) yields a representation function $f \in {\mathcal F}$ that is competitive with the <strong>best</strong> representation from ${\mathcal F}$ on downstream classification tasks, even with very few labeled examples per task.</p>
</blockquote>
<p>While the number of unlabeled data pairs needed to learn an approximate minimizer can be controlled using Rademacher complexity arguments (see paper), we show that the dream theorem is impossible as phrased: we can exhibit a simple class ${\mathcal F}$ where the contrastive objective does not yield representations even remotely competitive with the best in the class.
This should not be surprising and only suggests that further progress towards such a dream result would require making more assumptions than the above minimalistic ones.</p>
<p>Instead, our paper makes progress by showing that under the above framework, if the unsupervised loss happens to be small at the end of contrastive learning then the resulting representations perform well on downstream classification.</p>
<blockquote>
<p><strong>Simple Lemma:</strong> The average classification loss on downstream binary tasks is upper bounded by the unsupervised loss.
<script type="math/tex">L_{sup}(f)\le \alpha L_{un}(f),~\forall f\in\mathcal{F}\ \ \ \ \ \ (2)</script>
where $\alpha$ depends on $\rho$. ($\alpha\rightarrow 1$ when $|\mathcal{C}|\rightarrow\infty$, for uniform $\rho$)</p>
</blockquote>
<p>This says that the unsupervised loss function can be treated as a <strong>surrogate</strong> for the performance on downstream supervised tasks solved using linear classification, so minimizing it makes sense.
Furthermore, just a few labeled examples are needed to learn the linear classifiers in future downstream tasks.
Thus our minimalistic framework lets us show guarantees for contrastive learning and also highlights the labeled sample complexity benefits provided by it. For details as well as more finegrained analysis see <a href="https://arxiv.org/abs/1902.09229">the paper</a>.</p>
<h2 id="extensions-of-the-theoretical-analysis">Extensions of the theoretical analysis</h2>
<p>This conceptual framework not only allows us to reason about empirically successful variants of (1), but also leads to the design of new, theoretically grounded unsupervised objective functions.
Here we give a high level view; details are in <a href="https://arxiv.org/abs/1902.09229">our paper</a>.</p>
<p><em>A priori,</em> one might imagine that the log and exponentials in (1) have some information-theoretic interpretation; here we relate the functional form to the fact that logistic regression is going to be used in the downstream classification tasks.
Analogously, if the classification is done via hinge loss, then (2) is true for a different unsupervised loss that uses a hinge-like loss instead.
This objective, for instance, was used to learn image representations from videos by <a href="https://arxiv.org/abs/1505.00687">Wang and Gupta</a>.
Also, usually in practice $k>1$ negative samples are contrasted with each positive sample $(x,x^+)$ and the unsupervised objective looks like the $k$-class cross-entropy loss.
We prove a statement similar to (2) for this setting, where the supervised loss now is the average $(k+1)$-way classification loss.</p>
<p>Finally, the framework provides guidelines for designing new unsupervised objectives when <em>blocks</em> of similar data are available (e.g., sentences in a paragraph).
Replacing $f(x^+)$ and $f(x^-)$ in (1) with the average of the representations from the positive and the negative block respectively, we get a new objective which comes with stronger guarantees and better performance in practice.
We experimentally verify the effectiveness of this variant in our paper.</p>
<h2 id="experiments">Experiments</h2>
<p>We report some controlled experiments to verify the theory. Lacking a canonical multiclass problem for text, we constructed a new 3029-class labeled dataset where a class is one of 3029 articles from Wikipedia, and datapoints are one of $200$ sentences in these articles. Representations will be tested on a random binary classification task that involves two articles, where the labels of the data point is which of the two articles it belongs to. (A 10-way classification task is similarly defined.) Datapoints for the test tasks will be held out while training representations. The class of sentence representation ${\mathcal F}$ is a simple multilayer architecture one based on Gated Recurrent Unit (GRU).</p>
<p>The supervised method for learning representations trains a multiclass classifier on the 3029-way task and the representation is taken from the layer before the final softmax output. This was the gold standard in above discussions.</p>
<p>The unsupervised method is fed pairs of similar data points generated according to our theory: similar data points are just pairs of sentences sampled from the same article. Representations are learnt by minimizing the above unsupervised loss objectives.</p>
<p style="text-align:center;">
<img src="/assets/CURLexperiment.svg" width="60%" />
</p>
<p>The highlighted parts in the table show that the unsupervised representations compete well with the supervised representations on the average $k$-way classification task ($k=2, 10$).</p>
<p>Additionally, even though not covered by our theory, the representation also performs respectably on the full multiclass problem.
We find some support for a suggestion of our theory that the mean (centroid) of the unsupervised representations in each class should be good classifiers for average $k$-way supervised tasks. We find this to be true for unsupervised representations, and surprisingly for supervised representations as well.</p>
<p>The paper also has other experiments studying the effect of number of negative samples and larger blocks of similar data points, including experiments on the CIFAR-100 image dataset.</p>
<h2 id="conclusions">Conclusions</h2>
<p>While contrastive learning is a well-known <em>intuitive</em> algorithm, its practical success has been a mystery for theory.
Our conceptual framework lets us formally show guarantees for representations learnt using such algorithms.
While shedding light on such algorithms, the framework also lets us come up with and analyze variants of it.
It also provides insights into what guarantees are provable and shapes the search for new assumptions that would allow stronger guarantees.
While this is a first cut, possible extensions include imposing a metric structure among the latent classes.
Connections to meta-learning and transfer learning may also arise.
We hope that this framework influences and guides practical implementations in the future.</p>
Tue, 19 Mar 2019 12:00:00 -0700
http://offconvex.github.io/2019/03/19/CURL/
http://offconvex.github.io/2019/03/19/CURL/The search for biologically plausible neural computation: A similarity-based approach<p>This is the second post in a series reviewing recent progress in designing artificial neural networks (NNs) that resemble natural NNs not just superficially, but on a deeper, algorithmic level. In addition to serving as models of natural NNs, such networks can serve as general-purpose machine learning algorithms. Respecting biological constraints, viewed naively as a handicap in developing competitive general-purpose machine learning algorithms, can instead facilitate the development of artificial NNs by restricting the search space of possible algorithms.</p>
<p>In <a href="http://www.offconvex.org/2016/11/03/MityaNN1/">the previous post</a>, we focused on the constraints that must be met for an unsupervised algorithm to be biologically plausible. For an algorithm to be implementable as a NN, it must be formulated in the online setting. In the corresponding NN, synaptic weight updates must be local, i.e. depend on the activity of only two neurons that the synapse connects. Then, we demonstrated that deriving NNs for dimensionality reduction in a conventional way - by minimizing the reconstruction error - results in multi-neuron networks with biologically implausible non-local learning rules.</p>
<p>In this post, we propose a different objective function which we term similarity matching. From this objective function, we derive an online algorithm implementable by a NN with local learning rules. Then, we introduce other similarity-based algorithms which include more biological features such as different neuron classes and nonlinear activation functions. Finally, we review similarity-matching algorithms with state-of-the-art performance.</p>
<h2 id="similarity-matching-objective-function">Similarity-matching objective function</h2>
<p>We start by stating an objective function that will be used to derive NNs for linear dimensionality reduction. Let ${\bf x}_t\in \mathbb{R}^n$, $t = 1,\ldots T$, be a set of data points (inputs) and ${\bf y}_t\in \mathbb{R}^k$, $t = 1,\ldots T$, ($k < n$) be their learned representation (outputs). The similarity of a pair of inputs, ${\bf x}$<sub><small>$t$</small></sub> and ${\bf x}$<sub><small>$t’$</small></sub>, can be defined as their dot-product, ${\bf x}$<sub><small>$t$</small></sub>${}^\top {\bf x}$<sub><small>$t’$</small></sub>. Analogously, the similarity of a pair of outputs is ${\bf y}$<sub><small>$t$</small></sub>${}^\top {\bf y}$<sub><small>$t’$</small></sub>. Similarity matching, as its name suggests, learns a representation where the similarity between each pair of outputs matches that of the corresponding inputs:</p>
<script type="math/tex; mode=display">\min_{ {\bf y}_1,\ldots,{\bf y}_T} \frac{1}{T^2} \sum_{t=1}^T \sum_{t'=1}^T \left({\bf x}_t^\top {\bf x}_{t'} - {\bf y}_t^\top {\bf y}_{t'} \right)^2. \qquad\qquad (2.1)</script>
<p>This offline objective function, <a href="https://en.wikipedia.org/wiki/Multidimensional_scaling">previously employed for multidimensional scaling</a>, is optimized by the projections of inputs onto the principal subspace of their covariance, i.e. performing PCA up to an orthogonal rotation. Moreover, <a href="http://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum">(2.1) has no local minima other than the principal subspace solution</a>.</p>
<p>The similarity-matching objective (2.1) may seem like a strange choice for deriving an online algorithm implementable by a NN. In the online setting, inputs are streamed to the algorithm sequentially and each output must be computed before the next input arrives. Yet, in (2.1), pairs of inputs and outputs from different time points interact with each other. In addition, whereas ${\bf x}_t$ and ${\bf y}_t$ could be interpreted as inputs and outputs to a network, unlike in <a href="http://www.offconvex.org/2016/11/03/MityaNN1/">the reconstruction approach (1.4)</a>, synaptic weights do not appear explicitly in (2.1).</p>
<h2 id="variable-substitution-trick">Variable substitution trick</h2>
<p>Both of the above concerns can be resolved by a <a href="https://www.researchgate.net/publication/315570715_Why_Do_Similarity_Matching_Objectives_Lead_to_HebbianAnti-Hebbian_Networks">simple math trick akin to completing the square</a>. We first focus on the cross-term in (2.1), which we call similarity alignment. By re-ordering the variables and introducing a new variable, ${\bf W} \in \mathbb{R}^{k\times n}$, we obtain:</p>
<script type="math/tex; mode=display">- \frac{1}{T^2}\sum_{t=1}^T \sum_{t'=1}^T {\bf y}_{t}^\top {\bf y}_{t'} {\bf x}_t^\top {\bf x}_{t'}= - \frac{1}{T^2}\sum_{t=1}^T {\bf y}_{t}^\top \left[ \sum_{t'=1}^T {\bf y}_{t'} {\bf x}_{t'}^\top \right ] {\bf x}_t</script>
<script type="math/tex; mode=display">\qquad \qquad \qquad \qquad\qquad \qquad \qquad \quad = \min_{ {\bf W} \in \mathbb{R}^{k\times n}} -\frac{2}{T} \sum_{t=1}^T{\bf y}_t^\top {\bf W} {\bf x}_t + {\rm Tr } {\bf W}^\top {\bf W}.\; (2.2)</script>
<p>To prove the second identity, find optimal ${\bf W}$ by taking a derivative of the expression on the right with respect to ${\bf W}$ and setting it to zero, and then substitute the optimal ${\bf W}$ back into the expression. Similarly, for the quartic ${\bf y}_t$ term in (2.1):</p>
<script type="math/tex; mode=display">\frac{1}{T^2}\sum_{t=1}^T \sum_{t'=1}^T {\bf y}_{t}^\top {\bf y}_{t'} {\bf y}_t^\top {\bf y}_{t'}= \frac{1}{T^2}\sum_{t=1}^T {\bf y}_{t}^\top \left[ \sum_{t'=1}^T {\bf y}_{t'} {\bf y}_{t'}^\top \right ] {\bf y}_t</script>
<script type="math/tex; mode=display">\qquad \qquad \qquad \qquad\qquad \qquad \qquad \quad = \max_{ {\bf M} \in \mathbb{R}^{k\times k}} \frac{2}{T} \sum_{t=1}^T{\bf y}_t^\top {\bf M} {\bf y}_t - {\rm Tr } {\bf M}^\top {\bf M}.\quad(2.3)</script>
<p>By substituting (2.2) and (2.3) into (2.1) we get:</p>
<script type="math/tex; mode=display">\min_{ {\bf W}\in \mathbb{R}^{k\times n}}\max_{ {\bf M}\in \mathbb{R}^{k\times k}} \frac{1}{T} \sum_{t=1}^T \left[2 {\rm Tr}\left({\bf W}^\top{\bf W}\right) - {\rm Tr}\left({\bf M}^\top{\bf M}\right) + \min_{ {\bf y}_t\in \mathbb{R}^{k\times 1}} l_t({\bf W},{\bf M},{\bf y}_t)\right],\; (2.4)</script>
<p>where</p>
<script type="math/tex; mode=display">l_t({\bf W},{\bf M},{\bf y}_t)=-4{\bf x}_t^\top{\bf W}{\bf y}_t + 2{\bf y}_t^\top{\bf M}{\bf y}_t.\qquad\qquad (2.5)</script>
<p>In the resulting objective function, (2.4),(2.5), optimal outputs at different time steps can be computed independently, making the problem amenable to an online algorithm. The price paid for this simplification is the appearance of the minimax optimization problem in variables, ${\bf W}$ and ${\bf M}$. Minimization over ${\bf W}$ aligns output channels with the greatest variance directions of the input and maximization over ${\bf M}$ diversifies the output channels. The competition between the two in a gradient descent/ascent algorithm results in the principal subspace projection which is <a href="https://www.researchgate.net/publication/315570715_Why_Do_Similarity_Matching_Objectives_Lead_to_HebbianAnti-Hebbian_Networks">the only stable fixed point of the corresponding dynamics</a>.</p>
<h2 id="online-algorithm-and-neural-network">Online algorithm and neural network</h2>
<p>Now, we are ready to derive an algorithm for optimizing (2.1) online. First, by minimizing (2.5) with respect to ${\bf y}_t$ while keeping ${\bf W}$ and ${\bf M}$ fixed we get the dynamics for the output variables :</p>
<script type="math/tex; mode=display">\dot{\bf y}_t={\bf W} {\bf x}_t-{\bf M} {\bf y}_t.\qquad\qquad (2.6)</script>
<p>To find ${\bf y}_t$ after the presentation of the corresponding input, ${\bf x}_t$, (2.6) is iterated until convergence.</p>
<p>After the convergence of ${\bf y}_t$ we update ${\bf W}$ and ${\bf M}$ by gradient descent of (2.2) and gradient ascent of (2.3) respectively:</p>
<script type="math/tex; mode=display">W_{ij} \leftarrow W_{ij} + \eta \left(y_ix_j-W_{ij}\right), \qquad M_{ij} \leftarrow M_{ij} + \eta \left(y_iy_j-M_{ij}\right). \qquad (2.7)</script>
<p>Algorithm (2.6),(2.7), first derived <a href="https://www.researchgate.net/publication/273003026_A_HebbianAnti-Hebbian_Neural_Network_for_Linear_Subspace_Learning_A_Derivation_from_Multidimensional_Scaling_of_Streaming_Data">here</a>, can be naturally implemented by a biologically plausible NN, Figure 1. Here, activity (firing rate) of the upstream neurons corresponds to input variables. Output variables are computed by the dynamics of activity (2.6) in a single layer of neurons. Variables ${\bf W}$ and ${\bf M}$ are represented by the weights of synapses in feedforward and lateral connections respectively. The learning rules (2.7) are local, i.e. the weight update, $\Delta W_{ij}$, for the synapse between $j^{\rm th}$ input neuron and $i^{\rm th}$ output neuron depends only on the activities, $x_j$, of $j^{\rm th}$ input neuron and, $y_i$, of $i^{\rm th}$ output neuron, and the synaptic weight. In neuroscience, learning rules (2.7) for ${\bf W}$ and ${\bf M}$ are called Hebbian and anti-Hebbian respectively.</p>
<p><img src="https://drive.google.com/uc?export=view&id=18_7ApBztw00VwUn-Y81kOC4s8XTuXVVN" alt="" /></p>
<p><em>Figure 1: A Hebbian/Anti-Hebbian network derived from similarity matching.</em></p>
<p>To summarize, starting with the similarity-matching objective, we derived a Hebbian/anti-Hebbian NN for dimensionality reduction. The minimax objective can be viewed as a zero-sum game played by the weights of feedforward and lateral connections. This demonstrates that synapses with local updates can still collectively work together to optimize a global objective. A similar, although not identical, NN was proposed by Foldiak heuristically. The advantage of our normative approach is that the offline solution is known. Although no proof of convergence exists in the online setting, algorithm (2.6),(2.7) performs well in practice.</p>
<h2 id="other-similarity-based-objectives-and-linear-networks">Other similarity-based objectives and linear networks</h2>
<p>We used the same framework to derive NNs for other computational tasks and incorporating more biological features. As the algorithm (2.6),(2.7) and the NN in Figure 1 were derived from the similarity-matching objective (2.1), they project data onto the principal subspace but do not necessarily recover principal components <em>per se</em>. To derive PCA algorithms we modified the objective function (2.1), <a href="https://arxiv.org/abs/1511.09468">here</a> and <a href="https://arxiv.org/abs/1810.06966">here</a>, to encourage orthogonality of ${\bf W}$. Such algorithms are implemented by NNs of the same architecture as in Figure 1 but with slightly different learning rules.</p>
<p>Although the similarity-matching NN in Figure 1 relies on biologically plausible local learning rules, it lacks biological realism in several other ways. For example, computing output requires recurrent activity that must settle faster than the time scale of the input variation, which is unlikely in biology. To respect this biological constraint, <a href="https://arxiv.org/abs/1810.06966">we modified</a> the dimensionality reduction algorithm to avoid recurrency.</p>
<p>Another non-biological feature of the NN in Figure 1 is that the output neurons compete with each other by communicating via lateral connections. In biology, such interactions are not direct but mediated by interneurons. To reflect these observations, we modified the objective function by introducing a whitening constraint:</p>
<script type="math/tex; mode=display">\min_{ {\bf y}_1,\ldots,{\bf y}_T} - \frac{1}{T^2}\sum_{t=1}^T \sum_{t'=1}^T {\bf y}_{t}^\top {\bf y}_{t'} {\bf x}_t^\top {\bf x}_{t'}, \qquad {\rm s.t.} \quad \frac 1T \sum_{t=1}^T {\bf y}_t {\bf y}_t^\top = {\bf I}_k, \qquad (2.8)</script>
<p>where ${\bf I}$<sub><small>$k$</small></sub> is the $k$-by-$k$ identity matrix. Then, by representing the whitening constraint using Lagrange relaxation, <a href="http://papers.nips.cc/paper/5885-a-normative-theory-of-adaptive-dimensionality-reduction-in-neural-networks">we derived NNs</a> where interneurons appear naturally - their activity is modeled by the Lagrange multipliers, ${\bf z} _t^\top {\bf z} _{t’}$ (Figure 2):</p>
<script type="math/tex; mode=display">\min_{ {\bf y}_1,\ldots,{\bf y}_T} \max_{ {\bf z}_1,\ldots,{\bf z}_T } - \frac{1}{T^2}\sum_{t=1}^T \sum_{t'=1}^T {\bf y}_{t}^\top {\bf y}_{t'} {\bf x}_t^\top {\bf x}_{t'} + \frac{1}{T^2}\sum_{t=1}^T \sum_{t'=1}^T {\bf z}_{t}^\top {\bf z}_{t'} \left({\bf y}_{t}^\top {\bf y}_{t'} -\delta_{t,t'}\right). \; (2.9)</script>
<p>Notice how (2.9) contains the ${\bf y}$-${\bf z}$ similarity-alignment term similar to (2.2). We can now derive learning rules for the ${\bf y}$-${\bf z}$ connections using the variable substitution trick, leading to the network in Figure 2. For details of this and other NN derivations, see <a href="http://papers.nips.cc/paper/5885-a-normative-theory-of-adaptive-dimensionality-reduction-in-neural-networks">here</a>.</p>
<p><img src="https://drive.google.com/uc?export=view&id=1tYmjxDN2SUZY-8--uV_qGTb_gEIg0O8C" alt="" /></p>
<p><em>Figure 2: A biologically-plausible NN for whitening inputs, derived from a constrained similarity-alignment cost function.</em></p>
<h2 id="nonnegative-similarity-matching-objective-and-a-nonlinear-network">Nonnegative similarity-matching objective and a nonlinear network</h2>
<p>So far, we considered similarity-based NNs with linear neurons. However, biological neurons are not linear and many interesting computations require nonlinearity. A resolution to this discrepancy was suggested by the observation that the output of biological neurons is nonnegative (firing rate cannot be below zero). Hence, we modified the optimization problem by requiring that the output of the similarity-matching cost function (2.1) is nonnegative:</p>
<script type="math/tex; mode=display">\min_{ {\bf y}_1,\ldots,{\bf y}_T \geq 0} \frac{1}{T^2} \sum_{t=1}^T \sum_{t'=1}^T \left({\bf x}_t^\top {\bf x}_{t'} - {\bf y}_t^\top {\bf y}_{t'} \right)^2. \qquad\qquad (2.10)</script>
<p>Solutions of the optimization problem (2.10) are very different from PCA: They can <a href="https://arxiv.org/abs/1503.00680">cluster well-segregated data and extract sparse features from data</a>. Understanding the nature of these solutions will be the topic of the next post. For now, we note that (2.10) can be solved by the same online algorithm as (2.1) except that the output variables are projected onto the nonnegative domain. Such algorithm maps onto the same network as Figure 1 but with rectifying neurons (ReLUs), Figure 3A.</p>
<p><img src="https://drive.google.com/uc?export=view&id=1QahRt7mzirInGnmuSJh-4R4b88q9gyRq" alt="" /></p>
<p><em>Figure 3: A) A nonlinear Hebbian/Anti-Hebbian network derived from nonnegative similarity matching. B) Stacked network for NICA. NSM - nonnegative similarity-matching.</em></p>
<p>Another problem solved by similarity-based networks is <a href="https://www.researchgate.net/publication/317300210_Blind_Nonnegative_Source_Separation_Using_Biological_Neural_Networks">the nonnegative independent component analysis</a> (NICA) which can be used for blind source separation. The problem is to recover independent and nonnegative sources from observing only their linear mixture. <a href="https://www.researchgate.net/publication/3342750_Conditions_for_nonnegative_independent_component_analysis">Plumbley showed</a> that NICA can be solved in two steps, Figure 4. First, whiten the data to obtain an orthogonal rotation of the sources. Second, find an orthogonal rotation of the whitened sources that yields a nonnegative output, Figure 4. The first step can be implemented by the whitening network in Figure 2. The second step can be implemented by the nonnegative similarity-matching network, Figure 3A, because an orthogonal rotation does not affect dot-product similarities. Therefore, <a href="https://www.researchgate.net/publication/317300210_Blind_Nonnegative_Source_Separation_Using_Biological_Neural_Networks">NICA is solved by stacking the whitening and the nonnegative similarity-matching networks</a>, Figure 3B.</p>
<p><img src="https://drive.google.com/uc?export=view&id=16uJTV1QQpwcZcueXLCqNJieKzyWk0ZYd" alt="" /></p>
<p><em>Figure 4: Illustration of the nonnegative independent component analysis algorithm. Two source channels (left) are linearly transformed to a two-dimensional mixture, which are the inputs to the algorithm (middle). Whitening (right) yields an orthogonal rotation of the sources. Sources are then recovered by solving the nonnegative similarity-matching problem. Green and red plus signs track two source vectors across mixing and whitening stages.</em></p>
<h2 id="similarity-based-algorithms-as-general-purpose-tools">Similarity-based algorithms as general-purpose tools</h2>
<p>While the derivation of the similarity-matching algorithm was motivated by constraints imposed by biology, the resulting algorithm performs well on large-scale data. <a href="https://arxiv.org/abs/1808.02083">A recent paper</a> introduced an efficient modification of the similarity-matching algorithm and demonstrated its competitiveness with the state-of-the-art principal subspace projection algorithms in both processing speed and convergence rate. A package with implementations of these algorithms is <a href="https://github.com/flatironinstitute/online_psp">here</a> and <a href="https://github.com/flatironinstitute/online_psp_matlab">here</a>.</p>
<p>In this blog post, we introduced linear and non-linear similarity-matching NNs that can serve as models of biological NNs and as general-purpose machine-learning tools. In the next post, we will discuss the nature of the solutions to nonnegative similarity-based networks.</p>
Mon, 03 Dec 2018 15:30:00 -0800
http://offconvex.github.io/2018/12/03/MityaNN2/
http://offconvex.github.io/2018/12/03/MityaNN2/Understanding optimization in deep learning by analyzing trajectories of gradient descent<p>Neural network optimization is fundamentally non-convex, and yet simple gradient-based algorithms seem to consistently solve such problems.
This phenomenon is one of the central pillars of deep learning, and forms a mystery many of us theorists are trying to unravel.
In this post I’ll survey some recent attempts to tackle this problem, finishing off with a discussion on my <a href="https://arxiv.org/pdf/1810.02281.pdf">new paper with Sanjeev Arora, Noah Golowich and Wei Hu</a>, which for the case of gradient descent over deep linear neural networks, provides a guarantee for convergence to global minimum at a linear rate.</p>
<h2 id="landscape-approach-and-its-limitations">Landscape Approach and Its Limitations</h2>
<p>Many papers on optimization in deep learning implicitly assume that a rigorous understanding will follow from establishing geometric properties of the loss <em>landscape</em>, and in particular, of <em>critical points</em> (points where the gradient vanishes).
For example, through an analogy with the spherical spin-glass model from condensed matter physics, <a href="http://proceedings.mlr.press/v38/choromanska15.pdf">Choromanska et al. 2015</a> argued for what has become a colloquial conjecture in deep learning:</p>
<blockquote>
<p><strong>Landscape Conjecture:</strong>
In neural network optimization problems, suboptimal critical points are very likely to have negative eigenvalues to their Hessian.
In other words, there are almost <em>no poor local minima</em>, and nearly all <em>saddle points are strict</em>.</p>
</blockquote>
<p>Strong forms of this conjecture were proven for loss landscapes of various simple problems involving <strong>shallow</strong> (two layer) models, e.g. <a href="https://papers.nips.cc/paper/6271-global-optimality-of-local-search-for-low-rank-matrix-recovery.pdf">matrix sensing</a>, <a href="https://papers.nips.cc/paper/6048-matrix-completion-has-no-spurious-local-minimum.pdf">matrix completion</a>, <a href="http://proceedings.mlr.press/v40/Ge15.pdf">orthogonal tensor decomposition</a>, <a href="https://arxiv.org/pdf/1602.06664.pdf">phase retrieval</a>, and <a href="http://proceedings.mlr.press/v80/du18a/du18a.pdf">neural networks with quadratic activation</a>.
There was also work on establishing convergence of gradient descent to global minimum when the Landscape Conjecture holds, as described in the excellent posts on this blog by <a href="http://www.offconvex.org/2016/03/22/saddlepoints/">Rong Ge</a>, <a href="http://www.offconvex.org/2016/03/24/saddles-again/">Ben Recht</a> and <a href="http://www.offconvex.org/2017/07/19/saddle-efficiency/">Chi Jin and Michael Jordan</a>.
They describe how gradient descent can arrive at a second order local minimum (critical point whose Hessian is positive semidefinite) by escaping all strict saddle points, and how this process is efficient given that perturbations are added to the algorithm.
Note that under the Landscape Conjecture, i.e. when there are no poor local minima and non-strict saddles, second order local minima are also global minima.</p>
<p style="text-align:center;">
<img src="/assets/optimization-beyond-landscape-points.png" width="100%" alt="Local minima and saddle points" />
</p>
<p>However, it has become clear that the landscape approach (and the Landscape Conjecture) cannot be applied as is to <strong>deep</strong> (three or more layer) networks, for several reasons.
First, deep networks typically induce non-strict saddles (e.g. at the point where all weights are zero, see <a href="https://papers.nips.cc/paper/6112-deep-learning-without-poor-local-minima.pdf">Kawaguchi 2016</a>).
Second, a landscape perspective largely ignores algorithmic aspects that empirically are known to greatly affect convergence with deep networks — for example the <a href="http://proceedings.mlr.press/v28/sutskever13.html">type of initialization</a>, or <a href="http://proceedings.mlr.press/v37/ioffe15.pdf">batch normalization</a>.
Finally, as I argued in my <a href="http://www.offconvex.org/2018/03/02/acceleration-overparameterization/">previous blog post</a>, based upon <a href="http://proceedings.mlr.press/v80/arora18a/arora18a.pdf">work with Sanjeev Arora and Elad Hazan</a>, adding (redundant) linear layers to a classic linear model can sometimes accelerate gradient-based optimization, without any gain in expressiveness, and despite introducing non-convexity to a formerly convex problem.
Any landscape analysis that relies on properties of critical points alone will have difficulty explaining this phenomenon, as through such lens, nothing is easier to optimize than a convex objective with a single critical point which is the global minimum.</p>
<h2 id="a-way-out">A Way Out?</h2>
<p>The limitations of the landscape approach for analyzing optimization in deep learning suggest that it may be abstracting away too many important details.
Perhaps a more relevant question than “is the landscape graceful?” is “what is the behavior of specific optimizer <strong>trajectories</strong> emanating from specific initializations?”.</p>
<p style="text-align:center;">
<img src="/assets/optimization-beyond-landscape-trajectories.png" width="66%" alt="Different trajectories lead to qualitatively different results" />
</p>
<p>While the trajectory-based approach is seemingly much more burdensome than landscape analyses, it is already leading to notable progress.
Several recent papers (e.g. <a href="http://proceedings.mlr.press/v70/brutzkus17a/brutzkus17a.pdf">Brutzkus and Globerson 2017</a>; <a href="https://papers.nips.cc/paper/6662-convergence-analysis-of-two-layer-neural-networks-with-relu-activation.pdf">Li and Yuan 2017</a>; <a href="http://proceedings.mlr.press/v70/zhong17a/zhong17a.pdf">Zhong et al. 2017</a>; <a href="http://proceedings.mlr.press/v70/tian17a/tian17a.pdf">Tian 2017</a>; <a href="https://openreview.net/pdf?id=rJ33wwxRb">Brutzkus et al. 2018</a>; <a href="http://proceedings.mlr.press/v75/li18a/li18a.pdf">Li et al. 2018</a>; <a href="https://arxiv.org/pdf/1806.00900.pdf">Du et al. 2018</a>; <a href="http://romaincouillet.hebfree.org/docs/conf/nips_GDD.pdf">Liao et al. 2018</a>) have adopted this strategy, successfully analyzing different types of shallow models.
Moreover, trajectory-based analyses are beginning to set foot beyond the realm of the landscape approach — for the case of linear neural networks, they have successfully established convergence of gradient descent to global minimum under <strong>arbitrary depth</strong>.</p>
<h2 id="trajectory-based-analyses-for-deep-linear-neural-networks">Trajectory-Based Analyses for Deep Linear Neural Networks</h2>
<p>Linear neural networks are fully-connected neural networks with linear (no) activation.
Specifically, a depth $N$ linear network with input dimension $d_0$, output dimension $d_N$, and hidden dimensions $d_1,d_2,\ldots,d_{N-1}$, is a linear mapping from $\mathbb{R}^{d_0}$ to $\mathbb{R}^{d_N}$ parameterized by $x \mapsto W_N W_{N-1} \cdots W_1 x$, where $W_j \in \mathbb{R}^{d_j \times d_{j-1}}$ is regarded as the weight matrix of layer $j$.
Though trivial from a representational perspective, linear neural networks are, somewhat surprisingly, complex in terms of optimization — they lead to non-convex training problems with multiple minima and saddle points.
Being viewed as a theoretical surrogate for optimization in deep learning, the application of gradient-based algorithms to linear neural networks is receiving significant attention these days.</p>
<p>To my knowledge, <a href="https://arxiv.org/pdf/1312.6120.pdf">Saxe et al. 2014</a> were the first to carry out a trajectory-based analysis for deep (three or more layer) linear networks, treating gradient flow (gradient descent with infinitesimally small learning rate) minimizing $\ell_2$ loss over whitened data.
Though a very significant contribution, this analysis did not formally establish convergence to global minimum, nor treat the aspect of computational complexity (number of iterations required to converge).
The recent work of <a href="http://proceedings.mlr.press/v80/bartlett18a.html">Bartlett et al. 2018</a> makes progress towards addressing these gaps, by applying a trajectory-based analysis to gradient descent for the special case of linear residual networks, i.e. linear networks with uniform width across all layers ($d_0=d_1=\cdots=d_N$) and identity initialization ($W_j=I$, $\forall j$).
Considering different data-label distributions (which boil down to what they refer to as “targets”), Bartlett et al. demonstrate cases where gradient descent provably converges to global minimum at a linear rate — loss is less than $\epsilon>0$ from optimum after $\mathcal{O}(\log\frac{1}{\epsilon})$ iterations — as well as situations where it fails to converge.</p>
<p>In a <a href="https://arxiv.org/pdf/1810.02281.pdf">new paper with Sanjeev Arora, Noah Golowich and Wei Hu</a>, we take an additional step forward in virtue of the trajectory-based approach.
Specifically, we analyze trajectories of gradient descent for any linear neural network that does not include “bottleneck layers”, i.e. whose hidden dimensions are no smaller than the minimum between the input and output dimensions ($d_j \geq \min\{d_0,d_N\}$, $\forall j$), and prove convergence to global minimum, at a linear rate, provided that initialization meets the following two conditions:
<em>(i)</em> <em>approximate balancedness</em> — $W_{j+1}^\top W_{j+1} \approx W_j W_j^\top$, $\forall j$;
and <em>(ii)</em> <em>deficiency margin</em> — initial loss is smaller than the loss of any rank deficient solution.
We show that both conditions are necessary, in the sense that violating any one of them may lead to a trajectory that fails to converge.
Approximate balancedness at initialization is trivially met in the special case of linear residual networks, and also holds for the customary setting of initialization via small random perturbations centered at zero.
The latter also leads to deficiency margin with positive probability.
For the case $d_N=1$, i.e. scalar regression, we provide a random initialization scheme under which both conditions are met, and thus convergence to global minimum at linear rate takes place, with constant probability.</p>
<p>Key to our analysis is the observation that if weights are initialized to be approximately balanced, they will remain that way throughout the iterations of gradient descent.
In other words, trajectories taken by the optimizer adhere to a special characterization:</p>
<blockquote>
<p><strong>Trajectory Characterization:</strong> <br /></p>
<p><script type="math/tex">W_{j+1}^\top(t) W_{j+1}(t) \approx W_j(t) W_j^\top(t), \quad j=1,\ldots,N-1, \quad t=0,1,\ldots</script>,</p>
</blockquote>
<p>which means that throughout the entire timeline, all layers have (approximately) the same set of singular values, and the left singular vectors of each layer (approximately) coincide with the right singular vectors of the layer that follows.
We show that this regularity implies steady progress for gradient descent, thereby demonstrating that even in cases where the loss landscape is complex as a whole (includes many non-strict saddle points), it may be particularly well-behaved around the specific trajectories taken by the optimizer.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Tackling the question of optimization in deep learning through the landscape approach, i.e. by analyzing the geometry of the objective independently of the algorithm used for training, is conceptually appealing.
However this strategy suffers from inherent limitations, predominantly as it requires the entire objective to be graceful, which seems to be too strict of a demand.
The alternative approach of taking into account the optimizer and its initialization, and focusing on the landscape only along the resulting trajectories, is gaining more and more traction.
While landscape analyses have thus far been limited to shallow (two layer) models only, the trajectory-based approach has recently treated arbitrarily deep models, proving convergence of gradient descent to global minimum at a linear rate.
Much work however remains to be done, as this success covered only linear neural networks.
I expect the trajectory-based approach to be key in developing our formal understanding of gradient-based optimization for deep non-linear networks as well.</p>
<p><a href="http://www.cohennadav.com/">Nadav Cohen</a></p>
Wed, 07 Nov 2018 04:00:00 -0800
http://offconvex.github.io/2018/11/07/optimization-beyond-landscape/
http://offconvex.github.io/2018/11/07/optimization-beyond-landscape/Simple and efficient semantic embeddings for rare words, n-grams, and language features<p>Distributional methods for capturing meaning, such as word embeddings, often require observing many examples of words in context. But most humans can infer a reasonable meaning from very few or even a single occurrence. For instance, if we read “Porgies live in shallow temperate marine waters,” we have a good idea that a <em>porgy</em> is a fish. Since language corpora often have a long tail of “rare words,” it is an interesting problem to imbue NLP algorithms with this capability. This is especially important for n-grams (i.e., ordered n-tuples of words, like “ice cream”), many of which occur rarely in the corpus.</p>
<p>Here we describe a simple but principled approach called <em>à la carte</em> embeddings, described in our <a href="http://aclweb.org/anthology/P18-1002">ACL’18 paper</a> with Yingyu Liang, Tengyu Ma, and Brandon Stewart. It also easily extends to learning embeddings of arbitrary language features such as word-senses and $n$-grams. The paper also combines these with our recent <a href="http://www.offconvex.org/2018/06/25/textembeddings/">deep-learning-free text embeddings</a> to get simple deep-learning free text embeddings with even better performance on downstream classification tasks, quite competitive with deep learning approaches.</p>
<h2 id="inducing-word-embedding-from-their-contexts-a-surprising-linear-relationship">Inducing word embedding from their contexts: a surprising linear relationship</h2>
<p>Suppose a single occurrence of a word $w$ is surrounded by a sequence $c$ of words. What is a reasonable guess for the word embedding $v_w$ of $w$? For convenience, we will let $u_w^c$ denote the average of the word embeddings of words in $c$. Anybody who knows the word2vec method may reasonably guess the following.</p>
<blockquote>
<p><strong>Guess 1:</strong> Up to scaling, $u_w^c$ is a good estimate for $v_w$.</p>
</blockquote>
<p>Unfortunately, this totally fails. Even taking thousands of occurrences of $w$, the average of such estimates stays far from the ground truth embedding $v_w$. The following discovery should therefore be surprising (read below for a theoretical justification):</p>
<blockquote>
<p><strong>Theorem 1</strong> (From <a href="https://transacl.org/ojs/index.php/tacl/article/view/1346">this TACL’18 paper</a>): There is a single matrix $A$ (depending only upon the text corpus) such that $A u_w^c$ is a good estimate for $v_w$.</p>
</blockquote>
<p>Note that the best such $A$ can be found via linear regression by minimizing the average $|Au_w^c -v_w|^2$ over occurrences of frequent words $w$, for which we already have word embeddings.</p>
<p>Once such an $A$ has been learnt from frequent words, the induction of embeddings for new words works very well. As we receive more and more occurrences of $w$ the average of $Au_w^c$ over all sentences containing $w$ has cosine similarity $>0.9$ with the true word embedding $v_w$ (this holds for GloVe as well as word2vec).</p>
<p>Thus the learnt $A$ gives a way to induce embeddings for new words from a few or even a single occurrence. We call this the <em>à la carte</em> embedding of $w$, because we don’t need to pay the <em>prix fixe</em> of re-running GloVe or word2vec on the entire corpus each time a new word is needed.</p>
<h3 id="testing-embeddings-for-rare-words">Testing embeddings for rare words</h3>
<p>Using Stanford’s <a href="https://nlp.stanford.edu/~lmthang/morphoNLM/">Rare Words</a> dataset we created the
<a href="http://nlp.cs.princeton.edu/CRW/"><em>Contextual Rare Words</em></a> dataset where, along with word pairs and human-rated scores, we also provide contexts (i.e., few usages) for the rare words.</p>
<p>We compare the performance of our method with alternatives such as <a href="http://www.offconvex.org/2018/06/17/textembeddings/">top singular component removal and frequency down-weighting</a> and find that <em>à la carte</em> embedding consistently outperforms other methods and requires far fewer contexts to match their best performance.
Below we plot the increase in Spearman correlation with human ratings as the tested algorithms are given more samples of the words in context. We see that given only 8 occurences of the word, the <em>a la carte</em> method outperforms other baselines that’re given 128 occurences.</p>
<p style="text-align:center;">
<img src="/assets/ALCcrwplot.svg" width="60%" />
</p>
<p>Now we turn to the task mentioned in the opening para of this post. <a href="http://aclweb.org/anthology/D17-1030">Herbelot and Baroni</a> constructed a “nonce” dataset consisting of single-word concepts and their Wikipedia definitions, to test algorithms that “simulate the process by which a competent speaker encounters a new word in known contexts.” They tested various methods, including a modified version of word2vec.
As we show in the table below, <em>à la carte</em> embedding outperforms all their methods in terms of the average rank of the target vector’s similarity with the constructed vector. The true word embedding is among the closest 165 or so word vectors to our embedding.
(Note that the vocabulary size exceeds 200K, so this is considered a strong performance.)</p>
<p style="text-align:center;">
<img src="/assets/ALCnonce.svg" width="50%" />
</p>
<h2 id="a-theory-of-induced-embeddings-for-general-features">A theory of induced embeddings for general features</h2>
<p>Why should the matrix $A$ mentioned above exist in the first place?
Sanjeev, Yingyu, and Tengyu’s <a href="https://transacl.org/ojs/index.php/tacl/article/view/1346">TACL’18</a> paper together with Yuanzhi Li and Andrej Risteski gives a justification via a latent-variable model of corpus generation that is a modification of their earlier model described in <a href="https://transacl.org/ojs/index.php/tacl/article/view/742">TACL’16</a> (see also this <a href="http://www.offconvex.org/2016/02/14/word-embeddings-2/">blog post</a>) The basic idea is to consider a random walk over an ellipsoid instead of the unit square.
Under this modification of the rand-walk model, whose approximate MLE objective is similar to that of GloVe, their first theorem shows the following:</p>
<script type="math/tex; mode=display">\exists~A\in\mathbb{R}^{d\times d}\textrm{ s.t. }v_w=A\mathbb{E} \left[\frac{1}{n}\sum\limits_{w'\in c}v_{w'}\bigg|w\in c\right]=A\mathbb{E}v_w^\textrm{avg}~\forall~w</script>
<p>where the expectation is taken over possible contexts $c$.</p>
<p>This result also explains the linear algebraic structure of the embeddings of polysemous words (words having multiple possible meanings, such as <em>tie</em>) discussed in an earlier <a href="http://www.offconvex.org/2016/07/10/embeddingspolysemy/">post</a>.
Assuming for simplicity that $tie$ only has two meanings (<em>clothing</em> and <em>game</em>), it is easy to see that its word embedding is a linear transformation of the sum of the average context vectors of its two senses:</p>
<script type="math/tex; mode=display">v_w=A\mathbb{E}v_w^\textrm{avg}=A\mathbb{E}\left[v_\textrm{clothing}^\textrm{avg}+v_\textrm{game}^\textrm{avg}\right]=A\mathbb{E}v_\textrm{clothing}^\textrm{avg}+A\mathbb{E}v_\textrm{game}^\textrm{avg}</script>
<p>The above also shows that we can get a reasonable estimate for the vector of the sense <em>clothing</em>, and, by extension many other features of interest, by setting $v_\textrm{clothing}=A\mathbb{E}v_\textrm{clothing}^\textrm{avg}$.
Note that this linear method also subsumes other context representations, such as removing the <a href="http://www.offconvex.org/2018/06/17/textembeddings/">top singular component or down-weighting frequent directions</a>.</p>
<h3 id="n-gram-embeddings">$n$-gram embeddings</h3>
<p>While the theory suggests existence of a linear transform between word embeddings and their context embeddings, one could also use this linear transform to induce embeddings for other kinds of linguistic features in context.
We test this hypothesis by inducing embeddings for $n$-grams by using contexts from a large text corpus and word embeddings trained on the same corpus.
A qualitative evaluation of the $n$-gram embeddings is done by finding the closest words to it in terms of cosine similarity between the embeddings.
As evident from the below figure, <em>à la carte</em> bigram embeddings capture the meaning of the phrase better than some other compositional and learned bigram embeddings.</p>
<p style="text-align:center;">
<img src="/assets/ALCngram_quality.png" width="65%" />
</p>
<h3 id="sentence-embeddings">Sentence embeddings</h3>
<p>We also use these $n$-gram embeddings to construct sentence embeddings, similarly to <a href="http://www.offconvex.org/2018/06/25/textembeddings/">DisC embeddings</a>, to evaluate on classification tasks.
A sentence is embedded as the concatenation of sums of embeddings for $n$-gram in the sentence for use in downstream classification tasks.
Using this simple approach we can match the performance of other linear and LSTM representations, even obtaining state-of-the-art results on some of them. Note that Logeswaran and Lee is a contemporary paper that uses deep nets.</p>
<p style="text-align:center;">
<img src="/assets/ALCngram_clf.svg" width="80%" />
</p>
<h2 id="discussion">Discussion</h2>
<p>Our <em>à la carte</em> method is simple, almost elementary, and yet gives results competitive with many other feature embedding methods and also beats them in many cases.
Can one do zero-shot learning of word embeddings, i.e. inducing embeddings for a words/features without any context?
Character level methods such as <a href="https://fasttext.cc/">fastText</a> can do this and it is a good problem to incorporate character level information into the <em>à la carte</em> approach (the few things we tried didn’t work so far).</p>
<p>The <em>à la carte</em> code is <a href="https://github.com/NLPrinceton/ALaCarte">available here</a>, allowing you to re-create the results described.</p>
Tue, 18 Sep 2018 02:00:00 -0700
http://offconvex.github.io/2018/09/18/alacarte/
http://offconvex.github.io/2018/09/18/alacarte/When Recurrent Models Don't Need to be Recurrent<p>In the last few years, deep learning practitioners have proposed a litany of
different sequence models. Although recurrent neural networks were once the
tool of choice, now models like the autoregressive
<a href="https://deepmind.com/blog/wavenet-generative-model-raw-audio/">Wavenet</a> or the
<a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Transformer</a>
are replacing RNNs on a diverse set of tasks. In this post, we explore the
trade-offs between recurrent and feed-forward models. Feed-forward models can
offer improvements in training stability and speed, while recurrent models are
strictly more expressive. Intriguingly, this added expressivity does not seem to
boost the performance of recurrent models. Several groups have shown
feed-forward networks can match the results of the best recurrent models on
benchmark sequence tasks. This phenomenon raises an interesting question for
theoretical investigation:</p>
<blockquote>
<p>When and why can feed-forward networks replace recurrent neural networks
without a loss in performance?</p>
</blockquote>
<p>We discuss several proposed answers to this question and highlight our
<a href="https://arxiv.org/abs/1805.10369">recent work</a> that offers an explanation in
terms of a fundamental stability property.</p>
<h1 id="a-tale-of-two-sequence-models">A Tale of Two Sequence Models</h1>
<h2 id="recurrent-neural-networks">Recurrent Neural Networks</h2>
<p>The many variants of recurrent models all have a similar form. The model
maintains a state $h_t$ that summarizes the past sequence of inputs. At each
time step $t$, the state is updated according to the equation
[
h_{t+1} = \phi(h_t, x_t),
]
where $x_t$ is the input at time $t$, $\phi$ is a differentiable map, and $h_0$
is an initial state. In a vanilla recurrent neural network, the model is
parameterized by matrices $W$ and $U$, and the state is updated according to
[
h_{t+1} = \tanh(Wh_t + Ux_t).
]
In practice, the <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Long Short-Term Memory
(LSTM)</a> network is
more frequently used. In either case, to make predictions, the state is passed
to a function $f$, and the model predicts $y_t = f(h_t)$. Since the state $h_t$
is a function of all of the past inputs $x_0, \dots, x_t$, the prediction $y_t$
depends on the entire history $x_0, \dots, x_t$ as well.</p>
<p>A recurrent model can also be represented graphically.</p>
<p style="text-align:center;">
<img src="/assets/approx_recurrent/recurrent_net.png" width="500px" height="250px" />
</p>
<p>Recurrent models are fit to data using backpropagation. However, backpropagating
gradients from time step $T$ to time step $0$ often requires infeasibly large
amounts of memory, so essentially every implementation of a recurrent model
<em>truncates</em> the model and only backpropagates gradient $k$ times steps.</p>
<figure>
<p style="text-align:center;">
<img src="/assets/approx_recurrent/truncated_backprop.png" />
</p>
<figcaption>
<small>
Source: <a href="https://r2rt.com/styles-of-truncated-backpropagation.html">
https://r2rt.com/styles-of-truncated-backpropagation.html </a>
</small>
</figcaption>
</figure>
<p>In this setup, the predictions of the recurrent model still depend on the entire
history $x_0, \dots, x_T$. However, it’s not clear how this training procedure
affects the model’s ability to learn long-term patterns, particularly those that
require more than $k$ steps.</p>
<h2 id="autoregressive-feed-forward-models">Autoregressive, Feed-Forward Models</h2>
<p>Instead of making predictions from a state that depends on the entire history,
an autoregressive model directly predicts $y_t$ using only the $k$ most recent
inputs, $x_{t-k+1}, \dots, x_{t}$. This corresponds to a strong <em>conditional
independence</em> assumption. In particular, a feed-forward model assumes the target
only depends on the $k$ most recent inputs. Google’s
<a href="https://arxiv.org/abs/1609.03499">WaveNet</a> nicely illustrates this general
principle.</p>
<figure>
<p style="text-align:center;">
<img src="https://storage.googleapis.com/deepmind-live-cms/documents/BlogPost-Fig2-Anim-160908-r01.gif" />
</p>
<figcaption>
<small>
Source: <a href="https://deepmind.com/blog/wavenet-generative-model-raw-audio/">
https://deepmind.com/blog/wavenet-generative-model-raw-audio/</a>
</small>
</figcaption>
</figure>
<p>In contrast to an RNN, the limited context of a feed-forward model means that it
cannot capture patterns that extend more than $k$ steps. However, using
techniques like dilated-convolutions, one can make $k$ quite large.</p>
<h1 id="why-care-about-feed-forward-models">Why Care About Feed-Forward Models?</h1>
<p>At the outset, recurrent models appear to be a strictly more flexible and
expressive model class than feed-forward models. After all, feed-forward
networks make a strong conditional independence assumption that recurrent models
don’t make. Even if feed-forward models are less expressive, there are still
several reasons one might prefer a feed-forward network.</p>
<ul>
<li><strong>Parallelization</strong>: Convolutional feed-forward models are easier to <a href="https://arxiv.org/abs/1705.03122">parallelize
at training time</a>.
There’s no hidden state to update and maintain, and
therefore no sequential dependencies between outputs. This allows very
efficient implementations of training on modern hardware.</li>
<li><strong>Trainability</strong>: Training deep convolutional neural networks is the
bread-and-butter of deep learning. Whereas recurrent models are often more
finicky and difficult to <a href="https://arxiv.org/abs/1211.5063">optimize</a>,
significant effort has gone into designing architectures and software to
efficiently and reliably train deep feed-forward networks.</li>
<li><strong>Inference Speed</strong>: In some cases, feed-forward models can be significantly
more light-weight and perform <a href="https://arxiv.org/abs/1211.5063">inference faster than similar recurrent
systems</a>. In other cases,
particularly for long sequences, autoregressive inference is a large
bottleneck and requires <a href="https://arxiv.org/abs/1702.07825">significant engineering
work</a> or <a href="https://arxiv.org/abs/1711.10433">significant
cleverness</a> to overcome.</li>
</ul>
<h1 id="feed-forward-models-can-outperform-recurrent-models">Feed-Forward Models Can Outperform Recurrent Models</h1>
<p>Although it appears trainability and parallelization for feed-forward models
comes at the price of reduced accuracy, there have been several recent examples
showing that feed-forward networks can actually achieve the same accuracies as
their recurrent counterparts on benchmark tasks.</p>
<ul>
<li>
<p><strong>Language Modeling.</strong>
In language modeling, the goal is to predict the next word in a document given
all of the previous words. Feed-forward models make predictions using only the
$k$ most recent words, whereas recurrent models can potentially use the entire
document. The <a href="https://arxiv.org/abs/1612.08083">Gated-Convolutional Language
Model</a> is a feed-forward autoregressive models
that is competitive with <a href="https://arxiv.org/abs/1602.02410">large LSTM baseline
models</a>. Despite using a truncation length of
$k=25$, the model outperforms a large LSTM on the
<a href="https://einstein.ai/research/the-wikitext-long-term-dependency-language-modeling-dataset">Wikitext-103</a>
benchmark, which is designed to reward models that capture long-term
dependencies. On the <a href="http://www.statmt.org/lm-benchmark/">Billion Word
Benchmark</a>, the model is slightly worse
than the largest LSTM, but is faster to train and uses fewer resources.</p>
</li>
<li>
<p><strong>Machine Translation.</strong>
The goal in machine translation is to map sequences of English words to
sequences of, say, French words. Feed-forward models make translations using
only $k$ words of the sentence, whereas recurrent models can leverage the entire
sentence. Within the deep learning world, variants of the LSTM-based <a href="https://arxiv.org/abs/1409.0473">Sequence
to Sequence with Attention</a> model, particularly
<a href="https://arxiv.org/abs/1609.08144">Google Neural Machine Translation</a>, were
superseded first by a fully <a href="https://arxiv.org/abs/1705.03122">convolutional sequence to
sequence</a> model and then by the
<a href="https://arxiv.org/abs/1706.03762">Transformer</a>.<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></p>
</li>
</ul>
<figure>
<p style="text-align:center;">
<img src="https://raw.githubusercontent.com/facebookresearch/fairseq/master/fairseq.gif" />
</p>
<figcaption>
<small>
Source: <a href="https://github.com/facebookresearch/fairseq/blob/master/fairseq.gif">
https://github.com/facebookresearch/fairseq/blob/master/fairseq.gif </a>
</small>
</figcaption>
</figure>
<ul>
<li>
<p><strong>Speech Synthesis.</strong>
In speech synthesis, one seeks to generate a realistic human speech signal.
Feed-forward models are limited to the past $k$ samples, whereas recurrent
models can use the entire history. Upon publication, the feed-forward,
autoregressive <a href="https://arxiv.org/abs/1609.03499">WaveNet</a> was a substantial
improvement over LSTM-RNN parametric models.</p>
</li>
<li>
<p><strong>Everthing Else.</strong>
Recently <a href="https://arxiv.org/abs/1803.01271">Bai et al.</a> proposed a generic
feed-forward model leveraging dilated convolutions and showed it outperforms
recurrent baselines on tasks ranging from synthetic copying tasks to music
generation.</p>
</li>
</ul>
<h1 id="how-can-feed-forward-models-outperform-recurrent-ones">How Can Feed-Forward Models Outperform Recurrent Ones?</h1>
<p>In the examples above, feed-forward networks achieve results on par with or
better than recurrent networks. This is perplexing since recurrent models
seem to be more powerful a priori. One explanation for this phenomenon is
given by <a href="https://arxiv.org/abs/1612.08083">Dauphin et al.</a>:</p>
<blockquote>
<p>The unlimited context offered by recurrent models is not strictly necessary
for language modeling.</p>
</blockquote>
<p>In other words, it’s possible you don’t need a large amount of context to do
well on the prediction task on average. <a href="https://arxiv.org/abs/1612.02526">Recent theoretical
work</a> offers some evidence in favor of this view.</p>
<p>Another explanation is given by <a href="https://arxiv.org/abs/1803.01271">Bai et al.</a>:</p>
<blockquote>
<p>The “infinite memory” advantage of RNNs is largely absent in practice.</p>
</blockquote>
<p>As Bai et al. report, even in experiments explicitly requiring long-term
context, RNN variants were unable to learn long sequences. On the Billion Word
Benchmark, an <a href="https://arxiv.org/abs/1703.10724">intriguing Google Technical
Report</a> suggests an LSTM $n$-gram model with
$n=13$ words of memory is as good as an LSTM with arbitrary context.</p>
<p>This evidence leads us to conjecture: <strong>Recurrent models <em>trained in practice</em>
are effectively feed-forward.</strong> This could happen either because truncated
backpropagation time cannot learn patterns significantly longer than $k$ steps,
or, more provocatively, because models <em>trainable by gradient descent</em> cannot
have long-term memory.</p>
<p>In <a href="https://arxiv.org/abs/1805.10369">our recent paper</a>, we study the gap
between recurrent and feed-forward models trained using gradient descent. We
show if the recurrent model is <em>stable</em> (meaning the gradients can not explode),
then the model can be well-approximated by a feed-forward network for the
purposes of both <em>inference and training.</em> In other words, we show feed-forward
and stable recurrent models trained by gradient descent are <em>equivalent</em> in the
sense of making identical predictions at test-time. Of course, not all models
trained in practice are stable. We also give empirical evidence the stability
condition can be imposed on certain recurrent models without loss in
performance.</p>
<h1 id="conclusion">Conclusion</h1>
<p>Despite some initial attempts, there is still much to do to understand
why feed-forward models are competitive with recurrent ones and
shed light onto the trade-offs between sequence models. How much memory is
really needed to perform well on common sequence benchmarks? What are the
expressivity trade-offs between truncated RNNs (which can be considered
feed-forward) and the convolutional models that are in popular use? Why can
feed-forward networks perform as well as unstable RNNs in practice?</p>
<p>Answering these questions is a step towards building a theory that can both
explain the strengths and limitations of our current methods and give guidance
about how to choose between different classes of models in concrete settings.</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>The Transformer isn’t strictly a feed-forward model in the style described above (since it doesn’t make the $k$ step conditional independence assumption), but is not really a recurrent model because it doesn’t maintain a hidden state. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
Fri, 27 Jul 2018 01:00:00 -0700
http://offconvex.github.io/2018/07/27/approximating-recurrent/
http://offconvex.github.io/2018/07/27/approximating-recurrent/Deep-learning-free Text and Sentence Embedding, Part 2<p>This post continues <a href="http://www.offconvex.org/2018/06/17/textembeddings/">Sanjeev’s post</a> and describes further attempts to construct elementary and interpretable text embeddings.
The previous post described the <a href="https://openreview.net/pdf?id=SyK00v5xx">the SIF embedding</a>, which uses a simple weighted combination of word embeddings combined with some mild “denoising” based upon singular vectors, yet outperforms many deep learning based methods, including <a href="https://arxiv.org/pdf/1506.06726.pdf">Skipthought</a>, on certain downstream NLP tasks such as sentence semantic similarity and entailment.
See also this <a href="http://nlp.town/blog/sentence-similarity/">independent study by Yves Peirsman</a>.</p>
<p>However, SIF embeddings embeddings ignore word order (similar to classic <em>Bag of Words</em> models in NLP), which leads to unexciting performance on many other downstream classification tasks.
(Even the denoising via SVD, which is crucial in similarity tasks, can sometimes reduces performance on other tasks.)
Can we design a text embedding with the simplicity and transparency of SIF while also incorporating word order information?
Our <a href="https://openreview.net/pdf?id=B1e5ef-C-">ICLR’18 paper</a> with Kiran Vodrahalli does this, and achieves strong empirical performance and also some surprising theoretical guarantees stemming from the <a href="https://en.wikipedia.org/wiki/Compressed_sensing">theory of compressed sensing</a>.
It is competitive with all pre-2018 LSTM-based methods on standard tasks.
Even better, it is much faster to compute, since it uses pretrained (GloVe) word vectors and simple linear algebra.</p>
<p style="text-align:center;">
<img src="/assets/unsupervised_pipeline.png" width="50%" alt="Pipeline" />
</p>
<h2 id="incorporating-local-word-order-n-gram-embeddings">Incorporating local word order: $n$-gram embeddings</h2>
<p><em>Bigrams</em> are ordered word-pairs that appear in the sentence, and $n$-grams are ordered $n$-tuples.
A document with $k$ words has $k-1$ bigrams and $k-n+1$ $n$-grams.
The <em>Bag of n-gram (BonG) representation</em> of a document refers to a long vector whose each entry is indexed by all possible $n$-grams, and contains the number of times the corresponding $n$-gram appears in the document.
Linear classifiers trained on BonG representations are a <a href="https://www.aclweb.org/anthology/P12-2018">surprisingly strong baseline for document classification tasks</a>.
While $n$-grams don’t directly encode long-range dependencies in text, one hopes that a fair bit of such information is implicitly present.</p>
<p>A trivial idea for incorporating $n$-grams into SIF embeddings would be to treat $n$-grams like words, and compute word embeddings for them using either GloVe and word2vec.
This runs into the difficulty that the number of distinct $n$-grams in the corpus gets very large even for $n=2$ (let alone $n=3$), making it almost impossible to solve word2vec or GloVe.
Thus one gravitates towards a more <em>compositional</em> approach.</p>
<blockquote>
<p><strong>Compositional $n$-gram embedding:</strong> Represent $n$-gram $g=(w_1,\dots,w_n)$ as the element-wise product $v_g=v_{w_1}\odot\cdots\odot v_{w_n}$ of the embeddings of its constituent words.</p>
</blockquote>
<p>Note that due to the element-wise multiplication we actually represent unordered $n$-gram information, not ordered $n$-grams (the performance for order-preserving methods is about the same).
Now we are ready to define our <em>Distributed Co-occurrence (DisC) embeddings</em>.</p>
<blockquote>
<p>The <strong>DisC embedding</strong> of a piece of text is just a concatenation for $(v_1, v_2, \ldots)$ where $v_n$ is the sum of the $n$-gram embeddings of all $n$-grams in the document (for $n=1$ this is just the sum of word embeddings).</p>
</blockquote>
<p>Note that DisC embeddings leverage classic Bag-of-n-Gram information as well as the power of word embeddings.
For instance, the sentences <em>“Loved this movie!”</em> and <em>“I enjoyed the film.”</em> share no $n$-gram information for any $n$, but their DisC embeddings are fairly similar.
Thus if the first example comes with a label, it gives the learner some idea of how to classify the second.
This can be useful especially in settings with few labeled examples; e.g. DisC outperform BonG on the Stanford Sentiment Treebank (SST) task, which has only 6,000 labeled examples.
DisC embeddings also beat SIF and a standard LSTM-based method, Skipthoughts.
On the much larger IMDB testbed, BonG still reigns at top (although DisC is not too far behind).</p>
<div style="text-align:center;">
<img src="/assets/clfperf_sst_imdb.png" width="70%" alt="Performance on SST and IMDB" />
</div>
<p>Skip-thoughts does match or beat our DisC embeddings on some other classification tasks, but that’s still not too shabby an outcome for such a simple method. (By contrast, LSTM methods can take days or weeks of training, and are quite slow to evaluate at test time on a new piece of text.)</p>
<div style="text-align:center;">
<img src="/assets/sentenceembedtable.jpg" width="70%" alt="Performance on various classification tasks" />
</div>
<h2 id="some-theoretical-analysis-via-compressed-sensing">Some theoretical analysis via compressed sensing</h2>
<p>A linear SIF-like embedding represents a document with Bag-of-Words vector $x$ as
<script type="math/tex">\sum_w \alpha_w x_w v_w,</script>
where $v_w$ is the embedding of word $w$ and $\alpha_w$ is a scaling term.
In other words, it represents document $x$ as $A x$ where $A$ is the matrix with as many columns as the number of words in the language, and the column corresponding to word $w$ is $\alpha_w A$.
Note that $x$ has many zero coordinates corresponding to words that don’t occur in the document; in other words it’s a <em>sparse</em> vector.</p>
<p>The starting point of our DisC work was the realization that perhaps the reason SIF-like embeddings work reasonably well is that they <em>preserve</em> the Bag-of-words information, in the sense that it may be possible to <em>easily recover</em> $x$ from $A$.
This is not an outlandish conjecture at all, because <a href="https://en.wikipedia.org/wiki/Compressed_sensing"><em>compressed sensing</em></a> does exactly this when $x$ is suitably sparse and matrix $A$ has some nice properties such as RIP or incoherence.
A classic example is when $A$ is a random matrix, which in our case corresponds to using random vectors as word embeddings.
Thus one could try to use random word embeddings instead of GloVe vectors in the construction and see what happens!
Indeed, we find that so long as we raise the dimension of the word embeddings, then text embeddings using random vectors do indeed converge to the performance of BonG representations.</p>
<p>This is a surprising result, as compressed sensing does not imply this per se, since the ability to reconstruct the BoW vector from its compressed version doesn’t directly imply that the compressed version gives the same performance as BoW on linear classification tasks.
However, a result of <a href="https://pdfs.semanticscholar.org/627c/14fe9097d459b8fd47e8a901694198be9d5d.pdf">Calderbank, Jafarpour, & Schapire</a> shows that the compressed sensing condition that implies optimal recovery also implies good performance on linear classification under compression. Intuitively, this happens because of two facts.</p>
<p><script type="math/tex">\mbox{1) Optimum linear classifier $c^*$ is convex combination of datapoints.} \quad c^* = \sum_{i}\alpha_i x_i.</script>
<script type="math/tex">% <![CDATA[
\mbox{(2) RIP condition implies} <Ax, Ax'> \approx <x, x'>~\mbox{for $k$-sparse}~x, x'. %]]></script></p>
<p>Furthermore, by extending these ideas to the $n$-gram case, we show that our DisC embeddings computed using random word vectors, which can be seen as a linear compression of the BonG representation, can do as well as the original BonG representation on linear classification tasks. To do this we prove that the “sensing” matrix $A$ corresponding to DisC embeddings satisfy the <em>Restricted Isometry Property (RIP)</em> introduced in the seminal paper of <a href="https://statweb.stanford.edu/~candes/papers/DecodingLP.pdf">Candes & Tao</a>. The theorem relies upon <a href="http://www.cis.pku.edu.cn/faculty/vision/zlin/A%20Mathematical%20Introduction%20to%20Compressive%20Sensing.pdf">compressed sensing results for bounded orthonormal systems</a> and says that then the performance of DisC embeddings on linear classification tasks approaches that of BonG vectors as we increase the dimension.
Please see our paper for details of the proof.</p>
<p>It is worth noting that our idea of composing objects (words) represented by random vectors to embed structures ($n$-grams/documents) is closely related to ideas in neuroscience and neural coding proposed by <a href="http://www2.fiit.stuba.sk/~kvasnicka/CognitiveScience/6.prednaska/plate.ieee95.pdf">Tony Plate</a> and <a href="http://www.rctn.org/vs265/kanerva09-hyperdimensional.pdf">Pentti Kanerva</a>.
They also were interested in how these objects and structures could be recovered from the representations;
we take the further step of relating the recoverability to performance on a downstream linear classification task.
Text classification over compressed BonG vectors has been proposed before by <a href="https://papers.nips.cc/paper/4932-compressive-feature-learning.pdf">Paskov, West, Mitchell, & Hastie</a>, albeit with a more complicated compression that does not achieve a low-dimensional representation (dimension >100,000) due to the use of classical lossless algorithms rather than linear projection.
Our work ties together these ideas of composition and compression into a simple text representation method with provable guarantees.</p>
<h2 id="a-surprising-lower-bound-on-the-power-of-lstm-based-text-representations">A surprising lower bound on the power of LSTM-based text representations</h2>
<p>The above result also leads to a new theorem about deep learning: <em>text embeddings computed using low-memory LSTMs can do at least as well as BonG representations on downstream classification tasks</em>.
At first glance this result may seem uninteresting: surely it’s no surprise that the field’s latest and greatest method is at least as powerful as its oldest?
But in practice, most papers on LSTM-based text embeddings make it a point to compare to performance of BonG baseline, and <em>often are unable to improve upon that baseline</em>!
Thus empirically this new theorem had not been clear at all! (One reason could be that our theory requires the random embeddings to be somewhat higher dimensional than the LSTM work had considered.)</p>
<p>The new theorem follows from considering an LSTM that uses random vectors as word embeddings and computes the DisC embedding in one pass over the text. (For details see our appendix.)</p>
<p>We empirically tested the effect of dimensionality by measuring performance of DisC on IMDb sentiment classification.
As our theory predicts, the accuracy of DisC using random word embeddings converges to that of BonGs as dimensionality increases. (In the figure below “Rademacher vectors” are those with entries drawn randomly from $\pm1$.) Interestingly we also find that DisC using pretrained word embeddings like GloVe reaches BonG performance at much smaller dimensions, an unsurprising but important point that we will discuss next.</p>
<div style="text-align:center;">
<img src="/assets/imdbperf_uni_bi.png" width="60%" />
</div>
<h2 id="unexplained-mystery-higher-performance-of-pretrained-word-embeddings">Unexplained mystery: higher performance of pretrained word embeddings</h2>
<p>While compressed sensing theory is a good starting point for understanding the power of linear text embeddings, it leaves some mysteries.
Using pre-trained embeddings (such as GloVe) in DisC gives higher performance than random embeddings, both in recovering the BonG information out of the text embedding, as well as in downstream tasks. However, pre-trained embeddings do not satisfy some of the nice properties assumed in compressed sensing theory such as RIP or incoherence, since those properties forbid pairs of words having similar embeddings.</p>
<p>Even though the matrix of embeddings does not satisfy these classical compressed sensing properties, we find that using Basis Pursuit, a sparse recovery approach related to LASSO with provable guarantees for RIP matrices, we can recover Bag-of-Words information better using GloVe-based text embeddings than from embeddings using random word vectors (measuring success via the $F_1$-score of the recovered words — higher is better).</p>
<div style="text-align:center;">
<img src="/assets/recovery.png" width="60%" />
</div>
<p>Note that random embeddings are better than pretrained embeddings at recovering words from random word salad (the right-hand image).
This suggests that pretrained embeddings are specialized — thanks to their training on a text corpus — to do well only on real text rather than a random collection of words.
It would be nice to give a mathematical explanation for this phenomenon.
We suspect that this should be possible using a result of <a href="http://www.pnas.org/content/pnas/102/27/9446.full.pdf">Donoho & Tanner</a>, which we use to show that words in a document can be recovered from the sum of word vectors if and only if there is a hyperplane containing the vectors for words in the document with the vectors for all other words on one side of it.
Since co-occurring words will have similar embeddings, that should make it easier to find such a hyperplane separating words in a document from the rest of the words and hence would ensure good recovery.</p>
<p>However, even if this could be made more rigorous, it would only imply sparse recovery, not good performance on classification tasks.
Perhaps assuming a generative model for text, like the RandWalk model discussed in an <a href="https://www.offconvex.org/2016/02/14/word-embeddings-2/">earlier post</a>, could help move this theory forward.</p>
<h2 id="discussion">Discussion</h2>
<p>Could we improve the performance of such simple embeddings even further?
One promising idea is to define better $n$-gram embeddings than the simple compositional embeddings defined in DisC.
An independent <a href="https://arxiv.org/abs/1703.02507">NAACL’18 paper</a> of Pagliardini, Gupta, & Jaggi proposes a text embedding similar to DisC in which unigram and bigram embeddings are trained specifically to be added together to form sentence embeddings, also achieving good results, albeit not as good as DisC.
(Of course, their training time is higher than ours.)
In our upcoming <a href="https://arxiv.org/abs/1805.05388">ACL’18 paper</a> with Yingyu Liang, Tengyu Ma, & Brandon Stewart we give a very simple and efficient method to induce embeddings for $n$-grams as well as other rare linguistic features that improves upon DisC and beats skipthought on several other benchmarks.
This will be described in a future blog post.</p>
<p>Sample code for constructing and evaluating DisC embeddings is <a href="https://github.com/NLPrinceton/text_embedding">available</a>, as well as <a href="https://github.com/NLPrinceton/sparse_recovery">solvers</a> for recreating the sparse recovery results for word embeddings.</p>
Mon, 25 Jun 2018 03:00:00 -0700
http://offconvex.github.io/2018/06/25/textembeddings/
http://offconvex.github.io/2018/06/25/textembeddings/Deep-learning-free Text and Sentence Embedding, Part 1<p>Word embeddings (see my old <a href="http://www.offconvex.org/2015/12/12/word-embeddings-1/">post1</a> and
<a href="http://www.offconvex.org/2016/02/14/word-embeddings-2/">post2</a>) capture the idea that one can express “meaning” of words using a vector, so that the cosine of the angle between the vectors captures semantic similarity. (“Cosine similarity” property.) Sentence embeddings and text embeddings try to achieve something similar: use a fixed-dimensional vector to represent a small piece of text, say a sentence or a small paragraph. The performance of such embeddings can be tested via the Sentence Textual Similarity (STS) datasets (see the <a href="http://ixa2.si.ehu.es/stswiki/index.php/Main_Page">wiki page</a>), which contain sentence pairs humanly-labeled with similarity ratings.</p>
<p style="text-align:center;">
<img src="/assets/textembeddingvectorslide.jpg" width="30%" alt="What are text embeddings." />
</p>
<p>A general hope behind computing text embeddings is that they can be learnt using a large <em>unlabeled</em> text corpus (similar to word embeddings) and then allow good performance on downstream classification tasks with few <em>labeled</em> examples. Thus the overall pipeline could look like this:</p>
<p style="text-align:center;">
<img src="/assets/textembeddingpipeline.jpg" width="80%" alt="How are text embeddings used in downstream classification task." />
</p>
<p>Computing such representations is a form of <a href="http://www.offconvex.org/2017/06/26/unsupervised1/">representation learning as well as unsupervised learning</a>. This post will be an introduction to <strong>extremely simple</strong> ways of computing sentence embeddings, which on many standard tasks, beat many state-of-the-art deep learning methods. This post is based upon <a href="https://openreview.net/pdf?id=SyK00v5xx">my ICLR’17 paper on SIF embeddings</a> with Yingyu Liang and Tengyu Ma.</p>
<h2 id="existing-methods">Existing methods</h2>
<p><a href="https://dl.acm.org/citation.cfm?id=2133826">Topic modeling</a> is a classic technique for unsupervised learning on text and it also yields a vector representation for a paragraph (or longer document), specifically, the vector of “topics” occuring in this document and their relative proportions. Unfortunately, topic modeling is not accurate at producing good representations at the sentence or short paragraph level, and furthermore there appears to be no variant of topic modeling that leads to the good cosine similarity property that we desire.</p>
<p><em>Recurrent neural net</em> is the default deep learning technique to train a <a href="https://www.tensorflow.org/tutorials/recurrent">language model</a>. It scans the text from left to right, maintaining a fixed-dimensional vector-representation of the text it has seen so far. It’s goal is to use this representation to predict the next word at each time step, and the training objective is to maximise log-likelihood of the data (or similar). Thus for example, a well-trained model when given a text fragment <em>“I went to the cafe and ordered a ….”</em> would assign high probability to <em>“coffee”, “croissant”</em> etc. and low probability to <em>“puppy”</em>. Myriad variations of such language models exist, many using biLSTMs which have some long-term memory and can scan the text forward and backwards. Lately biLSTMs have been replaced by convolutional architectures with attention mechanism; see for instance <a href="http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf">this paper</a>.</p>
<p>One obtains a text representation by peeking at the internal representation (i.e., node activations) at the top layer of this deep model. After all, when the model is scanning through text, its ability to predict the next word must imply that this internal representation implicitly captures a gist of all it has seen, reflecting rules of grammar, common-sense etc. (e.g., that you don’t order a puppy at a cafe). Some notable modern efforts along such lines are <a href="https://arxiv.org/abs/1506.01057">Hierarchichal Neural Autoencoder of Li et al.</a> as well as <a href="https://arxiv.org/abs/1502.06922">Palangi et al</a>, and <a href="https://arxiv.org/abs/1506.06726"><em>Skipthought</em> of Kiros et al.</a>.</p>
<p>As with all deep learning models, one wishes for interpretability: what information exactly did the machine choose to put into the text embedding? Besides <a href="https://people.csail.mit.edu/beenkim/papers/BeenK_FinaleDV_ICML2017_tutorial.pdf">the usual reasons for seeking interpretability</a>, in an NLP context it may help us leverage additional external resources such as <a href="https://wordnet.princeton.edu/">WordNet</a> in the task. Other motivations include
transfer learning/domain adaptation (to solve classification tasks for a small text corpus, leverage text embeddings trained on a large unrelated corpus).</p>
<h2 id="surprising-power-of-simple-linear-representations">Surprising power of simple linear representations</h2>
<p>In practice, many NLP applications rely on a simple sentence embedding: the average of the embeddings of the words in it. This makes some intuitive sense, because recall that the <a href="https://arxiv.org/pdf/1310.4546.pdf">Word2Vec paper</a> uses the following expression (in the their simpler CBOW word embedding)</p>
<script type="math/tex; mode=display">\Pr[w~|~w_1,w_2, w_3, w_4, w_5] \propto \exp(v_w \cdot (\frac{1}{5} \sum_i v_{w_i}). \qquad (1)</script>
<p>which suggests that the sense of a sequence of words is captured via simple average of word vectors.</p>
<p>While this simple average has only fair performance in capturing sentence similarity via cosine similarity, it can be quite powerful in downstream classification tasks (after passing through a single layer neural net) as shown in a
surprising paper of <a href="https://arxiv.org/abs/1511.08198">Wieting et al. ICLR’16</a>.</p>
<h2 id="better-linear-representation-sif-embeddings">Better linear representation: SIF embeddings</h2>
<p>My <a href="https://openreview.net/pdf?id=SyK00v5xx">ICLR’17 paper</a> with Yingyu Liang and Tengyu Ma improved such simple averaging using our <strong>SIF</strong> embeddings. They’re motivated by the empirical observation that word embeddings have various pecularities stemming from the training method, which tries to capture word cooccurence probabilities using vector inner product, and words sometimes occur out of context in documents. These anomalies cause the average of word vectors to have nontrivial components along semantically meaningless directions. SIF embeddings try to combat this in two ways, which I describe intuitively first, followed by more theoretical justification.</p>
<p><strong>Idea 1: Nonuniform weighting of words.</strong>
Conventional wisdom in information retrieval holds that “frequent words carry less signal.” Usually this is captured via <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF weighting</a>, which assigns weightings to words inversely proportional to their frequency. We introduce a new variant we call <em>Smoothed Inverse Frequency</em> (SIF) weighting,
which assigns to word $w$ a weighting $\alpha_w = a/(a+ p_w)$ where $p_w$ is the frequency of $w$ in the corpus and $a$ is a hyperparameter. Thus the embedding of a piece of text is $\sum_w \alpha_w v_w$ where the sum is over words in it.
(Aside: word frequencies can be estimated from any sufficiently large corpus; we find embedding quality to be not too dependent upon this.)</p>
<p>On a related note, we found that folklore understanding of word2vec, viz., expression (1), is <em>false.</em> A dig into the code reveals a resampling trick that is tantamount to a weighted average quite similar to our SIF weighting. (See Section 3.1 in our paper for a discussion.)</p>
<p><strong>Idea 2: Remove component from top singular direction.</strong>
The next idea is to modify the above weighted average by removing the component in a special direction, corresponding to the top singular direction set of weighted embeddings of a smallish sample of sentences from the domain (if doing domain adaptation, component is computed using sentences of the target domain). The paper notes that the direction corresponding to the top singular vector tends to contain information related to grammar and stop words, and removing the component in this subspace really cleans up the text embedding’s ability to express meaning.</p>
<h2 id="theoretical-justification">Theoretical justification</h2>
<p>A notable part of our paper is to give a theoretical justification for this weighting using a generative model for text similar to one used in our <a href="http://aclweb.org/anthology/Q16-1028">word embedding paper in TACL’16</a> as described in <a href="http://www.offconvex.org/2016/02/14/word-embeddings-2/">my old post</a>.
That model tries to give the causative relationship between word meanings and their cooccurence probabilities. It thinks of corpus generation as a dynamic process, where the $t$-th word is produced at step $t$. The model says that the process is driven by the random walk of a <em>discourse</em> vector $c_t \in \Re^d$. It is a unit vector whose direction in space represents <em>what is being talked about.</em>
Each word has a (time-invariant) latent vector $v_w \in \Re^d$ that captures its correlations with the discourse vector. We model this bias with a loglinear word production model:</p>
<script type="math/tex; mode=display">\Pr[w~\mbox{emitted at time $t$}~|~c_t] \propto \exp(c_t\cdot v_w). \qquad (2)</script>
<p>The discourse vector does a slow geometric random walk over the unit sphere in $\Re^d$. Thus $c_{t+1}$ is obtained by a small random displacement from $c_t$. Since expression (2) places much higher probability on words that are clustered around $c_t$, and $c_t$ moves slowly. If the discourse vector moves slowly, then we can assume a single discourse vector gave rise to the entire sentence or short paragraph. Thus given a sentence, a plausible vector representation of its “meaning” is a <em>max a posteriori</em> (MAP) estimate of the discourse vector that generated it.</p>
<p>Such models have been empirically studied for a while, but our paper gave a theoretical analysis, and showed that various subcases imply standard word embedding methods such as word2vec and GloVe. For example, it shows that MAP estimate of the discourse vector is the simple average of the embeddings of the preceding $k$ words – in other words, the average word vector!</p>
<p>This model is clearly simplistic and our ICLR’17 paper suggests two correction terms, intended to account for words occuring out of context, and to allow some common words (<em>“the”, “and”, “but”</em> etc.) appear often regardless of the discourse. We first introduce an additive term $\alpha p(w)$ in the log-linear model, where $p(w)$ is the unigram probability (in the entire corpus) of word and $\alpha$ is a scalar. This allows words to occur even if their vectors have very low inner products with $c_s$.
Secondly, we introduce a common discourse vector $c_0\in \Re^d$ which serves as a correction term for the most frequent discourse that is often related to syntax. It boosts the co-occurrence probability of words that have a high component along $c_0$.(One could make other correction terms, which are left to future work.) To put it another way, words that need to appear a lot out of context can do so by having a component along $c_0$, and the size of this component controls its probability of appearance out of context.</p>
<p>Concretely, given the discourse vector $c_s$ that produces sentence $s$, the probability of a word $w$ is emitted in the sentence $s$ is modeled as follows, where $\tilde{c}_{s} = \beta c_0 + (1-\beta) c_s, c_0 \perp c_s$,
$\alpha$ and $\beta$ are scalar hyperparameters:</p>
<script type="math/tex; mode=display">% <![CDATA[
\Pr[w \mid s] = \alpha p(w) + (1-\alpha) \frac{\exp(<\tilde{c}_{s}, v_w>)}{Z_{\tilde{c,s}}}, %]]></script>
<p>where</p>
<script type="math/tex; mode=display">% <![CDATA[
Z_{\tilde{c,s}} = \sum_{w} \exp(<\tilde{c}_{s}, v_w>) %]]></script>
<p>is the normalizing constant (the partition function). We see that the model allows a word $w$ unrelated to the discourse $c_s$ to be emitted for two reasons: a) by chance from the term $\alpha p(w)$; b) if $w$ is correlated with the common direction $c_0$.</p>
<p>The paper shows that the MAP estimate of the $c_s$ vector corresponds to the SIF embeddings described earlier, where the top singular vector used in their construction is an estimate of the $c_0$ vector in the model.</p>
<h2 id="empirical-performance">Empirical performance</h2>
<p>The performance of this embedding scheme appears in the figure below. Note that Wieting et al. had already shown that their method (which is semi-supervised, relying upon a large unannotated corpus and a small annotated corpus) beats many LSTM-based methods. So this table only compares to their work; see the papers for comparison with more past work.</p>
<p style="text-align:center;">
<img src="/assets/textembedexperiments.jpg" width="80%" alt="Performance of our embedding on downstream classification tasks" />
</p>
<p>For other performance results please see the paper.</p>
<h2 id="next-post">Next post</h2>
<p>In the next post, I will sketch improvements to the above embedding in two of our new papers. Special guest appearance: Compressed Sensing (aka Sparse Recovery).</p>
<p>The SIF embedding package is available <a href="https://github.com/PrincetonML/SIF">from our github page</a></p>
Sun, 17 Jun 2018 03:00:00 -0700
http://offconvex.github.io/2018/06/17/textembeddings/
http://offconvex.github.io/2018/06/17/textembeddings/Limitations of Encoder-Decoder GAN architectures<p>This is yet another post about <a href="http://www.offconvex.org/2017/03/15/GANs/">Generative Adversarial Nets (GANs)</a>, and based upon our new <a href="https://openreview.net/forum?id=BJehNfW0-">ICLR’18 paper</a> with Yi Zhang. A quick recap of the story so far. GANs are an unsupervised method in deep learning to learn interesting distributions (e.g., images of human faces), and also have a plethora of uses for image-to-image mappings in computer vision. Standard GANs training is motivated using this task of distribution learning, and is designed with the idea that given large enough deep nets and enough training examples, as well as accurate optimization, GANs will learn the full distribution.</p>
<p><a href="http://www.offconvex.org/2017/03/30/GANs2/">Sanjeev’s previous post</a> concerned <a href="https://arxiv.org/abs/1703.00573">his co-authored ICML’17 paper</a> which called this intuition into question when the deep nets have finite capacity. It shows that the training objective has near-equilibria where the discriminator is fooled —i.e., training objective is good—but the generator’s distributions has very small support, i.e. shows <em>mode collapse.</em> This is a failure of the model, and raises the question whether such bad equilibria are found in real-life training. A <a href="http://www.offconvex.org/2017/07/07/GANs3/">second post</a> showed empirical evidence that they do, using the birthday-paradox test.</p>
<p>The current post concerns our <a href="https://arxiv.org/abs/1711.02651">new result</a> (part of our upcoming <a href="https://openreview.net/forum?id=BJehNfW0-">ICLR paper</a>) which shows that bad equilibria exist also in more recent GAN architectures based on simultaneously learning an <em>encoder</em> and <em>decoder</em>. This should be surprising because many researchers believe that encoder-decoder architectures fix many issues with GANs, including mode collapse.</p>
<p>As we will see, encoder-decoder GANs seem very powerful. In particular, the proof of the previously mentioned <a href="http://www.offconvex.org/2017/03/30/GANs2/">negative result</a> utterly breaks down for this architecture. But, we then discovered a cute argument that shows encoder-decoder GANs can have poor solutions, featuring not only mode collapse but also encoders that map images to nonsense (more precisely Gaussian noise). This is the worst possible failure of the model one could imagine.</p>
<h2 id="encoder-decoder-architectures">Encoder-decoder architectures</h2>
<p>Encoders and decoders have long been around in machine learning in various forms – especially deep learning. Speaking loosely, underlying all of them are two basic assumptions: <br />
(1) Some form of the so-called <a href="https://mitpress.mit.edu/sites/default/files/titles/content/9780262033589_sch_0001.pdf"><em>manifold assumption</em></a> which asserts that high-dimensional data such as real-life images lie (roughly) on a low-dimensional manifold. (“Manifold” should be interpreted rather informally – sometimes this intuition applies only very approximately sometimes it’s meant in a “distributional” sense, etc.) <br />
(2) The low-dimensional structure is “meaningful”: if we think of an image $x$ as a high-dimensional vector and its “code” $z$ as its coordinates on the low-dimensional manifold, the code $z$ is thought of as a “high-level” descriptor of the image.</p>
<p>With the above two points in mind, an <em>encoder</em> maps the image to its code, and a <em>decoder</em> computes the reverse map. (We also discussed encoders and decoders in <a href="http://www.offconvex.org/2017/06/27/unsupervised1/">our earlier post on representation learning</a> in a more general setup.)</p>
<p style="text-align:center;">
<img src="/assets/BIGAN_manifold2.jpg" width="80%" alt="Manifold structure" />
</p>
<h2 id="encoder-decoder-gans">Encoder-Decoder GANs</h2>
<p>These were introduced by <a href="https://arxiv.org/abs/1606.00704">Dumoulin et al.(ALI)</a> and <a href="https://arxiv.org/abs/1605.09782">Donahue et al.(BiGAN)</a>. They involve two competitors: Player 1 involves a discriminator net $D$ that is given an input of the form (image, code) and it outputs a number in the interval $[0,1]$, which denotes its “satisfaction level” with this input. Player 2 trains a decoder net $G$ (also called <em>generator</em> in the GANs setting) and an encoder net $E$.</p>
<p>Recall that in the standard GAN, discriminator tries to distinguish real images from images generated by the generator $G$. Here
discriminator’s input is an image and its code. Specifically, Player 1 is trying to train its net to distinguish between the following two settings, and Player 2 is trying to make sure the two settings look indistinguishable to Player 1’s net.</p>
<p><script type="math/tex">\mbox{Setting 1: presented with}~(x, E(x))~\mbox{where $x$ is random real image}.</script>
<script type="math/tex">\mbox{Setting 2: presented with}~(G(z), z)~\mbox{where $z$ is random code}.</script></p>
<p>(Here it is assumed that a random code is a vector with i.i.d gaussian coordinates, though one could consider other distributions.)</p>
<p style="text-align:center;">
<img src="/assets/BIGAN_2settings_v2.jpg" width="80%" alt="Two settings which discriminator net has to distinguish between" />
</p>
<p>The hoped-for equilibrium obviously is one where generator and encoder are inverses of each other: $E(G(z)) \approx z$ and $G(E(x)) \approx x$, and the joint distributions $(z,G(z))$ and $(E(x), x)$ roughly match.
The underlying intuition is that if this happens, Player 1 must’ve produced a “meaningful” representation $E(x)$ for the images – and this should improve the quality of the generator as well.
Indeed, <a href="https://arxiv.org/abs/1606.00704">Dumoulin et al.(ALI)</a> provide some small-scale empirical examples on mixtures of Gaussians for which encoder-decoder architectures seem to ameliorate the problem of mode collapse.</p>
<p>The above papers prove that when the encoder/decoder/discriminator have infinite capacity, the desired solution is indeed an equilibrium. However, we’ll see that things are very different when capacities are finite.</p>
<h2 id="finite-capacity-discriminators-are-weak">Finite-capacity discriminators are weak</h2>
<p>Say a generator/encoder pair $(G,E)$ $\epsilon$-<em>fools</em> a decoder $D$ if</p>
<script type="math/tex; mode=display">|E_{x} D(x, E(x)) - E_{z} D(G(z), z)| \leq \epsilon</script>
<p>In other words, $D$ has roughly similar output in Settings 1 and 2.</p>
<p>Our theorem applies when the distribution consists of realistic images, as explained later. We show the following:</p>
<blockquote>
<p>(Informal theorem) If the discriminator $D$ has capacity (i.e. number of parameters) at most $p$, then there is an encoder $E$ of capacity $\ll p$ and generator $G$ of slightly larger capacity than $p$ such that $(G, E)$ can $\epsilon$-fool every such $D$. Furthermore, the generator exhibits mode collapse: its distribution is essentially supported on a bit more than $p$ images, and the encoder $E$ just outputs white noise (i.e. does not extract any “meaningful” representation) given an image.</p>
</blockquote>
<p>(Note that such a $(G, E)$ represents an $\epsilon$-approximate equilibrium, in the sense that player 1 cannot gain more than $\epsilon$ in the distinguishing probability by switching its discriminator. )</p>
<p>It is important that the encoder’s capacity is much less than $p$, and thus the theorem allows a discriminator that is able to simulate $E$ if it needed, and in particular verify for a random seed $z$ that $E(G(z)) \approx z$. The theorem says that even the ability to conduct such a verification cannot give it power to force encoder to produce meaningful codes. This is a counterintuitive aspect of the result. The main difficulty in the proof (which stumped us for a bit) was how to exhibit such an equilibrium where $E$ is a small net.</p>
<p>This is ensured by a simple assumption. We assume the image distribution is mildly “noised”: say, every 100th pixel is replaced by Gaussian noise. To a human, such an image would of course be indistinguishable from a real image. (NB: Our proof could be carried out via some other assumptions to the effect that images have an innate stochastic/noise component that is efficiently extractable by a small neural network. But let’s keep things clean.) When noise $\eta$ is thus added to an image $x$, we denote the resulting image as $x \odot \eta$.</p>
<p>Now the encoder will be rather trivial: given the noised image $x \odot \eta$, output $\eta$. Clearly, such an encoder does not in any sense capture “meaning” in the image. It is also implementable by a tiny single-layer net, as required by the theorem.</p>
<h3 id="construction-of-generator">Construction of generator</h3>
<p>As usual in the GAN literature, we will assume the discriminator is $L$-<a href="https://www.encyclopediaofmath.org/index.php/Lipschitz_constant">Lipschitz</a>. This can be a loose upperbound, since only $\log L$ enters quantitatively in the proof.</p>
<p>The generator $G(z)$ in the theorem statement memorizes a hash function that partitions the set of all seeds/codes $z$ into $m$ equal-sized blocks; it also memorizes a “pool” of $m := p \log^2(pL)/ \epsilon^2$ unnoised images $\tilde{x}_1, \tilde{x}_2, \dots, \tilde{x}_m$. When presented with a random seed $z$, the generator computes the block of the partition that $z$ lies in, and then produces the image $\tilde{x}_i \odot z$, where $i$ is the block $z$ belongs to. (See the Figure below.)</p>
<p style="text-align:center;">
<img src="/assets/BIGAN_construction_2.jpg" width="50%" alt="The bad generator construction" />
</p>
<p>Now we have to prove that such a memorizing generator exists that $\epsilon$-fools all discriminators of capacity $p$. This is shown by the <a href="https://en.wikipedia.org/wiki/Probabilistic_method">probabilistic method</a>: we describe a distribution over generators $G$ that works “in expectation”, and subsequently use concentration bounds to prove there exists at least one generator that does the job.</p>
<p>The distribution on $G$’s is straightforward: we select the pool of (unnoised) images
$\tilde{x}_1, \tilde{x}_2, .., \tilde{x}_m$ at random. Why is this distribution for $G$ sensible? Notice the following simple fact:</p>
<script type="math/tex; mode=display">E_{G} E_{z} D(G(z), z) = E_{\tilde{x}, z} D(\tilde{x} \odot z, z) = E_{x} D(x, E(x)) \hspace{2cm} (3)</script>
<p>In other words, the “expected” encoder correctly matches the expectation of $D(x, E(x))$, so that the discriminator is fooled “in expectation”.
This of course is not enough: we need some kind of concentration argument to show a particular $G$ works against <em>all possible discriminators</em>, which will ultimately use the fact that the discriminator $D$ has a small capacity and small Lipschitz constant. (Think covering number arguments in learning theory.)</p>
<p>Towards that, another useful observation: if $q$ is the uniform distribution over sets $T= {z_1, z_2,\dots, z_m}$, s.t. each $z_i$ is independently sampled from the conditional distribution inside the $i$-th block of the partition of the noise space, by the law of total expectation one can see that
<script type="math/tex">E_{z} D(G(z), z) = E_{T \sim q} \frac{1}{m} \sum_{i=1}^m D(G(z_i), z_i)</script>
The right hand side is an average of terms, each of which is a bounded function of mutually independent random variables – so, by e.g. McDiarmid’s inequality it concentrates around it’s expectation, which by (3) is exactly $E_{z} D(G(z), z)$.</p>
<p>To finish the argument off, we use the fact that due to Lipschitzness and the bound on the number of parameters, the “effective” number of distinct discriminators is small, so we can union bound over them. (Formally, this translates to an epsilon-net + union bound argument. This also gives rise to the value of $m$ used in the construction.)</p>
<h2 id="takeaway">Takeaway</h2>
<p>The result should be interpreted as saying that possibly the theoretical foundations of GANs need more work. The current way of thinking about them as distribution learners may not be the right way to formalize them. Furthermore, one has to take care about transfering notions invented for distribution learning, such as encoders and decoders, over into the GANs setting. Finally there is an empirical question whether any of the <a href="https://deephunt.in/the-gan-zoo-79597dc8c347">myriad GANS variations</a> can avoid mode collapse.</p>
Mon, 12 Mar 2018 03:00:00 -0700
http://offconvex.github.io/2018/03/12/bigan/
http://offconvex.github.io/2018/03/12/bigan/Can increasing depth serve to accelerate optimization?<p>“How does depth help?” is a fundamental question in the theory of deep learning. Conventional wisdom, backed by theoretical studies (e.g. <a href="http://proceedings.mlr.press/v49/eldan16.pdf">Eldan & Shamir 2016</a>; <a href="http://proceedings.mlr.press/v70/raghu17a/raghu17a.pdf">Raghu et al. 2017</a>; <a href="http://proceedings.mlr.press/v65/lee17a/lee17a.pdf">Lee et al. 2017</a>; <a href="http://proceedings.mlr.press/v49/cohen16.pdf">Cohen et al. 2016</a>; <a href="http://proceedings.mlr.press/v65/daniely17a/daniely17a.pdf">Daniely 2017</a>; <a href="https://openreview.net/pdf?id=B1J_rgWRW">Arora et al. 2018</a>), holds that adding layers increases expressive power. But often this expressive gain comes at a price –optimization is harder for deeper networks (viz., <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing/exploding gradients</a>). Recent works on “landscape characterization” implicitly adopt this worldview (e.g. <a href="https://papers.nips.cc/paper/6112-deep-learning-without-poor-local-minima.pdf">Kawaguchi 2016</a>; <a href="https://openreview.net/pdf?id=ryxB0Rtxx">Hardt & Ma 2017</a>; <a href="http://proceedings.mlr.press/v38/choromanska15.pdf">Choromanska et al. 2015</a>; <a href="http://openaccess.thecvf.com/content_cvpr_2017/papers/Haeffele_Global_Optimality_in_CVPR_2017_paper.pdf">Haeffele & Vidal 2017</a>; <a href="https://arxiv.org/pdf/1605.08361.pdf">Soudry & Carmon 2016</a>; <a href="https://arxiv.org/pdf/1712.08968.pdf">Safran & Shamir 2017</a>). They prove theorems about local minima and/or saddle points in the objective of a deep network, while implicitly assuming that the ideal landscape would be convex (single global minimum, no other critical point). My <a href="https://arxiv.org/pdf/1802.06509.pdf">new paper</a> with Sanjeev Arora and Elad Hazan makes the counterintuitive suggestion that sometimes, increasing depth can <em>accelerate</em> optimization.</p>
<p>Our work can also be seen as one more piece of evidence for a nascent belief that <em>overparameterization</em> of deep nets may be a good thing. By contrast, classical statistics discourages training a model with more parameters than necessary <a href="https://www.rasch.org/rmt/rmt222b.htm">as this can lead to overfitting</a>.</p>
<h2 id="ell_p-regression">$\ell_p$ Regression</h2>
<p>Let’s begin by considering a very simple learning problem - scalar linear regression with $\ell_p$ loss (our theory and experiments will apply to $p>2$):</p>
<script type="math/tex; mode=display">\min_{\mathbf{w}}~L(\mathbf{w}):=\frac{1}{p}\sum_{(\mathbf{x},y)\in{S}}(\mathbf{x}^\top\mathbf{w}-y)^p</script>
<p>$S$ here stands for a training set, consisting of pairs $(\mathbf{x},y)$ where $\mathbf{x}$ is a vector representing an instance and $y$ is a (numeric) scalar standing for its label; $\mathbf{w}$ is the parameter vector we wish to learn. Let’s convert the linear model to an extremely simple “depth-2 network”, by replacing the vector $\mathbf{w}$ with a vector $\mathbf{w_1}$ times a scalar $\omega_2$. Clearly, this is an overparameterization that does not change expressiveness, but yields the (non-convex) objective:</p>
<script type="math/tex; mode=display">\min_{\mathbf{w_1},\omega_2}~L(\mathbf{w_1},\omega_2):=\frac{1}{p}\sum_{(\mathbf{x},y)\in{S}}(\mathbf{x}^\top\mathbf{w_1}\omega_2-y)^p</script>
<p>We show in the paper, that if one applies gradient descent over $\mathbf{w_1}$ and $\omega_2$, with small learning rate and near-zero initialization (as customary in deep learning), the induced dynamics on the overall (<em>end-to-end</em>) model $\mathbf{w}=\mathbf{w_1}\omega_2$ can be written as follows:</p>
<script type="math/tex; mode=display">\mathbf{w}^{(t+1)}\leftarrow\mathbf{w}^{(t)}-\rho^{(t)}\nabla{L}(\mathbf{w}^{(t)})-\sum_{\tau=1}^{t-1}\mu^{(t,\tau)}\nabla{L}(\mathbf{w}^{(\tau)})</script>
<p>where $\rho^{(t)}$ and $\mu^{(t,\tau)}$ are appropriately defined (time-dependent) coefficients.
Thus the seemingly benign addition of a single multiplicative scalar turned plain gradient descent into a scheme that somehow has a memory of past gradients —the key feature of <a href="https://distill.pub/2017/momentum/">momentum</a> methods— as well as a time-varying learning rate. While theoretical analysis of the precise benefit of momentum methods is never easy, a simple experiment with $p=4$, on <a href="https://archive.ics.uci.edu/ml/index.php">UCI Machine Learning Repository</a>’s <a href="https://archive.ics.uci.edu/ml/datasets/gas+sensor+array+drift+dataset">“Gas Sensor Array Drift at Different Concentrations” dataset</a>, shows the following effect:</p>
<p style="text-align:center;">
<img src="/assets/acc_oprm_L4_exp.png" width="40%" alt="L4 regression experiment" />
</p>
<p>Not only did the overparameterization accelerate gradient descent, but it has done so more than two well-known, explicitly designed acceleration methods – <a href="http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf">AdaGrad</a> and <a href="https://arxiv.org/pdf/1212.5701.pdf">AdaDelta</a> (the former did not really provide a speedup in this experiment). We observed similar speedups in other settings as well.</p>
<p>What is happening here? Can non-convex objectives corresponding to deep networks be easier to optimize than convex ones?
Is this phenomenon common or is it limited to toy problems as above?
We take a first crack at addressing these questions…</p>
<h2 id="overparameterization-decoupling-optimization-from-expressiveness">Overparameterization: Decoupling Optimization from Expressiveness</h2>
<p>A general study of the effect of depth on optimization entails an inherent difficulty - deeper networks may seem to converge faster due to their superior expressiveness.
In other words, if optimization of a deep network progresses more rapidly than that of a shallow one, it may not be obvious whether this is a result of a true acceleration phenomenon, or simply a byproduct of the fact that the shallow model cannot reach the same loss as the deep one.
We resolve this conundrum by focusing on models whose representational capacity is oblivious to depth - <em>linear neural networks</em>, the subject of many recent studies.
With linear networks, adding layers does not alter expressiveness; it manifests itself only in the replacement of a matrix parameter by a product of matrices - an overparameterization.
Accordingly, if this leads to accelerated convergence, one can be certain that it is not an outcome of any phenomenon other than favorable properties of depth for optimization.</p>
<h2 id="implicit-dynamics-of-depth">Implicit Dynamics of Depth</h2>
<p>Suppose we are interested in learning a linear model parameterized by a matrix $W$, through minimization of some training loss $L(W)$.
Instead of working directly with $W$, we replace it by a depth $N$ linear neural network, i.e. we overparameterize it as $W=W_{N}W_{N-1}\cdots{W_1}$, with $W_j$ being weight matrices of individual layers.
In the paper we show that if one applies gradient descent over $W_{1}\ldots{W}_N$, with small learning rate $\eta$, and with the condition:</p>
<script type="math/tex; mode=display">W_{j+1}^\top W_{j+1} = W_j W_j^\top</script>
<p>satisfied at optimization commencement (note that this approximately holds with standard near-zero initialization), the dynamics induced on the overall end-to-end mapping $W$ can be written as follows:</p>
<script type="math/tex; mode=display">W^{(t+1)}\leftarrow{W}^{(t)}-\eta\sum_{j=1}^{N}\left[W^{(t)}(W^{(t)})^\top\right]^\frac{j-1}{N}\nabla{L}(W^{(t)})\left[(W^{(t)})^\top{W}^{(t)}\right]^\frac{N-j}{N}</script>
<p>We validate empirically that this analytically derived update rule (over classic linear model) indeed complies with deep network optimization, and take a series of steps to theoretically interpret it.
We find that the transformation applied to the gradient $\nabla{L}(W)$ (multiplication from the left by $[WW^\top]^\frac{j-1}{N}$, and from the right by $[W^\top{W}]^\frac{N-j}{N}$, followed by summation over $j$) is a particular preconditioning scheme, that promotes movement along directions already taken by optimization.
More concretely, the preconditioning can be seen as a combination of two elements:</p>
<ul>
<li>an adaptive learning rate that increases step sizes away from initialization; and</li>
<li>a “momentum-like” operation that stretches the gradient along the azimuth taken so far.</li>
</ul>
<p>An important point to make is that the update rule above, referred to hereafter as the <em>end-to-end update rule</em>, does not depend on widths of hidden layers in the linear neural network, only on its depth ($N$).
This implies that from an optimization perspective, overparameterizing using wide or narrow networks has the same effect - it is only the number of layers that matters.
Therefore, acceleration by depth need not be computationally demanding - a fact we clearly observe in our experiments (previous figure for example shows acceleration by orders of magnitude at the price of a single extra scalar parameter).</p>
<p style="text-align:center;">
<img src="/assets/acc_oprm_update_rule.png" width="60%" alt="End-to-end update rule" />
</p>
<h2 id="beyond-regularization">Beyond Regularization</h2>
<p>The end-to-end update rule defines an optimization scheme whose steps are a function of the gradient $\nabla{L}(W)$ and the parameter $W$.
As opposed to many acceleration methods (e.g. <a href="https://distill.pub/2017/momentum/">momentum</a> or <a href="https://arxiv.org/pdf/1412.6980.pdf">Adam</a>) that explicitly maintain auxiliary variables, this scheme is memoryless, and by definition born from gradient descent over something (overparameterized objective).
It is therefore natural to ask if we can represent the end-to-end update rule as gradient descent over some regularization of the loss $L(W)$, i.e. over some function of $W$.
We prove, somewhat surprisingly, that the answer is almost always negative - as long as the loss $L(W)$ does not have a critical point at $W=0$, the end-to-end update rule, i.e. the effect of overparameterization, cannot be attained via <em>any</em> regularizer.</p>
<h2 id="acceleration">Acceleration</h2>
<p>So far, we analyzed the effect of depth (in the form of overparameterization) on optimization by presenting an equivalent preconditioning scheme and discussing some of its properties.
We have not, however, provided any theoretical evidence in support of acceleration (faster convergence) resulting from this scheme.
Full characterization of the scenarios in which there is a speedup goes beyond the scope of our paper.
Nonetheless, we do analyze a simple $\ell_p$ regression problem, and find that whether or not increasing depth accelerates depends on the choice of $p$:
for $p=2$ (square loss) adding layers does not lead to a speedup (in accordance with previous findings by <a href="https://arxiv.org/pdf/1312.6120.pdf">Saxe et al. 2014</a>);
for $p>2$ it can, and this may be attributed to the preconditioning scheme’s ability to handle large plateaus in the objective landscape.
A number of experiments, with $p$ equal to 2 and 4, and depths ranging between 1 (classic linear model) and 8, support this conclusion.</p>
<h2 id="non-linear-experiment">Non-Linear Experiment</h2>
<p>As a final test, we evaluated the effect of overparameterization on optimization in a non-idealized (yet simple) deep learning setting - the <a href="https://github.com/tensorflow/models/tree/master/tutorials/image/mnist">convolutional network tutorial for MNIST built into TensorFlow</a>.
We introduced overparameterization by simply placing two matrices in succession instead of the matrix in each dense layer.
With an addition of roughly 15% in number of parameters, optimization accelerated by orders of magnitude:</p>
<p style="text-align:center;">
<img src="/assets/acc_oprm_cnn_exp.png" width="40%" alt="TensorFlow MNIST CNN experiment" />
</p>
<p>We note that similar experiments on other convolutional networks also gave rise to a speedup, but not nearly as prominent as the above.
Empirical characterization of conditions under which overparameterization accelerates optimization in non-linear settings is potentially an interesting direction for future research.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Our work provides insight into benefits of depth in the form of overparameterization, from the perspective of optimization.
Many open questions and problems remain.
For example, is it possible to rigorously analyze the acceleration effect of the end-to-end update rule (analogously to, say, <a href="http://www.cis.pku.edu.cn/faculty/vision/zlin/1983-A%20Method%20of%20Solving%20a%20Convex%20Programming%20Problem%20with%20Convergence%20Rate%20O(k%5E(-2))_Nesterov.pdf">Nesterov 1983</a> or <a href="http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf">Duchi et al. 2011</a>)?
Treatment of non-linear deep networks is of course also of interest, as well as more extensive empirical evaluation.</p>
<p><a href="http://www.cohennadav.com/">Nadav Cohen</a></p>
Fri, 02 Mar 2018 05:00:00 -0800
http://offconvex.github.io/2018/03/02/acceleration-overparameterization/
http://offconvex.github.io/2018/03/02/acceleration-overparameterization/Proving generalization of deep nets via compression<p>This post is about <a href="https://arxiv.org/abs/1802.05296">my new paper with Rong Ge, Behnam Neyshabur, and Yi Zhang</a> which offers some new perspective into the generalization mystery for deep nets discussed in
<a href="http://www.offconvex.org/2017/12/08/generalization1/">my earlier post</a>. The new paper introduces an elementary compression-based framework for proving generalization bounds. It shows that deep nets are highly noise stable, and consequently, compressible. The framework also gives easy proofs (sketched below) of some papers that appeared in the past year.</p>
<p>Recall that the <strong>basic theorem</strong> of generalization theory says something like this: if training set had $m$ samples then the <em>generalization error</em> —defined as the difference between error on training data and test data (aka held out data)— is of the order of $\sqrt{N/m}$. Here $N$ is the number of <em>effective parameters</em> (or <em>complexity measure</em>) of the net; it is at most the actual number of trainable parameters but could be much less. (For ease of exposition this post will ignore nuisance factors like $\log N$ etc. which also appear in the these calculations.) The mystery is that networks with millions of parameters have low generalization error even when $m =50K$ (as in CIFAR10 dataset), which suggests that the number of true parameters is actually much less than $50K$. The papers <a href="https://arxiv.org/abs/1706.08498">Bartlett et al. NIPS’17</a> and <a href="https://openreview.net/forum?id=Skz_WfbCZ">Neyshabur et al. ICLR’18</a>
try to quantify the complexity measure using very interesting ideas like Pac-Bayes and Margin (which influenced our paper). But ultimately the quantitative estimates are fairly vacuous —orders of magnitude <em>more</em> than the number of <em>actual parameters.</em> By contrast our new estimates are several orders of magnitude better, and on the verge of being interesting. See the following bar graph on a log scale. (All bounds are listed ignoring “nuisance factors.” Number of trainable parameters is included only to indicate scale.)</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/acompare.png" width="75%" alt="comparison of bounds from various recent papers" />
</p>
<h2 id="the-compression-approach">The Compression Approach</h2>
<p>The compression approach takes a deep net $C$ with $N$ trainable parameters and tries to compress it to another one $\hat{C}$ that has (a) much fewer parameters $\hat{N}$ than $C$ and (b) has roughly the same training error as $C$.</p>
<p>Then the above basic theorem guarantees that so long as the number of training samples exceeds $\hat{N}$, then $\hat{C}$ <em>does</em> generalize well (even if $C$ doesn’t). An extension of this approach says that the same conclusions holds if we let the compression algorithm to depend upon an arbitrarily long <em>random string</em> provided this string is fixed in advance of seeing the training data. We call this <em>compression with respect to fixed string</em> and rely upon it.</p>
<p>Note that the above approach proves good generalization of the compressed $\hat{C}$, not the original $C$. (I suspect the ideas may extend to proving good generalization of the original $C$; the hurdles seem technical rather than inherent.) Something similar was true of earlier approaches using PAC-Bayes bounds, which also prove the generalization of some net related to $C$, not of $C$ itself. (Hence the tongue-in-cheek title of the classic reference <a href="http://www.cs.cmu.edu/~jcl/papers/nn_bound/not_bound.pdf">Langford-Caruana2002</a>.)</p>
<p>Of course, in practice deep nets are well-known to be compressible using a slew of ideas—by factors of 10x to 100x; see <a href="https://arxiv.org/abs/1710.09282">the recent survey</a>. However, usually such compression involves <em>retraining</em> the compressed net. Our paper doesn’t consider retraining the net (since it involves reasoning about the loss landscape) but followup work should look at this.</p>
<h2 id="flat-minima-and-noise-stability">Flat minima and Noise Stability</h2>
<p>Modern generalization results can be seen as proceeding via some formalization of a <em>flat minimum</em> of the loss landscape. This was suggested in 1990s as the source of good generalization <a href="http://www.bioinf.jku.at/publications/older/3304.pdf">Hochreiter and Schmidhuber 1995</a>. Recent empirical work of <a href="https://arxiv.org/abs/1609.04836">Keskar et al 2016</a> on modern deep architectures finds that flatness does correlate with better generalization, though the issue is complicated, as discussed in an upcoming post by Behnam Neyshabur.</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/aflatminima.png" width="65%" alt="Flat vs sharp minima" />
</p>
<p>Here’s the intuition why a flat minimum should generalize better, as originally articulated by <a href="http://www.cs.toronto.edu/~fritz/absps/colt93.pdf">Hinton and Camp 1993</a>. Crudely speaking, suppose a flat minimum is one that occupies “volume” $\tau$ in the landscape. (The flatter the minimum, the higher $\tau$ is.) Then the number of <em>distinct</em> flat minima in the landscape is at most $S =\text{total volume}/\tau$. Thus one can number the flat minima from $1$ to $S$, implying that a flat minimum can be represented using $\log S$ bits. The above-mentioned <em>basic theorem</em> implies that flat minima generalize if the number of training samples $m$ exceeds $\log S$.</p>
<p>PAC-Bayes approaches try to formalize the above intuition by defining a flat minimum as follows: it is a net $C$ such that adding appropriately-scaled gaussian noise to all its trainable parameters does not greatly affect the training error. This allows quantifying the “volume” above in terms of probability/measure (see
<a href="http://www.cs.princeton.edu/courses/archive/fall17/cos597A/lecnotes/generalize.pdf">my lecture notes</a> or <a href="https://arxiv.org/abs/1703.11008">Dziugaite-Roy</a>) and yields some explicit estimates on sample complexity. However, obtaining good quantitative estimates from this calculation has proved difficut, as seen in the bar graph earlier.</p>
<p>We formalize “flat minimum” using noise stability of a slightly different form. Roughly speaking, it says that if we inject appropriately scaled gaussian noise at the output of some layer, then this noise gets attenuated as it propagates up to higher layers. (Here “top” direction refers to the output of the net.) This is obviously related to notions like dropout, though it arises also in nets that are not trained with dropout. The following figure illustrates how noise injected at a certain layer of VGG19 (trained on CIFAR10) affects the higher layer. The y-axis denote the magnitude of the noise ($\ell_2$ norm) as a multiple of the vector being computed at the layer, and shows how a single noise vector quickly attenuates as it propagates up the layers.</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/attenuate.jpg" width="65%" alt="How noises attenuates as it travels up the layers of VGG." />
</p>
<p>Clearly, computation of the trained net is highly resistant to noise. (This has obvious implications for biological neural nets…)
Note that the training involved no explicit injection of noise (eg dropout). Of course, stochastic gradient descent <em>implicitly</em> adds noise to the gradient, and it would be nice to investigate more rigorously if the noise stability arises from this or from some other source.</p>
<h2 id="noise-stability-and-compressibility-of-single-layer">Noise stability and compressibility of single layer</h2>
<p>To understand why noise-stable nets are compressible, let’s first understand noise stability for a single layer in the net, where we ignore the nonlinearity. Then this layer is just a linear transformation, i.e., matrix $M$.</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/alinear.png" width="40%" alt="matrix M describing a single layer" />
</p>
<p>What does it mean that this matrix’s output is stable to noise? Suppose the vector at the previous layer is a unit vector $x$. This is the output of the lower layers on an actual sample, so $x$ can be thought of as the “signal” for the current layer. The matrix converts $x$ into $Mx$. If we inject a noise vector $\eta$ of unit norm at $x$ then the output must become $M(x +\eta)$. We say $M$ is noise stable for input $x$ if such noising affects the output very little, which implies the norm of $Mx$ is much higher than that of $M \eta$.
The former is at most $\sigma_{max}(M)$, the largest singular value of $M$. The latter is approximately
$(\sum_i \sigma_i(M)^2)^{1/2}/\sqrt{h}$ where $\sigma_i(M)$ is the $i$th singular value of $M$ and $h$ is dimension of $Mx$. The reason is that gaussian noise divides itself evenly across all directions, with variance in each direction $1/h$.
We conclude that:
<script type="math/tex">(\sigma_{max}(M))^2 \gg \frac{1}{h} \sum_i (\sigma_i(M)^2),</script></p>
<p>which implies that the matrix has an uneven distribution of singular values. Ratio of left side and right side is called the <a href="https://nickhar.wordpress.com/2012/02/29/lecture-15-low-rank-approximation-of-matrices/"><em>stable rank</em></a> and is at most the linear algebraic rank. Furthermore, the above analysis suggests that the “signal” $x$ is <em>correlated</em> with the singular directions corresponding to the higher singular values, which is at the root of the noise stability.</p>
<p>Our experiments on VGG and GoogleNet reveal that the higher layers of deep nets—where most of the net’s parameters reside—do indeed exhibit a highly uneven distribution of singular values, and that the signal aligns more with the higher singular directions. The figure below describes layer 10 in VGG19 trained on CIFAR10.</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/aspectrumlayer10.png" width="45%" alt="distribution of singular values of matrix at layer 10 of VGG19" />
</p>
<h2 id="compressing-multilayer-net">Compressing multilayer net</h2>
<p>The above analysis of noise stability in terms of singular values cannot hold across multiple layers of a deep net, because the mapping becomes nonlinear, thus lacking a notion of singular values. Noise stability is therefore formalized using the <a href="https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant">Jacobian</a> of this mapping, which is the matrix describing how the output reacts to tiny perturbations of the input. Noise stability says that this nonlinear mapping passes signal (i.e., the vector from previous layers) much more strongly than it does a noise vector.</p>
<p>Our compression algorithm applies a randomized transformation to the matrix of each layer (aside: note the use of randomness, which fits in our “compressing with fixed string” framework) that relies on the low stable rank condition at each layer. This compression introduces error in the layer’s output, but the vector describing this error is “gaussian-like” due to the use of randomness in the compression. Thus this error gets attenuated by higher layers.</p>
<p>Details can be found in the paper. All noise stability properties formalized there are later checked in the experiments section.</p>
<h2 id="simpler-proofs-of-existing-generalization-bounds">Simpler proofs of existing generalization bounds</h2>
<p>In the paper we also use our compression framework to give elementary (say, 1-page) proofs of the previous generalization bounds from the past year. For example, the paper of <a href="https://openreview.net/forum?id=Skz_WfbCZ">Neyshabur et al.</a> shows the following is an upper bound on the generalization error where $A_i$ is the matrix describing the $i$th layer.</p>
<p style="text-align:center;">
<img src="/assets/saddle_eff/aexpression1.png" width="50%" alt="Expression for effective number of parameters in Neyshabur et al" />
</p>
<p>Comparing to the <em>basic theorem</em>, we realize the numerator corresponds to the number of effective parameters. The second part of the expression is the sum of stable ranks of the layer matrices, and is a natural measure of complexity. The first part is product of spectral norms (= top singular value) of the layer matrices, which happens to be an upper bound on the Lipschitz constant of the entire network. (Lipschitz constant of a mapping $f$ in this context is a constant $L$ such that $f(x) \leq L c\dot |x|$.)
The reason this is the Lipschitz constant is that if an input $x$ is presented at the bottom of the net, then each successive layer can multiply its norm by at most the top singular value, and the ReLU nonlinearity can only decrease norm since its only action is to zero out some entries.</p>
<p>Having decoded the above expression, it is clear how to interpret it as an analysis of a (deterministic) compression of the net. Compress each layer by zero-ing out (in the <a href="https://en.wikipedia.org/wiki/Singular-value_decomposition">SVD</a>) singular values less than some threshold $t|A|$, which we hope turns it into a low rank matrix. (Recall that a matrix with rank $r$ can be expressed using $2nr$ parameters.) A simple computation shows that the number of remaining singular values is at most the stable rank divided by $t^2$. How do we set $t$? The truncation introduces error in the layer’s computation, which gets propagated through the higher layers and magnified at most by the Lipschitz constant. We want to make this propagated error small, which can be done by making $t$ inversely proportional to the Lipschitz constant. This leads to the above bound on the number of effective parameters.</p>
<p>This proof sketch also clarifies how our work improves upon the older works: they are also (implicitly) compressing the deep net, but their analysis of how much compression is possible is much more pessimistic because they assume the network transmits noise at peak efficiency given by the Lipschitz constant.</p>
<h2 id="extending-the-ideas-to-convolutional-nets">Extending the ideas to convolutional nets</h2>
<p>Convolutional nets could not be dealt with cleanly in the earlier papers. I must admit that handling convolution stumped us as too for a while. A layer in a convolutional net applies the same filter to all patches in that layer. This <em>weight sharing</em> means that the full layer matrix already has a fairly compact representation, and it seems challenging to compress this further. However, in nets like VGG and GoogleNet, the higher layers use rather large filter matrices (i.e., they use a large number of channels), and one could hope to compress these individual filter matrices.</p>
<p>Let’s discuss the two naive ideas. The first is to compress the filter independently in different patches. This unfortunately is not a compression at all, since each copy of the filter then comes with its own parameters. The second idea is to do a single compression of the filter and use the compressed copy in each patch. This messes up the error analysis because the errors introduced due to compression in the different copies are now correlated, whereas the analysis requires them to be more like gaussian.</p>
<p>The idea we end up using is to compress the filters using $k$-wise independence (an idea from <a href="https://en.wikipedia.org/wiki/K-independent_hashing">theory of hashing schemes</a>), where $k$ is roughly logarithmic in the number of training samples.</p>
<h2 id="concluding-thoughts">Concluding thoughts</h2>
<p>While generalization theory can seem merely academic at times —since in practice held-out data establishes generalizaton— I hope you see from the above account that understanding generalization can give some interesting insights into what is going on in deep net training. Insights about noise stability of trained deep nets have obvious interest for study of biological neural nets. (See also the classic <a href="http://fab.cba.mit.edu/classes/862.16/notes/computation/vonNeumann-1956.pdf">von Neumann ideas</a> on noise resilient computation.)</p>
<p>At the same time, I suspect that compressibility is only one part of the generalization mystery, and that we are still missing some big idea. I don’t see how to use the above ideas to demonstrate that the effective number of parameters in VGG19 is as low as $50k$, as seems to be the case. I suspect doing so will force us to understand the structure of the data (in this case, real-life images) which the above analysis mostly ignores. The only property of data used is that the deep net aligns itself better with data than with noise.</p>
Sat, 17 Feb 2018 08:00:00 -0800
http://offconvex.github.io/2018/02/17/generalization2/
http://offconvex.github.io/2018/02/17/generalization2/