Semanlink - Zero shot AND Arxiv Doc

> We address the problem of distance metric learning (DML), defined as learning a distance consistent with a notion of semantic similarity...
> Traditionnaly, supervision is expressed in the form of sets of points that follow
an ordinal relationship – an anchor point x is similar to
a set of positive points Y , and dissimilar to a set of negative
points Z, and a loss defined over these distances is minimized.
> Triplet-Based methods are challenging to optimize (a main issue is the need for finding informative triplets).
>
> We propose to **optimize the triplet loss on a different space of triplets, consisting of an anchor data point and similar and dissimilar proxy points which are learned as well**. These proxies approximate the original data points, so that a triplet loss over the proxies is a tight upper bound of the original loss.

Mentioned in this [blog post](/doc/2020/01/training_a_speaker_embedding_fr):

> "**Proxy based triplet learning**": instead of generating triplets, we learn an embedding for each class and use the learnt embedding as a proxy for triplets as part of the training. In other words, we can train end to end without the computationally expensive step of resampling triplets after each network update.

Near the conclusion:

> Our formulation of Proxy-NCA loss produces a loss very
similar to the standard cross-entropy loss used in classification.
However, we arrive at our formulation from a different
direction: we are not interested in the actual classifier and
indeed discard the proxies once the model has been trained.
Instead, the proxies are auxiliary variables, enabling more
effective optimization of the embedding model parameters.
**As such, our formulation not only enables us to surpass the
state of the art in zero-shot learning, but also offers an explanation
to the effectiveness of the standard trick of training
a classifier, and using its penultimate layer’s output as the
embedding.**

2020-02-09 About