Au sujet de ce document
- sl:arxiv_author :
- sl:arxiv_firstAuthor : Nikunj Saunshi
- sl:arxiv_num : 2202.14037
- sl:arxiv_published : 2022-02-28T18:59:20Z
- sl:arxiv_summary : Contrastive learning is a popular form of self-supervised learning that
encourages augmentations (views) of the same input to have more similar
representations compared to augmentations of different inputs. Recent attempts
to theoretically explain the success of contrastive learning on downstream
classification tasks prove guarantees depending on properties of {\em
augmentations} and the value of {\em contrastive loss} of representations. We
demonstrate that such analyses, that ignore {\em inductive biases} of the
function class and training algorithm, cannot adequately explain the success of
contrastive learning, even {\em provably} leading to vacuous guarantees in some
settings. Extensive experiments on image and text domains highlight the
ubiquity of this problem -- different function classes and algorithms behave
very differently on downstream tasks, despite having the same augmentations and
contrastive losses. Theoretical analysis is presented for the class of linear
representations, where incorporating inductive biases of the function class
allows contrastive learning to work with less stringent conditions compared to
prior analyses.@en
- sl:arxiv_title : Understanding Contrastive Learning Requires Incorporating Inductive Biases@en
- sl:arxiv_updated : 2022-02-28T18:59:20Z
- sl:bookmarkOf : https://arxiv.org/abs/2202.14037
- sl:creationDate : 2022-03-05
- sl:creationTime : 2022-03-05T11:25:53Z
Documents with similar tags (experimental)
Préférences...