@prefix rdf: . @prefix sl: . @prefix skos: . @prefix rdfs: . @prefix tag: . @prefix foaf: . @prefix dc: . tag:bert_and_sentence_embeddings a sl:Tag ; skos:prefLabel "BERT + Sentence Embeddings" . tag:anisotropy_in_lm_space a sl:Tag ; rdfs:isDefinedBy ; skos:broader tag:language_model ; skos:prefLabel "Anisotropy in LM space" ; skos:related tag:sif_embeddings ; foaf:page tag:anisotropy_in_lm_space.html . dc:title "[2011.05864] On the Sentence Embeddings from Pre-trained Language Models" ; sl:comment "> **the sentence\r\nembeddings from the pre-trained language\r\nmodels without fine-tuning have been\r\nfound to poorly capture semantic meaning of\r\nsentences.**\r\n>\r\n> We find that **BERT always induces\r\na non-smooth anisotropic semantic space of\r\nsentences**, which harms its performance of\r\nsemantic similarity. To address this issue,\r\nwe propose to transform the anisotropic sentence\r\nembedding distribution to a smooth and\r\nisotropic Gaussian distribution through normalizing\r\nflows that are learned with an unsupervised\r\nobjective\r\n\r\n> normalizing flows (Dinh et al., 2015): invertible function parameterized by neural networks.\r\n> **During\r\ntraining, only the flow network is optimized\r\nwhile the BERT parameters remain unchanged**\r\n\r\n> When combined with external supervision from\r\nnatural language inference tasks (Bowman et al.,\r\n2015; Williams et al., 2018), our method outperforms\r\nthe [Sentence-BERT](tag:sbert) embeddings\r\n\r\n[GitHub](https://github.com/bohanli/BERT-flow)\r\n" ; sl:creationDate "2021-04-19" ; sl:tag tag:emnlp_2020 , tag:bert_and_sentence_embeddings , tag:arxiv_doc , tag:anisotropy_in_lm_space . tag:language_model a sl:Tag ; skos:prefLabel "Language Model" . tag:nlp_baidu a sl:Tag ; skos:prefLabel "NLP@Baidu" . dc:title "Isotropy in the Contextual Embedding Space: Clusters and Manifolds | OpenReview" ; sl:creationDate "2021-10-26" ; sl:tag tag:nlp_baidu , tag:anisotropy_in_lm_space . tag:emnlp_2020 a sl:Tag ; skos:prefLabel "EMNLP 2020" . tag:arxiv_doc a sl:Tag ; skos:prefLabel "Arxiv Doc" . tag:sif_embeddings a sl:Tag ; skos:prefLabel "SIF embeddings" .