Semanlink - AI teams

Parents:

Artificial Intelligence

AI teams

Related Tags:

Descendants

AI@Amazon
- NLP@Amazon
AI@Apple
AI@Facebook
- Facebook FAIR
  - faiss
  - ParlAI
- NLP@Facebook
  - BlenderBot
  - BLINK
  - CamemBERT
  - FastText
  - ParlAI
  - Poincaré Embeddings
  - StarSpace
- StarSpace
  [1511.03643] Unifying distillation and privileged information
  Tags:
  A framework to learn from multiple machines and data representations, unifying two techniques that enable machines to learn from other machines: [distillation](tag:knowledge_distillation) ([Hinton et al., 2015](doc:2020/04/1503_02531_distilling_the_kno)) and privileged information (Vapnik & Izmailov, 2015)
  
  2020-05-31 About
  [1912.08422] Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation
  Tags:
  distilling structured knowledge from a differentiable path-based recommendation model. > proposed framework can achieve state-of-the-art recommendation performance and meanwhile provide interpretable recommendation reasons
  
  2020-05-12 About
  [2003.08505] A Metric Learning Reality Check
  Tags:
  2020-05-10 About
  One-track minds: Using AI for music source separation
  Tags:
  - AI@Facebook
  - Music source separation
  2020-03-08 About
  Online speech recognition with wav2letter@anywhere
  Tags:
  - AI@Facebook
  - Speech-to-Text
  2020-02-12 About
  Improving long-form question answering by compressing search results
  Tags:
  Facebook AI has developed a new method that improves the performance of long-form question answering models by compressing the text that’s used to train them, turning 300,000-word sections into 10,000-word linearized knowledge graphs
  
  2019-10-24 About
AI@Google
- DeepMind
  - AlphaFold
  - Alphago
    - AlphaZero
  - Chichilla
- Gemini
- Google AI Blog
- Google Brain
- Google car
- NLP@Google
  - ALBERT
  - Bard
  - BERT
    - ALBERT
    - BERT + KB
      - BLP
        CKB
        
        raphaelsty/ckb: Contextual knowledge bases
        
        Tags:
        
        CKB
        GitHub project
        Raphaël Sourty
        Semanlink Tag Finder
        
        Une implémentation de [BLP](tag:blp) [[2010.03496] Inductive Entity Representations from Text via Link Prediction](doc:2020/11/2010_03496_inductive_entity_r)
        
        2020-11-09 About
        
        Daniel Daza sur Twitter : "BioBLP, a method for learning embeddings on multimodal knowledge graphs...."
        
        Tags:
        
        BLP
        Tweet
        
        2023-06-07 About
        
        [2010.03496] Inductive Entity Representations from Text via Link Prediction
        
        Tags:
        
        Arxiv Doc
        Attention + Knowledge Graphs
        BLP
        Discuté avec Raphaël
        Entity embeddings
        Good
        Link Prediction
        Text-Aware KG embedding
        TheWebConf 2021
        
        BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training) > a method for **learning representations of entities**, that uses a **pre-trained Transformer** based architecture as an entity encoder, and **link prediction training on a knowledge graph with textual entity descriptions**. > using entity descriptions, an entity encoder is trained for link prediction in a knowledge graph. The encoder can then be used without fine-tuning to obtain features for entity classification and information retrieval Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their objective targeted exclusively for link prediction (and not an objective that combines language modeling and link prediction as Kepler) performs better than Kepler's more complex one.
        
        2020-11-03 About
      - KnowBert
        
        [1909.04164] Knowledge Enhanced Contextual Word Representations
        
        Tags:
        
        Allen Institute for AI (AI2)
        Arxiv Doc
        Contextualized word representations
        EMNLP 2019
        Entities and LM
        Good
        Grounded Language Learning
        KD-MKB biblio
        KG-augmented Language Models
        KnowBert
        Knowledge-augmented Language Models
        Knowledge-driven embeddings
        Knowledge Graphs in NLP
        Multiple Knowledge Bases
        
        General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
        
        2020-05-13 About
      - LUKE
        
        [2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models
        
        Tags:
        
        Arxiv Doc
        Entities and LM
        Ikuya Yamada
        LUKE
        Masked entity prediction
        Multilingual NLP
        
        [Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_) > Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**. > > the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features. [Github](https://github.com/studio-ousia/luke) > Entity representations are known to enhance language models in mono-lingual settings (Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019: [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al., 2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context)) presumably by introducing real-world knowledge. We show that using entity representations facilitates cross-lingual transfer by providing languageindependent features. > > Multilingual extension of LUKE. The model is trained with the multilingual masked language modeling (MLM) task as well as the masked entity prediction (MEP) task with Wikipedia entity embeddings > We investigate two ways of using the entity representations in cross-lingual transfer tasks: > 1. perform entity linking for the input text, and append the detected entity tokens to the input sequence. The entity tokens are expected to provide language independent features to the model > 2. use the entity [MASK] token from the MEP task as a languageindependent feature extractor.
        
        2022-04-17 About
        
        Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."
        
        Tags:
        
        Entities and LM
        Ikuya Yamada
        LUKE
        Multilingual Language Models
        Tweet
        
        [[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](doc:2022/04/2110_08151_mluke_the_power_o) > mLUKE, an extension of [LUKE](tag:luke) based on 1M Wikidata entity embeddings shared across languages > mLUKE solves downstream tasks by using its language-agnostic entity embeddings as inputs. > entity representations are shared across languages during pretraining -> they are much more language-agnostic than word representations
        
        2022-04-13 About
        
        Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter
        
        Tags:
        
        LUKE
        Named Entity Recognition
        Tweet
        
        2022-03-15 About
        
        [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
        
        Tags:
        
        Arxiv Doc
        BERT + KB
        EMNLP 2020
        Entity embeddings
        Ikuya Yamada
        LUKE
        Self-Attention
        Text-Aware KG embedding
        
        > LUKE is based on bidirectional Transformer, treats words and entities in a text as independent tokens, and outputs contextualized representations of them. The representations can be used to address downstream tasks similarly to BERT. [src](https://twitter.com/ikuyamada/status/1312947499141750786) > LUKE is trained using a novel pretraining task that involves predicting randomly masked words (equivalent to BERT’s masked language model) and entities in an entity-annotated corpus obtained from Wikipedia. (Hum, ça me rappelle quelque chose) > LUKE also uses a new *entity-aware* self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores. [github](https://github.com/studio-ousia/luke), [at Hugging Face](https://twitter.com/AkariAsai/status/1389428550298525696), [doc](https://huggingface.co/transformers/model_doc/luke.html), [tweet](https://twitter.com/ikuyamada/status/1392742990586683392?s=20)
        
        2020-11-26 About
    - BERT + Sentence Embeddings
      - Sentence-BERT
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."
        
        Tags:
        
        Adapter modules (LM finetuning)
        EMNLP 2023
        Few-shot text classification
        Sentence-BERT: Domain adaptation
        Sentence embeddings: Domain adaptation
        SetFit (SBERT fine-tuning)
        Tweet
        
        AdaSent is an approach to creating domain-specialized sentence encoders for few-shot sentence classification > Reusable general sentence adapter across domains > AdaSent decouples DAPT (Domain Adaptative Pre-Training) & SEPT (Sentence Embedding Pre-Training) **by storing the sentence encoding abilities into an adapter**, which is trained only once in the general domain and plugged into various DAPT-ed PLMs [Github](https://github.com/UKPLab/AdaSent)
        
        2023-12-09 About
        
        Daniel Vila Suero sur Twitter : "Data quality is key for LLMs, but we're building Open Source LLMs with data of "unknown" quality... Introducing Alpaca GarbageCollector..."
        
        Tags:
        
        Alpaca
        Dataset quality
        SetFit (SBERT fine-tuning)
        Tweet
        
        > a cross-lingual SetFit model to identify potential bad instructions in Alpaca-like datasets
        
        2023-04-05 About
        
        ValueError "invalid literal for int() with base 10" in trainer.evaluate (dataset created from pandas) · Issue #228 · huggingface/setfit
        
        Tags:
        
        fps' post
        SetFit (SBERT fine-tuning)
        
        see <https://github.com/huggingface/setfit/blob/main/notebooks/zero-shot-classification.ipynb> > Note: some datasets on the Hugging Face Hub don't have a ClassLabel feature for the label column. In these cases, you should compute the candidate labels manually by first computing the id2label mapping as follows:
        
        2022-12-13 About
        
        Lewis Tunstall sur Twitter : "The SetFit library for few-shot learning with Sentence Transformers now supports *multi-label text classification*..."
        
        Tags:
        
        Multi-label Text classification
        SetFit (SBERT fine-tuning)
        Tweet
        
        Multilabel support [github issue](https://github.com/huggingface/setfit/issues/65)
        
        2022-10-14 About
        
        huggingface/setfit: Efficient few-shot learning with Sentence Transformers
        
        Tags:
        
        GitHub project
        SetFit (SBERT fine-tuning)
        
        2022-10-12 About
        
        [2209.11055] Efficient Few-Shot Learning Without Prompts
        
        Tags:
        
        Arxiv Doc
        Few-shot learning
        Moshe Wasserblat
        Nils Reimers
        SetFit (SBERT fine-tuning)
        
        [tweet](https://twitter.com/_akhaliq/status/1573109469646561280?s=20&t=RTpK9dh90az0zT1Xg2ohpQ): > So if I have 4 classes and say 2 labels per class, I would first fine tune an ST on these 4 pairs and then vectorize the 8 total examples for fine-tuning the classifier
        
        2022-09-23 About
        
        Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)
        
        Tags:
        
        Few-shot text classification
        Moshe Wasserblat
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        Finetuning d'un SBERT sur une tâche de classification (in fine, produit un SBERT) > **Few-shot text classification based on fine-tuning a Sentence Transformer with task-specific data** that can easily be implemented with the sentence-transformers library > Surprisingly, we did not find any work that performed an end-to-end ST fine-tuning for text classification in a Siamese manner. [COLAB](https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb) [Nils Reimers sur Twitter](doc:2022/03/nils_reimers_sur_twitter_gre)
        
        2022-03-31 About
        
        Nils Reimers sur Twitter : "Great post on SetFit"
        
        Tags:
        
        Nils Reimers
        SetFit (SBERT fine-tuning)
        Tweet
        
        About [Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat](doc:2022/03/sentence_transformer_fine_tunin) > - Outperforms GPT-3 in few-shot text-classification (50 labeled examples, secret test set) > - 1600 times smaller > - Can be run on your CPU > - No limitation on the number of training examples > - Just few lines of code needed
        
        2022-03-31 About
        
        LlamaIndex 🦙 sur X : "Fine-tuning Embedding Models for RAG with LoRA'
        
        Tags:
        
        LlamaIndex
        LoRA
        sBERT fine-tuning
        
        2024-04-23 About
        
        [2008.11228] A simple method for domain adaptation of sentence embeddings
        
        Tags:
        
        Arxiv Doc
        Domain adaptation (NLP)
        sBERT fine-tuning
        Sentence-BERT: Domain adaptation
        Universal Sentence Encoder
        
        2022-04-01 About
        
        [2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer
        
        Tags:
        
        ACL 2020
        Arxiv Doc
        Low-Resource NLP
        NLP@Amazon
        NLP@Microsoft
        sBERT fine-tuning
        
        > Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pre-trained model with those from a simple sentence embedding model trained only on the target data, can improve over the performance of FT for few-sample tasks
        
        2022-03-31 About
        
        Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)
        
        Tags:
        
        Few-shot text classification
        Moshe Wasserblat
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        Finetuning d'un SBERT sur une tâche de classification (in fine, produit un SBERT) > **Few-shot text classification based on fine-tuning a Sentence Transformer with task-specific data** that can easily be implemented with the sentence-transformers library > Surprisingly, we did not find any work that performed an end-to-end ST fine-tuning for text classification in a Siamese manner. [COLAB](https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb) [Nils Reimers sur Twitter](doc:2022/03/nils_reimers_sur_twitter_gre)
        
        2022-03-31 About
        
        Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium
        
        Tags:
        
        CamemBERT
        NLP: French
        NLP sample code
        sBERT fine-tuning
        
        2022-03-31 About
        
        [2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning
        
        Tags:
        
        Arxiv Doc
        Few-shot text classification
        sBERT fine-tuning
        Sentence-BERT
        Siamese networks
        
        > the problem of building text classifiers with little or no training data. > > In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. (cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis) > In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative. > > We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings) > The drop in quality can be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)** [Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)
        
        2022-03-30 About
        
        Sentence-BERT: Domain adaptation
        GPL (Generative Pseudo Labeling)
        
        Domain Adaptation with Generative Pseudo-Labeling (GPL) | Pinecone
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Pinecone
        
        2023-04-09 About
        
        [2205.11498] Domain Adaptation for Memory-Efficient Dense Retrieval
        
        Tags:
        
        Arxiv Doc
        Dense retriever
        Domain adaptation
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        
        Refers to [Binary Passage Retriever (BPR)](doc:2021/06/2106_00882_efficient_passage_)
        
        2022-09-26 About
        
        Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Multilingual NLP
        
        2022-06-02 About
        
        Nils Reimers sur Twitter : "GPL goes multi-lingual..."
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Tweet
        
        [Domain transfer with GGPL: German Generative Pseudo Labeling](doc:2022/06/domain_transfer_with_ggpl_germ)
        
        2022-06-01 About
        
        Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Tweet
        
        2022-04-27 About
        
        Domain Adaptation — Sentence-Transformers documentation
        
        Tags:
        
        Documentation
        Domain adaptation (NLP)
        GPL (Generative Pseudo Labeling)
        Sentence-BERT: Domain adaptation
        TSDAE
        
        2022-03-31 About
        
        NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        J'y étais
        Neural Search
        Nils Reimers
        Unsupervised Domain Adaptation (NLP)
        
        2022-03-09 About
        
        [2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
        
        Tags:
        
        Arxiv Doc
        Dense Passage Retrieval
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Synthetic QA data
        Unsupervised Domain Adaptation (NLP)
        
        An unsupervised domain adaptation technique for dense retrieval models 1. synthetic queries are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer) encoder-decoder) 2. the generated queries are used for mining negative passages (retrieving the most similar paragraphs using an existing dense retrieval model == hard negatives!) 3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted dense retriever (using method described in [Hofstätter et al., 2020](doc:2021/12/2010_02666_improving_efficien)) [Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl), by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans) Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598)) > - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data. > - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.
        
        2021-12-15 About
        
        sBERT fine-tuning
        
        UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."
        
        Tags:
        
        Adapter modules (LM finetuning)
        EMNLP 2023
        Few-shot text classification
        Sentence-BERT: Domain adaptation
        Sentence embeddings: Domain adaptation
        SetFit (SBERT fine-tuning)
        Tweet
        
        AdaSent is an approach to creating domain-specialized sentence encoders for few-shot sentence classification > Reusable general sentence adapter across domains > AdaSent decouples DAPT (Domain Adaptative Pre-Training) & SEPT (Sentence Embedding Pre-Training) **by storing the sentence encoding abilities into an adapter**, which is trained only once in the general domain and plugged into various DAPT-ed PLMs [Github](https://github.com/UKPLab/AdaSent)
        
        2023-12-09 About
        
        [2008.11228] A simple method for domain adaptation of sentence embeddings
        
        Tags:
        
        Arxiv Doc
        Domain adaptation (NLP)
        sBERT fine-tuning
        Sentence-BERT: Domain adaptation
        Universal Sentence Encoder
        
        2022-04-01 About
        
        Domain Adaptation — Sentence-Transformers documentation
        
        Tags:
        
        Documentation
        Domain adaptation (NLP)
        GPL (Generative Pseudo Labeling)
        Sentence-BERT: Domain adaptation
        TSDAE
        
        2022-03-31 About
        
        Unsupervised Training for Sentence Transformers | Pinecone
        
        Tags:
        
        Pinecone
        Sentence-BERT: Domain adaptation
        TSDAE
        Unsupervised Sentence Embedding Learning
        
        Blog post about [[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](doc:2021/09/2104_06979_tsdae_using_trans) > Fine-tuning with TSDAE simply cannot compete in terms of performance against supervised methods. However, **the point and value of TSDAE is that it allows us to fine-tune models for use-cases where we have no data**. Specific domains with unique terminology or low resource languages.
        
        2021-11-24 About
        
        SentenceBERT and Words
        
        Few-Shot Text Classification (Cloudera 2020)
        
        Tags:
        
        Few-shot text classification
        SentenceBERT and Words
        
        > Sentence-BERT has been optimized… well, for sentences! It’s reasonable to suspect that SBERT’s representations of single words or short phrases like “Business” or “Science & Technology” won’t be as semantically relevant as representations derived from a word-level method, like word2vec or GloVe
        
        2022-11-24 About
        
        skeskinen/bert.cpp: ggml implementation of BERT
        
        Tags:
        
        .cpp
        BERT
        Frugal LLMs
        Quantization
        Sentence-BERT
        Sentence Embeddings
        
        > ggml inference of BERT neural net architecture with pooling and normalization from SentenceTransformers (sbert.net). High quality sentence embeddings in pure C++ (with C API). > > The main goal of bert.cpp is to run the BERT model using **4-bit integer quantization on CPU**
        
        2023-05-09 About
        
        Multilingual Sentence Transformers | Pinecone
        
        Tags:
        
        Multilingual embeddings
        Pinecone
        Sentence-BERT
        
        Focus on **Multilingual Knowledge Distillation** > recent method introduced by Nils Reimers and Iryna Gurevych in 2020 > The teacher model is an already fine-tuned sentence transformer used for creating embeddings in a single language (most likely English). The student model is a transformer that has been pretrained on a multilingual corpus.
        
        2023-01-13 About
        
        Unsupervised Learning — Sentence-Transformers documentation
        
        Tags:
        
        Sentence-BERT
        TSDAE
        Unsupervised Sentence Embedding Learning
        
        > In our paper TSDAE we compare approaches for sentence embedding tasks, and in GPL we compare them for semantic search tasks (given a query, find relevant passages). While the unsupervised approach achieve acceptable performances for sentence embedding tasks, they perform poorly for semantic search tasks.
        
        2022-08-20 About
        
        Train and Fine-Tune Sentence Transformers Models
        
        Tags:
        
        Sentence-BERT
        
        2022-08-13 About
        
        sentence bert model in onnx format · Issue #46 · UKPLab/sentence-transformers
        
        Tags:
        
        ONNX
        Raphaël Sourty
        Sentence-BERT
        
        2022-06-13 About
        
        [2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain
        
        Tags:
        
        Arxiv Doc
        Knowledge Graph + Deep Learning
        Question Answering
        Sentence-BERT
        
        2022-06-11 About
        
        [2205.04260] EASE: Entity-Aware Contrastive Learning of Sentence Embedding
        
        Tags:
        
        Arxiv Doc
        Contrastive Learning
        Entities
        Entities and LM
        Entity type
        Ikuya Yamada
        Sentence-BERT
        Sentence Embeddings
        
        > we explore a type of supervision that has been under-explored in the literature: entity hyperlink annotations from Wikipedia. > > entities have been shown to be a strong indicator of text semantics > > a method for mining hard negatives based on the entity type Uses wikipedia2vec > the reliance on Wikipedia for training data may limit the application of the models to specific domains (e.g., general or encyclopedia domains). To apply EASE to other domains, one may need to annotate text from the domain either manually or automatically.
        
        2022-05-11 About
        
        [2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning
        
        Tags:
        
        Arxiv Doc
        Few-shot text classification
        sBERT fine-tuning
        Sentence-BERT
        Siamese networks
        
        > the problem of building text classifiers with little or no training data. > > In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. (cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis) > In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative. > > We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings) > The drop in quality can be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)** [Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)
        
        2022-03-30 About
        
        Document Matching for Job Descriptions | Semantic Scholar (2021)
        
        Tags:
        
        NLP@Stanford
        NLP + Human Resources
        Sentence-BERT
        
        > We train a document encoder to match online job descriptions to one of many standardized job roles from Singapore’s Skills Framework. The encoder generates semantically meaningful document encodings from textual descriptions of job roles, which are then compared using Cosine Similarity to determine matching. During training, we implement the methodology used by Sentence-BERT, fine tuning pre-trained BERT models using a siamese network architecture on labelled document pairs.
        
        2022-03-09 About
        
        Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter
        
        Tags:
        
        Annotations (ML)
        Chatbots
        Nils Reimers
        Sentence-BERT
        Spectral clustering
        Tutorial
        Tweet
        
        2022-02-19 About
        
        Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."
        
        Tags:
        
        Nils Reimers
        Sentence-BERT
        Short Text Clustering
        Tweet
        
        Clustering millions of sentences to optimize the ML-workflow
        
        2022-02-19 About
        
        sentence-transformers/fast_clustering.py at master · UKPLab/sentence-transformers
        
        Tags:
        
        Clustering
        Nils Reimers
        Sentence-BERT
        
        > This is a more complex example on performing clustering on large scale dataset. This examples find in a large set of sentences local communities, i.e., groups of sentences that are highly similar. You can freely configure the threshold what is considered as similar. A high threshold will only find extremely similar sentences, a lower threshold will find more sentence that are less similar. A second parameter is 'min_community_size': Only communities with at least a certain number of sentences will be returned. The method for finding the communities is extremely fast, for clustering 50k sentences it requires only 5 seconds (plus embedding comuptation). In this example, we download a large set of questions from Quora and then find similar questions in this set.
        
        2022-02-18 About
        
        gsarti/scibert-nli · Hugging Face
        
        Tags:
        
        Hugging Face
        NLP for scientific documents
        Sentence-BERT
        
        SciBERT fine-tuned on the SNLI and the MultiNLI datasets using the sentence-transformers library to produce universal sentence embeddings
        
        2022-01-29 About
        
        Semantic Search — Sentence-Transformers documentation
        
        Tags:
        
        Documentation
        Semantic Search
        Sentence-BERT
        
        **symmetric** semantic search vs **asymmetric** semantic search > - Suitable models for symmetric semantic search: Pre-Trained Sentence Embedding > - Suitable models for asymmetric semantic search: Pre-Trained MS MARCO Models
        
        2022-01-29 About
        
        Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers
        
        Tags:
        
        Cross-Encoders
        Sentence-BERT
        
        > so would it be a good idea to finetune a SBERT model on a cross-encoder task? > > The SBERT models are regular transformers model and hence can be used as base for cross encoders. Sometimes it could be helpful, otherwise it is better to use the original models. ([Nils Reimers](tag:nils_reimers))
        
        2021-12-17 About
        
        Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science
        
        Tags:
        
        BERT
        Cross-Encoders
        Data Augmentation
        Sentence-BERT
        Sentence Similarity
        
        Data Augmentation Method to improve SBERT Bi-Encoders for Pairwise Sentence Scoring Tasks (Semantic sentence tasks)
        
        2021-12-17 About
        
        Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine
        
        Tags:
        
        Semantic Search
        Sentence-BERT
        Similarity queries / Vector search
        Weaviate
        Wikipedia
        
        2021-12-05 About
        
        Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople
        
        Tags:
        
        Extractive Text Summarization
        GitHub project
        Sentence-BERT
        
        Unsupervised Extractive Text Summarization and Semantic Search [Github](https://github.com/Hellisotherpeople/CX_DB8)
        
        2021-12-03 About
        
        How to Fine-Tune Sentence-BERT for Question Answering | Capital One
        
        Tags:
        
        Question Answering
        Sentence-BERT
        Tutorial
        
        > tutorial on using the sentence-transformers library to fine-tune Sentence-BERT for question matching
        
        2021-11-21 About
        
        Multilingual Sentence Transformers | Pinecone
        
        Tags:
        
        Knowledge distillation
        Multilingual Language Models
        Pinecone
        Sentence-BERT
        Transformers
        
        How to make a text encoder multilingual using sentence transformers and multilingual knowledge distillation.
        
        2021-11-04 About
        
        Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone
        
        Tags:
        
        Multiple negatives ranking (MNR) loss
        Pinecone
        Sentence-BERT
        Sentence Embeddings
        
        > the world of sentence embeddings was ignited with the introduction of SBERT in 2019. Since then, many more sentence transformers have been introduced. These models quickly made the original SBERT obsolete. How did these newer sentence transformers manage to outperform SBERT so quickly? The answer is **multiple negatives ranking (MNR) loss**. > In short; **fine-tune your models with MNR loss, and do it with the [sentence-transformers](tag:sbert) library**. (mentionned in a [tweet](https://twitter.com/Nils_Reimers/status/1453001422400856086) by [Nils Reimers](tag:nils_reimers))
        
        2021-10-27 About
        
        Sentence Embeddings and Transformers | Pinecone
        
        Tags:
        
        Pinecone
        Sentence-BERT
        
        2021-10-23 About
        
        Sahajtomar/french_semantic · Hugging Face
        
        Tags:
        
        CamemBERT
        Discuté avec Raphaël
        huggingface / transformers
        NLP: French
        Sentence-BERT
        
        2021-10-14 About
        
        Semantic Search with S-BERT is all you need
        
        Tags:
        
        Howto
        Semantic Search
        Sentence-BERT
        
        > SentenceTransformers is designed in such way that fine-tuning your own sentence / text embeddings models is easy.
        
        2021-06-05 About
        
        Nils Reimers sur Twitter : "SBERT Release v1.1.0"
        
        Tags:
        
        Nils Reimers
        Sentence-BERT
        Tweet
        
        2021-04-22 About
        
        Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."
        
        Tags:
        
        Information retrieval
        Nils Reimers
        Sentence-BERT
        Tweet
        
        2021-04-17 About
        
        SentenceTransformers Documentation
        
        Tags:
        
        Documentation
        Sentence-BERT
        
        2021-03-25 About
        
        Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)
        
        Tags:
        
        Hugging Face
        Natural Language Inference
        Sentence-BERT
        Text Classification
        Zero-Shot Learning
        Zero-shot Text Classifier
        
        > state-of-the-art NLP models for sequence classification without large annotated training sets. Simple idea: use a single model (eg. [Sentence-BERT](tag:sbert)) to embed both the text data and the class names into the same space. Pb: Sentence-BERT is designed to learn effective sentence-level, not single- or multi-word representations like our class names -> the label embeddings may not be as semantically salient as word-level embedding methods (i.e. word2vec). Solution 1: Learn a projection from sentence level embeddings of words to word2vec embeddings, use it for encoding when learning classifier. Can be adapted to few short learning Solution 2: "Classification as [#Natural Language Inference](tag:nli)". > A method which not only embeds sequences and labels into the same latent space where their distance can be measured, but that can actually tell us something about the compatibility of two distinct sequences out of the box.
        
        2021-02-23 About
        
        UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet
        
        Tags:
        
        GitHub project
        huggingface / transformers
        Sentence-BERT
        XLNet
        
        [paper](doc:2019/08/_1908_10084_sentence_bert_sen)
        
        2020-07-14 About
        
        Richer Sentence Embeddings using Sentence-BERT — Part I
        
        Tags:
        
        Sentence-BERT
        
        Simplistic (and often used) methods for sentence embeddings with BERT are too simplistic to be good (avearaging the word vectors, or using the \[CLS\] special vector (start of sequence). [About this paper](/doc/2019/08/_1908_10084_sentence_bert_sen)
        
        2020-01-06 About
        
        [1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
        
        Tags:
        
        Arxiv Doc
        EMNLP 2019
        huggingface / transformers
        Nearest neighbor search
        Sentence-BERT
        Sentence Similarity
        Siamese networks
        
        > Sentence-BERT (SBERT), a modification of the pretrained BERT network that use siamese and triplet network structures to derive **semantically meaningful sentence embeddings** that can be compared using cosine-similarity. Important because - BERT ist unsuitable for semantic similarity search as well as for unsupervised tasks like clustering. - simple methods such as using the CLS token give low quality sentence embeddings However, the purpose of SBERT sentence embeddings are **not to be used for transfer learning for other tasks**. [Related blog post](/doc/2020/01/richer_sentence_embeddings_usin); [Github](https://github.com/UKPLab/sentence-transformers)
        
        2019-08-28 About
    - BERT-based entity linking
    - BERT fine-tuning
      - BERT fine-tuning: instabilities
        
        On Stability of Few-Sample Transformer Fine-Tuning | Kaggle
        
        Tags:
        
        BERT fine-tuning: instabilities
        Fine-tuning
        Kaggle
        
        [[2006.05987] Revisiting Few-sample BERT Fine-tuning](doc:2022/03/2006_05987_revisiting_few_sam)
        
        2022-08-29 About
        
        [2006.05987] Revisiting Few-sample BERT Fine-tuning
        
        Tags:
        
        ADAM (ML)
        Arxiv Doc
        BERT fine-tuning
        BERT fine-tuning: instabilities
        NLP@Stanford
        
        > A study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. > The most commonly used optimizer for fine-tuning BERT is BERTADAM, a modified version of the ADAM first-order stochastic optimization method. It differs from the original ADAM algorithm (Kingma & Ba, 2014) in omitting a bias correction step. > > ... We observe that the bias correction omission influences the learning rate, especially early in the fine-tuning process, and is one of the primary reasons for instability in fine-tuning BERT and this is bad when finetuning with less than 10K samples. Pb included in many > open source libraries, including the official implementation huggingface’s Transformers How to solve pb in HuggingFace? > HuggingFace Transformers AdamW has correct_bias parameter set to True by default. Still it's worth noting the importance this parameter serves. [src](doc:2022/08/on_stability_of_few_sample_tran)
        
        2022-03-21 About
        
        On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2021)
        
        Tags:
        
        BERT fine-tuning
        BERT fine-tuning: instabilities
        Catastrophic forgetting
        Vanishing gradient
        
        > **an analysis of the fine-tuning instability of BERT-based models and a simple method to fix it** > > Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. > > 2 potential reasons identified in (Devlin et al., 2019; Lee et al., 2020; Dodge et al., 2020) : > - catastrophic forgetting > - small size of the fine-tuning datasets. > > we show that both hypotheses fail to explain the fine-tuning instability, which is caused by optimization difficulties / **vanishing gradients**). > > A simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches. > > [Github](https://github.com/uds-lsv/bert-stable-fine-tuning)
        
        2021-10-30 About
    - Bertology
    - BERTopic
    - CamemBERT
    - DistilBERT
    - Domain-Specific BERT
    - KeyBERT
    - KnowBert
      An efficient long-text semantic retrieval approach via utilizing presentation learning on short-text | Complex & Intelligent Systems (2023)
      
      Tags:
      
      BERT
      Information retrieval
      Long documents
      
      long-text retrieval model based on BERT (called LTR-BERT)
      
      2024-01-31 About
      Rethinking Query Expansion for BERT Reranking | Advances in Information Retrieval (2020)
      
      Tags:
      
      BERT
      Information retrieval
      Query Expansion
      
      using BERT for Information Retrieval: > We find that traditional word-based query expansion is not entirely applicable
      
      2023-10-29 About
      [2002.06275] TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
      
      Tags:
      
      Arxiv Doc
      BERT
      Knowledge distillation
      Neural Search
      NLP@Microsoft
      
      2023-08-27 About
      skeskinen/bert.cpp: ggml implementation of BERT
      
      Tags:
      
      .cpp
      BERT
      Frugal LLMs
      Quantization
      Sentence-BERT
      Sentence Embeddings
      
      > ggml inference of BERT neural net architecture with pooling and normalization from SentenceTransformers (sbert.net). High quality sentence embeddings in pure C++ (with C API). > > The main goal of bert.cpp is to run the BERT model using **4-bit integer quantization on CPU**
      
      2023-05-09 About
      Classifying long textual documents (up to 25 000 tokens) using BERT | by Sinequa | (2020)
      
      Tags:
      
      BERT
      Long documents
      Representation of text + metadata
      Sinequa
      Text Classification
      
      > long text + additional textual metadata (such as title, abstract …) and categories (location, authors …).
      
      2023-04-07 About
      Prompt Tuning BERT🎯:CommonLit Readability | Kaggle
      
      Tags:
      
      BERT
      NLP sample code
      Prompting/In-context learning
      
      > Prompt-tuning is a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks.Soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Finally, we show that conditioning,a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.
      
      2022-09-16 About
      Active Learning for BERT: An Empirical Study - ACL Anthology
      
      Tags:
      
      Active learning
      BERT
      EMNLP 2020
      Imbalanced Data
      NLP@IBM
      Text Classification
      
      > The use of Actice Learning (AL) with deep pre-trained models has so far received little consideration. > > We study the potential of (i) various AL strategies; (ii) in conjunction with BERT, (iii) within a highly challenging – yet common – real-world scenario of class imbalance and scarce labeled data. focused on binary classification > AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. [Github](https://github.com/IBM/low-resource-text-classification-framework)
      
      2022-09-02 About
      Using BERT For Classifying Documents with Long Texts | by Armand Olivares | Medium
      
      Tags:
      
      BERT
      Long documents
      
      2022-06-29 About
      [1909.00426] Global Entity Disambiguation with BERT
      
      Tags:
      
      Arxiv Doc
      BERT
      Entity Disambiguation
      Ikuya Yamada
      
      2022-04-18 About
      MaartenGr/BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.
      
      Tags:
      
      BERT
      BERTopic
      GitHub project
      Topic Modeling
      
      > topic modeling technique that leverages 🤗 transformers and [c-TF-IDF](https://github.com/MaartenGr/cTFIDF) to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions. refers to [Top2Vec](doc:2022/03/ddangelov_top2vec_top2vec_lear) [youtube](https://www.youtube.com/watch?v=Qub3PrFvauI) [tweet](https://twitter.com/JayAlammar/status/1594681648121102336?s=20&t=R0G_LrajK9WBtzypwXtD7Q)
      
      2022-03-10 About
      [2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
      
      Tags:
      
      Arxiv Doc
      BERT
      Phrase embeddings
      
      2022-02-25 About
      Domain Transfer with BERT | Pinecone
      
      Tags:
      
      BERT
      Domain adaptation (NLP)
      Pinecone
      
      2022-01-04 About
      Making the Most of Data: Augmentation with BERT | Pinecone
      
      Tags:
      
      BERT
      Cross-Encoders
      Pinecone
      Synthetic data (NLP)
      
      2021-12-18 About
      Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science
      
      Tags:
      
      BERT
      Cross-Encoders
      Data Augmentation
      Sentence-BERT
      Sentence Similarity
      
      Data Augmentation Method to improve SBERT Bi-Encoders for Pairwise Sentence Scoring Tasks (Semantic sentence tasks)
      
      2021-12-17 About
      [2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles
      
      Tags:
      
      Arxiv Doc
      BERT
      BM25
      Embeddings in Information Retrieval
      Semantic Search
      
      2021-04-12 About
      [1902.00751] Parameter-Efficient Transfer Learning for NLP
      
      Tags:
      
      Adapter modules (LM finetuning)
      Arxiv Doc
      BERT
      Continual Learning
      NLP@Google
      Transfer learning in NLP
      
      **Adapter tuning for NLP**. A strategy for tuning a large text model on several downstream tasks, that permits training on tasks sequentially, and that adds only a small number of additional parameters per task. New modules added between layers of a pre-trained network. Parameters of the original network are frozen and therefore may be shared by many tasks. [GitHub google-research/adapter-bert](https://github.com/google-research/adapter-bert)
      
      2021-04-11 About
      exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology
      
      Tags:
      
      BERT
      Biomedical NLP
      Domain adaptation (NLP)
      Domain-Specific BERT
      EMNLP 2020
      NLP@Harvard
      
      **Focus on the Embedding of Domain-specific Vocabulary.** > exBERT adds a new domain-specific vocabulary and the corresponding embedding layer, as well as a small extension module to the original unmodified model > a pretraining method allowing **low-cost embedding of domain-specific vocabulary in the context of an existing large pre-trained model such as BERT** > exBERT... explicitly incorporates the new domain’s vocabulary, while being able to **reuse the original pre-trained model’s weights as is** to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting its embeddings for the original vocabulary with new embeddings for the domain-specific vocabulary via **a learned small “extension” module**. **The output of the original and extension modules are combined via a trainable weighted sum operation** In a way similar to concept developed in > [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm. [Github](https://github.com/cgmhaicenter/exBERT)
      
      2021-04-11 About
      [1901.04085] Passage Re-ranking with BERT
      
      Tags:
      
      Arxiv Doc
      BERT
      Learning to rank
      NLP@Facebook
      Ranking (information retrieval)
      
      a simple re-implementation of BERT for query-based passage re-ranking ["Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"](doc:2021/03/rodrigo_nogueira_sur_twitter_)
      
      2021-03-26 About
      Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"
      
      Tags:
      
      BERT
      Learning to rank
      Ranking (information retrieval)
      Slides
      Tutorial
      Tweet
      
      2021-03-09 About
      kamalkraj/BERT-NER: Pytorch-Named-Entity-Recognition-with-BERT
      
      Tags:
      
      BERT
      GitHub project
      Named Entity Recognition
      
      Use google BERT to do CoNLL-2003 NER !
      
      2021-02-07 About
      [1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
      
      Tags:
      
      Arxiv Doc
      BERT
      BERT + KB
      Discuté avec Raphaël
      Entity embeddings
      Text, KG and embeddings
      
      > way of **injecting factual knowledge about entities into the pretrained BERT model**. (Feeding entity vectors into BERT as if they were wordpiece vectors without additional encoder pretraining) > > **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**. > > Our vector space alignment strategy is inspired by cross-lingual word vector alignment Related work on Entity-enhanced BERT: > ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle that BERT be adapted to entity vectors. They introduce new encoder layers to feed pretrained entity vectors into the Transformer, and they require additional pretraining to integrate the new parameters. In contrast, E-BERT’s design principle is that entity vectors be adapted to BERT. > > Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_) (Wang et al., 2019c) and K-Adapter (Wang et al., 2020)... Their factual knowledge does not stem from entity vectors – instead, they are trained in a multi-task setting on relation classification and knowledge base completion. Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)
      
      2021-01-12 About
      Keyword Extraction with BERT | Towards Data Science
      
      Tags:
      
      BERT
      Keyword/keyphrase extraction
      
      A minimal method for extracting keywords and keyphrases. [GitHub](https://github.com/MaartenGr/KeyBERT/) > uses BERT-embeddings and simple cosine similarity to find the sub-phrases in a document that are the most similar to the document itself.
      
      2020-12-06 About
      Salmon Run: Word Sense Disambiguation using BERT as a Language Model
      
      Tags:
      
      BERT
      huggingface / transformers
      NLP sample code
      Word-sense disambiguation
      
      2020-12-01 About
      Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science
      
      Tags:
      
      BERT
      huggingface / transformers
      Question Answering
      Tutorial
      
      A guide to choosing and benchmarking BERT models for question answering
      
      2020-10-04 About
      BERT Word Embeddings Tutorial · Chris McCormick
      
      Tags:
      
      BERT
      Tutorial
      Word embeddings
      
      2020-07-06 About
      [1909.03193] KG-BERT: BERT for Knowledge Graph Completion
      
      Tags:
      
      Arxiv Doc
      BERT
      Discuté avec Raphaël
      Knowledge Graph Completion
      Pre-Trained Language Models
      Transformers
      
      Pre-trained language models for knowledge graph completion. **Triples are treated as textual sequences**. (Hum, j'ai déjà vu ça quelque part. Ah, peut-être [RDF2VEC](tag:rdf2vec)? // TODO à voir) Takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model > we first treat entities, relations and triples as textual sequences and turn knowledge graph completion into a sequence classification problem. We then fine-tune BERT model on these sequences for predicting the plausibility of a triple or a relation. The method [GitHub](https://github.com/yao8839836/kg-bert)
      
      2020-03-22 About
      [1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph
      
      Tags:
      
      Arxiv Doc
      BERT
      Knowledge Graph + Deep Learning
      Knowledge Graphs in NLP
      Language Model
      
      a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge (Summarized in [Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)](doc:2022/03/domain_adaptation_of_word_embed), p43)
      
      2020-03-08 About
      Unsupervised NER using BERT - Hands-on NLP model review - Quora
      
      Tags:
      
      BERT
      Named Entity Recognition
      
      [GitHub](https://github.com/ajitrajasekharan/unsupervised_NER)
      
      2020-03-06 About
      [2002.12327] A Primer in BERTology: What we know about how BERT works
      
      Tags:
      
      Arxiv Doc
      BERT
      Bertology
      Survey / Review
      
      (article praised on [twitter](https://twitter.com/dennybritz/status/1233343170596917248?s=20) by D Britz and Y. Goldberg)
      
      2020-02-28 About
      [2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia
      
      Tags:
      
      Arxiv Doc
      BERT
      Conditional random fields
      Named Entity Recognition
      Wikipedia
      
      2020-02-27 About
      Distilling BERT models with spaCy - Towards Data Science (2019)
      
      Tags:
      
      BERT
      Knowledge distillation
      spaCy
      Transfer learning in NLP
      Yves Peirsman
      
      2020-02-15 About
      [2002.02925] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
      
      Tags:
      
      Arxiv Doc
      BERT
      Knowledge distillation
      NLP@Microsoft
      
      approach to compress BERT by progressive module replacing. > Compared to the previous knowledge distillation approaches for BERT compression, our approach leverages only one loss function and one hyper-parameter [Github](https://github.com/JetRunner/BERT-of-Theseus)
      
      2020-02-10 About
      Canwen Xu sur Twitter : "WTF? We brutally dismember BERT and replace all his organs?"
      
      Tags:
      
      BERT
      Knowledge distillation
      NLP@Microsoft
      
      [paper](/doc/2020/02/_2002_02925_bert_of_theseus_c)
      
      2020-02-10 About
      Building a Search Engine with BERT and TensorFlow - Towards Data Science
      
      Tags:
      
      BERT
      Nearest neighbor search
      NLP and Search
      Search Engines
      TensorFlow
      
      [somewhat related](/doc/2020/01/elasticsearch_meets_bert_build)
      
      2020-01-12 About
      Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT
      
      Tags:
      
      BERT
      ElasticSearch
      GitHub project
      NLP and Search
      Text Similarity
      
      - Links to [this ES blog post](/doc/2020/01/text_similarity_search_in_elast) - [somewhat related](/doc/2020/01/building_a_search_engine_with_b)
      
      2020-01-10 About
      [2003.05473] Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNNL 2019)
      
      Tags:
      
      Arxiv Doc
      BERT
      Bertology
      Discuté avec Raphaël
      End-To-End Entity Linking
      Good
      
      Training BERT-base-uncased on English Wikipedia and then fine-tuned and evaluating it on an entity linking (EL) benchmark (EL implemented as a token classification over the entity vocabulary) > BERT+Entity is a straightforward extension on top of BERT, i.e. we initialize BERT with the publicly available weights from the BERT-base-uncased model and add an output classification layer on top of the architecture. Given a contextualized token, the classifier computes the probability of an entity link for each entry in the entity vocabulary. Can BERT’s architecture learn all entity linking steps jointly? To answer: > an extreme simplification of the **entity linking setup that works surprisingly well**: simply cast it as **a per token classification over the entire entity vocabulary** (over 700K classes in our case). > the model is the first that performs entity linking without any pipeline or any heuristics, compared to all prior approaches. We found that with our approach we can learn additional entity knowledge in BERT that helps in entity linking. **However, we also found that almost none of the downstream tasks really required entity knowledge**. ### Related work - > [Durrett and Klein (2014)](/doc/2020/01/a_joint_model_for_entity_analys) were the first to propose jointly modelling Mention detection, Candidate generation and Entity disambiguation in a graphical model and could show that each of those steps are interdependent and benefit from a joint objective This paper uses neural techniques instead of CRF. - > [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) (2016, 2017) was the first to investigate neural text representations and entity linking, but their approach is limited to ED. cf. [#Wikipedia2Vec](tag:wikipedia2vec). Compare with [newer work by Yamada](doc:2020/09/1909_01259_neural_attentive_b)
      
      2020-01-09 About
      Named Entity Recognition with Bert – Depends on the definition
      
      Tags:
      
      BERT
      Hugging Face
      Named Entity Recognition
      NLP sample code
      
      > how you can finetune the Bert model to do state-of-the art named entity recognition Same author: [NER with Lime](/doc/2020/01/interpretable_named_entity_reco)
      
      2020-01-09 About
      [1902.10909] BERT for Joint Intent Classification and Slot Filling
      
      Tags:
      
      Arxiv Doc
      BERT
      Intent classification and slot filling
      NLP@Alibaba
      
      > Experimental results show that our proposed joint BERT model outperforms BERT models modeling intent classification and slot filling separately, demonstrating the efficacy of exploiting the relationship between the two tasks. Adding a CRF on top of the model doesn't improve the results.
      
      2020-01-09 About
      Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019
      
      Tags:
      
      BERT
      Contextualized word representations
      ELMo
      NLP@Stanford
      
      2020-01-05 About
      BERT is now part of Google Search, so let’s understand how it reasons
      
      Tags:
      
      BERT
      Bertology
      NLP: negation
      
      2019-10-31 About
      Meet ALBERT: a new ‘Lite BERT’ from Google & Toyota with State of the Art NLP performance and 18x fewer parameters.
      
      Tags:
      
      ALBERT
      BERT
      NLP@Google
      Toyota
      
      2019-10-01 About
      Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT
      
      Tags:
      
      BERT
      Hugging Face
      Knowledge distillation
      
      2019-08-28 About
      [1905.07129] ERNIE: Enhanced Language Representation with Informative Entities
      
      Tags:
      
      ACL 2019
      Arxiv Doc
      BERT
      Discuté avec Raphaël
      ERNIE
      Knowledge Graphs in NLP
      Text, KG and embeddings
      Transformers
      
      > We argue that informative entities in **KGs can enhance language representation with external knowledge**. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. > ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks [GitHub](https://github.com/thunlp/ERNIE) WARNING, there is another ERNIE (by [NLP@Baidu](tag:nlp_baidu)): Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. This doesn't happen when you choose François-Paul as the name for your child.
      
      2019-08-05 About
      What is XLNet and why it outperforms BERT - Towards Data Science
      
      Tags:
      
      BERT
      XLNet
      
      2019-08-02 About
      BERT's success in some benchmarks tests may be simply due to the exploitation of spurious statistical cues in the dataset. Without them it is no better then random. : MachineLearning
      
      Tags:
      
      BERT
      Bertology
      
      2019-07-24 About
      [1907.07355] Probing Neural Network Comprehension of Natural Language Arguments
      
      Tags:
      
      Arxiv Doc
      BERT
      Bertology
      
      what has BERT learned about argument comprehension? [Comments](/doc/2019/07/bert_s_success_in_some_benchmar)
      
      2019-07-24 About
      [1906.04341] What Does BERT Look At? An Analysis of BERT's Attention
      
      Tags:
      
      Arxiv Doc
      Attention mechanism
      BERT
      Bertology
      Chris Manning
      What's encoded by a NN
      
      2019-06-21 About
      NLP: Contextualized word embeddings from BERT – Towards Data Science
      
      Tags:
      
      BERT
      
      2019-06-12 About
      [1906.02715] Visualizing and Measuring the Geometry of BERT
      
      Tags:
      
      Arxiv Doc
      BERT
      Bertology
      Geometry of language embeddings
      NLP@Google
      Thèse IRIT-Renault: biblio
      Tree embeddings
      
      > At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses. We also present empirical descriptions of syntactic representations in both attention matrices and individual word embeddings, as well as a mathematical argument to explain the geometry of these representations
      
      2019-06-07 About
      "I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"
      
      Tags:
      
      BERT
      Humour
      Naive Bayes classifier
      Rada Mihalcea
      
      [Jeremy Howard's answer](https://forums.fast.ai/t/nlp-challenge-project/44153)
      
      2019-06-06 About
      Introducing FastBert — A simple Deep Learning library for BERT Models
      
      Tags:
      
      BERT
      fast.ai
      huggingface / transformers
      
      2019-05-23 About
      Robust Language Representation Learning via Multi-task Knowledge Distillation - Microsoft Research
      
      Tags:
      
      BERT
      Ensemble learning
      NLP@Microsoft
      
      Related to [this](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1901.11504).
      
      2019-05-19 About
      [1904.08398] DocBERT: BERT for Document Classification
      
      Tags:
      
      Arxiv Doc
      BERT
      Text Classification
      
      2019-04-18 About
      huggingface/pytorch-pretrained-BERT: The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL.
      
      Tags:
      
      BERT
      GitHub project
      PyTorch
      
      2019-03-15 About
      [1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding
      
      Tags:
      
      Arxiv Doc
      BERT
      KD-MKB biblio
      Multi-task learning
      NLP@Microsoft
      NLU
      Pre-Trained Language Models
      
      outperforms BERT in nine of eleven benchmark NLP tasks
      
      2019-02-17 About
      Using BERT for state-of-the-art pre-training for natural language processing
      
      Tags:
      
      BERT
      Howto
      
      2019-02-14 About
      Jacob Devlin talks about BERT at the Stanford NLP seminar
      
      Tags:
      
      BERT
      Slides
      
      Includes new results such as the effect of the masking strategy, using synthetic training data,...
      
      2019-02-11 About
      Romain Vial (Hyperlex) at Paris NLP meetup, slides
      
      Tags:
      
      BERT
      Good
      Information extraction
      Knowledge Graph Completion
      Knowledge Graph Construction
      Long documents
      NLP + juridique
      Paris NLP meetup
      Slides
      
      > Hyperlex is a contract analytics and management solution powered by artificial intelligence. Hyperlex helps companies manage and make the most of their contract portfolio by identifying relevant information and data to manage key contractual commitments. > Take-home message: > > - Sentence representation starts to be well understood empirically > - Large document representation is still an open (and interesting) problem!
      
      2019-01-24 About
      The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar
      
      Tags:
      
      BERT
      Contextualized word representations
      ELMo
      Transfer learning in NLP
      ULMFiT
      
      2018-12-03 About
      Google AI Blog: Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing
      
      Tags:
      
      BERT
      
      2018-11-05 About
      GitHub - google-research/bert: TensorFlow code and pre-trained models for BERT
      
      Tags:
      
      BERT
      GitHub
      
      Code and pretrained weights for BERT. Includes scripts to reproduce results. BERT-Base can be fine-tuned on a standard GPU; for BERT-Large, a Cloud TPU is required
      
      2018-11-05 About
      [1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
      
      Tags:
      
      Arxiv Doc
      BERT
      Language Model
      Neural machine translation
      Sequence-to-sequence learning
      Transformers
      
      **The "Devlin et al 2019" paper** [Paper Dissected](https://datasciencetoday.net/index.php/en-us/nlp/211-paper-dissected-bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding-explained)
      
      2018-10-12 About
  - Dialogflow
  - Gemini
  - Google LaMDA
  - PaLM
    - PaLM 2
  - SGNN
  - T5 (Text-To-Text Transfer Transformer)
    - Flan-T5
  - Universal Sentence Encoder
  - Word2vec
    - Doc2Vec
    - Word2vec: howto
- Pix2Struct
- Reformer
- TensorFlow
  - Tensor2Tensor
  - TensorFlow 2.0
- VertexAI
  - Google Document AI
AI@Meta
- AI@Facebook
- NLP@Meta
  - LLaMA
    - Alpaca
    - LLaMa.cpp
    - Llama 2
    - LLama3
  - NLP@Facebook
    Zeyuan Allen-Zhu sur X : " surprisingly, when pre-training good data (e.g., Wiki) together with "junks" (e.g., Common Crawl), LLM's capacity on good data may decrease by 20x times!"
    Tags:
    
    LLM pre-training
    LLM Tips
    NLP@Meta
    Simple idea
    Twitter thread
    > A simple fix: add domain tokens to your data; LLMs can auto-detect domains rich in knowledge and prioritize."
    
    2024-04-10 About
    Philipp Schmid sur X : "Can we combine multiple fine-tuned LLMs into a single MoE?..."
    Tags:
    
    LLM Composition
    Mixture of Experts
    NLP@Meta
    > 1. Select pre-trained LLM as the seed model > 2. Fine-tune individual LLMs on dedicated task, domain or language > 3. Combine feedforward parameters of LLMs (2) in MoE layers and average the remaining parameters > 4. Fine-Tune combined MoE to learn token-level routing (assuming experts are frozen), allowing the model to activate the appropriate experts for different inputs selectively.
    
    2024-03-30 About
    Getting started with Llama 2 - AI at Meta
    Tags:
    
    Llama 2
    NLP@Meta
    Tutorial
    2023-10-28 About
    [2310.01352] RA-DIT: Retrieval-Augmented Dual Instruction Tuning
    Tags:
    
    Arxiv Doc
    Instruction tuning
    NLP@Meta
    Retrieval augmented LM
    > Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by **retrofitting any LLM with retrieval capabilities**.
    
    2023-10-07 About
    [2308.13418] Nougat: Neural Optical Understanding for Academic Documents
    Tags:
    
    Arxiv Doc
    Document AI
    NLP@Meta
    OCR
    new generative model from @MetaAI trained to extract text from academic PDFs without needing traditional OCR engines. [Tweet](https://twitter.com/_philschmid/status/1703321340504166494)
    
    2023-09-17 About
    [2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
    Tags:
    
    Arxiv Doc
    LM: tokenization
    LMs: context length
    Long documents
    NLP@Meta
    Transformers
    > these results establish the viability of tokenization-free autoregressive sequence modeling at scale
    
    2023-07-01 About
    Google "We Have No Moat, And Neither Does OpenAI"
    Tags:
    
    Cannibalizing LLMs
    Good
    Google
    LLM
    LoRA
    NLP@Google
    NLP@Meta
    OpenAI
    Open Source
    Open-source LLM
    > low-cost public involvement was enabled by a vastly cheaper mechanism for fine tuning called low rank adaptation ()[LoRA](tag:lora) > **Part of what makes LoRA so effective is that ... it’s stackable.** > > By contrast, training giant models from scratch not only throws away the pretraining, but also any iterative improvements that have been made on top. > LoRA updates are very cheap to produce (~$100) for the most popular model sizes. > Many of these projects are saving time by training on small, highly curated datasets... > These datasets are built using synthetic methods (e.g. filtering the best responses from an existing model) and scavenging from other projects > Directly Competing With Open Source Is a Losing Proposition > Paradoxically, the one clear winner in all of this is Meta. Because the leaked model was theirs ([LLaMA](tag:llama)), they have effectively garnered an entire planet's worth of free labor. Since most open source innovation is happening on top of their architecture, there is nothing stopping them from directly incorporating it into their products.
    
    2023-05-04 About
    [2303.14177] Scaling Expert Language Models with Unsupervised Domain Discovery
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Highlight
    Language Model
    NLP@Meta
    > a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora. Our method > > - clusters a corpus into sets of related documents, > - trains a separate expert language model on each cluster, > - and combines them in a sparse ensemble for inference. > > Our technique outperforms dense baselines on multiple corpora and few-shot tasks, and our analysis shows that specializing experts to meaningful clusters is key to these gains.
    
    2023-03-27 About
    Guillaume Lample sur Twitter : "Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters..."
    Tags:
    
    Foundation Models
    Guillaume Lample
    Language Models: size
    LLaMA
    LLM
    NLP@Meta
    Publicly available datasets
    Tweet
    > LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B. > > The weights for all models are open and available > > trained on at least 1T tokens, > > Unlike Chinchilla, PaLM, or GPT-3, we only use datasets publicly available, > > We also briefly tried instruction finetuning LLaMA-13B is competitive with GPT-3, despite being 10x smaller. But that's not really open-source [github](https://github.com/facebookresearch/llama) "The license prohibits using the models or any data produced by the models for any type of commercial or production purpose."
    
    2023-02-25 About
    [2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Instruction tuning
    NLP@Meta
    Text Embeddings
    > INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. > every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). [Documentation](https://instructor-embedding.github.io) ; [At Hugging Face](doc:2023/02/hkunlp_instructor_xl_·_hugging_) ex of use [here](https://postgresml.org/blog/generating-llm-embeddings-with-open-source-models-in-postgresml)
    
    2023-02-17 About
    [2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools
    Tags:
    
    Arxiv Doc
    NLP@Meta
    Self-Supervised Learning (NLP)
    Tool-Augmented LLMs
    Toolformer
    > Toolformer, **a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction**. > fulfills the following desiderata: > - The use of tools should be learned in a self-supervised way without requiring large amounts of human annotations >- The LM should be able to decide for itself when and how to use which tool. > Approach based on the recent idea of using large LMs with incontext learning (Brown et al., 2020) to generate entire datasets from scratch. > > Given just a handful of human-written examples of how an API can be used, > - we let a LM annotate a huge language modeling dataset with potential API calls. > - We then use a self-supervised loss to determine which of these API calls actually help the model in predicting future tokens. >- Finally, we finetune the LM itself on the API calls that it considers useful. [Jay Hack @mathemagic1an sur twitter](https://twitter.com/mathemagic1an/status/1624870248221663232): > from a small seed set of human inputs (essentially demonstrating usage of APIs), the training set for this behavior is generated by the LLM itself. > > So what does this mean? We've found a promising way to tightly integrate arbitrary APIs with our best-performing models.
    
    2023-02-13 About
    Akari Asai sur Twitter : "Can we solely rely on LLMs’ memories (eg replace search w ChatGPT)? Probably not... Our analysis shows how retrieval is complementary to LLMs’ parametric knowledge..."
    Tags:
    
    ChatGPT
    ChatGPT+Search
    Information retrieval
    Language Models + Knowledge
    NLP@Meta
    Not Encoding Factual Knowledge in Language Model
    Tweet
    2022-12-21 About
AI@Microsoft
- NLP@Microsoft
  - Bhaskar Mitra
  - E5
  - LoRA
    - DoRA
    - QLoRA
  - Microsoft Concept Graph
  - Semantic Kernel
- Sebastien Bubeck
AI@Stanford
- NLP@Stanford
  - Alpaca
  - ColBERT
  - Coursera: NLP class
  - cs224n
  - GloVe
  - Stanford classifier
  - Stanford NER
  - Stanford POS Tagger
  - Stanza
NLP Teams
- Allen Institute for AI (AI2)
  - AllenNLP
  - ELMo
  - KnowBert
    Self-RAG: Learning to Retrieve, Generate and Critique through Self-Reflection
    Tags:
    
    Allen Institute for AI (AI2)
    Discuté avec Raphaël
    RAG (Retrieval-Augmented Generation)
    Tool-Augmented LLMs
    2023-10-20 About
    [2307.13269] LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Cross-Task Generalization
    LoRA
    > a framework devised for the purposive assembly of LoRA modules trained on diverse given tasks > With just a few examples from a novel task, LoraHub enables the fluid combination of multiple LoRA modules, eradicating the need for human expertise. Notably, the composition requires neither additional model parameters nor gradients. > > Our empirical results... suggest that LoraHub can effectively mimic the performance of in-context learning in few-shot scenarios, excluding the necessity of in-context examples alongside each inference input. > The model architecture can be either encoder-decoder or decoder-only not encoder only!
    
    2023-08-08 About
    [2303.17651] Self-Refine: Iterative Refinement with Self-Feedback
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    LLM
    OpenAI GPT
    approach that allows LLMs to iteratively refine outputs and incorporate feedback along multiple dimensions to improve performance on diverse tasks.
    
    2023-04-03 About
    [2303.14177] Scaling Expert Language Models with Unsupervised Domain Discovery
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Highlight
    Language Model
    NLP@Meta
    > a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora. Our method > > - clusters a corpus into sets of related documents, > - trains a separate expert language model on each cluster, > - and combines them in a sparse ensemble for inference. > > Our technique outperforms dense baselines on multiple corpora and few-shot tasks, and our analysis shows that specializing experts to meaningful clusters is key to these gains.
    
    2023-03-27 About
    [2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Instruction tuning
    NLP@Meta
    Text Embeddings
    > INSTRUCTOR is a single embedder that can generate text embeddings tailored to different downstream tasks and domains, without any further training. > every text input is embedded together with instructions explaining the use case (e.g., task and domain descriptions). [Documentation](https://instructor-embedding.github.io) ; [At Hugging Face](doc:2023/02/hkunlp_instructor_xl_·_hugging_) ex of use [here](https://postgresml.org/blog/generating-llm-embeddings-with-open-source-models-in-postgresml)
    
    2023-02-17 About
    Embedding Recycling: Making Language Model Development More Sustainable | AI2 Blog
    Tags:
    
    Allen Institute for AI (AI2)
    Embeddings
    Language Model
    2023-02-17 About
    [2209.01975] Selective Annotation Makes Language Models Better Few-Shot Learners
    Tags:
    
    Allen Institute for AI (AI2)
    Annotations (ML)
    Arxiv Doc
    Deep Active Learning
    Few-shot learning
    Prompting/In-context learning
    > This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. > > Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. an unsupervised, graph-based selective annotation method, vote-k, to select diverse, representative examples to annotate
    
    2022-09-07 About
    [2106.10199] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    BERT fine-tuning
    Parameter-efficient fine-tuning (PEFT)
    Yoav Goldberg
    > BitFit, a sparse-finetuning method where only the bias-terms of the model (or a subset of them) are being modified. We show that **with small-to-medium training data, applying BitFit on pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model.** > **these findings support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge** -- ???!!! > The focus on modifying a small group of parameters eases deployment, as the vast majority of the parameters of the model are shared between various NLP tasks [GitHub](https://github.com/benzakenelad/BitFit)
    
    2022-09-01 About
    [1902.06006] Contextual Word Representations: A Contextual Introduction
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Good
    Word embeddings: introduction
    > the story of how we put words into computers
    
    2022-07-08 About
    [2205.08012] CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Knowledge Graphs
    Link Prediction
    [Tara Safavi sur Twitter : "CascadER, a new knowledge graph (KG) link prediction method leveraging structured relations + unstructured text..."](doc:2022/07/tara_safavi_sur_twitter_casc)
    
    2022-07-07 About
    (((ل()(ل() 'yoav))))👾 sur Twitter : "... another step in understanding how transformer-based LMs work..."
    Tags:
    
    Allen Institute for AI (AI2)
    Language Model
    Transformers
    Tweet
    Yoav Goldberg
    > Very succinctly, we show that individual columns in the feedforward matrices at different layers contribute to shifting the prediction towards specific concepts, *which we can interpret*. > We show that a token representation can be viewed as a changing distribution over the output vocabulary
    
    2022-03-30 About
    SPIKE: Extractive Search from Allen Institute for AI
    Tags:
    
    Allen Institute for AI (AI2)
    Extractive Search
    > sentence-level, context-aware, and linguistically informed extractive search system.
    
    2022-02-22 About
    [2107.12708] QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    NLP datasets
    Question Answering
    Reading Comprehension
    Sebastian Ruder
    Survey / Review
    recommandé par [Sebastian Ruder](tag:sebastian_ruder)
    
    2021-08-06 About
    [2106.04612] Neural Extractive Search
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Cognitive Search
    Neural Search
    Search
    Yoav Goldberg
    how to extend a search paradigm we call “**extractive search**” with neural similarity techniques. > some information needs require extracting and aggregating sub-sentence information (words, phrases, or entities) from multiple documents (e.g. a list of all the risk factors for a specific disease and their number of mentions, or a comprehensive table of startups and CEOs). > extractive search combines document selection with information extraction. **The query is extended with capture slots**: these are **search terms that act as variables, whose values should be extracted**. > The user is then presented with the matched documents, each annotated with the corresponding captured spans, as well as aggregate information over the captured spans Conclusion : > We presented a system for neural extractive search. While we found our system to be useful for scientific search, it also has clear limitations and areas for improvement, both in terms of accuracy (only 72.2% of the returned results are relevant, both the alignment and similarity models generalize well to some relations but not to others), and in terms of scale [Video of demo](https://www.youtube.com/watch?v=TtqWi2GgB5A&t=1832s)
    
    2021-06-23 About
    [1807.04905] Ultra-Fine Entity Typing
    Tags:
    
    ACL 2018
    Allen Institute for AI (AI2)
    Arxiv Doc
    Entity type prediction
    > a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e.g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity
    
    2021-06-22 About
    Semantic Scholar | TLDR Feature
    Tags:
    
    Allen Institute for AI (AI2)
    Research papers
    Scientific information extraction
    Text Summarization
    [Semantic Scholar](doc:2020/11/semantic_scholar_%7C_ai_powered_r) | AI-Powered Research Tool
    
    2020-11-24 About
    TL;DR: This AI summarizes research papers so you don’t have to
    Tags:
    
    Allen Institute for AI (AI2)
    NLP for scientific documents
    Research papers
    Text Summarization
    Blog post about [Semantic Scholar | TLDR](doc:2020/11/semantic_scholar_%7C_tldr_feature)
    
    2020-11-24 About
    [1909.04164] Knowledge Enhanced Contextual Word Representations
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Contextualized word representations
    EMNLP 2019
    Entities and LM
    Good
    Grounded Language Learning
    KD-MKB biblio
    KG-augmented Language Models
    KnowBert
    Knowledge-augmented Language Models
    Knowledge-driven embeddings
    Knowledge Graphs in NLP
    Multiple Knowledge Bases
    General method to **embed multiple knowledge bases into pre-trained language models** (KB in the sense as fixed collection of entity nodes) > The key idea is to explicitly model entity spans in the input text and use an **entity linker** to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. > Then, update contextual word representations via a form of **word-to-entity attention**. > In contrast to previous approaches, the entity linkers and self-supervised language modeling objective are jointly trained end-to-end in a multitask setting that **combines a small amount of entity linking supervision with a large amount of raw text**.
    
    2020-05-13 About
    [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    KD-MKB biblio
    KG-augmented Language Models
    Knowledge Graphs and NLP
    > a **neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context**. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. > The KGLM maintains a dynamically growing local knowledge graph, a subset of the knowledge graph that contains entities that have already been mentioned in the text, and their related entities.
    
    2020-05-11 About
    Isabel Cachola sur Twitter : "TLDR: Extreme Summarization of Scientific Documents"
    Tags:
    
    Allen Institute for AI (AI2)
    Text Summarization
    Tweet
    > a new automatic summarization task with high source compression requiring expert background knowledge and complex language understanding
    
    2020-05-03 About
    [2004.05150] Longformer: The Long-Document Transformer
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Long documents
    > **Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length**. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length. Longformer's attention mechanism is a **drop-in replacement** for the standard self-attention and **combines a local windowed attention with a task motivated global attention**.
    
    2020-04-13 About
    [2002.05867] Transformers as Soft Reasoners over Language
    Tags:
    
    Allen Institute for AI (AI2)
    Arxiv Doc
    Knowledge Representation
    Reasoning
    Rules
    Transformers
    > AI has long pursued the goal of having systems reason over *explicitly provided* knowledge, but building suitable representations has proved challenging. Here we explore whether transformers can similarly learn to reason (or emulate reasoning), but **using rules expressed in language, thus bypassing a formal representation**.
    
    2020-02-17 About
- Hugging Face
  - AutoNLP
  - Hugging Face: dev
  - huggingface / transformers
  - HuggingFace BigScience
- NLP@Alibaba
- NLP@Baidu
- NLP@ENS
- NLP@Facebook
- NLP@Google
- NLP@Harvard
- NLP@IBM
- NLP@Microsoft
- NLP@MIT
- NLP @ OpenAI
- NLP@Stanford
  The Berkeley NLP Group
  Tags:
  - Berkeley
  - NLP Teams
  2020-01-12 About

1 Document (Long List)

SCAI - Sorbonne Center for Artificial Intelligence

Tags:

2022-09-15 About

Properties

sl:creationDate : 2020-03-19
sl:creationTime : 2020-03-19T21:38:01Z
rdf:type : sl:Tag
skos:prefLabel : AI teams