Semanlink - NLP Teams

Parents:

NLP Teams

Related Tags:

Descendants

Allen Institute for AI (AI2)
- AllenNLP
- ELMo
- KnowBert
Hugging Face
- AutoNLP
- Hugging Face: dev
- huggingface / transformers
- HuggingFace BigScience
NLP@Alibaba
NLP@Baidu
NLP@ENS
NLP@Facebook
- BlenderBot
- BLINK
- CamemBERT
- FastText
- ParlAI
- Poincaré Embeddings
- StarSpace
NLP@Google
- ALBERT
- Bard
- BERT
  - ALBERT
  - BERT + KB
    - BLP
      - CKB
        
        raphaelsty/ckb: Contextual knowledge bases
        
        Tags:
        
        CKB
        GitHub project
        Raphaël Sourty
        Semanlink Tag Finder
        
        Une implémentation de [BLP](tag:blp) [[2010.03496] Inductive Entity Representations from Text via Link Prediction](doc:2020/11/2010_03496_inductive_entity_r)
        
        2020-11-09 About
    - KnowBert
    - LUKE
  - BERT + Sentence Embeddings
    - Sentence-BERT
      - sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."
        
        Tags:
        
        Adapter modules (LM finetuning)
        EMNLP 2023
        Few-shot text classification
        Sentence-BERT: Domain adaptation
        Sentence embeddings: Domain adaptation
        SetFit (SBERT fine-tuning)
        Tweet
        
        AdaSent is an approach to creating domain-specialized sentence encoders for few-shot sentence classification > Reusable general sentence adapter across domains > AdaSent decouples DAPT (Domain Adaptative Pre-Training) & SEPT (Sentence Embedding Pre-Training) **by storing the sentence encoding abilities into an adapter**, which is trained only once in the general domain and plugged into various DAPT-ed PLMs [Github](https://github.com/UKPLab/AdaSent)
        
        2023-12-09 About
        
        Daniel Vila Suero sur Twitter : "Data quality is key for LLMs, but we're building Open Source LLMs with data of "unknown" quality... Introducing Alpaca GarbageCollector..."
        
        Tags:
        
        Alpaca
        Dataset quality
        SetFit (SBERT fine-tuning)
        Tweet
        
        > a cross-lingual SetFit model to identify potential bad instructions in Alpaca-like datasets
        
        2023-04-05 About
        
        ValueError "invalid literal for int() with base 10" in trainer.evaluate (dataset created from pandas) · Issue #228 · huggingface/setfit
        
        Tags:
        
        fps' post
        SetFit (SBERT fine-tuning)
        
        see <https://github.com/huggingface/setfit/blob/main/notebooks/zero-shot-classification.ipynb> > Note: some datasets on the Hugging Face Hub don't have a ClassLabel feature for the label column. In these cases, you should compute the candidate labels manually by first computing the id2label mapping as follows:
        
        2022-12-13 About
        
        Lewis Tunstall sur Twitter : "The SetFit library for few-shot learning with Sentence Transformers now supports *multi-label text classification*..."
        
        Tags:
        
        Multi-label Text classification
        SetFit (SBERT fine-tuning)
        Tweet
        
        Multilabel support [github issue](https://github.com/huggingface/setfit/issues/65)
        
        2022-10-14 About
        
        huggingface/setfit: Efficient few-shot learning with Sentence Transformers
        
        Tags:
        
        GitHub project
        SetFit (SBERT fine-tuning)
        
        2022-10-12 About
        
        [2209.11055] Efficient Few-Shot Learning Without Prompts
        
        Tags:
        
        Arxiv Doc
        Few-shot learning
        Moshe Wasserblat
        Nils Reimers
        SetFit (SBERT fine-tuning)
        
        [tweet](https://twitter.com/_akhaliq/status/1573109469646561280?s=20&t=RTpK9dh90az0zT1Xg2ohpQ): > So if I have 4 classes and say 2 labels per class, I would first fine tune an ST on these 4 pairs and then vectorize the 8 total examples for fine-tuning the classifier
        
        2022-09-23 About
        
        Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)
        
        Tags:
        
        Few-shot text classification
        Moshe Wasserblat
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        Finetuning d'un SBERT sur une tâche de classification (in fine, produit un SBERT) > **Few-shot text classification based on fine-tuning a Sentence Transformer with task-specific data** that can easily be implemented with the sentence-transformers library > Surprisingly, we did not find any work that performed an end-to-end ST fine-tuning for text classification in a Siamese manner. [COLAB](https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb) [Nils Reimers sur Twitter](doc:2022/03/nils_reimers_sur_twitter_gre)
        
        2022-03-31 About
        
        Nils Reimers sur Twitter : "Great post on SetFit"
        
        Tags:
        
        Nils Reimers
        SetFit (SBERT fine-tuning)
        Tweet
        
        About [Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat](doc:2022/03/sentence_transformer_fine_tunin) > - Outperforms GPT-3 in few-shot text-classification (50 labeled examples, secret test set) > - 1600 times smaller > - Can be run on your CPU > - No limitation on the number of training examples > - Just few lines of code needed
        
        2022-03-31 About
        
        LlamaIndex 🦙 sur X : "Fine-tuning Embedding Models for RAG with LoRA'
        
        Tags:
        
        LlamaIndex
        LoRA
        sBERT fine-tuning
        
        2024-04-23 About
        
        [2008.11228] A simple method for domain adaptation of sentence embeddings
        
        Tags:
        
        Arxiv Doc
        Domain adaptation (NLP)
        sBERT fine-tuning
        Sentence-BERT: Domain adaptation
        Universal Sentence Encoder
        
        2022-04-01 About
        
        [2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer
        
        Tags:
        
        ACL 2020
        Arxiv Doc
        Low-Resource NLP
        NLP@Amazon
        NLP@Microsoft
        sBERT fine-tuning
        
        > Fine-tuning (FT) pre-trained sentence embedding models on small datasets has been shown to have limitations. In this paper we show that concatenating the embeddings from the pre-trained model with those from a simple sentence embedding model trained only on the target data, can improve over the performance of FT for few-sample tasks
        
        2022-03-31 About
        
        Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)
        
        Tags:
        
        Few-shot text classification
        Moshe Wasserblat
        sBERT fine-tuning
        SetFit (SBERT fine-tuning)
        
        Finetuning d'un SBERT sur une tâche de classification (in fine, produit un SBERT) > **Few-shot text classification based on fine-tuning a Sentence Transformer with task-specific data** that can easily be implemented with the sentence-transformers library > Surprisingly, we did not find any work that performed an end-to-end ST fine-tuning for text classification in a Siamese manner. [COLAB](https://colab.research.google.com/github/MosheWasserb/SetFit/blob/main/SetFit_SST_2.ipynb) [Nils Reimers sur Twitter](doc:2022/03/nils_reimers_sur_twitter_gre)
        
        2022-03-31 About
        
        Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium
        
        Tags:
        
        CamemBERT
        NLP: French
        NLP sample code
        sBERT fine-tuning
        
        2022-03-31 About
        
        [2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning
        
        Tags:
        
        Arxiv Doc
        Few-shot text classification
        sBERT fine-tuning
        Sentence-BERT
        Siamese networks
        
        > the problem of building text classifiers with little or no training data. > > In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks. (cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis) > In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative. > > We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings) > The drop in quality can be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)** [Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)
        
        2022-03-30 About
      - Sentence-BERT: Domain adaptation
        GPL (Generative Pseudo Labeling)
        
        Domain Adaptation with Generative Pseudo-Labeling (GPL) | Pinecone
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Pinecone
        
        2023-04-09 About
        
        [2205.11498] Domain Adaptation for Memory-Efficient Dense Retrieval
        
        Tags:
        
        Arxiv Doc
        Dense retriever
        Domain adaptation
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        
        Refers to [Binary Passage Retriever (BPR)](doc:2021/06/2106_00882_efficient_passage_)
        
        2022-09-26 About
        
        Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Multilingual NLP
        
        2022-06-02 About
        
        Nils Reimers sur Twitter : "GPL goes multi-lingual..."
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Tweet
        
        [Domain transfer with GGPL: German Generative Pseudo Labeling](doc:2022/06/domain_transfer_with_ggpl_germ)
        
        2022-06-01 About
        
        Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Tweet
        
        2022-04-27 About
        
        Domain Adaptation — Sentence-Transformers documentation
        
        Tags:
        
        Documentation
        Domain adaptation (NLP)
        GPL (Generative Pseudo Labeling)
        Sentence-BERT: Domain adaptation
        TSDAE
        
        2022-03-31 About
        
        NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"
        
        Tags:
        
        GPL (Generative Pseudo Labeling)
        J'y étais
        Neural Search
        Nils Reimers
        Unsupervised Domain Adaptation (NLP)
        
        2022-03-09 About
        
        [2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval
        
        Tags:
        
        Arxiv Doc
        Dense Passage Retrieval
        GPL (Generative Pseudo Labeling)
        Nils Reimers
        Synthetic QA data
        Unsupervised Domain Adaptation (NLP)
        
        An unsupervised domain adaptation technique for dense retrieval models 1. synthetic queries are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer) encoder-decoder) 2. the generated queries are used for mining negative passages (retrieving the most similar paragraphs using an existing dense retrieval model == hard negatives!) 3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted dense retriever (using method described in [Hofstätter et al., 2020](doc:2021/12/2010_02666_improving_efficien)) [Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl), by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans) Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598)) > - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data. > - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.
        
        2021-12-15 About
        
        sBERT fine-tuning
        
        UKP Lab sur X : "a lightweight solution for few-shot domain-specific sentence classification: AdaSent!..."
        
        Tags:
        
        Adapter modules (LM finetuning)
        EMNLP 2023
        Few-shot text classification
        Sentence-BERT: Domain adaptation
        Sentence embeddings: Domain adaptation
        SetFit (SBERT fine-tuning)
        Tweet
        
        AdaSent is an approach to creating domain-specialized sentence encoders for few-shot sentence classification > Reusable general sentence adapter across domains > AdaSent decouples DAPT (Domain Adaptative Pre-Training) & SEPT (Sentence Embedding Pre-Training) **by storing the sentence encoding abilities into an adapter**, which is trained only once in the general domain and plugged into various DAPT-ed PLMs [Github](https://github.com/UKPLab/AdaSent)
        
        2023-12-09 About
        
        [2008.11228] A simple method for domain adaptation of sentence embeddings
        
        Tags:
        
        Arxiv Doc
        Domain adaptation (NLP)
        sBERT fine-tuning
        Sentence-BERT: Domain adaptation
        Universal Sentence Encoder
        
        2022-04-01 About
        
        Domain Adaptation — Sentence-Transformers documentation
        
        Tags:
        
        Documentation
        Domain adaptation (NLP)
        GPL (Generative Pseudo Labeling)
        Sentence-BERT: Domain adaptation
        TSDAE
        
        2022-03-31 About
        
        Unsupervised Training for Sentence Transformers | Pinecone
        
        Tags:
        
        Pinecone
        Sentence-BERT: Domain adaptation
        TSDAE
        Unsupervised Sentence Embedding Learning
        
        Blog post about [[2104.06979] TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning](doc:2021/09/2104_06979_tsdae_using_trans) > Fine-tuning with TSDAE simply cannot compete in terms of performance against supervised methods. However, **the point and value of TSDAE is that it allows us to fine-tune models for use-cases where we have no data**. Specific domains with unique terminology or low resource languages.
        
        2021-11-24 About
      - SentenceBERT and Words
        
        Few-Shot Text Classification (Cloudera 2020)
        
        Tags:
        
        Few-shot text classification
        SentenceBERT and Words
        
        > Sentence-BERT has been optimized… well, for sentences! It’s reasonable to suspect that SBERT’s representations of single words or short phrases like “Business” or “Science & Technology” won’t be as semantically relevant as representations derived from a word-level method, like word2vec or GloVe
        
        2022-11-24 About
  - BERT-based entity linking
  - BERT fine-tuning
    - BERT fine-tuning: instabilities
  - Bertology
  - BERTopic
  - CamemBERT
  - DistilBERT
  - Domain-Specific BERT
  - KeyBERT
  - KnowBert
    An efficient long-text semantic retrieval approach via utilizing presentation learning on short-text | Complex & Intelligent Systems (2023)
    Tags:
    
    BERT
    Information retrieval
    Long documents
    long-text retrieval model based on BERT (called LTR-BERT)
    
    2024-01-31 About
    Rethinking Query Expansion for BERT Reranking | Advances in Information Retrieval (2020)
    Tags:
    
    BERT
    Information retrieval
    Query Expansion
    using BERT for Information Retrieval: > We find that traditional word-based query expansion is not entirely applicable
    
    2023-10-29 About
    [2002.06275] TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
    Tags:
    
    Arxiv Doc
    BERT
    Knowledge distillation
    Neural Search
    NLP@Microsoft
    2023-08-27 About
    skeskinen/bert.cpp: ggml implementation of BERT
    Tags:
    
    .cpp
    BERT
    Frugal LLMs
    Quantization
    Sentence-BERT
    Sentence Embeddings
    > ggml inference of BERT neural net architecture with pooling and normalization from SentenceTransformers (sbert.net). High quality sentence embeddings in pure C++ (with C API). > > The main goal of bert.cpp is to run the BERT model using **4-bit integer quantization on CPU**
    
    2023-05-09 About
    Classifying long textual documents (up to 25 000 tokens) using BERT | by Sinequa | (2020)
    Tags:
    
    BERT
    Long documents
    Representation of text + metadata
    Sinequa
    Text Classification
    > long text + additional textual metadata (such as title, abstract …) and categories (location, authors …).
    
    2023-04-07 About
    Prompt Tuning BERT🎯:CommonLit Readability | Kaggle
    Tags:
    
    BERT
    NLP sample code
    Prompting/In-context learning
    > Prompt-tuning is a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks.Soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Finally, we show that conditioning,a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.
    
    2022-09-16 About
    Active Learning for BERT: An Empirical Study - ACL Anthology
    Tags:
    
    Active learning
    BERT
    EMNLP 2020
    Imbalanced Data
    NLP@IBM
    Text Classification
    > The use of Actice Learning (AL) with deep pre-trained models has so far received little consideration. > > We study the potential of (i) various AL strategies; (ii) in conjunction with BERT, (iii) within a highly challenging – yet common – real-world scenario of class imbalance and scarce labeled data. focused on binary classification > AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. [Github](https://github.com/IBM/low-resource-text-classification-framework)
    
    2022-09-02 About
    Using BERT For Classifying Documents with Long Texts | by Armand Olivares | Medium
    Tags:
    
    BERT
    Long documents
    2022-06-29 About
    [1909.00426] Global Entity Disambiguation with BERT
    Tags:
    
    Arxiv Doc
    BERT
    Entity Disambiguation
    Ikuya Yamada
    2022-04-18 About
    MaartenGr/BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.
    Tags:
    
    BERT
    BERTopic
    GitHub project
    Topic Modeling
    > topic modeling technique that leverages 🤗 transformers and [c-TF-IDF](https://github.com/MaartenGr/cTFIDF) to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions. refers to [Top2Vec](doc:2022/03/ddangelov_top2vec_top2vec_lear) [youtube](https://www.youtube.com/watch?v=Qub3PrFvauI) [tweet](https://twitter.com/JayAlammar/status/1594681648121102336?s=20&t=R0G_LrajK9WBtzypwXtD7Q)
    
    2022-03-10 About
    [2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
    Tags:
    
    Arxiv Doc
    BERT
    Phrase embeddings
    2022-02-25 About
    Domain Transfer with BERT | Pinecone
    Tags:
    
    BERT
    Domain adaptation (NLP)
    Pinecone
    2022-01-04 About
    Making the Most of Data: Augmentation with BERT | Pinecone
    Tags:
    
    BERT
    Cross-Encoders
    Pinecone
    Synthetic data (NLP)
    2021-12-18 About
    Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science
    Tags:
    
    BERT
    Cross-Encoders
    Data Augmentation
    Sentence-BERT
    Sentence Similarity
    Data Augmentation Method to improve SBERT Bi-Encoders for Pairwise Sentence Scoring Tasks (Semantic sentence tasks)
    
    2021-12-17 About
    [2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles
    Tags:
    
    Arxiv Doc
    BERT
    BM25
    Embeddings in Information Retrieval
    Semantic Search
    2021-04-12 About
    [1902.00751] Parameter-Efficient Transfer Learning for NLP
    Tags:
    
    Adapter modules (LM finetuning)
    Arxiv Doc
    BERT
    Continual Learning
    NLP@Google
    Transfer learning in NLP
    **Adapter tuning for NLP**. A strategy for tuning a large text model on several downstream tasks, that permits training on tasks sequentially, and that adds only a small number of additional parameters per task. New modules added between layers of a pre-trained network. Parameters of the original network are frozen and therefore may be shared by many tasks. [GitHub google-research/adapter-bert](https://github.com/google-research/adapter-bert)
    
    2021-04-11 About
    exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology
    Tags:
    
    BERT
    Biomedical NLP
    Domain adaptation (NLP)
    Domain-Specific BERT
    EMNLP 2020
    NLP@Harvard
    **Focus on the Embedding of Domain-specific Vocabulary.** > exBERT adds a new domain-specific vocabulary and the corresponding embedding layer, as well as a small extension module to the original unmodified model > a pretraining method allowing **low-cost embedding of domain-specific vocabulary in the context of an existing large pre-trained model such as BERT** > exBERT... explicitly incorporates the new domain’s vocabulary, while being able to **reuse the original pre-trained model’s weights as is** to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting its embeddings for the original vocabulary with new embeddings for the domain-specific vocabulary via **a learned small “extension” module**. **The output of the original and extension modules are combined via a trainable weighted sum operation** In a way similar to concept developed in > [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm. [Github](https://github.com/cgmhaicenter/exBERT)
    
    2021-04-11 About
    [1901.04085] Passage Re-ranking with BERT
    Tags:
    
    Arxiv Doc
    BERT
    Learning to rank
    NLP@Facebook
    Ranking (information retrieval)
    a simple re-implementation of BERT for query-based passage re-ranking ["Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"](doc:2021/03/rodrigo_nogueira_sur_twitter_)
    
    2021-03-26 About
    Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"
    Tags:
    
    BERT
    Learning to rank
    Ranking (information retrieval)
    Slides
    Tutorial
    Tweet
    2021-03-09 About
    kamalkraj/BERT-NER: Pytorch-Named-Entity-Recognition-with-BERT
    Tags:
    
    BERT
    GitHub project
    Named Entity Recognition
    Use google BERT to do CoNLL-2003 NER !
    
    2021-02-07 About
    [1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
    Tags:
    
    Arxiv Doc
    BERT
    BERT + KB
    Discuté avec Raphaël
    Entity embeddings
    Text, KG and embeddings
    > way of **injecting factual knowledge about entities into the pretrained BERT model**. (Feeding entity vectors into BERT as if they were wordpiece vectors without additional encoder pretraining) > > **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**. > > Our vector space alignment strategy is inspired by cross-lingual word vector alignment Related work on Entity-enhanced BERT: > ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle that BERT be adapted to entity vectors. They introduce new encoder layers to feed pretrained entity vectors into the Transformer, and they require additional pretraining to integrate the new parameters. In contrast, E-BERT’s design principle is that entity vectors be adapted to BERT. > > Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_) (Wang et al., 2019c) and K-Adapter (Wang et al., 2020)... Their factual knowledge does not stem from entity vectors – instead, they are trained in a multi-task setting on relation classification and knowledge base completion. Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)
    
    2021-01-12 About
    Keyword Extraction with BERT | Towards Data Science
    Tags:
    
    BERT
    Keyword/keyphrase extraction
    A minimal method for extracting keywords and keyphrases. [GitHub](https://github.com/MaartenGr/KeyBERT/) > uses BERT-embeddings and simple cosine similarity to find the sub-phrases in a document that are the most similar to the document itself.
    
    2020-12-06 About
    Salmon Run: Word Sense Disambiguation using BERT as a Language Model
    Tags:
    
    BERT
    huggingface / transformers
    NLP sample code
    Word-sense disambiguation
    2020-12-01 About
    Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science
    Tags:
    
    BERT
    huggingface / transformers
    Question Answering
    Tutorial
    A guide to choosing and benchmarking BERT models for question answering
    
    2020-10-04 About
    BERT Word Embeddings Tutorial · Chris McCormick
    Tags:
    
    BERT
    Tutorial
    Word embeddings
    2020-07-06 About
    [1909.03193] KG-BERT: BERT for Knowledge Graph Completion
    Tags:
    
    Arxiv Doc
    BERT
    Discuté avec Raphaël
    Knowledge Graph Completion
    Pre-Trained Language Models
    Transformers
    Pre-trained language models for knowledge graph completion. **Triples are treated as textual sequences**. (Hum, j'ai déjà vu ça quelque part. Ah, peut-être [RDF2VEC](tag:rdf2vec)? // TODO à voir) Takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model > we first treat entities, relations and triples as textual sequences and turn knowledge graph completion into a sequence classification problem. We then fine-tune BERT model on these sequences for predicting the plausibility of a triple or a relation. The method [GitHub](https://github.com/yao8839836/kg-bert)
    
    2020-03-22 About
    [1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph
    Tags:
    
    Arxiv Doc
    BERT
    Knowledge Graph + Deep Learning
    Knowledge Graphs in NLP
    Language Model
    a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge (Summarized in [Domain adaptation of word embeddings through the exploitation of in-domain corpora and knowledge bases (PhD Thesis 2021)](doc:2022/03/domain_adaptation_of_word_embed), p43)
    
    2020-03-08 About
    Unsupervised NER using BERT - Hands-on NLP model review - Quora
    Tags:
    
    BERT
    Named Entity Recognition
    [GitHub](https://github.com/ajitrajasekharan/unsupervised_NER)
    
    2020-03-06 About
    [2002.12327] A Primer in BERTology: What we know about how BERT works
    Tags:
    
    Arxiv Doc
    BERT
    Bertology
    Survey / Review
    (article praised on [twitter](https://twitter.com/dennybritz/status/1233343170596917248?s=20) by D Britz and Y. Goldberg)
    
    2020-02-28 About
    [2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia
    Tags:
    
    Arxiv Doc
    BERT
    Conditional random fields
    Named Entity Recognition
    Wikipedia
    2020-02-27 About
    Distilling BERT models with spaCy - Towards Data Science (2019)
    Tags:
    
    BERT
    Knowledge distillation
    spaCy
    Transfer learning in NLP
    Yves Peirsman
    2020-02-15 About
    [2002.02925] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
    Tags:
    
    Arxiv Doc
    BERT
    Knowledge distillation
    NLP@Microsoft
    approach to compress BERT by progressive module replacing. > Compared to the previous knowledge distillation approaches for BERT compression, our approach leverages only one loss function and one hyper-parameter [Github](https://github.com/JetRunner/BERT-of-Theseus)
    
    2020-02-10 About
    Canwen Xu sur Twitter : "WTF? We brutally dismember BERT and replace all his organs?"
    Tags:
    
    BERT
    Knowledge distillation
    NLP@Microsoft
    [paper](/doc/2020/02/_2002_02925_bert_of_theseus_c)
    
    2020-02-10 About
    Building a Search Engine with BERT and TensorFlow - Towards Data Science
    Tags:
    
    BERT
    Nearest neighbor search
    NLP and Search
    Search Engines
    TensorFlow
    [somewhat related](/doc/2020/01/elasticsearch_meets_bert_build)
    
    2020-01-12 About
    Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT
    Tags:
    
    BERT
    ElasticSearch
    GitHub project
    NLP and Search
    Text Similarity
    - Links to [this ES blog post](/doc/2020/01/text_similarity_search_in_elast) - [somewhat related](/doc/2020/01/building_a_search_engine_with_b)
    
    2020-01-10 About
    [2003.05473] Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNNL 2019)
    Tags:
    
    Arxiv Doc
    BERT
    Bertology
    Discuté avec Raphaël
    End-To-End Entity Linking
    Good
    Training BERT-base-uncased on English Wikipedia and then fine-tuned and evaluating it on an entity linking (EL) benchmark (EL implemented as a token classification over the entity vocabulary) > BERT+Entity is a straightforward extension on top of BERT, i.e. we initialize BERT with the publicly available weights from the BERT-base-uncased model and add an output classification layer on top of the architecture. Given a contextualized token, the classifier computes the probability of an entity link for each entry in the entity vocabulary. Can BERT’s architecture learn all entity linking steps jointly? To answer: > an extreme simplification of the **entity linking setup that works surprisingly well**: simply cast it as **a per token classification over the entire entity vocabulary** (over 700K classes in our case). > the model is the first that performs entity linking without any pipeline or any heuristics, compared to all prior approaches. We found that with our approach we can learn additional entity knowledge in BERT that helps in entity linking. **However, we also found that almost none of the downstream tasks really required entity knowledge**. ### Related work - > [Durrett and Klein (2014)](/doc/2020/01/a_joint_model_for_entity_analys) were the first to propose jointly modelling Mention detection, Candidate generation and Entity disambiguation in a graphical model and could show that each of those steps are interdependent and benefit from a joint objective This paper uses neural techniques instead of CRF. - > [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) (2016, 2017) was the first to investigate neural text representations and entity linking, but their approach is limited to ED. cf. [#Wikipedia2Vec](tag:wikipedia2vec). Compare with [newer work by Yamada](doc:2020/09/1909_01259_neural_attentive_b)
    
    2020-01-09 About
    Named Entity Recognition with Bert – Depends on the definition
    Tags:
    
    BERT
    Hugging Face
    Named Entity Recognition
    NLP sample code
    > how you can finetune the Bert model to do state-of-the art named entity recognition Same author: [NER with Lime](/doc/2020/01/interpretable_named_entity_reco)
    
    2020-01-09 About
    [1902.10909] BERT for Joint Intent Classification and Slot Filling
    Tags:
    
    Arxiv Doc
    BERT
    Intent classification and slot filling
    NLP@Alibaba
    > Experimental results show that our proposed joint BERT model outperforms BERT models modeling intent classification and slot filling separately, demonstrating the efficacy of exploiting the relationship between the two tasks. Adding a CRF on top of the model doesn't improve the results.
    
    2020-01-09 About
    Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019
    Tags:
    
    BERT
    Contextualized word representations
    ELMo
    NLP@Stanford
    2020-01-05 About
    BERT is now part of Google Search, so let’s understand how it reasons
    Tags:
    
    BERT
    Bertology
    NLP: negation
    2019-10-31 About
    Meet ALBERT: a new ‘Lite BERT’ from Google & Toyota with State of the Art NLP performance and 18x fewer parameters.
    Tags:
    
    ALBERT
    BERT
    NLP@Google
    Toyota
    2019-10-01 About
    Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT
    Tags:
    
    BERT
    Hugging Face
    Knowledge distillation
    2019-08-28 About
    [1905.07129] ERNIE: Enhanced Language Representation with Informative Entities
    Tags:
    
    ACL 2019
    Arxiv Doc
    BERT
    Discuté avec Raphaël
    ERNIE
    Knowledge Graphs in NLP
    Text, KG and embeddings
    Transformers
    > We argue that informative entities in **KGs can enhance language representation with external knowledge**. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. > ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks [GitHub](https://github.com/thunlp/ERNIE) WARNING, there is another ERNIE (by [NLP@Baidu](tag:nlp_baidu)): Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. This doesn't happen when you choose François-Paul as the name for your child.
    
    2019-08-05 About
    What is XLNet and why it outperforms BERT - Towards Data Science
    Tags:
    
    BERT
    XLNet
    2019-08-02 About
    BERT's success in some benchmarks tests may be simply due to the exploitation of spurious statistical cues in the dataset. Without them it is no better then random. : MachineLearning
    Tags:
    
    BERT
    Bertology
    2019-07-24 About
    [1907.07355] Probing Neural Network Comprehension of Natural Language Arguments
    Tags:
    
    Arxiv Doc
    BERT
    Bertology
    what has BERT learned about argument comprehension? [Comments](/doc/2019/07/bert_s_success_in_some_benchmar)
    
    2019-07-24 About
    [1906.04341] What Does BERT Look At? An Analysis of BERT's Attention
    Tags:
    
    Arxiv Doc
    Attention mechanism
    BERT
    Bertology
    Chris Manning
    What's encoded by a NN
    2019-06-21 About
    NLP: Contextualized word embeddings from BERT – Towards Data Science
    Tags:
    
    BERT
    2019-06-12 About
    [1906.02715] Visualizing and Measuring the Geometry of BERT
    Tags:
    
    Arxiv Doc
    BERT
    Bertology
    Geometry of language embeddings
    NLP@Google
    Thèse IRIT-Renault: biblio
    Tree embeddings
    > At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses. We also present empirical descriptions of syntactic representations in both attention matrices and individual word embeddings, as well as a mathematical argument to explain the geometry of these representations
    
    2019-06-07 About
    "I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"
    Tags:
    
    BERT
    Humour
    Naive Bayes classifier
    Rada Mihalcea
    [Jeremy Howard's answer](https://forums.fast.ai/t/nlp-challenge-project/44153)
    
    2019-06-06 About
    Introducing FastBert — A simple Deep Learning library for BERT Models
    Tags:
    
    BERT
    fast.ai
    huggingface / transformers
    2019-05-23 About
    Robust Language Representation Learning via Multi-task Knowledge Distillation - Microsoft Research
    Tags:
    
    BERT
    Ensemble learning
    NLP@Microsoft
    Related to [this](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1901.11504).
    
    2019-05-19 About
    [1904.08398] DocBERT: BERT for Document Classification
    Tags:
    
    Arxiv Doc
    BERT
    Text Classification
    2019-04-18 About
    huggingface/pytorch-pretrained-BERT: The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL.
    Tags:
    
    BERT
    GitHub project
    PyTorch
    2019-03-15 About
    [1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding
    Tags:
    
    Arxiv Doc
    BERT
    KD-MKB biblio
    Multi-task learning
    NLP@Microsoft
    NLU
    Pre-Trained Language Models
    outperforms BERT in nine of eleven benchmark NLP tasks
    
    2019-02-17 About
    Using BERT for state-of-the-art pre-training for natural language processing
    Tags:
    
    BERT
    Howto
    2019-02-14 About
    Jacob Devlin talks about BERT at the Stanford NLP seminar
    Tags:
    
    BERT
    Slides
    Includes new results such as the effect of the masking strategy, using synthetic training data,...
    
    2019-02-11 About
    Romain Vial (Hyperlex) at Paris NLP meetup, slides
    Tags:
    
    BERT
    Good
    Information extraction
    Knowledge Graph Completion
    Knowledge Graph Construction
    Long documents
    NLP + juridique
    Paris NLP meetup
    Slides
    > Hyperlex is a contract analytics and management solution powered by artificial intelligence. Hyperlex helps companies manage and make the most of their contract portfolio by identifying relevant information and data to manage key contractual commitments. > Take-home message: > > - Sentence representation starts to be well understood empirically > - Large document representation is still an open (and interesting) problem!
    
    2019-01-24 About
    The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar
    Tags:
    
    BERT
    Contextualized word representations
    ELMo
    Transfer learning in NLP
    ULMFiT
    2018-12-03 About
    Google AI Blog: Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing
    Tags:
    
    BERT
    2018-11-05 About
    GitHub - google-research/bert: TensorFlow code and pre-trained models for BERT
    Tags:
    
    BERT
    GitHub
    Code and pretrained weights for BERT. Includes scripts to reproduce results. BERT-Base can be fine-tuned on a standard GPU; for BERT-Large, a Cloud TPU is required
    
    2018-11-05 About
    [1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
    Tags:
    
    Arxiv Doc
    BERT
    Language Model
    Neural machine translation
    Sequence-to-sequence learning
    Transformers
    **The "Devlin et al 2019" paper** [Paper Dissected](https://datasciencetoday.net/index.php/en-us/nlp/211-paper-dissected-bert-pre-training-of-deep-bidirectional-transformers-for-language-understanding-explained)
    
    2018-10-12 About
- Dialogflow
- Gemini
- Google LaMDA
- PaLM
  - PaLM 2
- SGNN
- T5 (Text-To-Text Transfer Transformer)
  - Flan-T5
- Universal Sentence Encoder
- Word2vec
  - Doc2Vec
  - Word2vec: howto
NLP@Harvard
NLP@IBM
NLP@Microsoft
- Bhaskar Mitra
- E5
- LoRA
  - DoRA
  - QLoRA
- Microsoft Concept Graph
- Semantic Kernel
NLP@MIT
NLP @ OpenAI
NLP@Stanford
- Alpaca
- ColBERT
- Coursera: NLP class
- cs224n
- GloVe
- Stanford classifier
- Stanford NER
- Stanford POS Tagger
- Stanza

1 Document (Long List)

The Berkeley NLP Group

Tags:

2020-01-12 About

Properties

sl:creationDate : 2020-01-12
sl:creationTime : 2020-01-12T10:52:37Z
rdf:type : sl:Tag
skos:altLabel : NLP Groups
skos:prefLabel : NLP Teams