Semanlink - NLP Teams AND Deep Learning

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 A propos

Aran Komatsuzaki sur Twitter : "JaxPruner: A concise library for sparsity research An open-source JAX-based pruning and sparse training library for machine learning research repo"

Tags:

2023-04-28 A propos

Domain Adaptation with Generative Pseudo-Labeling (GPL) | Pinecone

Tags:

2023-04-09 A propos

Classifying long textual documents (up to 25 000 tokens) using BERT | by Sinequa | (2020)

Tags:

2023-04-07 A propos

Daniel Vila Suero sur Twitter : "Data quality is key for LLMs, but we're building Open Source LLMs with data of "unknown" quality... Introducing Alpaca GarbageCollector..."

Tags:

2023-04-05 A propos

[2303.17651] Self-Refine: Iterative Refinement with Self-Feedback

Tags:

2023-04-03 A propos

Alpaca Finetuning of Llama on a 24G Consumer GPU

Tags:

2023-03-27 A propos

tloen/alpaca-lora: Instruct-tune LLaMA on consumer hardware

Tags:

2023-03-22 A propos

LLM Zoo at Home: LLaMA & Alpaca | bergis universe of software, hardware and ideas

Tags:

2023-03-20 A propos

[2112.05682] Self-attention Does Not Need O(n^2) Memory

Tags:

2023-02-27 A propos

Maria Khalusova @maria@recsys.social sur Twitter : "Did you know that you can tweak the text output generated by a LLM without changing any of the trainable parameters?..."

Tags:

2023-02-23 A propos

stanfordnlp/dspy: 𝗗𝗦𝗣: Demonstrate-Search-Predict. A framework for composing retrieval and language models for knowledge-intensive NLP.

Tags:

2023-02-18 A propos

[2108.08877] Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Tags:

2023-02-17 A propos

[2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Tags:

2023-02-17 A propos

Prompting, Instruction Finetuning, and RLHF (CS224N)

Tags:

2023-02-16 A propos

Maarten Grootendorst sur Twitter : "The v0.14 release of BERTopic is here. Fine-tune your topic keywords and labels with models from @OpenAI, @huggingface, @CohereAI, @spacy_io, and @LangChainAI... An overview thread"

Tags:

2023-02-15 A propos

[2203.14465] STaR: Bootstrapping Reasoning With Reasoning

Tags:

2023-02-07 A propos

Google announces ChatGPT rival Bard, with wider availability in ‘coming weeks’ - The Verge

Tags:

2023-02-07 A propos

Ramsri Goutham Golla sur Twitter : "The most practical open-source competitor to @OpenAI 's GPT-3 is Google's Flan-T5 Here are 5 Flan-T5 resources to try out easily, deploy, or fine-tune it! 🧵" / Twitter

Tags:

2023-02-04 A propos

BioGPT

Tags:

2023-02-02 A propos

The Flan Collection: Advancing open source methods for instruction tuning – Google AI Blog

Tags:

2023-02-02 A propos

Shayne Longpre sur Twitter : "What’s the best completely public competitor to #ChatGPT? Flan-T5 beats all public models we tested..."

Tags:

2023-02-01 A propos

An empirical analysis of compute-optimal large language model training

Tags:

2023-01-26 A propos

Characterizing Emergent Phenomena in Large Language Models – Google AI Blog

Tags:

2023-01-26 A propos

[2301.04709] Causal Abstraction for Faithful Model Interpretation

Tags:

2023-01-14 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2023-01-13 A propos

Where does hugging face's transformers save models? - Stack Overflow

Tags:

2023-01-04 A propos

ValueError "invalid literal for int() with base 10" in trainer.evaluate (dataset created from pandas) · Issue #228 · huggingface/setfit

Tags:

2022-12-13 A propos

shikhar sur Twitter : "Instead of asking whether tree structure should be baked into NNs, our new paper asks if transformers already have a tendency to learn tree structured computations when trained on language, and if this structure is predictive of generalization! "

Tags:

2022-12-09 A propos

Few-Shot Text Classification (Cloudera 2020)

Tags:

2022-11-24 A propos

Meet CoAuthor, an Experiment in Human-AI Collaborative Writing

Tags:

2022-11-13 A propos

[2202.06991] Transformer Memory as a Differentiable Search Index

Tags:

2022-10-25 A propos

Stephanie Chan sur Twitter : "Transformer inductive biases..."

Tags:

2022-10-14 A propos

Lewis Tunstall sur Twitter : "The SetFit library for few-shot learning with Sentence Transformers now supports *multi-label text classification*..."

Tags:

2022-10-14 A propos

huggingface/setfit: Efficient few-shot learning with Sentence Transformers

Tags:

2022-10-12 A propos

MaartenGr/KeyBERT: Minimal keyword extraction with BERT

Tags:

2022-10-06 A propos

[2205.11498] Domain Adaptation for Memory-Efficient Dense Retrieval

Tags:

2022-09-26 A propos

Thomas Wolf sur Twitter : "you can divide the size of any model in 🤗 transformers: model.int8()"

Tags:

2022-09-26 A propos

[2209.11055] Efficient Few-Shot Learning Without Prompts

Tags:

2022-09-23 A propos

merve sur Twitter : "@huggingface transformers includes a new pipeline called Document Question Answering. This is a pipeline you can use to extract information from PDFs!...

Tags:

2022-09-20 A propos

[2208.01066] What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

Tags:

2022-09-17 A propos

PromptBERT improving BERT sentence embeddings with prompts - Ethan Kim

Tags:

BERT + Sentence Embeddings

2022-09-16 A propos

[2201.04337] PromptBERT: Improving BERT Sentence Embeddings with Prompts

Tags:

2022-09-16 A propos

Prompt Tuning BERT🎯:CommonLit Readability | Kaggle

Tags:

2022-09-16 A propos

[2010.00711] A Survey of the State of Explainable AI for Natural Language Processing

Tags:

2022-09-08 A propos

[2209.01975] Selective Annotation Makes Language Models Better Few-Shot Learners

Tags:

2022-09-07 A propos

Active Learning for BERT: An Empirical Study - ACL Anthology

Tags:

2022-09-02 A propos

Ankur Goyal sur Twitter : "DocQuery, a new #opensource query engine for analyzing documents using large language models (LLMs)..."

Tags:

2022-09-01 A propos

[2106.10199] BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

Tags:

2022-09-01 A propos

On Stability of Few-Sample Transformer Fine-Tuning | Kaggle

Tags:

2022-08-29 A propos

Unsupervised Learning — Sentence-Transformers documentation

Tags:

2022-08-20 A propos

Train and Fine-Tune Sentence Transformers Models

Tags:

Sentence-BERT

2022-08-13 A propos

Sylvain Gugger sur Twitter : "Load any HuggingFace model in Int8 precision and save half the memory..."

Tags:

2022-08-11 A propos

adapter-hub/adapter-transformers: Huggingface Transformers + Adapters

Tags:

2022-07-22 A propos

[2205.00820] Entity-aware Transformers for Entity Search

Tags:

2022-07-12 A propos

[2008.12813] HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Tags:

2022-06-30 A propos

Unveiling Transformers with LEGO - YouTube

Tags:

2022-06-30 A propos

Using BERT For Classifying Documents with Long Texts | by Armand Olivares | Medium

Tags:

2022-06-29 A propos

Chris Olah sur Twitter : "I'm excited to *finally* be making progress on understanding the first MLP layer in large transformer LMs. I've tried really hard and prior to SoLU had little success." / Twitter

Tags:

2022-06-27 A propos

ELS-RD/transformer-deploy: Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

Tags:

2022-06-13 A propos

sentence bert model in onnx format · Issue #46 · UKPLab/sentence-transformers

Tags:

2022-06-13 A propos

Hugging Face Transformer Inference Under 1 Millisecond Latency | by Michaël Benesty | Towards Data Science

Tags:

2022-06-13 A propos

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 A propos

Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team

Tags:

2022-06-02 A propos

Nils Reimers sur Twitter : "GPL goes multi-lingual..."

Tags:

2022-06-01 A propos

[2205.08184] SKILL: Structured Knowledge Infusion for Large Language Models

Tags:

2022-05-18 A propos

[2205.05131] Unifying Language Learning Paradigms

Tags:

2022-05-12 A propos

BERTopic: The Future of Topic Modeling | Pinecone

Tags:

2022-05-12 A propos

[2205.04260] EASE: Entity-Aware Contrastive Learning of Sentence Embedding

Tags:

2022-05-11 A propos

[2203.08913] Memorizing Transformers

Tags:

2022-05-07 A propos

Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."

Tags:

2022-04-27 A propos

[1909.00426] Global Entity Disambiguation with BERT

Tags:

2022-04-18 A propos

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 A propos

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 A propos

[2008.11228] A simple method for domain adaptation of sentence embeddings

Tags:

2022-04-01 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

Sentence Transformer Fine-Tuning (SetFit): Outperforming GPT-3 on few-shot Text-Classification while being 1600 times smaller | by Moshe Wasserblat (2021-12)

Tags:

2022-03-31 A propos

Nils Reimers sur Twitter : "Great post on SetFit"

Tags:

2022-03-31 A propos

Sentence Embedding Fine-tuning for the French Language | by La Javaness R&D | Feb, 2022 | Medium

Tags:

2022-03-31 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

[2203.14655] Few-Shot Learning with Siamese Networks and Label Tuning

Tags:

> the problem of building text classifiers with little or no training data.
>
> In recent years, an approach based on neural textual entailment models has been found to give strong results on a diverse range of tasks.

(cf. #[NLI](tag:nli), using the input text as the premise and the text representing the label as the hypothesis)

> In this work, we show that **with proper pre-training, Siamese Networks that embed texts and labels** offer a competitive alternative.
>
> We introduce **label tuning: fine-tuning the label embeddings only**. While giving lower performance than model fine-tuning (which updates all params of the model), this approach has the architectural advantage that a single encoder can be shared by many different tasks (we only fine-tune the label embeddings)
> The drop in quality can
be compensated by using a variant of **[Knowledge distillation](tag:knowledge_distillation)**

[Github](https://tinyurl.com/label-tuning), [Tweet](doc:2022/03/thomas_muller_sur_twitter_pa)

2022-03-30 A propos

(((ل()(ل() 'yoav))))👾 sur Twitter : "... another step in understanding how transformer-based LMs work..."

Tags:

2022-03-30 A propos

[2101.12294] Combining pre-trained language models and structured knowledge

Tags:

2022-03-25 A propos

[2006.05987] Revisiting Few-sample BERT Fine-tuning

Tags:

2022-03-21 A propos

NLP | How to add a domain-specific vocabulary (new tokens) to a subword tokenizer already trained like BERT WordPiece | by Pierre Guillou | Medium

Tags:

2022-03-18 A propos

Retraining roberta-base using the RoBERTa MLM Procedure | Medium

Tags:

2022-03-18 A propos

Studio Ousia sur Twitter : "Now using LUKE is easier than ever!" / Twitter

Tags:

2022-03-15 A propos

MaartenGr/BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.

Tags:

2022-03-10 A propos

Document Matching for Job Descriptions | Semantic Scholar (2021)

Tags:

2022-03-09 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

[2109.06304] Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Tags:

2022-02-25 A propos

Nils Reimers sur Twitter : "Creating intent classes for chatbots is challenging This tutorial shows how to use sentence-transformers to find potentially overlapping intent classes and how to improve your data annotation work." / Twitter

Tags:

2022-02-19 A propos

Nils Reimers sur Twitter : "how to use the fast clustering algorithm from sentence-transformers..."

Tags:

2022-02-19 A propos

sentence-transformers/fast_clustering.py at master · UKPLab/sentence-transformers

Tags:

2022-02-18 A propos

gsarti/scibert-nli · Hugging Face

Tags:

2022-01-29 A propos

Semantic Search — Sentence-Transformers documentation

Tags:

2022-01-29 A propos

[2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling

Tags:

2022-01-19 A propos

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 A propos

Domain Transfer with BERT | Pinecone

Tags:

2022-01-04 A propos

Anthropic sur Twitter : "a mathematical framework for trying to reverse engineer transformer language models..."

Tags:

2021-12-23 A propos

Making the Most of Data: Augmentation with BERT | Pinecone

Tags:

2021-12-18 A propos

Using pretrained SBERT model in cross-encoder · Issue #726 · UKPLab/sentence-transformers

Tags:

2021-12-17 A propos

Advance BERT model via transferring knowledge from Cross-Encoders to Bi-Encoders | by Chien Vu | Towards Data Science

Tags:

2021-12-17 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

Unsupervised_Extractive_Summarization - a Hugging Face Space by Hellisotherpeople

Tags:

2021-12-03 A propos

Unsupervised Training for Sentence Transformers | Pinecone

Tags:

2021-11-24 A propos

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 A propos

Extractive Question Answering - Hugging Face transformers doc

Tags:

2021-11-18 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines (2021)

Tags:

2021-10-30 A propos

Next-Gen Sentence Embeddings with Multiple Negatives Ranking Loss | Pinecone

Tags:

2021-10-27 A propos

Sentence Embeddings and Transformers | Pinecone

Tags:

2021-10-23 A propos

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 A propos

Seth Stafford sur Twitter : "Here’s a nice paper (ICLR spotlight) on how to apply masking in LM training..."

Tags:

2021-10-16 A propos

Sahajtomar/french_semantic · Hugging Face

Tags:

2021-10-14 A propos

Omer Levy sur Twitter : "What if I told you that fine-tuning T5-Large (0.8B params) on a couple hundred examples could outperform GPT-3 (175B params) on a bunch of tasks?"

Tags:

2021-10-13 A propos

Google AI Blog: Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer (2020)

Tags:

2021-10-13 A propos

How I almost won an NLP competition without knowing any Machine Learning - DEV Community

Tags:

2021-08-11 A propos

PaintTransformer - a Hugging Face Space by akhaliq

Tags:

2021-08-11 A propos

Transformer models - Hugging Face Course

Tags:

2021-06-15 A propos

Semantic Search with S-BERT is all you need

Tags:

2021-06-05 A propos

Transformers Pipelines.ipynb - Colaboratory

Tags:

2021-05-26 A propos

Nils Reimers sur Twitter : EasyNMT Easy-to-use (3 lines of code), state-of-the-art neural machine translations

Tags:

2021-04-27 A propos

Nils Reimers sur Twitter : "SBERT Release v1.1.0"

Tags:

2021-04-22 A propos

[2011.05864] On the Sentence Embeddings from Pre-trained Language Models

Tags:

2021-04-19 A propos

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Tags:

2021-04-18 A propos

Nils Reimers sur Twitter : "New models for Neural Information Retrieval..."

Tags:

2021-04-17 A propos

[2007.12603] IR-BERT: Leveraging BERT for Semantic Search in Background Linking for News Articles

Tags:

2021-04-12 A propos

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 A propos

[1902.00751] Parameter-Efficient Transfer Learning for NLP

Tags:

2021-04-11 A propos

exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology

Tags:

**Focus on the Embedding of Domain-specific Vocabulary.**

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 A propos

How many data points is a prompt worth?

Tags:

2021-04-08 A propos

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 A propos

SentenceTransformers Documentation

Tags:

2021-03-25 A propos

New pipeline for zero-shot text classification - 🤗Transformers - Hugging Face Forums

Tags:

2021-03-15 A propos

Hugging Face sur Twitter : "Fine-Tuning @facebookai's Wav2Vec2 for Speech Recognition is now possible in Transformers

Tags:

2021-03-12 A propos

Rodrigo Nogueira sur Twitter : "Slides of our WSDM 2021 tutorial "Pretrained Transformers for Text Ranking: BERT and Beyond"

Tags:

2021-03-09 A propos

asahi417/tner: Language model finetuning on NER

Tags:

2021-03-04 A propos

Zero-shot classifier distillation at master · huggingface/transformers

Tags:

2021-02-23 A propos

Zero-Shot Learning in Modern NLP | Joe Davison Blog (2020-05)

Tags:

2021-02-23 A propos

Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"

Tags:

2021-02-23 A propos

Retrieval Augmented Generation with Huggingface Transformers and Ray | Distributed Computing with Ray

Tags:

2021-02-11 A propos

kamalkraj/BERT-NER: Pytorch-Named-Entity-Recognition-with-BERT

Tags:

2021-02-07 A propos

[1911.03681] E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT

Tags:

> way of **injecting factual knowledge about entities into the pretrained BERT model**.

(Feeding entity vectors
into BERT as if they
were wordpiece vectors without additional encoder
pretraining)

>
> **We align [Wikipedia2Vec](tag:wikipedia2vec) entity vectors (Yamada et al., 2016) with BERT's native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors**. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to [ERNIE](tag:ernie) (Zhang et al., 2019) and [KnowBert](tag:knowbert) (Peters et al., 2019), but it **requires no expensive further pretraining of the BERT encoder**.
>
> Our vector space alignment strategy is inspired by
cross-lingual word vector alignment

Related work on Entity-enhanced BERT:

> ([ERNIE](doc:2019/08/_1905_07129_ernie_enhanced_la) and [Knowbert](doc:2020/05/1909_04164_knowledge_enhanced)) are based on the design principle
that BERT be adapted to entity vectors. They introduce
new encoder layers to feed pretrained entity
vectors into the Transformer, and they require additional
pretraining to integrate the new parameters.
In contrast, E-BERT’s design principle is that entity
vectors be adapted to BERT.
>
> Two other knowledge-enhanced MLMs are [KEPLER](doc:2020/11/1911_06136_kepler_a_unified_)
(Wang et al., 2019c) and K-Adapter (Wang
et al., 2020)... Their factual knowledge
does not stem from entity vectors – instead, they
are trained in a multi-task setting on relation classification
and knowledge base completion.

Not to be cofounded with [[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce](doc:2020/12/2009_02835_e_bert_a_phrase_a)

2021-01-12 A propos

google/tapas-base-finetuned-wtq · Hugging Face

Tags:

2020-12-17 A propos

[2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce

Tags:

2020-12-14 A propos

Supporting content decision makers with machine learning | Dec, 2020 | Netflix TechBlog

Tags:

2020-12-11 A propos

Keyword Extraction with BERT | Towards Data Science

Tags:

2020-12-06 A propos

Salmon Run: Word Sense Disambiguation using BERT as a Language Model

Tags:

2020-12-01 A propos

Domain-Specific BERT Models · Chris McCormick

Tags:

Domain-Specific BERT

2020-12-01 A propos

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 A propos

raphaelsty/ckb: Contextual knowledge bases

Tags:

2020-11-09 A propos

[2010.03496] Inductive Entity Representations from Text via Link Prediction

Tags:

BLP "BERT for Link Prediction". Central idea: **training an entity encoder with a
link prediction objective** (using the textual descriptions of entities when computing entity representations - hence not failing with entities unknown in training)

> a method for **learning representations
of entities**, that uses a **pre-trained Transformer** based
architecture as an entity encoder, and
**link prediction training on a knowledge graph
with textual entity descriptions**.

> using entity descriptions,
an entity encoder is trained for link prediction in
a knowledge graph. The encoder can then be used
without fine-tuning to obtain features for entity classification
and information retrieval

Cites [Xie et al](doc:2020/10/representation_learning_of_know) and [Kepler](doc:2020/11/1911_06136_kepler_a_unified_). They claim that their
objective targeted exclusively for link prediction (and not an objective that combines language modeling
and link prediction as Kepler)
performs better than Kepler's more complex one.

2020-11-03 A propos

Sylvain Gugger sur Twitter : "Training a transformer model for text classification..."

Tags:

2020-10-19 A propos

Guillaume Lample sur Twitter : "Last year, we showed that you can outperform a 24-layer transformer in language modeling with just...

Tags:

2020-10-10 A propos

Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science

Tags:

2020-10-04 A propos

Elastic Transformers. Making BERT stretchy — Scalable… | by Mihail Dungarov | Sep, 2020 | Medium

Tags:

2020-09-08 A propos

Hugging Face sur Twitter : "No labeled data? No problem. The 🤗 Transformers master branch now includes a built-in pipeline for zero-shot text classification...

Tags:

2020-08-12 A propos

UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet

Tags:

2020-07-14 A propos

How to use BERT for finding similar sentences or similar news? · Issue #876 · huggingface/transformers

Tags:

2020-07-12 A propos

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 A propos

Tags:

2020-07-10 A propos

[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Tags:

> a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones.

> a **neural language model which learns to access information
in a symbolic knowledge graph.**

> This
model builds on the recently-proposed [Entities as
Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020),
which extends the same transformer (Vaswani
et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities.
>
> After training EaE, the embedding associated
with an entity will (ideally) capture information
about the textual context in which that
entity appears, and by inference, the entity’s semantic
properties
>
> we include an additional
memory called a fact memory, which encodes
triples from a symbolic KB.
>
> This combination results in a
neural language model which learns to access information
in a the symbolic knowledge graph.

TODO:

- read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself")
- compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer)
- how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)?

2020-07-09 A propos

BERT Word Embeddings Tutorial · Chris McCormick

Tags:

2020-07-06 A propos

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 A propos

[2006.09462] Selective Question Answering under Domain Shift

Tags:

2020-06-30 A propos

Patrick von Platen sur Twitter : "Today, @huggingface is the start of our Reformer series..."

Tags:

2020-06-29 A propos

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 A propos

[1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling

Tags:

2020-05-11 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 A propos

BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Tags:

2020-03-28 A propos

[1909.03193] KG-BERT: BERT for Knowledge Graph Completion

Tags:

2020-03-22 A propos

[1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph

Tags:

2020-03-08 A propos

Unsupervised NER using BERT - Hands-on NLP model review - Quora

Tags:

2020-03-06 A propos

[2002.12327] A Primer in BERTology: What we know about how BERT works

Tags:

2020-02-28 A propos

[2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia

Tags:

2020-02-27 A propos

FastHugs | ntentional

Tags:

2020-02-19 A propos

[2002.05867] Transformers as Soft Reasoners over Language

Tags:

2020-02-17 A propos

Hugging Face: How to train a new language model from scratch using Transformers and Tokenizers

Tags:

2020-02-16 A propos

Distilling BERT models with spaCy - Towards Data Science (2019)

Tags:

2020-02-15 A propos

Hugging Face sur Twitter : DistilBERT-cased for Question Answering w/ just 3 lines of javascript

Tags:

2020-02-14 A propos

[2002.02925] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Tags:

2020-02-10 A propos

Canwen Xu sur Twitter : "WTF? We brutally dismember BERT and replace all his organs?"

Tags:

2020-02-10 A propos

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 A propos

Semantic Text Matching for Long-Form Documents (2019)

Tags:

2020-01-23 A propos

Building a Search Engine with BERT and TensorFlow - Towards Data Science

Tags:

2020-01-12 A propos

Elasticsearch meets BERT: Building Search Engine with Elasticsearch and BERT

Tags:

2020-01-10 A propos

NLP's Clever Hans Moment has Arrived

Tags:

2020-01-10 A propos

[2003.05473] Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (CoNNL 2019)

Tags:

Training BERT-base-uncased on English Wikipedia and then fine-tuned and evaluating it
on an entity linking (EL) benchmark (EL implemented as a token classification over the entity vocabulary)

> BERT+Entity is a straightforward extension on top
of BERT, i.e. we initialize BERT with the publicly
available weights from the BERT-base-uncased
model and add an output classification layer on
top of the architecture. Given a contextualized token,
the classifier computes the probability of an
entity link for each entry in the entity vocabulary.

Can BERT’s architecture learn all entity
linking steps jointly? To answer:

> an extreme
simplification of the **entity linking setup that
works surprisingly well**: simply cast it as **a
per token classification over the entire entity
vocabulary** (over 700K classes in our case).

> the model
is the first that performs entity linking without any
pipeline or any heuristics, compared to all prior
approaches. We found that with our approach we
can learn additional entity knowledge in BERT that
helps in entity linking. **However, we also found
that almost none of the downstream tasks really
required entity knowledge**.

### Related work

- > [Durrett and Klein (2014)](/doc/2020/01/a_joint_model_for_entity_analys) were the first to propose
jointly modelling Mention detection, Candidate generation and Entity disambiguation in a graphical
model and could show that each of those steps are
interdependent and benefit from a joint objective

This paper uses neural techniques instead of CRF.

- > [Yamada](/showprop.do?pptyuri=http%3A%2F%2Fwww.semanlink.net%2F2001%2F00%2Fsemanlink-schema%23arxiv_author&pptyval=Ikuya%2BYamada) (2016, 2017) was the first to
investigate neural text representations and entity
linking, but their approach is limited to ED.

cf. [#Wikipedia2Vec](tag:wikipedia2vec). Compare with [newer work by Yamada](doc:2020/09/1909_01259_neural_attentive_b)

2020-01-09 A propos

Named Entity Recognition with Bert – Depends on the definition

Tags:

2020-01-09 A propos

Joint Intent Classification and Slot Filling with Transformers (Jupyter Notebook Viewer)

Tags:

2020-01-09 A propos

[1902.10909] BERT for Joint Intent Classification and Slot Filling

Tags:

2020-01-09 A propos

Advancing Natural Language Processing (NLP) for Enterprise Domains

Tags:

2020-01-07 A propos

Richer Sentence Embeddings using Sentence-BERT — Part I

Tags:

Sentence-BERT

2020-01-06 A propos

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 A propos

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 A propos

Named Entity Recognition with Pytorch Transformers – Pierre-Yves Vandenbussche

Tags:

2019-12-11 A propos - Copie locale

[1912.01412] Deep Learning for Symbolic Mathematics

Tags:

2019-12-09 A propos

[1905.11852] EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction

Tags:

2019-12-05 A propos

Fastai with Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)

Tags:

2019-11-30 A propos

huggingface/transformers: 🤗 Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch.

Tags:

2019-11-16 A propos

CamemBERT

Tags:

CamemBERT

2019-11-10 A propos

[1911.01464] Emerging Cross-lingual Structure in Pretrained Language Models

Tags:

2019-11-06 A propos

BERT is now part of Google Search, so let’s understand how it reasons

Tags:

2019-10-31 A propos

[1802.07044] The Description Length of Deep Learning Models

Tags:

2019-10-11 A propos

Meet ALBERT: a new ‘Lite BERT’ from Google & Toyota with State of the Art NLP performance and 18x fewer parameters.

Tags:

2019-10-01 A propos

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 A propos

Evolution of Representations in the Transformer (2019)

Tags:

2019-09-16 A propos

[1909.01066] Language Models as Knowledge Bases?

Tags:

2019-09-05 A propos

Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT

Tags:

2019-08-28 A propos

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 A propos

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 A propos

What is XLNet and why it outperforms BERT - Towards Data Science

Tags:

2019-08-02 A propos

A2N: Attending to Neighbors for Knowledge Graph Inference - ACL 2019

Tags:

2019-07-31 A propos

BERT's success in some benchmarks tests may be simply due to the exploitation of spurious statistical cues in the dataset. Without them it is no better then random. : MachineLearning

Tags:

2019-07-24 A propos

[1907.07355] Probing Neural Network Comprehension of Natural Language Arguments

Tags:

2019-07-24 A propos

[1907.05242] Large Memory Layers with Product Keys

Tags:

> **a structured memory which can be easily integrated into a neural network.** The memory is very large by design and therefore significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on **product keys**, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time.

> a key-value memory layer that can increase model capacity for a negligible computational cost. A 12-layer transformer with a memory outperforms a 24-layer transformer, and is 2x faster!

[Implementation](/doc/2019/08/product_key_memory_pkm_minima)

TODO: compare with [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_)

2019-07-13 A propos

[1906.04341] What Does BERT Look At? An Analysis of BERT's Attention

Tags:

2019-06-21 A propos

NLP: Contextualized word embeddings from BERT – Towards Data Science

Tags:

BERT

2019-06-12 A propos

[1906.02715] Visualizing and Measuring the Geometry of BERT

Tags:

2019-06-07 A propos

"I made a bet that a Naive Bayes classifier would work as well on humor recognition as a neural net with fine-tuned Bert embeddings. I won"

Tags:

2019-06-06 A propos

Introducing FastBert — A simple Deep Learning library for BERT Models

Tags:

2019-05-23 A propos

Robust Language Representation Learning via Multi-task Knowledge Distillation - Microsoft Research

Tags:

2019-05-19 A propos

[1905.05950] BERT Rediscovers the Classical NLP Pipeline

Tags:

2019-05-18 A propos

[1904.08398] DocBERT: BERT for Document Classification

Tags:

2019-04-18 A propos

Introduction to PyTorch Code Examples

Tags:

2019-04-03 A propos

huggingface/pytorch-pretrained-BERT: The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL.

Tags:

2019-03-15 A propos

[1902.11269] Efficient Contextual Representation Learning Without Softmax Layer

Tags:

**how to accelerate contextual representation learning**.

> Contextual representation models are difficult to train due to the large parameter sizes and high computational complexity

> We find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size.
Therefore, we redesign the learning objectiv.
> Specifically, the proposed approach bypasses the softmax layer by performing language modeling with dimension reduction, and allows the models to leverage pre-trained word embeddings.
Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary.
When applied to ELMo, our method achieves a 4 times speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks.

**decouples learning contexts and words**

> Instead of using
a softmax layer to predict the distribution of the
missing word, we utilize and extend the SEMFIT
layer (Kumar and Tsvetkov, 2018) to **predict the
embedding of the missing word**.

2019-03-02 A propos

[1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding

Tags: