Semanlink - AI@Google AND Knowledge-augmented Language Models

[2304.01982] Rethinking the Role of Token Retrieval in Multi-Vector Retrieval

Tags:

2023-04-05 A propos

Yi Tay sur Twitter : "Don't retrieve, recite!..."

Tags:

2022-10-06 A propos

[2205.08184] SKILL: Structured Knowledge Infusion for Large Language Models

Tags:

2022-05-18 A propos

Improving Language Models by Retrieving from Trillions of Tokens | DeepMind

Tags:

2021-12-09 A propos

[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training

Tags:

**Augment language model pre-training with a retriever module**, which
is trained using the masked language modeling objective.

> To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents

Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model."

[Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat)

[Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)

2020-12-12 A propos

Google AI Blog: REALM: Integrating Retrieval into Language Representation Models

Tags:

2020-08-13 A propos

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 A propos

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 A propos

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 A propos