Semanlink - Question Answering ; Not Encoding Factual Knowledge in Language Model AND Knowledge-augmented Language Models

Question Answering ; Not Encoding Factual Knowledge in Language Model AND Knowledge-augmented Language Models

Common descendants

5 Documents

deepset sur Twitter : " Generative models have taken the world of NLP by storm. But LLMs do not know about your personal data. This makes personal assistants, enterprise knowledge management and many other applications challenging. Retrieval augmented pipelines are the answer"

Tags:

2023-02-02 About

[2207.06300] Re2G: Retrieve, Rerank, Generate

Tags:

2022-07-14 About

Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack

Tags:

2022-01-12 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 About