Semanlink - Favoris ; Semantic technology AND Question Answering

> **approach for training dense retrieval models that does not require any labeled training data**. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples.
>
> ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents).
>
> It uses a new document-retrieval autoencoding scheme, where
> 1. an input question is used to retrieve a set of evidence documents, and
> 2. the documents are then used to compute the probability of reconstructing the original question.
>
> Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning.

[Tweet](doc:2022/07/devendra_singh_sachan_sur_twitt)

> Given an
input question, ART first retrieves a small set
of possible evidences documents. It then recon
structs
the original question by attending to these
documents
>
> The
key idea in ART is to consider the retrieved documents
as a noisy representation of the original
question and question reconstruction probability
as a way of denoising that provides soft-labels for
how likely each document is to have been the correct
result

Refers to [[IZACARD 2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering](doc:2020/12/2012_04584_distilling_knowled)

2022-07-06 A propos

Devendra Singh Sachan sur Twitter : "ART (Autoencoding-based Retriever Training), an unsupervised method to train a dense retriever that only uses questions and a collection of unpaired documents as the training data."

Tags:

2022-07-06 A propos

[2205.15952] Knowledge Graph -- Deep Learning: A Case Study in Question Answering in Aviation Safety Domain

Tags:

2022-06-11 A propos

Understanding Semantic Search and Question Answering | deepset

Tags:

2022-06-03 A propos

Domain transfer with GGPL: German Generative Pseudo Labeling 🥨 | by Matthias Richter | Jun, 2022 | ML6team

Tags:

2022-06-02 A propos

Nils Reimers sur Twitter : "GPL goes multi-lingual..."

Tags:

2022-06-01 A propos

[2012.12624] Learning Dense Representations of Phrases at Scale

Tags:

2022-05-11 A propos

Ramsri Goutham Golla sur Twitter : "Hi @Nils_Reimers For GPL you used "msmarco-distilbert-base-tas-b" model and ..."

Tags:

2022-04-27 A propos

Domain Adaptation — Sentence-Transformers documentation

Tags:

2022-03-31 A propos

NAVER LABS Europe : "@Nils_Reimers of @huggingface on 'Unsupervised domain adaptation for neural search'"

Tags:

2022-03-09 A propos

[2004.11892] Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering

Tags:

2022-02-11 A propos

The Quick Guide to SQuAD

Tags:

SQuAD

2022-02-03 A propos

Xikun Zhang sur Twitter : GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

Tags:

2022-01-28 A propos

Modern Question Answering Systems Explained

Tags:

Haystack

2022-01-27 A propos

Haystack Annotation Tool

Tags:

Haystack Annotation Tool

2022-01-27 A propos

allenai/macaw: Multi-angle c(q)uestion answering

Tags:

2022-01-22 A propos

Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack

Tags:

2022-01-12 A propos

How to build a chatbot that reads all your data to find the right answer - Xatkit

Tags:

2022-01-11 A propos

[1906.00300] Latent Retrieval for Weakly Supervised Open Domain Question Answering

Tags:

2022-01-11 A propos

[1904.08375] Document Expansion by Query Prediction

Tags:

2022-01-05 A propos

AK sur Twitter : "You Only Need One Model for Open-domain Question Answering..."

Tags:

2021-12-23 A propos

[2112.07577] GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Tags:

An unsupervised domain adaptation technique for dense retrieval models

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 A propos

Nils Reimers sur Twitter : "Do dense retrieval models work out-of-the-box for your specific domain? Often the answer was No😢..."

Tags:

2021-12-15 A propos

Multi-domain Multilingual Question Answering

Tags:

2021-12-08 A propos

[1909.06356] Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

Tags:

2021-12-08 A propos

ZhangShiyue/QGforQA

Tags:

2021-12-08 A propos

awslabs/unsupervised-qa: Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering

Tags:

2021-12-08 A propos

facebookresearch/DrQA: Reading Wikipedia to Answer Open-Domain Questions

Tags:

2021-12-08 A propos

[1906.04980] Unsupervised Question Answering by Cloze Translation

Tags:

2021-12-08 A propos

facebookresearch/UnsupervisedQA: Unsupervised Question answering via Cloze Translation

Tags:

2021-12-07 A propos

Efficient Open-Domain Question Answering | Getting Started with Baselines

Tags:

2021-11-25 A propos

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 A propos

[1911.02655] Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks

Tags:

2021-11-19 A propos

[2108.13854] Contrastive Domain Adaptation for Question Answering using Limited Text Corpora

Tags:

2021-11-19 A propos

[1706.03610] Neural Domain Adaptation for Biomedical Question Answering

Tags:

2021-11-19 A propos

Extractive Question Answering - Hugging Face transformers doc

Tags:

2021-11-18 A propos

How to Build an Open-Domain Question Answering System?

Tags:

2021-11-18 A propos

Zexuan Zhong sur Twitter : ...Does this really mean dense models are better? No Our #EMNLP2021 paper shows dense retrievers even fail to answer simple entity-centric questions

Tags:

2021-10-07 A propos

raphaelsty/RetrieverReader: Fast API QA

Tags:

2021-10-04 A propos

Haystack (deepset)

Tags:

2021-09-20 A propos

Build NLP features into your product | deepset

Tags:

2021-09-20 A propos

[2107.12708] QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Tags:

2021-08-06 A propos

[2102.07043] Reasoning Over Virtual Knowledge Bases With Open Predicate Relations

Tags:

> a method for constructing **a virtual KB (VKB) trained entirely from text**

Open Predicate Query Language (OPQL): constructing a virtual knowledge base (VKB) that supports KB reasoning & open-domain QA, tackling the incompleteness of knowledge bases by constructing a virtual KB only from text

> OPQL constructs
a VKB by **encoding and indexing a set of
relation mentions** in a way that naturally enables
reasoning and can be trained without any structured
supervision.

> can be used
as an **external memory integrated into a language
model**

cf. this earlier paper [[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base](doc:2020/07/2002_10640_differentiable_rea). But does not require an initial structured KB for distant
supervision.

> The key idea in constructing the OPQL VKB is to use a
dual-encoder pre-training process, similar to 
[[1906.03158] Matching the Blanks: Distributional Similarity for Relation Learning](doc:2021/05/1906_03158_matching_the_blank)

Related work section refers to [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced). Also refers to [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_) (some authors in common)

2021-06-20 A propos

[2106.00882] Efficient Passage Retrieval with Hashing for Open-domain Question Answering

Tags:

2021-06-03 A propos

[2004.04906] Dense Passage Retrieval for Open-Domain Question Answering

Tags:

2021-06-03 A propos

[2103.12876] Complex Factoid Question Answering with a Free-Text Knowledge Graph

Tags:

2021-03-30 A propos

[1911.03876] Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering

Tags:

2021-02-08 A propos

The Natural Language Decathlon: Multitask Learning as Question Answering (slides)

Tags:

2021-01-26 A propos

QAnswer · Accessing your Knowledge via Natural Language

Tags:

2021-01-26 A propos

Event Extraction by Answering (Almost) Natural Questions

Tags:

2020-12-17 A propos

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 A propos

Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science

Tags:

2020-10-04 A propos

Aran Komatsuzaki sur Twitter : "Big Bird: Transformers for Longer Sequences..."

Tags:

2020-07-29 A propos

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 A propos

[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base

Tags:

2020-07-11 A propos

[2006.09462] Selective Question Answering under Domain Shift

Tags:

2020-06-30 A propos

Differentiable Reasoning over Text – Machine Learning Blog | ML@CMU | Carnegie Mellon University

Tags:

2020-05-16 A propos

Hugging Face sur Twitter : DistilBERT-cased for Question Answering w/ just 3 lines of javascript

Tags:

2020-02-14 A propos

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Tags:

> It has recently been observed that neural language
models trained on unstructured text can
implicitly store and retrieve knowledge using
natural language queries.

indeed, cf. Facebook's paper [Language Models as Knowledge Bases?](/doc/2019/09/_1909_01066_language_models_as)

> In this short paper,
we measure the practical utility of this
approach by fine-tuning pre-trained models to
answer questions without access to any external
context or knowledge.

> we show that a large language
model pre-trained on unstructured text can
attain competitive results on open-domain question
answering benchmarks without any access
to external knowledge

BUT:

>1. state-of-the-art results only with the largest model
which had 11 billion parameters.
>1. “open-book” models
typically provide some indication of what information
they accessed when answering a question
that provides a useful form of interpretability.
In contrast, our model distributes knowledge
in its parameters in an inexplicable way, which
precludes this form of interpretability.
>1. **the maximum-likelihood objective provides no guarantees as to whether
a model will learn a fact or not.**

So, what's the point? To be compared with this [IBM's paper](/doc/2019/09/_1909_04120_span_selection_pre): "a new pre-training task inspired by reading comprehension and an effort to avoid encoding general knowledge in the transformer network itself"

2020-02-11 A propos

Adam Roberts sur Twitter : "New preprint: How Much Knowledge Can You Pack into the Parameters of a Language Model?..."

Tags:

2020-02-11 A propos

[1912.08904] Macaw: An Extensible Conversational Information Seeking Platform

Tags:

2020-01-01 A propos

Unsupervised Learning with Text (AFIA 2019)

Tags:

2019-12-14 A propos

[1910.09760] Question Answering over Knowledge Graphs via Structural Query Patterns

Tags:

2019-11-06 A propos

Improving long-form question answering by compressing search results

Tags:

2019-10-24 A propos

Answering Complex Open-domain Questions at Scale | SAIL Blog

Tags:

2019-10-21 A propos

[1909.04120] Span Selection Pre-training for Question Answering

Tags:

> a **new pre-training task inspired by reading
comprehension** and an **effort to avoid encoding general knowledge in the transformer network itself**

Current transformer architectures store general knowledge -> large models, long pre-training time. Better to offload the requirement of general knowledge to a sparsely activated network.

"Span selection" as an additional auxiliary task: the query is a sentence drawn from a corpus
with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is
relevant as determined by a BM25 search, and answer-bearing (containing the answer
term). Unlike BERT’s cloze task, where the answer must be drawn from the model itself, the answer is found in a passage
using language understanding.

> **We hope to progress to a model of general purpose language modeling that uses an indexed long
term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.**

2019-09-18 A propos

Bringing the power of machine reading comprehension to specialized documents - Microsoft Research

Tags: