Semanlink - Question Answering ; Language Model AND NLP@Google

1. synthetic queries
are generated for each passage from the target corpus (using an existing pre-trained [T5](tag:text_to_text_transfer_transformer)
encoder-decoder)
2. the generated queries are used for mining negative
passages (retrieving the most similar
paragraphs using an existing dense retrieval
model == hard negatives!)
3. the query-passage pairs are labeled by a cross-encoder and used to train the domain-adapted
dense retriever (using method described in [Hofstätter et al.,
2020](doc:2021/12/2010_02666_improving_efficien))

[Nils Reimers sur Twitter](doc:2021/12/nils_reimers_sur_twitter_do_), [GitHub](https://github.com/UKPLab/gpl),  by the author of [TSDAE](doc:2021/09/2104_06979_tsdae_using_trans)

Claims to improve "Doc2Query" [Document Expansion by Query Prediction](doc:2022/01/1904_08375_document_expansion): ([src](https://twitter.com/KexinWang2049/status/1471435779415150598))

> - GPL: Uses doc2query to construct synthetic data and does knowledge distillation (i.e. training) on that data.
> - Doc2query: Generates queries to extend the documents and use BM25 on top of them w/o training.

2021-12-15 About

How to Fine-Tune Sentence-BERT for Question Answering | Capital One

Tags:

2021-11-21 About

Which flavor of BERT should you use for your QA task? | by Olesya Bondarenko | Towards Data Science

Tags:

2020-10-04 About

[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision

Tags:

2020-07-11 About

Hugging Face sur Twitter : DistilBERT-cased for Question Answering w/ just 3 lines of javascript

Tags:

2020-02-14 About

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Tags:

> It has recently been observed that neural language
models trained on unstructured text can
implicitly store and retrieve knowledge using
natural language queries.

indeed, cf. Facebook's paper [Language Models as Knowledge Bases?](/doc/2019/09/_1909_01066_language_models_as)

> In this short paper,
we measure the practical utility of this
approach by fine-tuning pre-trained models to
answer questions without access to any external
context or knowledge.

> we show that a large language
model pre-trained on unstructured text can
attain competitive results on open-domain question
answering benchmarks without any access
to external knowledge

BUT:

>1. state-of-the-art results only with the largest model
which had 11 billion parameters.
>1. “open-book” models
typically provide some indication of what information
they accessed when answering a question
that provides a useful form of interpretability.
In contrast, our model distributes knowledge
in its parameters in an inexplicable way, which
precludes this form of interpretability.
>1. **the maximum-likelihood objective provides no guarantees as to whether
a model will learn a fact or not.**

So, what's the point? To be compared with this [IBM's paper](/doc/2019/09/_1909_04120_span_selection_pre): "a new pre-training task inspired by reading comprehension and an effort to avoid encoding general knowledge in the transformer network itself"

2020-02-11 About

Adam Roberts sur Twitter : "New preprint: How Much Knowledge Can You Pack into the Parameters of a Language Model?..."

Tags:

2020-02-11 About