Semanlink - NLP@ENS

Parents:

NLP@ENS

Related Tags:

Descendants

7 Documents (Long List)

Evidence of a predictive coding hierarchy in the human brain listening to speech | Nature Human Behaviour

Tags:

> Language models still fail to match the language abilities of humans. **[Predictive coding theory](tag:predictive_coding.html)** offers a tentative explanation to this discrepancy: **while language models are optimized to predict nearby words, **the human brain would continuously predict a hierarchy of representations that spans multiple timescales**. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, **these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition.**

> our
results support the idea that, unlike current language algorithms, the
brain is not limited to predict word-level representations but rather
predicts multiple levels of representations.

> our study thus calls for systematically training algorithms to predict
multiple timescales and levels of representations

2023-03-03 About

[2012.15156] A Memory Efficient Baseline for Open Domain Question Answering

Tags:

2022-08-08 About

[2208.03299] Few-shot Learning with Retrieval Augmented Language Model

Tags:

2022-08-08 About

[2112.09118] Towards Unsupervised Dense Information Retrieval with Contrastive Learning

Tags:

2021-12-21 About

[2012.04584] Distilling Knowledge from Reader to Retriever for Question Answering

Tags:

> a method to train an information retrieval module for downstream tasks, **without using pairs of queries and documents as annotations**.

Uses two models (standard pipeline for open-domain QA):

- the first one retrieves documents from a large source of knowledge (the retriever)
- the second one processes the support documents to solve the task (the reader).

> First the retriever selects support passages in a large knowledge
source. Then these passages are processed by the reader, along with the question, to generate an
answer

Inspired by knowledge distillation: the reader model is the teacher and the retriever is the student.

> More precisely, we use a sequence-to-sequence model as the reader, and use
the attention activations over the input documents as synthetic labels to train the retriever. 
> (**train the retriever by learning to approximate the attention score of the reader**)

Refers to:

- [REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)
- [Dehghani: Neural Ranking Models with Weak Supervision](doc:?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1704.08803)

2020-12-11 About

From Random Grammars to Learning Language - Département de Physique de l'Ecole Normale supérieure

Tags:

2020-09-17 About

[1802.07044] The Description Length of Deep Learning Models

Tags:

2019-10-11 About

Properties

sl:creationDate : 2021-04-30
sl:creationTime : 2021-04-30T23:07:57Z
rdf:type : sl:Tag
skos:prefLabel : NLP@ENS