Semanlink - Yi Tay sur Twitter : "Don't retrieve, recite!..."

Impression

Recherche de Mot-clé

Recherche de Doc

Préférences...

Yi Tay sur Twitter : "Don't retrieve, recite!..."

Tags:

Au sujet de ce document

sl:bookmarkOf : https://twitter.com/YiTayML/status/1577702440312242188?s=20&t=bcQ6VwI7ttH26bleURBb_Q
sl:creationDate : 2022-10-06
sl:creationTime : 2022-10-06T01:47:13Z

Infos sur le fichier

Bookmark of: https://twitter.com/YiTayML/status/1577702440312242188?s=20&t=bcQ6VwI7ttH26bleURBb_Q

Documents with similar tags (experimental)

Peter J. Liu sur Twitter : "RLHF-alternative without RL"

Tags:

2023-05-18 A propos

Aran Komatsuzaki sur Twitter : "JaxPruner: A concise library for sparsity research An open-source JAX-based pruning and sparse training library for machine learning research repo"

Tags:

2023-04-28 A propos

[2303.16839] MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks

Tags:

2023-04-25 A propos

Jerry Liu sur Twitter : "To evaluate retrieval-augmented LLM’s, we can compare the query/response/source context to see if they match. We just added this to @gpt_index..."

Tags:

2023-04-11 A propos

David Chalmers sur Twitter : "what are some new and interesting results about the relative capacities of multimodal models and pure language models... (thinking about "do language models need sensory grounding for meaning and understanding?".)"

Tags:

2023-03-15 A propos

deepset sur Twitter : " Generative models have taken the world of NLP by storm. But LLMs do not know about your personal data. This makes personal assistants, enterprise knowledge management and many other applications challenging. Retrieval augmented pipelines are the answer"

Tags:

2023-02-02 A propos

fly51fly sur Twitter : "REPLUG: Retrieval-Augmented Black-Box Language Models..."

Tags:

2023-01-31 A propos

Uri Alon sur Twitter : "K-nearest neighbors language models (kNN-LMs; @ukhndlwl et al., ICLR'2020) improve the perplexity of standard LMs... Why?"

Tags:

2023-01-11 A propos

elvis sur Twitter : "NEW: Meta AI introduces OPT-IML, a large language model (175B) fine-tuned on 2000 NLP tasks. Uses instruction-tuning to improve zero-shot and few-shot generalization abilities...."

Tags:

2022-12-23 A propos

Ekin Akyürek @ NeurIPS sur Twitter : "How does in-context learning work?..."

Tags:

2022-12-01 A propos

Will Manidis sur Twitter : "Billions of hours of human potential every year are wasted on menial tasks. Data entry, form filling, basic knowledge work kind of stuff..."

Tags:

2022-10-26 A propos

Harrison Chase sur Twitter : "Introducing LangChain: a python package aimed at helping build LLM applications through composability..."

Tags:

2022-10-25 A propos

Stephanie Chan sur Twitter : "Transformer inductive biases..."

Tags:

2022-10-14 A propos

Prithviraj (Raj) Ammanabrolu sur Twitter : "The secret to aligning LMs to human preferences is reinforcement learning. ..."

Tags:

2022-10-06 A propos

Timo Schick sur Twitter : "PEER, a language model trained to incrementally write texts & collaborate w/ humans ..."

Tags:

2022-08-25 A propos

Andrej Karpathy sur Twitter : "For people wondering why, as a "vision person", I am interested in language models..."

Tags:

2022-07-18 A propos

[2203.08913] Memorizing Transformers

Tags:

2022-05-07 A propos

Papers with Code sur Twitter : "10 Recent Trends in Language Models In this thread..."

Tags:

2022-04-25 A propos

Tu Vu sur Twitter : "Enormous LMs like GPT-3 exhibit impressive few-shot performance, but w/ self-training a BERT base sized model can achieve much better results!

Tags:

2022-04-13 A propos

(((ل()(ل() 'yoav))))👾 sur Twitter : "... another step in understanding how transformer-based LMs work..."

Tags:

2022-03-30 A propos

Jason Weston sur Twitter : "SeeKeR: An open source search-augmented language model"

Tags:

2022-03-25 A propos

Improving Language Models by Retrieving from Trillions of Tokens | DeepMind

Tags:

2021-12-09 A propos

Hugging Face sur Twitter : "Transformers release of the Retrieval-Augmented Generation model in collaboration with @facebookai!"

Tags:

2021-02-23 A propos

[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training

Tags:

**Augment language model pre-training with a retriever module**, which
is trained using the masked language modeling objective.

> To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. **For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner**, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents

Hum, #TODO: parallel to be drawn with techniques in [KG-augmented Language Models](tag:knowledge_graph_augmented_language_models) which focus "on the problem of capturing declarative knowledge in the learned parameters of a language model."

[Google AI Blog Post](doc:2020/08/google_ai_blog_realm_integrat)

[Summary](https://joeddav.github.io/blog/2020/03/03/REALM.html) for the [Hugging Face awesome-papers reading group](doc:2021/03/huggingface_awesome_papers_pap)

2020-12-12 A propos

Guillaume Lample sur Twitter : "Last year, we showed that you can outperform a 24-layer transformer in language modeling with just...

Tags:

2020-10-10 A propos

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Tags:

> It has recently been observed that neural language
models trained on unstructured text can
implicitly store and retrieve knowledge using
natural language queries.

indeed, cf. Facebook's paper [Language Models as Knowledge Bases?](/doc/2019/09/_1909_01066_language_models_as)

> In this short paper,
we measure the practical utility of this
approach by fine-tuning pre-trained models to
answer questions without access to any external
context or knowledge.

> we show that a large language
model pre-trained on unstructured text can
attain competitive results on open-domain question
answering benchmarks without any access
to external knowledge

BUT:

>1. state-of-the-art results only with the largest model
which had 11 billion parameters.
>1. “open-book” models
typically provide some indication of what information
they accessed when answering a question
that provides a useful form of interpretability.
In contrast, our model distributes knowledge
in its parameters in an inexplicable way, which
precludes this form of interpretability.
>1. **the maximum-likelihood objective provides no guarantees as to whether
a model will learn a fact or not.**

So, what's the point? To be compared with this [IBM's paper](/doc/2019/09/_1909_04120_span_selection_pre): "a new pre-training task inspired by reading comprehension and an effort to avoid encoding general knowledge in the transformer network itself"

2020-02-11 A propos

Adam Roberts sur Twitter : "New preprint: How Much Knowledge Can You Pack into the Parameters of a Language Model?..."

Tags:

2020-02-11 A propos