Semanlink - Arxiv Doc ; NLP@Meta AND NLP: pretraining

[2310.01352] RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Tags:

2023-10-07 About

[2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Tags:

2023-07-01 About

[2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Tags:

2023-02-17 About

[2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools

Tags:

> Toolformer, **a model
trained to decide which APIs to call, when to
call them, what arguments to pass, and how to
best incorporate the results into future token
prediction**.

> fulfills the
following desiderata:
> - The use of tools should be learned in a
self-supervised way without requiring large
amounts of human annotations
>- The LM should be able to decide for itself when
and how to use which tool.

> Approach based
on the recent idea of using large LMs with incontext
learning (Brown et al., 2020) to generate
entire datasets from scratch.
>
> Given just a handful of human-written examples
of how an API can be used, 
> - we let a LM annotate
a huge language modeling dataset with potential
API calls. 
> - We then use a self-supervised loss to
determine which of these API calls actually help
the model in predicting future tokens. 
>- Finally, we
finetune the LM itself on the API calls that it considers
useful.

[Jay Hack @mathemagic1an sur twitter](https://twitter.com/mathemagic1an/status/1624870248221663232):

> from a small seed set of human inputs (essentially demonstrating usage of APIs), the training set for this behavior is generated by the LLM itself.
> 
> So what does this mean? We've found a promising way to tightly integrate arbitrary APIs with our best-performing models.

2023-02-13 About

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags: