Semanlink - Arxiv Doc ; Deep Learning AND NLP@Meta

[2310.01352] RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Tags:

2023-10-07 A propos

[2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Tags:

2023-07-01 A propos

[2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Tags:

2023-02-17 A propos

[2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools

Tags:

> Toolformer, **a model
trained to decide which APIs to call, when to
call them, what arguments to pass, and how to
best incorporate the results into future token
prediction**.

> fulfills the
following desiderata:
> - The use of tools should be learned in a
self-supervised way without requiring large
amounts of human annotations
>- The LM should be able to decide for itself when
and how to use which tool.

> Approach based
on the recent idea of using large LMs with incontext
learning (Brown et al., 2020) to generate
entire datasets from scratch.
>
> Given just a handful of human-written examples
of how an API can be used, 
> - we let a LM annotate
a huge language modeling dataset with potential
API calls. 
> - We then use a self-supervised loss to
determine which of these API calls actually help
the model in predicting future tokens. 
>- Finally, we
finetune the LM itself on the API calls that it considers
useful.

[Jay Hack @mathemagic1an sur twitter](https://twitter.com/mathemagic1an/status/1624870248221663232):

> from a small seed set of human inputs (essentially demonstrating usage of APIs), the training set for this behavior is generated by the LLM itself.
> 
> So what does this mean? We've found a promising way to tightly integrate arbitrary APIs with our best-performing models.

2023-02-13 A propos

[2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Tags:

2022-01-19 A propos

[1901.04085] Passage Re-ranking with BERT

Tags:

2021-03-26 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 A propos

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 A propos

[1912.01412] Deep Learning for Symbolic Mathematics

Tags:

2019-12-09 A propos

[1905.11852] EDUCE: Explaining model Decisions through Unsupervised Concepts Extraction

Tags:

2019-12-05 A propos

[1911.01464] Emerging Cross-lingual Structure in Pretrained Language Models

Tags:

2019-11-06 A propos

[1909.01066] Language Models as Knowledge Bases?

Tags:

2019-09-05 A propos

[1907.05242] Large Memory Layers with Product Keys

Tags:

> **a structured memory which can be easily integrated into a neural network.** The memory is very large by design and therefore significantly increases the capacity of the architecture, by up to a billion parameters with a negligible computational overhead. Its design and access pattern is based on **product keys**, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and computation efficiency both at training and test time.

> a key-value memory layer that can increase model capacity for a negligible computational cost. A 12-layer transformer with a memory outperforms a 24-layer transformer, and is 2x faster!

[Implementation](/doc/2019/08/product_key_memory_pkm_minima)

TODO: compare with [[2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge](doc:2020/07/2007_00849_facts_as_experts_)

2019-07-13 A propos

[1503.08895] End-To-End Memory Networks

Tags:

2018-10-23 A propos