Semanlink - Semantic Text Matching for Long-Form Documents (2019)

Semantic Text Matching for Long-Form Documents (2019)

Tags:

Au sujet de ce document

sl:bookmarkOf :
- https://jyunyu.csie.org/docs/pubs/www2019paper.pdf
- https://research.google/pubs/pub47856/
sl:creationDate : 2020-01-23
sl:creationTime : 2020-01-23T10:21:17Z

Infos sur le fichier

Bookmark of: https://jyunyu.csie.org/docs/pubs/www2019paper.pdf

Documents with similar tags (experimental)

[2401.18059] RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Tags:

2024-02-03 A propos

[2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Tags:

2023-07-01 A propos

Google "We Have No Moat, And Neither Does OpenAI"

Tags:

> low-cost public involvement was enabled
by a vastly cheaper mechanism for fine tuning called low
rank adaptation ()[LoRA](tag:lora)

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 A propos

Evidence of a predictive coding hierarchy in the human brain listening to speech | Nature Human Behaviour

Tags:

2023-04-16 A propos

shikhar sur Twitter : "Instead of asking whether tree structure should be baked into NNs, our new paper asks if transformers already have a tendency to learn tree structured computations when trained on language, and if this structure is predictive of generalization! "

Tags:

2022-12-09 A propos

[2202.06991] Transformer Memory as a Differentiable Search Index

Tags:

2022-10-25 A propos

Stephanie Chan sur Twitter : "Transformer inductive biases..."

Tags:

2022-10-14 A propos

Ilias Chalkidis sur Twitter : "Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification..."

Tags:

2022-10-13 A propos

[2010.06467] Pretrained Transformers for Text Ranking: BERT and Beyond

Tags:

2021-07-09 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

Aran Komatsuzaki sur Twitter : "Big Bird: Transformers for Longer Sequences..."

Tags: