Semanlink - Contextual Document Embedding

Parents:

Contextual Document Embedding

Related Tags:

Descendants

2 Documents (Long List)

jack morris sur X : "New state-of-the-art small text embedding model... (cde-small-v2)"

Tags:

2025-01-15 About

Philipp Schmid sur X : "Can we improve retrieval for RAG by learning from neighboring contexts? Contextual Document Embedding ..."

Tags:

> There is at least one notable benefit of statistical approaches that is lost by neural models. Statistical models can easily incorporate prior corpus statistics such as inverse document frequency (IDF), into their representation. This prior term imparts context-dependence onto the model, since it can be updated based on information specific to retrieval in a given domain at test time. We contrast this contextual formulation with neural document encoders that are by definition a function of the document itself. For example consider the following document:

[Tweet by author](https://x.com/jxmnop/status/1842236045074498026): "best BERT-sized text embedding model in the world", "a paradigm shift for text retrieval"

> a new contextual embedding architecture.
> 
> neighboring document information, during training and encoding, can create "context-aware" embeddings
>
> this requires changes to both the training and evaluation pipeline to incorporate *contextual tokens*

1. Cluster similar documents to identify neighboring documents for each one.
2. Extend Encoder to include information from these neighboring documents when generating embeddings.
3. Train the model using a contrastive learning objective that incorporates neighboring documents into the loss function.

(Point 1 reminds me of Raphaël's [raphaelsty/neural-tree: Tree-based indexes for neural-search](doc:2024/02/raphaelsty_neural_tree_tree_ba))

2024-10-11 About

Properties

sl:creationDate : 2024-10-11
sl:creationTime : 2024-10-11T00:38:58Z
rdf:type : sl:Tag
skos:prefLabel : Contextual Document Embedding