Semanlink - sl:creationDate : 2024-10-11

Search Tag:

Search Doc:

Preferences...

Tag Cloud

Contextual Document Embedding - Discuté avec Raphaël - Driverless car - ONNX - Sentence Transformers - Tesla, Inc - Twitter thread -

sl:creationDate : 2024-10-11

Contextual Document Embedding

Le robot-taxi, nouvel horizon de Tesla

Tags:

2024-10-11 About

Philipp Schmid sur X : "Can we improve retrieval for RAG by learning from neighboring contexts? Contextual Document Embedding ..."

Tags:

> There is at least one notable benefit of statistical approaches that is lost by neural models. Statistical models can easily incorporate prior corpus statistics such as inverse document frequency (IDF), into their representation. This prior term imparts context-dependence onto the model, since it can be updated based on information specific to retrieval in a given domain at test time. We contrast this contextual formulation with neural document encoders that are by definition a function of the document itself. For example consider the following document:

[Tweet by author](https://x.com/jxmnop/status/1842236045074498026): "best BERT-sized text embedding model in the world", "a paradigm shift for text retrieval"

> a new contextual embedding architecture.
> 
> neighboring document information, during training and encoding, can create "context-aware" embeddings
>
> this requires changes to both the training and evaluation pipeline to incorporate *contextual tokens*

1. Cluster similar documents to identify neighboring documents for each one.
2. Extend Encoder to include information from these neighboring documents when generating embeddings.
3. Train the model using a contrastive learning objective that incorporates neighboring documents into the loss function.

(Point 1 reminds me of Raphaël's [raphaelsty/neural-tree: Tree-based indexes for neural-search](doc:2024/02/raphaelsty_neural_tree_tree_ba))

2024-10-11 About

tomaarsen sur X : "📣 Sentence Transformers v3.2.0 is out..."

Tags:

2024-10-11 About