Twitter thread

Jeremy Howard sur X : "We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc..."

2024-12-21T17:13:36Z

Philipp Schmid sur X : "Can we improve retrieval for RAG by learning from neighboring contexts? Contextual Document Embedding ..."

2024-10-11T00:38:04Z

> There is at least one notable benefit of statistical approaches that is lost by neural models. Statistical models can easily incorporate prior corpus statistics such as inverse document frequency (IDF), into their representation. This prior term imparts context-dependence onto the model, since it can be updated based on information specific to retrieval in a given domain at test time. We contrast this contextual formulation with neural document encoders that are by definition a function of the document itself. For example consider the following document: [Tweet by author](https://x.com/jxmnop/status/1842236045074498026): "best BERT-sized text embedding model in the world", "a paradigm shift for text retrieval" > a new contextual embedding architecture. > > neighboring document information, during training and encoding, can create "context-aware" embeddings > > this requires changes to both the training and evaluation pipeline to incorporate *contextual tokens* 1. Cluster similar documents to identify neighboring documents for each one. 2. Extend Encoder to include information from these neighboring documents when generating embeddings. 3. Train the model using a contrastive learning objective that incorporates neighboring documents into the loss function. (Point 1 reminds me of Raphaël's [raphaelsty/neural-tree: Tree-based indexes for neural-search](doc:2024/02/raphaelsty_neural_tree_tree_ba))

So Yeon (Tiffany) Min sur X : "Embodied-RAG, a General Non-Parametric Method for Retrieval and Generation..."

2024-10-06T10:18:51Z

> A new framework that equips embodied agents with a non-parametric memory capable of autonomously constructing hierarchical knowledge for navigation and language generation. ([Ruslan Salakhutdinov](tag:ruslan_salakhutdinov) [tweet](https://x.com/rsalakhu/status/1842694504387916073)) > Hi robot, I'm dehydrated, can you take me somewhere? > How to apply non-parametric memory to every day experiences? > key challenges in building embodied memory > - Dense memory that logs everything is memory inefficient. > - Space is continuous, and locations are spatially correlated, in contrast to independent documents in the text domain. > During the retrieval/generation phase, we select K "chains" (a leaf node all the way up to the root node), that are closest to the query.

Jeremy Howard sur X : "FSDP/QDoRA with Llama3 : I believe that this combination is likely to create better task-specific models than anything else at any cost."

2024-04-23T22:22:44Z

Zeyuan Allen-Zhu sur X : " surprisingly, when pre-training good data (e.g., Wiki) together with "junks" (e.g., Common Crawl), LLM's capacity on good data may decrease by 20x times!"

2024-04-10T18:33:05Z

> A simple fix: add domain tokens to your data; LLMs can auto-detect domains rich in knowledge and prioritize."

raphaelsrty sur X : "My personal Knowledge Base made to the front page of HackerNews today..."

2024-03-23T11:53:35Z

Matteo Pagliardini sur X : "#DenseFormer, a simple modification that performs—after each transformer block—a weighted average of past representations."

2024-03-23T11:44:52Z

Benjamin Clavié sur X : "Introducing rerankers: a lightweight library to provide a unified way to use various reranking methods"

2024-03-16T10:28:38Z

Akshay 🚀 sur X : "Let's build a "Chat with your code" RAG application, step-by-step"

2024-03-09T11:55:54Z

Krista Opsahl-Ong sur X : "Got a pipeline with multiple prompts, like a DSPy program? ... Introducing MIPRO, a Multi-prompt Instruction Proposal Optimizer...."

2024-03-09T11:37:47Z

Hrishi sur X : "RAPTOR is... one of the very few [RAG architectures] that actively presumes and uses the structure in a document...."

2024-03-09T11:30:15Z

(thread by the person of [WalkingRAG](tag:walkingrag)) > The similarities between WalkingRAG and RAPTOR are that both attempt to capture relationships in the data into a higher structure using LLMs... This is a tree in RAPTOR's case, with WalkingRAG it's a graph.

Jerry Liu sur X : "To better augment LLMs with context, it makes a lot of sense to organize context not just as a flat list of text chunks, but as a hierarchy of high-level to low-level details. RAPTOR..."

2024-03-03T10:14:19Z

> To better augment LLMs with context, it makes a lot of sense to organize context not just as a flat list of text chunks, but as a hierarchy of high-level to low-level details. > > RAPTOR is a super simple but neat idea towards this direction. Hierarchically cluster and summarize the text into a tree (the clustering is important, allows semantically related concepts to be grouped together and doesn't purely rely on spatial positioning!). During query-time dynamically retrieve the most relevant context to the question.

Omar Khattab sur X : "A thread on late interaction, how it works efficiently, and why/where it's been shown to improve quality"

2024-02-05T22:47:40Z

Craig Macdonald sur X : "a thread of our main ColBERT research findings"

2024-02-01T08:34:16Z

Jo Kristian Bergum sur X : (on "why using ColBERT?")

2024-01-28T10:53:02Z

> The idea that you can accurately boil down the nuances of ~256 tokens (2/3rds of a page) into a single vector is a pretty wild proposition > Inspired by @lateinteraction , we hacked into the ColBERT model's contextualized late-interaction similarities to produce [interpretable snippets](https://x.com/jobergum/status/1750282246072746178?s=20)!

LlamaIndex 🦙 sur X : "Use RAG to build advanced text-to-SQL..."

2024-01-24T22:31:35Z

Jerry Liu sur X : "4 Levels of Agents for RAG..."

2024-01-23T20:35:24Z

Rachit Bansal sur X : "An LLM can be efficiently composed with specialized (L)LMs to enable new tasks"

2024-01-06T12:07:15Z

[[2401.02412] LLM Augmented LLMs: Expanding Capabilities through Composition](doc:2024/01/2401_02412_llm_augmented_llms) > CALM—Composition to Augment Language Models: > 1. Scales up LLMs on new tasks by *re-using* existing (L)LMs w/ very few new parameters & data, > 2. Keeps existing model weights intact, hence **preserves original capabilities**, > 3. Applies to diverse domains and settings. > Rather than a shallow combination, CALM introduces a small set of cross-attention parameters over models’ layer representations. Use-case example, Multilinguality: > We reuse an LM trained on a bunch of low-resource languages (LRLs) w/ an LLM that has never seen some of these LRLs.

Omar Khattab sur X : "...Let's use 30 lines of DSPy—without any hand-written prompts or any calls to OpenAI ($0)—to teach..."

2024-01-01T11:01:32Z

Omar Khattab sur X : "A🧵on beating the hardware lottery for retrieval: the internals of the late interaction stack. ColBERT..."

2023-12-29T11:40:34Z