]> >ready-to-use model capable of general question answering, showing robustness outside the domains it was trained on. It has been trained in "multi-angle" fashion, which means it can handle a flexible set of input and output "slots" (like question, answer, explanation) . 2022-01-22T00:25:31Z 2022-01-22 allenai/macaw: Multi-angle c(q)uestion answering 2022-01-11 2019-06-01T22:02:39Z 2022-01-11T11:06:38Z Recent work on open domain question answering (QA) assumes strong supervision of the supporting evidence and/or assumes a blackbox information retrieval (IR) system to retrieve evidence candidates. We argue that both are suboptimal, since gold evidence is not always available, and QA is fundamentally different from IR. We show for the first time that it is possible to jointly learn the retriever and reader from question-answer string pairs and without any IR system. In this setting, evidence retrieval from all of Wikipedia is treated as a latent variable. Since this is impractical to learn from scratch, we pre-train the retriever with an Inverse Cloze Task. We evaluate on open versions of five QA datasets. On datasets where the questioner already knows the answer, a traditional IR system such as BM25 is sufficient. On datasets where a user is genuinely seeking an answer, we show that learned retrieval is crucial, outperforming BM25 by up to 19 points in exact match. Kenton Lee Kenton Lee Ming-Wei Chang > The key insight of this work is that end-to-end learning is possible if we pre-train the retriever with an unsupervised Inverse Cloze Task (ICT). In ICT, a sentence is treated as a pseudo- question, and its context is treated as pseudo- evidence Kristina Toutanova [1906.00300] Latent Retrieval for Weakly Supervised Open Domain Question Answering 1906.00300 2019-06-27T21:06:12Z Latent Retrieval for Weakly Supervised Open Domain Question Answering How to Build a Semantic Search Engine With Transformers and Faiss | by Kostas Stathoulopoulos | Towards Data Science 2022-01-29T17:33:32Z 2022-01-29 Hugo, L'Homme Qui Rit 2022-01-26 2022-01-26T23:49:42Z > Quel est le père du privilège ? le hasard. Et quel est son fils ? l’abus Matei Zaharia Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Beyond reducing the cost of re-ranking the documents retrieved by a traditional model, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from a large document collection. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query. [2004.12832] ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT 2022-01-12 Omar Khattab 2022-01-12T00:15:40Z (The 1st Colbert paper) Omar Khattab 2020-06-04T05:28:21Z 2004.12832 ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT 2020-04-27T14:21:03Z 2022-01-29 gsarti/scibert-nli · Hugging Face 2022-01-29T15:52:08Z SciBERT fine-tuned on the SNLI and the MultiNLI datasets using the sentence-transformers library to produce universal sentence embeddings [1904.08375] Document Expansion by Query Prediction 1904.08375 Rodrigo Nogueira Kyunghyun Cho 2022-01-05T09:29:00Z Jimmy Lin Wei Yang "doc2query" > One technique to improve the retrieval effectiveness of a search engine is to **expand documents with terms that are related or representative of the documents' content**. From the perspective of a question answering system, this might comprise questions the document can potentially answer. Following this observation, we propose **a simple method that predicts which queries will be issued for a given document** and then expands it with those predictions with a vanilla sequence-to-sequence model, trained using datasets consisting of pairs of query and relevant documents. > > In a latency-critical regime, retrieval results alone (without re-ranking) approach the effectiveness of more computationally expensive neural re-rankers but are much faster [GitHub](https://github.com/nyu-dl/dl4ir-doc2query), Improved version [GitHub](https://github.com/castorini/docTTTTTquery) (using [T5](tag:text_to_text_transfer_transformer)) 2019-04-17T17:20:14Z One technique to improve the retrieval effectiveness of a search engine is to expand documents with terms that are related or representative of the documents' content.From the perspective of a question answering system, this might comprise questions the document can potentially answer. Following this observation, we propose a simple method that predicts which queries will be issued for a given document and then expands it with those predictions with a vanilla sequence-to-sequence model, trained using datasets consisting of pairs of query and relevant documents. By combining our method with a highly-effective re-ranking component, we achieve the state of the art in two retrieval tasks. In a latency-critical regime, retrieval results alone (without re-ranking) approach the effectiveness of more computationally expensive neural re-rankers but are much faster. 2022-01-05 Rodrigo Nogueira 2019-09-25T00:40:54Z Document Expansion by Query Prediction 2022-01-28 Xikun Zhang sur Twitter : GreaseLM: Graph REASoning Enhanced Language Models for Question Answering 2022-01-28T11:25:48Z Il faut cultiver notre jardin « Le jardin partagé est la forme archétypale de la société démocratique et écologique » 2022-01-09 2022-01-09T18:28:04Z 2022-01-27T22:49:43Z 2022-01-27 Modern Question Answering Systems Explained 2022-01-04T21:00:34Z 2022-01-04 Domain Transfer with BERT | Pinecone 2022-01-16T19:01:18Z 2022-01-16 Chinua Achebe and the Great African Novel | The New Yorker 2022-01-23T11:48:20Z > But the whites do not think this way-- they prefer to forget that everything they want already belongs to someone else. They think, 'Oh I am white, this must be mine.' And they believe it Tiehteti. **I have never seen a white person who did not look surprised when you killed them**." He shrugged. "Me, when I steal something, I expect the person to try to kill me, and I know the song I will sing when I die." > > Je n'ai pas besoin de te dire à quoi ressemblait cette terre, a-t-il dit. Et tu n'as pas besoin de me dire que c'est moi qui l'ai saccagée. C'est vrai, de mes propres mains et à jamais.... Mais c'est toute l'histoire de l'humanité. De la terre au sable, du fertile au stérile, des fruits aux épines. On ne sait faire que ça The Son (Meyer novel) - Wikipedia 2022-01-23 Representation learning is a critical ingredient for natural language processing systems. Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks. We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation. We show that SPECTER outperforms a variety of competitive baselines on the benchmark. 2004.07180 Iz Beltagy SPECTER: Document-level Representation Learning using Citation-informed Transformers Sergey Feldman Arman Cohan [2004.07180] SPECTER: Document-level Representation Learning using Citation-informed Transformers Arman Cohan 2022-01-29 Daniel S. Weld 2020-04-15T16:05:51Z > method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning. 2020-05-20T17:39:52Z 2022-01-29T15:18:20Z Doug Downey 2022-01-29T17:35:57Z 2022-01-29 Cape Fear (1991 film) - Wikipedia Les Nerfs à vif de Scorsese avec De Niro Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Vladimir Karpukhin 2020-05-22T21:34:34Z > We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. > > [REALM](tag:realm) and ORQA, two recently introduced models that combine masked language models with a differentiable retriever... have only explored open-domain extractive question answering. Here, we bring hybrid parametric and non-parametric memory to the “workhorse of NLP,” i.e. sequence-to-sequence (seq2seq) models. > > RAG models use the input sequence x to retrieve text documents z and use them as additional context when generating the target sequence > > **A key feature of our memory is that it is comprised of raw text rather distributed representations**, which makes the memory both (i) human-readable, lending a form of interpretability to our model, and (ii) human-writable, enabling us to dynamically update the model’s memory by editing the document index Tim Rocktäschel Patrick Lewis Naman Goyal Aleksandra Piktus Patrick Lewis Ethan Perez [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Heinrich Küttler Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline. Wen-tau Yih 2022-01-19 Mike Lewis 2021-04-12T15:42:18Z Fabio Petroni Douwe Kiela 2005.11401 2022-01-19T16:56:31Z Sebastian Riedel 2022-01-29 Fury (1936 film) - Wikipedia 2022-01-29T17:51:40Z The story of an innocent man (Spencer Tracy) who narrowly escapes being burned to death by a lynch mob and the revenge he then seeks. 2022-01-17T09:52:39Z 2022-01-17 sebastian-hofstaetter/teaching: Open-Source Information Retrieval Courses @ TU Wien How to build a chatbot that reads all your data to find the right answer - Xatkit 2022-01-11 2022-01-11T15:35:34Z avec Pierre Fresmay La Main du diable 2022-01-04 2022-01-04T01:06:21Z wikidata turned into a trivia card game [tweet](https://twitter.com/tom_j_watson/status/1482716797736529922) 2022-01-16 Wiki History Game 2022-01-16T23:00:53Z 2022-01-20T09:34:34Z 2022-01-20 « Vouloir repartir pour un nouveau round nucléaire engage très lourdement notre avenir » 2022-01-22 2022-01-22T13:12:09Z Vertex AI Workbench  |  Google Cloud [2009.02252] KILT: a Benchmark for Knowledge Intensive Language Tasks Majid Yazdani 2021-05-27T15:20:59Z 2022-01-23T18:25:25Z Jean Maillard Vassilis Plachouras Vladimir Karpukhin Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT. Fabio Petroni Fabio Petroni 2009.02252 Patrick Lewis 2020-09-04T15:32:19Z KILT: a Benchmark for Knowledge Intensive Language Tasks Sebastian Riedel James Thorne Tim Rocktäschel Aleksandra Piktus 2022-01-23 Angela Fan Nicola De Cao Yacine Jernite 2022-01-10T20:53:11Z 2022-01-10 Une bactérie inoculée au moustique pour lutter contre la dengue raphaelsty/cherche: Neural search > Cherche (search in French) allows you to create a neural search pipeline using retrievers and pre-trained language models as rankers. Cherche is meant to be used with small to medium sized corpora. 2022-01-11 2022-01-11T10:35:55Z Graph ML in 2022: Where Are We Now? /Users/fps/Sites/fps/2022/01/Graph ML in 2022.pdf 2022-01-28T11:23:42Z 2022-01-28 « Infliger des peines aux homosexuels, c’était ré-vol-tant » : en 1982, le plaidoyer de Robert Badinter pour la cause gay 2022-01-22T00:14:08Z 2022-01-22 REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained) - YouTube 2022-01-23T14:25:13Z 2022-01-23 - A new pretraining method - separate language and world knowledge - pre-training is MLM **symmetric** semantic search vs **asymmetric** semantic search > - Suitable models for symmetric semantic search: Pre-Trained Sentence Embedding > - Suitable models for asymmetric semantic search: Pre-Trained MS MARCO Models 2022-01-29T15:28:25Z 2022-01-29 Semantic Search — Sentence-Transformers documentation > "Knowledge Graph Induction", a system for slot filling based on advanced training strategies for both Dense Passage Retrieval (DPR) and Retrieval Augmented Generation (RAG) see [[1909.04120] Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) (same first author) [GitHub](https://github.com/IBM/kgi-slot-filling) Alfio Gliozzo 2021-08-31T15:51:27Z Michael Glass 2108.13934 2021-09-14T01:06:04Z 2022-01-19T17:14:49Z [2108.13934] Robust Retrieval Augmented Generation for Zero-shot Slot Filling Michael Glass 2022-01-19 Gaetano Rossiello Automatically inducing high quality knowledge graphs from a given collection of documents still remains a challenging problem in AI. One way to make headway for this problem is through advancements in a related task known as slot filling. In this task, given an entity query in form of [Entity, Slot, ?], a system is asked to fill the slot by generating or extracting the missing value exploiting evidence extracted from relevant passage(s) in the given document collection. The recent works in the field try to solve this task in an end-to-end fashion using retrieval-based language models. In this paper, we present a novel approach to zero-shot slot filling that extends dense passage retrieval with hard negatives and robust training procedures for retrieval augmented generation models. Our model reports large improvements on both T-REx and zsRE slot filling datasets, improving both passage retrieval and slot value generation, and ranking at the top-1 position in the KILT leaderboard. Moreover, we demonstrate the robustness of our system showing its domain adaptation capability on a new variant of the TACRED dataset for slot filling, through a combination of zero/few-shot learning. We release the source code and pre-trained models. Robust Retrieval Augmented Generation for Zero-shot Slot Filling Md Faisal Mahbub Chowdhury Deep Learning Images For Google Compute Engine, The Definitive Guide | by Viacheslav Kovalevskyi 2022-01-22 2022-01-22T12:59:33Z had been useful to me when creating VMs for deep learning some years ago cf. [Deep Learning Images For Google Compute Engine, The Definitive Guide | by Viacheslav Kovalevskyi](doc:2022/01/deep_learning_images_for_google) 2022-01-22T13:14:50Z 2022-01-22 Viacheslav Kovalevskyi – Medium 2021-08-02T17:14:01Z Colbert-QA 2020-07-01T23:50:58Z Christopher Potts 2007.00814 [2007.00814] Relevance-guided Supervision for OpenQA with ColBERT 2022-01-07 Matei Zaharia Omar Khattab Systems for Open-Domain Question Answering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions between questions and passages. We propose an efficient weak supervision strategy that iteratively uses ColBERT to create its own training data. This greatly improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA, and the resulting system attains state-of-the-art extractive OpenQA performance on all three datasets. Omar Khattab Relevance-guided Supervision for OpenQA with ColBERT 2022-01-07T18:39:10Z 2022-01-22 2022-01-22T13:13:50Z Conteneurs de deep learning  |  Google Cloud Colab+GCP Compute — how to link them together | by Gautham Senthilnathan | Medium 2022-01-14 2022-01-14T18:02:28Z de Duvivier avec Gérard Philippe, d'après Zola Pot-Bouille (film) — Wikipédia 2022-01-11T00:30:07Z 2022-01-11 Guess Who's Coming to Dinner - Wikipedia 2022-01-11T00:31:40Z 2022-01-11 avec Sydney Poitier, Kathrin Hepburn, Spencer Tracy 2022-01-27 2022-01-27T00:21:46Z Haystack Annotation Tool 2022-01-12T15:15:55Z Integrate ORQA and REALM for Open Domain Question Answering · Issue #312 · deepset-ai/haystack 2022-01-12 mntions [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)