]> Neuromorphic engineering aims to create computing hardware that mimics biological nervous systems. How it began. 2020-07-23 2020-07-23T00:08:46Z How we created neuromorphic engineering | Nature Electronics [1911.03903] A Re-evaluation of Knowledge Graph Completion Methods Soumya Sanyal Shikhar Vashishth 2020-07-08T19:32:34Z Knowledge Graph Completion (KGC) aims at automatically predicting missing links for large-scale knowledge graphs. A vast number of state-of-the-art KGC techniques have got published at top conferences in several research fields, including data mining, machine learning, and natural language processing. However, we notice that several recent papers report very high performance, which largely outperforms previous state-of-the-art methods. In this paper, we find that this can be attributed to the inappropriate evaluation protocol used by them and propose a simple evaluation protocol to address this problem. The proposed protocol is robust to handle bias in the model, which can substantially affect the final results. We conduct extensive experiments and report the performance of several existing methods using our protocol. The reproducible code has been made publicly available Partha Talukdar 2020-07-28 Zhiqing Sun A Re-evaluation of Knowledge Graph Completion Methods 2019-11-10T11:19:08Z 1911.03903 Zhiqing Sun Yiming Yang 2020-07-28T11:27:26Z 2020-07-04T11:34:35Z 2020-07-04 Aim to leverage both contextual representation of input text (deep LMs) and knowledge derived from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning)) Method: 1. retrieve potentially relevant KB entities and encode them into distributed representations that describe global graph-structured information 2. BERT encoder layer to capture context-aware representations of the sequence and attend to the KB embeddings using multi-level graph attention 3. integrate BERT embeddings and the KB embeddings to predict the slot type Contributions: 1. feasibility of applying lexical ontology to facilitate recognizing OOV words. First to consider the large-scale background knowledge for enhancing context-aware slot tagging models. 2. a knowledge integration mechanism that uses multi-level graph attention to model explicit lexical relations. 3.experiments on two benchmark datasets > our method makes a notable difference in a scenario where samples are linguistically diverse, and large vocab exists. (Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge) Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020) avec Charles Laughton Ruggles of Red Gap 2020-07-05 2020-07-05T23:45:26Z 2020-07-04 2020-07-04T13:32:55Z Journal des africanistes | Société des africanistes 2020-07-24 2020-07-24T23:43:00Z raphaelsty/mkb: Knowledge Base Embedding By Cooperative Knowledge Distillation Snorkel is a fundamentally new interface to ML without hand-labeled training data 2020-07-15T08:16:35Z 2020-07-15 The Top of My Todo List 2020-07-03 2020-07-03T00:49:53Z 2020-07-02T22:59:10Z 2020-07-02 VICE - How Police Secretly Took Over a Global Phone Network for Organised Crime Knowledge Graphs in Natural Language Processing @ ACL 2020 | by Michael Galkin 2020-07-12 2020-07-12T01:12:20Z 2020-07-23T00:50:05Z 2020-07-23 Le plan de la filière hydrogène pour accélérer le développement de la technologie [paper](doc:2019/08/_1908_10084_sentence_bert_sen) UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet 2020-07-14 2020-07-14T19:08:40Z links to [UKPLab/sentence-transformers](doc:2020/07/ukplab_sentence_transformers_s) [Another answer](https://github.com/huggingface/transformers/issues/2986) 2020-07-12T15:26:41Z How to use BERT for finding similar sentences or similar news? · Issue #876 · huggingface/transformers 2020-07-12 2020-07-02T15:45:01Z A recently approved Google patent sheds light on the search engine's process behind showing ranked lists of entities in the SERPs 2020-07-02 Ranked Entities in Search Results at Google Coronavirus : un rapport au vitriol des pompiers dénonce la gestion de la crise 2020-07-05 2020-07-05T18:58:56Z > La gestion de crise, c’est un métier, on ne la laisse pas aux directeurs administratifs et financiers 2020-07-28T16:12:03Z 2020-07-28 javascript - Partial matching a string against a regex - Stack Overflow This is a regex feature known as partial matching, it's available in several regex engines such as PCRE, Boost, Java but not in JavaScript. : [Regex - check if input still has chances to become matching](https://stackoverflow.com/questions/22483214/regex-check-if-input-still-has-chances-to-become-matching/22489941#22489941) (en java: regex utils hitEnd()) 2020-07-14T12:28:59Z 2020-07-14 Amnesty International dénonce l’espionnage d’un journaliste marocain par une technologie quasi indétectable Covid19 : pourquoi zéro mort au Vietnam ? 2020-07-15T23:32:29Z 2020-07-15 2020-07-02T15:12:42Z 2020-07-02 Au Sahel, des arbres et des bêches pour lutter contre l’avancée du désert 2020-07-03T00:59:55Z 2020-07-03 Abstract Wikipedia/July 2020 announcement Finding similar documents with transformers · Codegram 2020-07-10 2020-07-10T09:30:37Z Neat! Transformers as RNNs -- linearized attention helps produce transformer-like models with complexity linear (rather than quadratic) complexity in sequence length. Dr Simon Osindero sur Twitter : "Neat! Transformers as RNNs" 2020-07-02 2020-07-02T15:54:14Z 2020-07-27 2020-07-27T14:02:01Z Federico Errica 🇮🇹🇪🇺 sur Twitter : "Our “#Probabilistic #Learning on #Graphs via Contextual Architectures”..." > [M. Hirsch] a simplement confié son regret de ne pas avoir pu donner des masques aux soignants qui se déplaçaient en métro. 2020-07-07T09:22:34Z 2020-07-07 La « positive attitude » des directeurs d’hôpitaux face au coronavirus décontenance la commission d’enquête A collection of 300+ survey papers on NLP and ML 2020-07-18 2020-07-18T13:28:26Z 2020-06-30T19:46:10Z > Similarity search for Efficient Active Learning and Search (SEALS) In [Active Learning](tag:active_learning): instead of searching globally for the optimal examples to label, leverage the fact that data is often heavily skewed and expand the candidate pool with the nearest neighbors of the labeled set. > Our work attacks **both the labeling and computational costs of machine learning**...SEALS dramatically reduces the barrier to machine learning, enabling small teams or individuals to build accurate classifiers. **SEALS does, however, introduce another system component, a similarity search index, which adds some additional engineering complexity** to build, tune, and maintain. Fortunately, several highly optimized implementations like Annoy and [Faiss](doc:2020/06/facebookresearch_faiss_a_libra) work reasonably well out of the box. Roshan Sumbaly Cody Coleman 2007.00077 Similarity Search for Efficient Active Learning and Search of Rare Concepts 2020-07-02 Alexander C. Berg Cody Coleman Many active learning and search approaches are intractable for industrial settings with billions of unlabeled examples. Existing approaches, such as uncertainty sampling or information density, search globally for the optimal examples to label, scaling linearly or even quadratically with the unlabeled data. However, in practice, data is often heavily skewed; only a small fraction of collected data will be relevant for a given learning task. For example, when identifying rare classes, detecting malicious content, or debugging model performance, the ratio of positive to negative examples can be 1 to 1,000 or more. In this work, we exploit this skew in large training datasets to reduce the number of unlabeled examples considered in each selection round by only looking at the nearest neighbors to the labeled examples. Empirically, we observe that learned representations effectively cluster unseen concepts, making active learning very effective and substantially reducing the number of viable unlabeled examples. We evaluate several active learning and search techniques in this setting on three large-scale datasets: ImageNet, Goodreads spoiler detection, and OpenImages. For rare classes, active learning methods need as little as 0.31% of the labeled data to match the average precision of full supervision. By limiting active learning methods to only consider the immediate neighbors of the labeled data as candidates for labeling, we need only process as little as 1% of the unlabeled data while achieving similar reductions in labeling costs as the traditional global approach. This process of expanding the candidate pool with the nearest neighbors of the labeled set can be done efficiently and reduces the computational complexity of selection by orders of magnitude. Peter Bailis Edward Chou 2020-07-02T15:31:34Z 2020-06-30T19:46:10Z Matei Zaharia [2007.00077] Similarity Search for Efficient Active Learning and Search of Rare Concepts I. Zeki Yalniz Sean Culatana Pang Wei Koh Pang Wei Koh Thao Nguyen 2020-07-09T07:47:28Z Percy Liang [2007.04612] Concept Bottleneck Models Emma Pierson We seek to learn models that we can interact with using high-level concepts: if the model did not think there was a bone spur in the x-ray, would it still predict severe arthritis? State-of-the-art models today do not typically support the manipulation of concepts like "the existence of bone spurs", as they are trained end-to-end to go directly from raw input (e.g., pixels) to output (e.g., arthritis severity). We revisit the classic idea of first predicting concepts that are provided at training time, and then using these concepts to predict the label. By construction, we can intervene on these \emph{concept bottleneck models} by editing their predicted concept values and propagating these changes to the final prediction. On x-ray grading and bird identification, concept bottleneck models achieve competitive accuracy with standard end-to-end models, while enabling interpretation in terms of high-level clinical concepts ("bone spurs") or bird attributes ("wing color"). These models also allow for richer human-model interaction: accuracy improves significantly if we can correct model mistakes on concepts at test time. 2020-07-09T07:47:28Z Concept Bottleneck Models Yew Siang Tang Stephen Mussmann 2020-07-10T09:48:19Z > We seek to **learn models that we can interact with using high-level concepts**... > > We revisit the **classic idea of first predicting concepts that are provided at training time, and then using these concepts to predict the label**. By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction... These models allow for richer human-model interaction: accuracy improves significantly if we can correct model mistakes on concepts at test time. 2020-07-10 2007.04612 Been Kim 2020-07-06T14:51:33Z 2020-07-06 BERT Word Embeddings Tutorial · Chris McCormick 2020-07-31 L’aventure citoyenne des semences paysannes, « commun » nourricier 2020-07-31T15:50:11Z Sandstorm 2020-07-18 2020-07-18T13:31:12Z Self-host web-based productivity apps easily and securely. Sandstorm is an open source project built by a community of volunteers with the goal of making it really easy to run open source web applications 2020-07-03T17:44:02Z dicksontsai/stanford-nlp-local-extension: Chrome extension for sending content to localhost server running Stanford NLP tools. 2020-07-03 2020-07-07 2020-07-07T19:15:54Z awslabs/dgl-ke: package for learning large-scale knowledge graph embeddings. 2020-07-29T08:19:09Z Aran Komatsuzaki sur Twitter : "Big Bird: Transformers for Longer Sequences..." 2020-07-29 2020-07-14 Des militants catalans visés par un logiciel espion ultraperfectionné 2020-07-14T12:24:22Z Haitian Sun 2020-07-09T23:54:59Z > a neural language model that includes **an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge.**... **The model can be updated without re-training by manipulating its symbolic representations**. In particular this model allows us to add new facts and overwrite existing ones. > a **neural language model which learns to access information in a symbolic knowledge graph.** > This model builds on the recently-proposed [Entities as Experts](doc:2020/07/2004_07202_entities_as_expert) (EaE) language model (Févry et al., 2020), which extends the same transformer (Vaswani et al., 2017) architecture of BERT (Devlin et al., 2019) with an additional external memory for entities. > > After training EaE, the embedding associated with an entity will (ideally) capture information about the textual context in which that entity appears, and by inference, the entity’s semantic properties > > we include an additional memory called a fact memory, which encodes triples from a symbolic KB. > > This combination results in a neural language model which learns to access information in a the symbolic knowledge graph. TODO: - read again IBM's [Span Selection Pre-training for Question Answering](doc:2019/09/_1909_04120_span_selection_pre) ("an effort to avoid encoding general knowledge in the transformer network itself") - compare with [[1907.05242] Large Memory Layers with Product Keys](doc:2019/07/_1907_05242_large_memory_layer) - how does it relate with [[2002.08909] REALM: Retrieval-Augmented Language Model Pre-Training](doc:2020/12/2002_08909_realm_retrieval_a)? 2007.00849 Pat Verga Livio Baldini Soares 2020-07-02T03:05:41Z William W. Cohen Massive language models are the core of modern NLP modeling and have been shown to encode impressive amounts of commonsense and factual information. However, that knowledge exists only within the latent parameters of the model, inaccessible to inspection and interpretation, and even worse, factual information memorized from the training corpora is likely to become stale as the world changes. Knowledge stored as parameters will also inevitably exhibit all of the biases inherent in the source materials. To address these problems, we develop a neural language model that includes an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge. We show that this model dramatically improves performance on two knowledge-intensive question-answering tasks. More interestingly, the model can be updated without re-training by manipulating its symbolic representations. In particular this model allows us to add new facts and overwrite existing ones in ways that are not possible for earlier models. Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge [2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge 2020-07-02T03:05:41Z 2020-07-09 Tom Kwiatkowski Livio Baldini Soares 2020-07-11 Eunsol Choi Thibault Févry 2004.07202 [2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision Thibault Févry Entities as Experts: Sparse Memory Access with Entity Supervision Nicholas FitzGerald 2020-07-11T15:09:10Z 2020-04-15T17:00:05Z 2020-04-15T17:00:05Z We focus on the problem of capturing declarative knowledge in the learned parameters of a language model. We introduce a new model, Entities as Experts (EaE), that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EaE's entity representations are learned directly from text. These representations capture sufficient knowledge to answer TriviaQA questions such as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?". EaE outperforms a Transformer model with $30\times$ the parameters on this task. According to the Lama knowledge probes, EaE also contains more factual knowledge than a similar sized Bert. We show that associating parameters with specific entities means that EaE only needs to access a fraction of its parameters at inference time, and we show that the correct identification, and representation, of entities is essential to EaE's performance. We also argue that the discrete and independent entity representations in EaE make it more modular and interpretable than the Transformer architecture on which it is based. > We focus on the problem of **capturing declarative knowledge in the learned parameters of a language model**... > Entities as Experts (EaE) can access distinct memories of the entities mentioned in a piece of text; > To understand the motivation for distinct and independent entity representations: A traditional Transformer would need to build an internal representation of Charles Darwin from the words “Charles” and “Darwin”... Conversely, EAE can access a dedicated representation of “Charles Darwin”, which is a memory of all of the contexts in which this entity has previously been mentioned.... Having retrieved and re-integrated this memory it is much easier for EAE to relate the question to the answer > EaE's entity representations are learned directly from text. Correct identification, and representation, of entities is essential to EaE's performance Based on transformer architecture Extension: [Facts as Experts](doc:2020/07/2007_00849_facts_as_experts_) Pat Verga > We consider the task of answering complex multi-hop questions **using a corpus as a virtual knowledge base** (KB). In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus. At each step the module uses a combination of sparse-matrix TFIDF indices and a maximum inner product search (MIPS) on a **special index of contextual representations of the mentions**. This module is **differentiable**, so the full system can be trained end-to-end using gradient based methods, starting from natural language inputs. We also describe a pretraining scheme for the contextual representation encoder by generating hard negative examples using existing knowledge bases. [(Bhuwan Dhingra PhD Thesis)](doc:2020/07/end_to_end_learning_with_text_) 2020-02-25T03:13:32Z Ruslan Salakhutdinov Differentiable Reasoning over a Virtual Knowledge Base Bhuwan Dhingra 2020-02-25T03:13:32Z Manzil Zaheer Bhuwan Dhingra We consider the task of answering complex multi-hop questions using a corpus as a virtual knowledge base (KB). In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus. At each step the module uses a combination of sparse-matrix TFIDF indices and a maximum inner product search (MIPS) on a special index of contextual representations of the mentions. This module is differentiable, so the full system can be trained end-to-end using gradient based methods, starting from natural language inputs. We also describe a pretraining scheme for the contextual representation encoder by generating hard negative examples using existing knowledge bases. We show that DrKIT improves accuracy by 9 points on 3-hop questions in the MetaQA dataset, cutting the gap between text-based and KB-based state-of-the-art by 70%. On HotpotQA, DrKIT leads to a 10% improvement over a BERT-based re-ranking approach to retrieving the relevant passages required to answer a question. DrKIT is also very efficient, processing 10-100x more queries per second than existing multi-hop systems. William W. Cohen 2020-07-11 [2002.10640] Differentiable Reasoning over a Virtual Knowledge Base 2002.10640 2020-07-06T17:41:29Z 2020-07-06 End-to-End Learning with Text & Knowledge Bases (Bhuwan Dhingra PhD Thesis) > Th‘is thesis develops methods which leverage the strength of both neural and symbolic approaches. Specifically, we **augment raw text with symbolic structure about entities and their relations from a knowledge graph**, and learn task-speci€c neural embeddings of the combined data structure. We also develop algorithms for doing **multi-step reasoning over the embeddings in a di‚fferentiable manner**, leading to **end-to-end models for answering complex queries**. Along the way we develop variants of recurrent and graph neural networks suited to modeling textual and multi-relational data, respectively, and use transfer learning to improve generalization. Related paper by same author: [[2002.10640] Differentiable Reasoning over a Virtual Knowledge Base](doc:2020/07/2002_10640_differentiable_rea) Vidhisha Balachandran Graham Neubig 2020-07-11T14:03:19Z