Semanlink - Artificial Intelligence AND J'y étais

- Chit-chat (casual, non goal oriented) open-domain. Must be relevant to the context and diverse (informative) to be engaging.
- why creating a chat? to prove an AI can speak like a human, commercial reasons, link to services.

Task oriented vs non task oriented: this tutorial is about the second one.

Retrieval based vs generation based.

Basic knowledge of DL for chatbots:

- word embeddings
- sentence embeddings (CNN, RNN)
- dialogue modeling: seq-to-seq with attention

Response selection for retrieval based chatbots:

- single turn response selection (slides 37-57)
    - framework 1: matching with seq embeddings
    - framework 2: matching with message-response interaction (46)
    - extension of 1: KnowledgeMatching with External Knowledge (53)
    - extension of 2: RepresentationsMatching with Multiple Levels of Representations (54)
    - insights from comparison between 1 and 2 (57)
- multi turn response selection (62)
    - context is now: mess + history
    - again, 2 frameworks

Emerging directions (79):

- matching with better representations
    - Self-Attention (82)
    - fusing multiple types of repr. But how to fuse matters (83)
    - pre-training

Learning a matching model for response selection (84)

Generation based models for chatbots:

- single turn generarion (89)
    - Basic generation model
        - seq2seq
        - Attention
        - Bi-directional modeling
- multi turn generation
    - Contexts are important
    - Context sensitive models
    - Hierarchical context modeling
    - Latent variable modeling
    - Hierarchical memory networks

Diversity in conversations (99)

Content introducing (106)

Additional elements (113)

- Topics in cnversation
- Emotions

Persona in chat:

- Persona
- ...
- Knowledge
- Common sense

RL and Adversarial learning in conversations (125)

Evaluation (132)

Future trends:

- Reasoning in dialogues
- X-grounded dialogues

2018-11-06 About

Joint Models in NLP - Slides - Tutorial (EMNLP 2018) - Yue Zhang

Tags:

**Joint models: solve 2 tasks at once.**

Related tasks: POS tagging, NER, chuncking. Pipeline tasks

Motivations:

- reduce error propagation
- information exchange between tasks

Challenges:

- Joint learning
- Search

History: statistical models. 2 kinds:

- Graph-Based Methods
    - Traditional solution:
        - Score each candidate, select the highest-scored output
        - Search-space typically exponential
- Transition-Based Methods
    - Transition-Based systems: Automata
        - State: partial result during decoding, Action: operations that can be applied for state transition
        - Output constructed incrementally

- Deep learning based model
    - Neural transition based models
    - Neural graph-based models
        - Cross task
            - Seminal work: Collobert, Ronan, et al. "Natural language processing (almost) from scratch."
            - Not all tasks are mutually beneficial
            - Ramachandran, et al.  “Unsupervised pretraining for sequence to sequence learning.”
            - Peters, Matthew E., et al. "Deep contextualized word representations." (ELMo)
            - "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding."
            - ULMFIT
            - Correlation between multi-task learning and pretraining
        - Cross lingual
        - Cross domain
        - Cross standard

2018-11-06 About

PROCEEDINGS of the BlackboxNLP Workshop

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Analyzing and interpreting neural networks for NLP (Workshop's Home page)

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Writing code for Natural language processing Research

Tags:

2018-11-05 About

Transfer learning with language models

Tags:

2018-11-05 About

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Tags:

2018-11-04 About

Conference Schedule - EMNLP 2018

Tags:

EMNLP 2018

2018-11-04 About

Self-Governing Neural Networks for On-Device Short Text Classification - Sujith Ravi | Zornitsa Kozareva (2018)

Tags:

2018-11-02 About

EMNLP (2018) - ACL Anthology - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Tags:

EMNLP 2018

2018-11-02 About

Deep Latent-Variable Models for Natural Language - Tutorial - harvardnlp

Tags:

2018-11-01 About

Facebook Research at EMNLP – Facebook Research

Tags:

2018-11-01 About

Trying to Understand Recurrent Neural Networks for Language Processing (tweets)

Tags:

2018-11-01 About

Writing Code for NLP Research, AllenNLP's tutorial at #emnlp2018

Tags:

2018-10-31 About

Tutorials - EMNLP 2018

Tags:

2018-10-31 About

[1809.00782] Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text

Tags:

2018-09-06 About

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 About

2018 Conference on Empirical Methods in Natural Language Processing - EMNLP 2018

Tags:

EMNLP 2018

2018-08-23 About

Module google/universal-sentence-encoder | TensorFlow

Tags:

2018-05-23 About

Improving Word Embedding Compositionality using Lexicographic Definitions

Tags:

2018-05-10 About

That Makes Sense: Joint Sense Retrofitting from Contextual and Ontological Information

Tags:

post-processing method for generating low-dimensional sense embedding. Emploies the ontological and contextual information simultaneously.

(Poster at the Web Conf) [Github](https://github.com/y95847frank/Joint-Retrofitting)

Calcule des "sense embeddings", en partant de word embeddings pré-calculés (par ex avec word2vec), et de données de type lexicographiques (ex wordnet), en contraignant, pour un sens, la distance entre sense et word embedding.

Abstract:

> While recent word embedding models demonstrate their abilities to capture syntactic and semantic information, the demand for sense level embedding is getting higher. In this study, we propose a novel joint sense embedding learning model that retrofits the word representation into sense representation from contextual and ontological information. The experiments show the effectiveness and robustness of our model that outperforms previous approaches in four public available benchmark datasets.

> Given a trained word embedding and a lexical ontology that contains sense level relationships (e.g., synonym, hypernym, etc.), our model generates new sense vectors via constraining the distance between the sense vector and its word form vector, its sense neighbors and its contextual neighbors

[Influenced by](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1411.4166) (which post-processes and modifies word vectors to incorporate knowledge from semantic lexicons, while this creates new sense vectors)

2018-05-10 About

Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning

Tags:

2018-05-10 About

TUTORIAL: Graph-based Text Representations (SLIDES)

Tags:

2018-05-10 About

TUTORIAL: Graph-based Text Representations: Boosting Text Mining, NLP and Information Retrieval with Graphs

Tags:

2018-05-10 About

TUTORIAL: Representation Learning on Networks - TheWebConf 2018

Tags:

Network representation learning (NRL): Approaches that automatically learn to encode network structure into low-dimensional embeddings, using techniques based on deep learning and nonlinear dimensionality reduction

**Goal of representation learning for networks: efficient task-independant feature learning for ML in networks.** But it's hard. DL toolbox are designed for single sequences or grids (for instance CNN for images, RNN or word2vec are fixed size), but networks are far more complex!

from the abstract:

> In this tutorial, we will cover key advancements in NRL over the last decade, with an emphasis on fundamental advancements made in the last two years. We will discuss classic matrix factorization-based methods (e.g., Laplacian eigenmaps), random-walk based algorithms (e.g., DeepWalk and node2vec), as well as very recent advancements in graph convolutional networks (GCNs). We will cover methods to embed individual nodes (see [node embeddings](/tag/node_embeddings)) as well as approaches to embed entire (sub)graphs, and in doing so, we will present a unified framework for NRL.

2018-05-05 About

Smart-MD: Neural Paragraph Retrieval of Medical Topics

Tags:

2018-04-28 About

L’inventeur du Web exhorte à réguler l’intelligence artificielle

Tags:

2018-04-28 About

Improving the Compositionality of Word Embeddings (2017)

Tags:

(MS thesis, a [paper at TheWebConf 2018](/doc/?uri=https%3A%2F%2Fdoi.org%2F10.1145%2F3178876.3186007))

> This thesis explores a method to find better encodings of meaning a computer can work with. We specifically want to combine encodings of word meanings in such a way that a good encoding of their joint meaning is created. The act of combining multiple representations of meaning into a new representation of meaning is called semantic composition.

Analysis of four word embeddings (Word2Vec, GloVe, fastText and Paragram)  in terms of their semantic compositionality. A method to tune these embeddings towards better compositionality, using a simple neural network architecture with definitions and lemmas from WordNet.

> Since dictionary definitions are semantically similar to their associated lemmas, they are the ideal candidate for our tuning method, as well as evaluating for compositionality. Our architecture allows for the embeddings to be composed using simple arithmetic operations, which makes these embeddings specifically suitable for production applications such as web search and data mining. We also explore more elaborate and involved compositional models, such as recurrent composition and convolutional composition.

2018-02-13 About

RESEARCH TRACK: Web Content Analysis, Semantics and Knowledge

Tags:

[CFP](https://www2018.thewebconf.org/call-for-papers/research-tracks-cfp/web-content-analysis/)

> In previous years, ‘content analysis’ and ‘semantic and knowledge’ were in separate track. This year, we combined these tracks to emphasize the close relationship between these topics; **the use of content to curate knowledge and the use of knowledge to guide content analysis and intelligent usage**.

Some of the accepted papers:
### [Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN](https://doi.org/10.1145/3178876.3186005)

[Hierarchical Text Classification](/tag/nlp_hierarchical_text_classification): Text classification to a hierarchical taxonomy of topics, using graph representation of text, and CNN over this graph

Renvoie à ce qui a été vu dans le tutorial "Graph-based Text Representations"

from the abstract:

> a graph-CNN based deep learning model to first convert texts to graph-of-words, and then use graph convolution operations to convolve the word graph. Graph-of-words representation of texts has the advantage of capturing non-consecutive and long-distance semantics. CNN models have the advantage of learning different level of semantics. To further leverage the hierarchy of labels, we regularize the deep architecture with the dependency among labels

Conversion of text to graph: potentially given a single document

### [Weakly-supervised Relation Extraction by Pattern-enhanced Embedding Learning](https://doi.org/10.1145/3178876.3186024 )

Extraction de relations de corpus de textes de façon semi-supervisée, dans un contexte où on a peu de données labellisées décrivant les relations.

Par exemple, des données labellisées indique que le texte "Beijing, capital of China" correspond à la relation entre entités : ("Beijing", "Capital Of", "China), et on voudrait pouvoir extraire les entités et relations pertinentes à partir de texte tel que "Paris, France's capital,..."

Le papier décrit une méthode qui combine deux modules, l'un basé sur l'extraction automatique de patterns (par ex "[Head], Capital Of [Tail]") et l'autre sur la "sémantique distributionnelle" (du type "word embeddings"). Ces deux modules collaborent, le premier permettant de créer des instances de relations augmentant la base de connaissance sur lequel entrainer le second, et le second aidant le premier à déterminer des patterns informatifs ("co-entrainement")

### [Scalable Instance Reconstruction in Knowledge Bases via Relatedness Affiliated Embedding](https://doi.org/10.1145/3178876.3186017)

Knowledge base completion problem: usually, it is formulated as a link prediction problem, but not here. A novel knowledge embedding model ("Joint Modelling and Learning of Relatedness and Embedding")

### [Improving Word Embedding Compositionality using Lexicographic Definitions](https://doi.org/10.1145/3178876.3186007)

comment obtenir les meilleures représentations de texte à partir de représentations de mots (word embeddings) ? L'auteur utilise des ressources lexicographiques (wordnet) pour ses tests : l'embedding obtenu pour la définition d'un mot est-il proche de celui du mot ?

Le papier s'appuie sur une [thèse du même auteur](https://esc.fnwi.uva.nl/thesis/centraal/files/f1554608041.pdf), claire et bien écrite.

### [CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information](https://doi.org/10.1145/3178876.3186030)

Amélioration de l'extraction de triplets (nom phrase, property, nom phrase) à partir de texte en calculant des embeddings pour les "nom phrases" (~entités)

### [Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations](https://doi.org/10.1145/3178876.3186009)

Topic modeling for short texts, leveraging the word-context semantic correlations in the training

### [Towards Annotating Relational Data on the Web with Language Models](https://doi.org/10.1145/3178876.3186029)

### A paper by [David Blei](/tag/david_blei): (Dynamic Embeddings for Language Evolution)

2018-01-27 About

TUTORIAL: Representation Learning on Networks - TheWebConf 2018

Tags:

2018-01-27 About

WORKSHOP: BigNet @ WWW 2018 Workshop on Learning Representations for Big Networks

Tags:

2018-01-27 About

Topic modeling with network regularization

Tags:

2014-04-23 About

Using SKOS vocabularies for improving Web Search

Tags:

2013-05-30 About

Describing Customizable Products on the Web of Data, LDOW 2013

Tags:

2013-04-25 About

From Linked Data to Linked Entities: a Migration Path - Giovanni Bartolomeo, Stefano Salsano

Tags:

2012-04-20 About

LINDEN: Linking Named Entities with Knowledge Base via Semantic Knowledge

Tags:

2012-04-19 About

Factorizing Yago: scalable machine learning for the sw

Tags:

2012-04-18 About

A Spectrometry of Linked Data

Tags:

Entity mining is still a troublesome open problem. In past years many approaches allowed to automate the generation of equivalence links between references using schema matching or various heuristics based on the recognition of similar property values. In contrast, few of them considered the analysis of the network of equivalence links (“equivalence network”) as an indication of the likelihood and strength of the equivalence.

Could a URI reference (URIRef) be thought as exactly “attached” to its referent? Could it make sense to talk about entity “identifiers” or would it be better to talk about more ambiguous “references”, i.e., placeholders for any model that satisfies the formal semantics of the Semantic Web (Hayes)? Booth observes that the aforementioned question, which in the past has been often regarded as fundamental in the debate about identity on the Web, is relatively unimportant. As long as an entity, identified by whatsoever URIRef, is associated to at least one description containing machine understandable information, this information can be automatically processed and used by applications.

2012-04-16 About

NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud

Tags:

2012-04-16 About

Automated interlinking of speech radio archives

Tags:

2012-04-14 About

COMM: Core Ontology on Multimedia

Tags:

2008-04-23 About

Structured Objects in OWL: Representation and Reasoning. In Proc. of the 17th Int. World Wide Web Conference (WWW 2008), Beijing

Tags:

Very good presentation at WWW 2008. Nominated for the best paper award<br/>
Abstract: Applications of semantic technologies often require the representation of and reasoning with structured objects—that is, objects composed of parts connected in complex ways. Although OWL is a general and powerful language, its class descriptions and axioms cannot be used to describe arbitrarily connected structures. An OWL representation of structured objects can thus be underconstrained, which reduces the inferences that can be drawn and causes performance problems in reasoning. To address these problems, we extend OWL with description graphs, which allow for the description of structured objects in a simple and precise way. To represent conditional aspects of the domain, we also allow for SWRL-like rules over description graphs. Based on an observation about the nature of structured objects, we ensure decidability of our formalism. We also present a hypertableau-based decision procedure, which we implemented in the HermiT reasoner. To evaluate its performance, we have extracted description graphs from the GALEN and FMA ontologies, classified them successfully, and even detected a modeling error in GALEN.

2008-04-23 About