Semanlink - AI Event AND NLP

> The most
successful previous approaches like InferSent (Conneau
et al., 2017), Universial Sentence Encoder
(USE) (Cer et al., 2018) and SBERT (Reimers and
Gurevych, 2019) heavily relied on labeled data to
train sentence embedding models.
>
> TSDAE can
achieve up to 93.1% of the performance of indomain
supervised approaches. Further, we
show that TSDAE is **a strong domain adaptation
and pre-training method for sentence
embeddings**, significantly outperforming other
approaches like Masked Language Model.

> During training, TSDAE
encodes corrupted sentences into fixed-sized
vectors and requires the decoder to reconstruct the
original sentences from this sentence embedding.

- <https://www.sbert.net/examples/unsupervised_learning/TSDAE/README.html>
- [github](https://github.com/UKPLab/sentence-transformers/tree/master/examples/unsupervised_learning/TSDAE)
- [UKPLab/sentence-transformers: Sentence Embeddings with BERT & XLNet](doc:2020/07/ukplab_sentence_transformers_s)
- [twitter](https://twitter.com/KexinWang2049/status/1433361957579538432):

> **TSDAE can learn domain-specific sentence embeddings with unlabeled sentences**
>
> Most importantly, instead of STS (Semantic Textual Similarity), **we suggest evaluating unsupervised sentence embeddings on the domain-specific tasks&datasets, which is the real use case for them**. Actually, STS scores do not correlate with performance on specific tasks.

2021-09-01 About

[2010.02353] Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages

Tags:

2021-08-25 About

Knowledge Graphs in Natural Language Processing @ ACL 2021 | by Michael Galkin | Aug, 2021

Tags:

2021-08-06 About

[1807.04905] Ultra-Fine Entity Typing

Tags:

2021-06-22 About

[2106.04098] Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model

Tags:

2021-06-16 About

[2004.04906] Dense Passage Retrieval for Open-Domain Question Answering

Tags:

2021-06-03 About

Simple Unsupervised Keyphrase Extraction using Sentence Embeddings - ACL Anthology (2018)

Tags:

2021-05-31 About

[2009.12030] AutoETER: Automated Entity Type Representation for Knowledge Graph Embedding

Tags:

2021-05-17 About

[2011.05864] On the Sentence Embeddings from Pre-trained Language Models

Tags:

2021-04-19 About

exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources - ACL Anthology

Tags:

**Focus on the Embedding of Domain-specific Vocabulary.**

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 About

Journée TAL grand public – GDR TAL

Tags:

2021-01-08 About

Event Extraction by Answering (Almost) Natural Questions

Tags:

2020-12-17 About

TextGraphs 2020

Tags:

2020-12-13 About

Knowledge Base Embedding By Cooperative Knowledge Distillation - ACL Anthology

Tags:

KD-MKB paper

2020-12-05 About

[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Tags:

2020-11-26 About

Knowledge Graphs in NLP @ EMNLP 2020 | by Michael Galkin | Nov, 2020 | Medium

Tags:

2020-11-24 About

Text-Enhanced Representation Learning for Knowledge Graph (IJCAI 2016)

Tags:

2020-11-03 About

Towards Unsupervised Text Classification Leveraging Experts and Word Embeddings - (ACL 2019)

Tags:

2020-10-05 About

Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training - ACL Anthology (2019)

Tags:

2020-09-06 About

Knowledge Graphs in Natural Language Processing @ ACL 2020 | by Michael Galkin

Tags:

2020-07-12 About

Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (ACL Anthology 2020)

Tags:

Aim to leverage both contextual representation of input text (deep LMs)  and  knowledge derived
from curated KBs ([Wordnet](tag:wordnet)) to improve [slot tagging](tag:slot_tagging) in the presence of [out-of-vocab](tag:oov) words ([few-shot scenario](tag:few_shot_learning))

Method:

1. retrieve potentially relevant KB entities and
encode them into distributed representations that
describe global graph-structured information
2. BERT encoder
layer to capture context-aware representations of
the sequence and attend to the KB embeddings
using multi-level graph attention
3. integrate
BERT embeddings and the KB embeddings
to predict the slot type

Contributions:

1. feasibility of applying lexical ontology
to facilitate recognizing OOV words. First to consider the large-scale background
knowledge for enhancing context-aware
slot tagging models.
2. a knowledge integration mechanism that uses multi-level graph
attention to model explicit lexical relations.
3.experiments on two benchmark datasets

> our method makes a notable difference in a
scenario where samples are linguistically diverse,
and large vocab exists.

(Better improvements when using RNN than BERT, because BERT already contains a lot of background knowledge)

2020-07-04 About

Information Bottleneck for NLP (parsing & summarization)

Tags:

2020-06-29 About

[1910.00163] Specializing Word Embeddings (for Parsing) by Information Bottleneck

Tags:

2020-06-29 About

Representation Learning for Information Extraction from Form-like Documents – Google Research

Tags:

2020-06-15 About

[1909.04164] Knowledge Enhanced Contextual Word Representations

Tags:

2020-05-13 About

[1906.01195] Learning Attention-based Embeddings for Relation Prediction in Knowledge Graphs

Tags:

2020-04-30 About

BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Tags:

2020-03-28 About

Information Retrieval for HR (2018)

Tags:

2020-02-14 About

Siamese CNN for job–candidate matching (slides)

Tags:

2020-02-10 About

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 About

Advancing Natural Language Processing (NLP) for Enterprise Domains

Tags:

2020-01-07 About

Highlights from CoNLL and EMNLP 2019

Tags:

EMNLP 2019

2019-12-07 About

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (ACL 2019)

Tags:

2019-11-17 About

Graph Neural Networks for Natural Language Processing tutorial at EMNLP 2019

Tags:

2019-11-08 About

Language and Perception in Deep Learning - Florian Strub DeepMind, Univ. Lille, Inria

Tags:

2019-10-07 About

Paris NLP Season 4 Meetup #1 at Algolia

Tags:

2019-10-07 About

Evolution of Representations in the Transformer (2019)

Tags:

2019-09-16 About

[1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Tags:

2019-08-28 About

ACL 2019: Highlights and Trends - Maria Khvalchik - Medium

Tags:

ACL 2019

2019-08-13 About

4th Workshop on Representation Learning for NLP

Tags:

2019-08-12 About

Trends in Natural Language Processing: ACL 2019 In Review - Mihail Eric

Tags:

ACL 2019

2019-08-05 About

[1905.07129] ERNIE: Enhanced Language Representation with Informative Entities

Tags:

2019-08-05 About

Knowledge graphs in Natural Language Processing @ ACL 2019 - Michael Galkin

Tags:

2019-08-05 About

A2N: Attending to Neighbors for Knowledge Graph Inference - ACL 2019

Tags:

2019-07-31 About

NAACL 2019 Highlights

Tags:

2019-07-30 About

Nominations for ACL 2019 Best Paper Awards - ACL 2019

Tags:

ACL 2019

2019-07-25 About

Papers - ACL 2019

Tags:

NLP conference

2019-06-12 About

thunlp/OpenKE: An Open-Source Package for Knowledge Embedding (KE)

Tags:

2019-04-23 About

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction - ACL Anthology

Tags:

2019-02-09 About

Paris NLP Season 3 Meetup #3 at Doctrine – Paris NLP

Tags:

Paris NLP meetup

Trois présentations :

- une première décrivant l'utilisation de données textuelles dans le cadre de la conception d'une opération marketing (définition de l'orientation à donner à l'image d'une marque). Les techniques NLP utilisées sont simples, mais leur utilisation comme outil pour aider les créatifs du marketing est originale.

- une deuxième dans le domaine juridique, très intéressante en termes de techniques mises en oeuvre. L'application vise à la valorisation de bases de contrats (un corpus de textes, sur lequel on souhaite effectuer des recherches complexes), en s'appuyant à la fois sur les techniques récentes de représentation de textes, et sur un knowledge graph (ontologie de termes juridiques). Pour ce qui est de le représentation de textes, ils utilisent Google BERT. Ce que permet BERT, c'est une forme de transfert d'apprentissage : BERT est un réseau de neurones profond entrainé de façon non supervisée, par et chez Google, sur une énorme quantité de textes, de façon à emmagasiner une connaissance sur une langue ("pre-trained language model"). Ces données (c'est à dire ce réseau pré-entrainé) sont mises à disposition par Google. Chacun peut ainsi affiner l'entrainement du réseau sur son propre corpus de textes, et ses propres données labellisées concernant le problème qu'il souhaite effectivement résoudre (par exemple, dans le cas juridique, la reconnaissance d'entités dans les contrats).

Le speaker rapporte des résultats sensiblement améliorés par rapport à ce qu'ils obtenaient auparavant en ce qui concerne la qualité de la représentation de phrases, et des problèmes de type classification de phrases ou reconnaissance d'entités (il note que la représentation de textes longs reste un problème ouvert). Les temps d'entrainement de BERT sur leur problème ne sont pas exorbitants (il parle de quelques heures de GPU, pas de jours ou de semaines de TPU comme pour l'entrainement initial).

- le dernier speaker quant à lui a présenté deux papiers de recherche, justement sur les techniques au cœur de BERT ("Transformer architecture").

2019-01-29 About

Romain Vial (Hyperlex) at Paris NLP meetup, slides

Tags:

2019-01-24 About

[1901.02860] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Tags:

2019-01-11 About

10 Exciting Ideas of 2018 in NLP

Tags:

2018-12-19 About

Monolingual data in NMT

Tags:

2018-12-04 About

Paris NLP Season 3 Meetup #2 – Paris NLP

Tags:

2018-12-04 About

Paris NLP - blog

Tags:

Paris NLP meetup

2018-11-29 About

Highlights of EMNLP 2018 – Chris Zhu – Medium

Tags:

EMNLP 2018

2018-11-25 About

Interesting Stuff at EMNLP (part II) – Valentin Malykh – Medium

Tags:

EMNLP 2018

2018-11-25 About

Interesting Stuff in EMNLP (part I) – Valentin Malykh – Medium

Tags:

EMNLP 2018

2018-11-25 About

Google AI Blog: Google at EMNLP 2018

Tags:

2018-11-25 About

Associative Multichannel Autoencoder for Multimodal Word Representation (2018)

Tags:

2018-11-15 About

EMNLP 2018 Thoughts and Notes · Supernatural Language Processing

Tags:

EMNLP 2018

2018-11-13 About

Trying to Understand Recurrent Neural Networks for Language Processing (slides)

Tags:

2018-11-11 About

Word Mover's Embedding: From Word2Vec to Document Embedding (2018)

Tags:

2018-11-10 About

EMNLP 2018 Highlights: Inductive bias, cross-lingual learning, and more

Tags:

2018-11-08 About

Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization

Tags:

2018-11-06 About

Deep Chit-Chat: deep learning for chatbots (EMNLP 2018 Tutorial)

Tags:

by Dr Wei Wu (Microsoft Xiaolce - chatbot with 200 millions users in China) and Dr Rui Yan (Peking Univ)

- Chit-chat (casual, non goal oriented) open-domain. Must be relevant to the context and diverse (informative) to be engaging.
- why creating a chat? to prove an AI can speak like a human, commercial reasons, link to services.

Task oriented vs non task oriented: this tutorial is about the second one.

Retrieval based vs generation based.

Basic knowledge of DL for chatbots:

- word embeddings
- sentence embeddings (CNN, RNN)
- dialogue modeling: seq-to-seq with attention

Response selection for retrieval based chatbots:

- single turn response selection (slides 37-57)
    - framework 1: matching with seq embeddings
    - framework 2: matching with message-response interaction (46)
    - extension of 1: KnowledgeMatching with External Knowledge (53)
    - extension of 2: RepresentationsMatching with Multiple Levels of Representations (54)
    - insights from comparison between 1 and 2 (57)
- multi turn response selection (62)
    - context is now: mess + history
    - again, 2 frameworks

Emerging directions (79):

- matching with better representations
    - Self-Attention (82)
    - fusing multiple types of repr. But how to fuse matters (83)
    - pre-training

Learning a matching model for response selection (84)

Generation based models for chatbots:

- single turn generarion (89)
    - Basic generation model
        - seq2seq
        - Attention
        - Bi-directional modeling
- multi turn generation
    - Contexts are important
    - Context sensitive models
    - Hierarchical context modeling
    - Latent variable modeling
    - Hierarchical memory networks

Diversity in conversations (99)

Content introducing (106)

Additional elements (113)

- Topics in cnversation
- Emotions

Persona in chat:

- Persona
- ...
- Knowledge
- Common sense

RL and Adversarial learning in conversations (125)

Evaluation (132)

Future trends:

- Reasoning in dialogues
- X-grounded dialogues

2018-11-06 About

Joint Models in NLP - Slides - Tutorial (EMNLP 2018) - Yue Zhang

Tags:

**Joint models: solve 2 tasks at once.**

Related tasks: POS tagging, NER, chuncking. Pipeline tasks

Motivations:

- reduce error propagation
- information exchange between tasks

Challenges:

- Joint learning
- Search

History: statistical models. 2 kinds:

- Graph-Based Methods
    - Traditional solution:
        - Score each candidate, select the highest-scored output
        - Search-space typically exponential
- Transition-Based Methods
    - Transition-Based systems: Automata
        - State: partial result during decoding, Action: operations that can be applied for state transition
        - Output constructed incrementally

- Deep learning based model
    - Neural transition based models
    - Neural graph-based models
        - Cross task
            - Seminal work: Collobert, Ronan, et al. "Natural language processing (almost) from scratch."
            - Not all tasks are mutually beneficial
            - Ramachandran, et al.  “Unsupervised pretraining for sequence to sequence learning.”
            - Peters, Matthew E., et al. "Deep contextualized word representations." (ELMo)
            - "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding."
            - ULMFIT
            - Correlation between multi-task learning and pretraining
        - Cross lingual
        - Cross domain
        - Cross standard

2018-11-06 About

PROCEEDINGS of the BlackboxNLP Workshop

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Analyzing and interpreting neural networks for NLP (Workshop's Home page)

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Writing code for Natural language processing Research

Tags:

2018-11-05 About

Transfer learning with language models

Tags:

2018-11-05 About

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Tags:

2018-11-04 About

Conference Schedule - EMNLP 2018

Tags:

EMNLP 2018

2018-11-04 About

Self-Governing Neural Networks for On-Device Short Text Classification - Sujith Ravi | Zornitsa Kozareva (2018)

Tags:

2018-11-02 About

EMNLP (2018) - ACL Anthology - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Tags:

EMNLP 2018

2018-11-02 About

Deep Latent-Variable Models for Natural Language - Tutorial - harvardnlp

Tags:

2018-11-01 About

Facebook Research at EMNLP – Facebook Research

Tags:

2018-11-01 About

Trying to Understand Recurrent Neural Networks for Language Processing (tweets)

Tags:

2018-11-01 About

Writing Code for NLP Research, AllenNLP's tutorial at #emnlp2018

Tags:

2018-10-31 About

Tutorials - EMNLP 2018

Tags:

2018-10-31 About

Teaching Machines to Understand Natural Language (2018)

Tags:

2018-10-26 About

Small Data is Big in AI : Train-spotting at France is AI

Tags:

2018-10-26 About

Grounded Language Learning: Where Robotics and NLP Meet (IJCAI 2018)

Tags:

2018-10-26 About

[1503.08895] End-To-End Memory Networks

Tags:

2018-10-23 About

Paris NLP Season 3 Meetup #1 | Meetup

Tags:

2018-09-27 About

[1809.00782] Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text

Tags:

2018-09-06 About

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 About

2018 Conference on Empirical Methods in Natural Language Processing - EMNLP 2018

Tags: