Semanlink - Favoris ; Géographie AND NLP event

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 About

Journée TAL grand public – GDR TAL

Tags:

2021-01-08 About

BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Tags:

2020-03-28 About

Information Retrieval for HR (2018)

Tags:

2020-02-14 About

Siamese CNN for job–candidate matching (slides)

Tags:

2020-02-10 About

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 About

Language and Perception in Deep Learning - Florian Strub DeepMind, Univ. Lille, Inria

Tags:

2019-10-07 About

Paris NLP Season 4 Meetup #1 at Algolia

Tags:

2019-10-07 About

thunlp/OpenKE: An Open-Source Package for Knowledge Embedding (KE)

Tags:

2019-04-23 About

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction - ACL Anthology

Tags:

2019-02-09 About

Paris NLP Season 3 Meetup #3 at Doctrine – Paris NLP

Tags:

Paris NLP meetup

Trois présentations :

- une première décrivant l'utilisation de données textuelles dans le cadre de la conception d'une opération marketing (définition de l'orientation à donner à l'image d'une marque). Les techniques NLP utilisées sont simples, mais leur utilisation comme outil pour aider les créatifs du marketing est originale.

- une deuxième dans le domaine juridique, très intéressante en termes de techniques mises en oeuvre. L'application vise à la valorisation de bases de contrats (un corpus de textes, sur lequel on souhaite effectuer des recherches complexes), en s'appuyant à la fois sur les techniques récentes de représentation de textes, et sur un knowledge graph (ontologie de termes juridiques). Pour ce qui est de le représentation de textes, ils utilisent Google BERT. Ce que permet BERT, c'est une forme de transfert d'apprentissage : BERT est un réseau de neurones profond entrainé de façon non supervisée, par et chez Google, sur une énorme quantité de textes, de façon à emmagasiner une connaissance sur une langue ("pre-trained language model"). Ces données (c'est à dire ce réseau pré-entrainé) sont mises à disposition par Google. Chacun peut ainsi affiner l'entrainement du réseau sur son propre corpus de textes, et ses propres données labellisées concernant le problème qu'il souhaite effectivement résoudre (par exemple, dans le cas juridique, la reconnaissance d'entités dans les contrats).

Le speaker rapporte des résultats sensiblement améliorés par rapport à ce qu'ils obtenaient auparavant en ce qui concerne la qualité de la représentation de phrases, et des problèmes de type classification de phrases ou reconnaissance d'entités (il note que la représentation de textes longs reste un problème ouvert). Les temps d'entrainement de BERT sur leur problème ne sont pas exorbitants (il parle de quelques heures de GPU, pas de jours ou de semaines de TPU comme pour l'entrainement initial).

- le dernier speaker quant à lui a présenté deux papiers de recherche, justement sur les techniques au cœur de BERT ("Transformer architecture").

2019-01-29 About

Romain Vial (Hyperlex) at Paris NLP meetup, slides

Tags:

2019-01-24 About

10 Exciting Ideas of 2018 in NLP

Tags:

2018-12-19 About

Monolingual data in NMT

Tags:

2018-12-04 About

Paris NLP Season 3 Meetup #2 – Paris NLP

Tags:

2018-12-04 About

Paris NLP - blog

Tags:

Paris NLP meetup

2018-11-29 About

Highlights of EMNLP 2018 – Chris Zhu – Medium

Tags:

EMNLP 2018

2018-11-25 About

Interesting Stuff at EMNLP (part II) – Valentin Malykh – Medium

Tags:

EMNLP 2018

2018-11-25 About

Interesting Stuff in EMNLP (part I) – Valentin Malykh – Medium

Tags:

EMNLP 2018

2018-11-25 About

Google AI Blog: Google at EMNLP 2018

Tags:

2018-11-25 About

Associative Multichannel Autoencoder for Multimodal Word Representation (2018)

Tags:

2018-11-15 About

EMNLP 2018 Thoughts and Notes · Supernatural Language Processing

Tags:

EMNLP 2018

2018-11-13 About

Trying to Understand Recurrent Neural Networks for Language Processing (slides)

Tags:

2018-11-11 About

Word Mover's Embedding: From Word2Vec to Document Embedding (2018)

Tags:

2018-11-10 About

EMNLP 2018 Highlights: Inductive bias, cross-lingual learning, and more

Tags:

2018-11-08 About

Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization

Tags:

2018-11-06 About

Deep Chit-Chat: deep learning for chatbots (EMNLP 2018 Tutorial)

Tags:

by Dr Wei Wu (Microsoft Xiaolce - chatbot with 200 millions users in China) and Dr Rui Yan (Peking Univ)

- Chit-chat (casual, non goal oriented) open-domain. Must be relevant to the context and diverse (informative) to be engaging.
- why creating a chat? to prove an AI can speak like a human, commercial reasons, link to services.

Task oriented vs non task oriented: this tutorial is about the second one.

Retrieval based vs generation based.

Basic knowledge of DL for chatbots:

- word embeddings
- sentence embeddings (CNN, RNN)
- dialogue modeling: seq-to-seq with attention

Response selection for retrieval based chatbots:

- single turn response selection (slides 37-57)
    - framework 1: matching with seq embeddings
    - framework 2: matching with message-response interaction (46)
    - extension of 1: KnowledgeMatching with External Knowledge (53)
    - extension of 2: RepresentationsMatching with Multiple Levels of Representations (54)
    - insights from comparison between 1 and 2 (57)
- multi turn response selection (62)
    - context is now: mess + history
    - again, 2 frameworks

Emerging directions (79):

- matching with better representations
    - Self-Attention (82)
    - fusing multiple types of repr. But how to fuse matters (83)
    - pre-training

Learning a matching model for response selection (84)

Generation based models for chatbots:

- single turn generarion (89)
    - Basic generation model
        - seq2seq
        - Attention
        - Bi-directional modeling
- multi turn generation
    - Contexts are important
    - Context sensitive models
    - Hierarchical context modeling
    - Latent variable modeling
    - Hierarchical memory networks

Diversity in conversations (99)

Content introducing (106)

Additional elements (113)

- Topics in cnversation
- Emotions

Persona in chat:

- Persona
- ...
- Knowledge
- Common sense

RL and Adversarial learning in conversations (125)

Evaluation (132)

Future trends:

- Reasoning in dialogues
- X-grounded dialogues

2018-11-06 About

Joint Models in NLP - Slides - Tutorial (EMNLP 2018) - Yue Zhang

Tags:

**Joint models: solve 2 tasks at once.**

Related tasks: POS tagging, NER, chuncking. Pipeline tasks

Motivations:

- reduce error propagation
- information exchange between tasks

Challenges:

- Joint learning
- Search

History: statistical models. 2 kinds:

- Graph-Based Methods
    - Traditional solution:
        - Score each candidate, select the highest-scored output
        - Search-space typically exponential
- Transition-Based Methods
    - Transition-Based systems: Automata
        - State: partial result during decoding, Action: operations that can be applied for state transition
        - Output constructed incrementally

- Deep learning based model
    - Neural transition based models
    - Neural graph-based models
        - Cross task
            - Seminal work: Collobert, Ronan, et al. "Natural language processing (almost) from scratch."
            - Not all tasks are mutually beneficial
            - Ramachandran, et al.  “Unsupervised pretraining for sequence to sequence learning.”
            - Peters, Matthew E., et al. "Deep contextualized word representations." (ELMo)
            - "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding."
            - ULMFIT
            - Correlation between multi-task learning and pretraining
        - Cross lingual
        - Cross domain
        - Cross standard

2018-11-06 About

PROCEEDINGS of the BlackboxNLP Workshop

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Analyzing and interpreting neural networks for NLP (Workshop's Home page)

Tags:

BlackboxNLP (2018 workshop)

2018-11-06 About

Writing code for Natural language processing Research

Tags:

2018-11-05 About

Transfer learning with language models

Tags:

2018-11-05 About

Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction

Tags:

2018-11-04 About

Conference Schedule - EMNLP 2018

Tags:

EMNLP 2018

2018-11-04 About

Self-Governing Neural Networks for On-Device Short Text Classification - Sujith Ravi | Zornitsa Kozareva (2018)

Tags:

2018-11-02 About

EMNLP (2018) - ACL Anthology - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Tags:

EMNLP 2018

2018-11-02 About

Deep Latent-Variable Models for Natural Language - Tutorial - harvardnlp

Tags:

2018-11-01 About

Facebook Research at EMNLP – Facebook Research

Tags:

2018-11-01 About

Trying to Understand Recurrent Neural Networks for Language Processing (tweets)

Tags:

2018-11-01 About

Writing Code for NLP Research, AllenNLP's tutorial at #emnlp2018

Tags:

2018-10-31 About

Tutorials - EMNLP 2018

Tags:

2018-10-31 About

Paris NLP Season 3 Meetup #1 | Meetup

Tags:

2018-09-27 About

[1809.00782] Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text

Tags:

2018-09-06 About

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 About

2018 Conference on Empirical Methods in Natural Language Processing - EMNLP 2018

Tags: