Semanlink - Favoris ; Technical girls and guys AND Multilingual NLP

Omar Khattab sur X : "ColBERT in 81 languages by generalizing from English training! ..."

Tags:

2024-02-28 About

Nils Reimers sur Twitter : "Cross-Lingual Text-Classification just from English Data"

Tags:

2023-07-20 About

Nils Reimers sur Twitter : "Building search products that support many languages was always a nightmare..."

Tags:

2023-02-16 About

The State of Multilingual AI

Tags:

2022-11-14 About

UKPLab/EasyNMT: Easy to use, state-of-the-art Neural Machine Translation for 100+ languages

Tags:

2022-10-14 About

[2110.08151] mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models

Tags:

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 About

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 About

[2004.09813] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Tags:

2022-03-18 About

Nils Reimers sur Twitter : EasyNMT Easy-to-use (3 lines of code), state-of-the-art neural machine translations

Tags:

2021-04-27 About

Why You Should Do NLP Beyond English

Tags:

2020-08-01 About

[2004.14958] A Call for More Rigor in Unsupervised Cross-lingual Learning

Tags:

2020-05-02 About

Unsupervised Learning with Text (AFIA 2019)

Tags:

2019-12-14 About

Efficient multi-lingual language model fine-tuning · fast.ai NLP

Tags:

2019-10-22 About

Sebastian Ruder sur Twitter : "In the second part of the NLP and speech processing session @DeepIndaba, @alienelf presents her journey and work on machine translation for African languages with @LauraMartinus #DLIndaba2019"

Tags:

2019-08-29 About

Monolingual data in NMT

Tags:

2018-12-04 About

Paris NLP Season 3 Meetup #2 – Paris NLP

Tags:

2018-12-04 About

Using Monolingual Data in Neural Machine Translation: a Systematic Study

Tags:

2018-11-28 About

Unsupervised Machine Translation. G. Lample (slides)

Tags:

2018-09-29 About

Paris NLP Season 3 Meetup #1 | Meetup

Tags:

2018-09-27 About

facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings

Tags:

2017-12-22 About

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 About

Attention and Memory in Deep Learning and NLP – WildML

Tags:

cf. visual attention

In standard [#seq2seq](/tag/sequence_to_sequence_learning) NMT, the decoder is supposed to generate a translation solely based on the last hidden state of the encoder - which therefore must capture everything from the source sentence (it must be a sentence embedding). Not good. Hence the attention mechanism.

> we allow the decoder to “attend” to different parts of the source sentence at each step of the output generation. Importantly, we let the model learn what to attend to based on the input sentence and what it has produced so far

> each decoder output word now depends on a weighted combination of all the input states, not just the last state.

Possible to interpret what the model is doing by looking at the Attention weight matrix

Cost: We need to calculate an attention value for each combination of input and output word (-> attention is a bit of a misnomer: we look at everything in details before deciding what to focus on)

> attention mechanism is simply giving the network access to its internal memory, which is the hidden state of the encoder

> Unlike typical memory, the memory access mechanism here is soft, which means that the network retrieves a weighted combination of all memory locations, not a value from a single discrete location

2016-01-03 About

Scientists See Advances in Deep Learning, a Part of Artificial Intelligence - NYTimes.com

Tags:

2012-11-30 About