Semanlink - Favoris ; Artificial Intelligence AND Multilingual NLP

[Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."](doc:2022/04/ikuya_yamada_sur_twitter_is_)

> Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, **existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks**. In this study, we explore the **effectiveness of leveraging entity representations for downstream cross-lingual tasks**.
>
> the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features.

[Github](https://github.com/studio-ousia/luke)

> Entity representations are known to enhance
language models in mono-lingual settings
(Zhang et al., 2019: [ERNIE](tag:ernie.html); Peters et al., 2019:  [[1909.04164] Knowledge Enhanced Contextual Word Representations](doc:2020/05/1909_04164_knowledge_enhanced); Wang et al.,
2021 [[1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](doc:2020/11/1911_06136_kepler_a_unified_); Xiong et al., 2020; Yamada et al., 2020: [[2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](doc:2020/11/2010_01057_luke_deep_context))
presumably by introducing real-world knowledge.
We show that using entity representations facilitates
cross-lingual transfer by providing languageindependent
features.
>
> Multilingual extension of LUKE. The model is trained with the multilingual
masked language modeling (MLM) task as well
as the masked entity prediction (MEP) task with
Wikipedia entity embeddings

> We investigate two ways of using the entity representations
in cross-lingual transfer tasks:
> 1. perform
entity linking for the input text, and append
the detected entity tokens to the input sequence.
The entity tokens are expected to provide language independent
features to the model
> 2. use the entity
[MASK] token from the MEP task as a languageindependent
feature extractor.

2022-04-17 A propos

Ikuya Yamada sur Twitter : "Is entity representation effective to improve multilingual language models?..."

Tags:

2022-04-13 A propos

[2004.09813] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Tags:

2022-03-18 A propos

Multilingual Sentence Transformers | Pinecone

Tags:

2021-11-04 A propos

VaLaR NMT: Vastly Lacking Resources Neural Machine Translation (2019)

Tags:

2021-10-14 A propos

NMT Training through the Lens of SMT

Tags:

2021-09-07 A propos

[2010.12566] DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

Tags:

2021-09-06 A propos

Joey NMT’s documentation!

Tags:

Neural machine translation

2021-08-26 A propos

[2010.02353] Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages

Tags:

2021-08-25 A propos

[1911.02116] Unsupervised Cross-lingual Representation Learning at Scale

Tags:

2021-07-29 A propos

[2107.00676] A Primer on Pretrained Multilingual Language Models

Tags:

2021-07-13 A propos

[1906.05685] A Focus on Neural Machine Translation for African Languages

Tags:

2021-06-30 A propos

Nils Reimers sur Twitter : EasyNMT Easy-to-use (3 lines of code), state-of-the-art neural machine translations

Tags:

2021-04-27 A propos

[1806.06478] Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment

Tags:

2020-09-06 A propos

Why You Should Do NLP Beyond English

Tags:

2020-08-01 A propos

[2004.14958] A Call for More Rigor in Unsupervised Cross-lingual Learning

Tags:

2020-05-02 A propos

GitHub - OpenNMT/OpenNMT-py: Open Source Neural Machine Translation in PyTorch

Tags:

2020-01-17 A propos

Unsupervised Learning with Text (AFIA 2019)

Tags:

2019-12-14 A propos

[1911.01464] Emerging Cross-lingual Structure in Pretrained Language Models

Tags:

2019-11-06 A propos

Efficient multi-lingual language model fine-tuning · fast.ai NLP

Tags:

2019-10-22 A propos

Machine Translation for African Languages

Tags:

2019-09-02 A propos

Jade Abbott sur Twitter : "Calling all African NLPers! Goal: Let's publish a paper with NMT baselines for all African languages Slides

Tags:

2019-09-02 A propos

Sebastian Ruder sur Twitter : "In the second part of the NLP and speech processing session @DeepIndaba, @alienelf presents her journey and work on machine translation for African languages with @LauraMartinus #DLIndaba2019"

Tags:

2019-08-29 A propos

[1908.08983] A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource Named Entity Recognizers

Tags:

2019-08-28 A propos

[1812.04616] Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Tags:

2018-12-14 A propos

Monolingual data in NMT

Tags:

2018-12-04 A propos

Paris NLP Season 3 Meetup #2 – Paris NLP

Tags:

2018-12-04 A propos

Using Monolingual Data in Neural Machine Translation: a Systematic Study

Tags:

2018-11-28 A propos

Encoder-Decoder Recurrent Neural Network Models for Neural Machine Translation

Tags:

2018-11-07 A propos

[1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Tags:

2018-10-12 A propos

Found in translation: Building a language translator from scratch with deep learning - FloydHub

Tags:

Neural machine translation

2018-10-11 A propos

Unsupervised Machine Translation. G. Lample (slides)

Tags:

2018-09-29 A propos

Paris NLP Season 3 Meetup #1 | Meetup

Tags:

2018-09-27 A propos

Google AI Blog: Transformer: A Novel Neural Network Architecture for Language Understanding

Tags:

2018-08-17 A propos

The Illustrated Transformer – Jay Alammar

Tags:

2018-07-09 A propos

Who is doing interesting NLP research for low resource languages? - Quora

Tags:

Cross-lingual NLP

2018-07-03 A propos

Bringing low-resource languages and spoken dialects into play with Semi-Supervised Universal Neural Machine Translation - Microsoft Research

Tags:

2018-05-18 A propos

Under the hood: Multilingual embeddings | Engineering Blog | Facebook Code

Tags:

2018-05-11 A propos

GitHub - Babylonpartners/fastText_multilingual: Multilingual word vectors

Tags:

2018-05-11 A propos

facebookresearch/MUSE: A library for Multilingual Unsupervised or Supervised word Embeddings

Tags:

2017-12-22 A propos

Artificial intelligence goes bilingual—without a dictionary | Science | AAAS

Tags:

Machine translation

2017-12-01 A propos

DeepL Traducteur

Tags:

Machine translation

2017-11-21 A propos

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings (2015)

Tags:

2017-11-12 A propos

[1710.04087] Word Translation Without Parallel Data

Tags:

2017-10-14 A propos

TensorFlow Neural Machine Translation (seq2seq) Tutorial

Tags:

2017-09-18 A propos

Research Blog: Transformer: A Novel Neural Network Architecture for Language Understanding

Tags:

2017-09-01 A propos

DeepL

Tags:

2017-08-30 A propos

Attention and Memory in Deep Learning and NLP – WildML

Tags:

cf. visual attention

In standard [#seq2seq](/tag/sequence_to_sequence_learning) NMT, the decoder is supposed to generate a translation solely based on the last hidden state of the encoder - which therefore must capture everything from the source sentence (it must be a sentence embedding). Not good. Hence the attention mechanism.

> we allow the decoder to “attend” to different parts of the source sentence at each step of the output generation. Importantly, we let the model learn what to attend to based on the input sentence and what it has produced so far

> each decoder output word now depends on a weighted combination of all the input states, not just the last state.

Possible to interpret what the model is doing by looking at the Attention weight matrix

Cost: We need to calculate an attention value for each combination of input and output word (-> attention is a bit of a misnomer: we look at everything in details before deciding what to focus on)

> attention mechanism is simply giving the network access to its internal memory, which is the hidden state of the encoder

> Unlike typical memory, the memory access mechanism here is soft, which means that the network retrieves a weighted combination of all memory locations, not a value from a single discrete location

2016-01-03 A propos

Scientists See Advances in Deep Learning, a Part of Artificial Intelligence - NYTimes.com

Tags:

2012-11-30 A propos

Google Translator: The Universal Language

Tags:

2005-05-31 A propos