Semanlink - Favoris ; Géographie AND AI@Google

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 About

[2010.00402] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering

Tags:

2020-10-03 About

BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Tags:

2020-03-28 About

Paris NLP Season 4 Meetup #3 – Paris NLP (2020)

Tags:

2020-01-23 About

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 About

[1807.00082] Amanuensis: The Programmer's Apprentice

Tags:

**The use of natural language to facilitate communication
between the expert programmer and apprentice AI system.**

> an overview of the material covered in a course taught at Stanford in the spring quarter of 2018. The course draws upon **insight from cognitive and systems neuroscience to implement hybrid connectionist and symbolic reasoning systems** that leverage and extend the state of the art in machine learning **by integrating human and machine intelligence**. As a concrete example we focus on digital assistants that learn from continuous dialog with an expert software engineer while providing initial value as powerful analytical, computational and mathematical savants.

> [#Dehaene](/tag/stanislas_dehaene)'s work extends the [#Global Workspace Theory](/tag/global_workspace_theory) of Bernard Baars. Dehaene’s version of the theory combined with Yoshua Bengio’s concept of a [#consciousness prior](/tag/consciousness_prior.html) and deep reinforcement learning suggest a model for constructing and maintaining the cognitive states that arise and persist during complex problem solving.

2019-11-12 About

CamemBERT

Tags: