Semanlink - Language AND Université

> exBERT
adds a new domain-specific vocabulary and the corresponding
embedding layer, as well as a small
extension module to the original unmodified model

> a pretraining
method allowing **low-cost embedding of
domain-specific vocabulary in the context of an
existing large pre-trained model such as BERT**

> exBERT... explicitly incorporates
the new domain’s vocabulary, while being able to
**reuse the original pre-trained model’s weights as is**
to reduce required computation and training data. Specifically, exBERT extends BERT by augmenting
its embeddings for the original vocabulary with
new embeddings for the domain-specific vocabulary
via **a learned small “extension” module**. **The
output of the original and extension modules are
combined via a trainable weighted sum operation**

In a way similar to concept developed in

> [[1902.00751] Parameter-Efficient Transfer Learning for NLP](doc:2021/04/1902_00751_parameter_efficien), but not in the fine-tuning paradigm.

[Github](https://github.com/cgmhaicenter/exBERT)

2021-04-11 A propos

Pablo Castro sur Twitter : "Random finding of the day for word embeddings: vec("apple")-vec("apples") yields a vector close to ipad, ipod, etc. (apples removes the "fruitness" from apple)

Tags:

2020-12-18 A propos

How to allow deep learning on your data without revealing the data – Off the convex path

Tags:

2020-11-12 A propos

[2010.00402] From Trees to Continuous Embeddings and Back: Hyperbolic Hierarchical Clustering

Tags:

2020-10-03 A propos

A Study of multilabel text classification and the effect of label hierarchy (2015)

Tags:

2020-08-15 A propos

dicksontsai/stanford-nlp-local-extension: Chrome extension for sending content to localhost server running Stanford NLP tools.

Tags:

2020-07-03 A propos

[2006.09462] Selective Question Answering under Domain Shift

Tags:

2020-06-30 A propos

Softmax classifier (CS231n Convolutional Neural Networks for Visual Recognition)

Tags:

2020-06-04 A propos

BERT, ELMo, & GPT-2: How Contextual are Contextualized Word Representations? | SAIL Blog

Tags:

2020-03-28 A propos

The Berkeley NLP Group

Tags:

2020-01-12 A propos

Lecture 14 – Contextual Vectors | Stanford CS224U: Natural Language Understanding | Spring 2019

Tags:

2020-01-05 A propos

CS224U: Natural Language Understanding

Tags:

2020-01-05 A propos

[1911.00172] Generalization through Memorization: Nearest Neighbor Language Models

Tags:

2019-12-20 A propos

[1807.00082] Amanuensis: The Programmer's Apprentice

Tags:

**The use of natural language to facilitate communication
between the expert programmer and apprentice AI system.**

> an overview of the material covered in a course taught at Stanford in the spring quarter of 2018. The course draws upon **insight from cognitive and systems neuroscience to implement hybrid connectionist and symbolic reasoning systems** that leverage and extend the state of the art in machine learning **by integrating human and machine intelligence**. As a concrete example we focus on digital assistants that learn from continuous dialog with an expert software engineer while providing initial value as powerful analytical, computational and mathematical savants.

> [#Dehaene](/tag/stanislas_dehaene)'s work extends the [#Global Workspace Theory](/tag/global_workspace_theory) of Bernard Baars. Dehaene’s version of the theory combined with Yoshua Bengio’s concept of a [#consciousness prior](/tag/consciousness_prior.html) and deep reinforcement learning suggest a model for constructing and maintaining the cognitive states that arise and persist during complex problem solving.

2019-11-12 A propos

Answering Complex Open-domain Questions at Scale | SAIL Blog

Tags:

2019-10-21 A propos

CS224n: Natural Language Processing with Deep Learning Stanford / Winter 2019

Tags:

2019-09-30 A propos

CS224n: Natural Language Processing with Deep Learning

Tags:

2019-04-03 A propos

Introduction to PyTorch Code Examples

Tags:

2019-04-03 A propos

[1703.02507] Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features

Tags:

2019-03-25 A propos

[1902.09229] A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Tags:

2019-03-20 A propos

Contrastive Unsupervised Learning of Semantic Representations: A Theoretical Framework – Off the convex path (2019-03)

Tags:

2019-03-20 A propos

Word Embeddings: Explaining their properties – Off the convex path (2016)

Tags:

2019-03-20 A propos

nlp-notebooks/Simple Sentence Similarity.ipynb at master · nlptown/nlp-notebooks

Tags:

2019-02-07 A propos

StanfordNLP | StanfordNLP

Tags:

2019-01-30 A propos

The Stanford Question Answering Dataset

Tags:

2018-11-05 A propos

Deep Latent-Variable Models for Natural Language - Tutorial - harvardnlp

Tags:

2018-11-01 A propos

Machines that learn language more like kids do | MIT News

Tags:

2018-11-01 A propos

Grounded Language Learning and Understanding — MIT Media Lab (1999-2001)

Tags:

2018-10-26 A propos

Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline (Ethayarajh 2018)

Tags:

2018-10-08 A propos

Linear algebraic structure of word meanings – Off the convex path

Tags:

2018-09-20 A propos

A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018)

Tags:

A La Carte Embedding

2018-09-18 A propos

Simple and efficient semantic embeddings for rare words, n-grams, and language features – Off the convex path

Tags:

2018-09-18 A propos

Off the convex path

Tags:

2018-09-09 A propos

A Latent Variable Model Approach to PMI-based Word Embeddings (2016)

Tags:

2018-08-28 A propos

[1601.03764] Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Tags:

> Here it is shown that multiple word senses reside
in linear superposition within the word
embedding and simple sparse coding can recover
vectors that approximately capture the
senses

> Each extracted word sense is accompanied by one of about  2000 “discourse atoms” that gives a succinct description of which other words co-occur with that word sense.

> The success of the approach is mathematically explained using a variant of
the random walk on discourses model

("random walk": a generative model for language). Under the assumptions of this model,  there
exists a linear relationship between the vector of a
word w and the vectors of the words in its contexts (It is not the average of the words in w's context, but in a given corpus the matrix of the linear relationship does not depend on w. It can be estimated, and so we can compute the embedding of a word from the contexts it belongs to)

[Related blog post](/doc/?uri=https%3A%2F%2Fwww.offconvex.org%2F2016%2F07%2F10%2Fembeddingspolysemy%2F)

2018-08-28 A propos

Mathematics of Machine Learning: An introduction

Tags: