Semanlink - Entreprise AND Software

> the paper LoRA: Low-Rank Adaptation of Large Language Models proposes to decompose the weight changes, ΔW, into a lower-rank representation. (To be technically correct, LoRA does not decompose the matrices directly, but it learns the decomposed matrices via backpropagation).
>
> suppose ΔW is the weight update for an A × B weight matrix. Then, we can decompose the weight update matrix into two smaller matrices: ΔW = WA WB, where WA is an an A × r-dimensional matrix, and WB is an an r × B-dimensional matrix.

LORA and LLama :

> Lit-LLaMA repository a simple, readable reimplementation of Meta’s popular LLaMA model. Besides code for training and running LLaMA itself (with the original Meta LLaMA weights), it also contains code for finetuning LLaMA using LLaMA-Adapter and LoRA.

2023-07-27 About

What is low-rank adaptation (LoRA)? - TechTalks

Tags:

2023-07-27 About

LoRA for semantic similarity tasks

Tags:

2023-07-21 About

[2307.08621] Retentive Network: A Successor to Transformer for Large Language Models

Tags:

2023-07-20 About

Llama 2 is here - get it on Hugging Face

Tags:

2023-07-19 About

clem 🤗 sur Twitter : "Llama 2 by @Meta is already integrated with @huggingface transformers, TGI, inference endpoints, PEFT and much more..."

Tags:

2023-07-19 About

MSMARCO | MSMARCO-Question-Answering

Tags:

2023-07-14 About

Introducing Azure OpenAI Service On Your Data in Public Preview - Microsoft Community Hub

Tags:

2023-06-21 About

[2306.07174] Augmenting Language Models with Long-Term Memory

Tags:

2023-06-13 About

The Falcon has landed in the Hugging Face ecosystem

Tags:

2023-06-05 About

How to use Alpaca-LoRA to fine-tune a model like ChatGPT

Tags:

2023-05-26 About

[2305.15294] Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

Tags:

2023-05-26 About

State of GPT (Andrej Karpathy at Microsoft Build)

Tags:

2023-05-24 About

Bringing the power of AI to Windows 11

Tags:

Windows Copilot

2023-05-24 About

Tags:

2023-05-21 About

microsoft/azure-openai-in-a-day-workshop

Tags:

2023-05-21 About

DataChazGPT sur Twitter : "The new 𝚝𝚛𝚊𝚗𝚜𝚏𝚘𝚛𝚖𝚎𝚛𝚜.𝚝𝚘𝚘𝚕𝚜 library from @huggingface is insane! E.g. you can summarize and chat with a PDF in just 6 lines of code..."

Tags:

2023-05-14 About

python - How can I install Anaconda aside an existing pyenv installation on OSX? - Stack Overflow

Tags:

2023-05-13 About

Azure-Samples/openai-python-enterprise-logging

Tags:

2023-05-09 About

Text Translation cognitive skill - Azure Cognitive Search | Microsoft Learn

Tags:

2023-05-09 About

Google "We Have No Moat, And Neither Does OpenAI"

Tags:

> low-cost public involvement was enabled
by a vastly cheaper mechanism for fine tuning called low
rank adaptation ()[LoRA](tag:lora)

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 About

Jupyter cell debugging does not support "step into" the third party library code with "justmycode:false" · Issue #8146 · microsoft/vscode-jupyter

Tags:

2023-04-24 About

The Embedding Archives: Millions of Wikipedia Article Embeddings in Many Languages

Tags:

2023-04-21 About

Azure-Samples/azure-search-openai-demo: A sample app for the Retrieval-Augmented Generation pattern running in Azure

Tags:

2023-04-14 About

Physics of AI - YouTube

Tags:

- Intelligence has emerged: why? how?
- Let's study this with *controlled experiments* and *toy models*
- Clean and clear insights that peer slightly behind the magic curtain

emergence; rise of scale. Phase transition: fct(compute time for training: nb of params, dataset size)

(12:25) Transformers: the jump when compared to traditional NN: instead on oparating on single input X, operates on a set of inputs. (cf. training to say whether an image contains 2 elements of the same kind). **The aptitude to compare elements in the input sequence enables analogies** - the essence of reasoning.(Classical NN treats one high dimensional vector, tranformers hanlde sets of vectors. Attention layer replaces learned filters W of classical NN by the other elements in the sequence. Attention module compares part of input against other part of input. "Relative machines rather than absolute". Sets are a very powerful level of abstraction)

emergence
> The truth is that nobody has a clue what is going on"
>
> "**Something unknown is doing we don't know what**" (A. Eddington)

being inspired by physics' methodology: controled experiments and toy mathematical models

2023-03-28 About

[2106.09685] LoRA: Low-Rank Adaptation of Large Language Models

Tags:

2023-03-21 About

Tutorial: VS Code with Google Cloud AI Platform as a backend | by Kyle Ziegler | Medium

Tags:

2023-03-21 About

Jim Fan sur Twitter : "Do you know that DeepMind has actually open-sourced the heart of AlphaGo & AlphaZero?... "

Tags:

2023-02-15 About

Microsoft launches Teams Premium with features powered by OpenAI - The Verge

Tags:

2023-02-07 About

Ramsri Goutham Golla sur Twitter : "The most practical open-source competitor to @OpenAI 's GPT-3 is Google's Flan-T5 Here are 5 Flan-T5 resources to try out easily, deploy, or fine-tune it! 🧵" / Twitter

Tags:

2023-02-04 About

BioGPT

Tags:

2023-02-02 About

Microsoft, university researchers use AI to aid in study of ancient script on China’s “oracle bones” - Microsoft Stories Asia

Tags:

2023-01-30 About

Revolutionizing Document AI with Multimodal Document Foundation Models - Microsoft Research

Tags:

2023-01-30 About

[2206.02743] A Neural Corpus Indexer for Document Retrieval

Tags:

2023-01-18 About

[2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

Tags:

2023-01-12 About

Apache Jena - RDF Patch

Tags:

Jena

2023-01-03 About

[2212.02623] Unifying Vision, Text, and Layout for Universal Document Processing

Tags:

2022-12-07 About

Talking to Models: Stanford U & Microsoft Method Enables Developers to Correct Model Bugs via Natural Language Patches | Synced

Tags:

2022-11-27 About

HyperCard FAQ 3 - Chiclette, the chic palette

Tags:

2022-10-07 About

Materials for ACL-2022 tutorial: Knowledge-Augmented Methods for NLP

Tags:

2022-08-26 About

Sparql Secrets In Jena-Fuseki - DataScienceCentral.com

Tags:

Jena Fuseki

2022-07-26 About

[2008.12813] HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Tags:

2022-06-30 About

Unveiling Transformers with LEGO - YouTube

Tags:

2022-06-30 About

Lenka Zdeborova sur Twitter : "Amazing lecture of @SebastienBubeck giving intuition on transformers and how to approach them theoretically..."

Tags:

2022-06-30 About

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 About

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 About

[1911.02655] Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks

Tags:

2021-11-19 About

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 About

Linguistic Diversity

Tags:

2021-10-03 About

[2004.09095] The State and Fate of Linguistic Diversity and Inclusion in the NLP World

Tags:

2021-10-03 About

cmd - How do I kill the process currently using a port on localhost in Windows? - Stack Overflow

Tags:

2021-08-02 About

Calling your own JavaScript functions from SPARQL queries

Tags:

2021-06-07 About

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 About

Custom NLP Approaches to Data Anonymization | by Omri Mendels | Towards Data Science

Tags:

2021-02-01 About

Pablo Castro sur Twitter : "Knowledge mining using the knowledge store feature of #AzureSearch"

Tags:

2020-12-19 About

Representing Text for Joint Embedding of Text and Knowledge Bases (EMNLP 2015)

Tags:

2020-09-14 About

The Extreme Classification Repository

Tags:

2020-08-12 About

Everything you always wanted to know about extreme classification (but were afraid to ask) - Microsoft Research - 2019

Tags:

2020-08-12 About

SLEEC: Sparse Local Embeddings for Extreme Multi-label Classification (2015)

Tags:

2020-08-11 About

python - Scraping: SSL: CERTIFICATE_VERIFY_FAILED - Stack Overflow

Tags:

2020-06-25 About

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags:

2020-05-02 About

[2001.09522] TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network

Tags:

how to add a set of new concepts to an existing taxonomy.

[Tweet](https://twitter.com/mickeyjs6/status/1253772146142216194?s=20) [GitHub](https://github.com/mickeystroller/TaxoExpan)

> we study the taxonomy expansion task: given an
existing taxonomy and a set of new emerging concepts, we aim
to automatically expand the taxonomy to incorporate these new
concepts (without changing the existing relations in the given taxonomy).

> To the best of our knowledge, this is the first study on **how to
expand an existing directed acyclic graph (as we model a taxonomy
as a DAG) using self-supervised learning**.

Self-supervised framework, the existing taxonomy being used as training data: it learns a model to predict whether a query concept is the direct hyponym of an anchor concept.

> 2 techniques:
>
> 1. a **position-enhanced graph neural network that encodes the local structure of an anchor concept** in the existing taxonomy,
> 2. a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.

Regarding 1: uses [GNN](/tag/graph_neural_networks.html) to model the "ego network" of concepts (potential “siblings”
and “grand parents” of the query concept).

> Regular
GNNs fail to distinguish nodes with different relative positions to
the query (i.e., some nodes are grand parents of the query while
the others are siblings of the query). To address this limitation, we
present a simple but effective enhancement to inject such position
information into GNNs using position embedding. We show that
such embedding can be easily integrated with existing GNN architectures
(e.g., [GCN](/tag/graph_convolutional_networks) and GAT) and significantly boosts the
prediction performance

Regarding point 2: uses InfoNCE loss, cf. [Contrastive Predictive Coding](/doc/?uri=https%3A%2F%2Farxiv.org%2Fabs%2F1807.03748)

> Instead of predicting
whether each individual ⟨query concept, anchor concept⟩ pair
is positive or not, we first group all pairs sharing the same query
concept into a single training instance and learn a model to select
the positive pair among other negative ones from the group.

(Hum, ça me rappelle quelque chose)

> assume each concept (in existing taxonomy + set of new concepts) has an initial embedding
vector learned from some text associated with this concept.

To keep things tractable, only attempts to find a single parent node of each new concept.

2020-04-25 About

[2002.11402] Detecting Potential Topics In News Using BERT, CRF and Wikipedia

Tags:

2020-02-27 About

[2002.02925] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Tags:

2020-02-10 About

Canwen Xu sur Twitter : "WTF? We brutally dismember BERT and replace all his organs?"

Tags:

2020-02-10 About

[2001.01447] Improving Entity Linking by Modeling Latent Entity Type Information

Tags:

2020-01-09 About

[1912.08904] Macaw: An Extensible Conversational Information Seeking Platform

Tags:

2020-01-01 About

Towards a Conscious AI: A Computer Architecture inspired by Neuroscience - Microsoft Research

Tags:

2019-11-12 About

Apache POI - the Java API for Microsoft Documents

Tags:

2019-10-20 About

Neural Models for Information Retrieval (2017)

Tags:

2019-08-18 About

Deep InfoMax: Learning good representations through mutual information maximization - Microsoft Research

Tags:

2019-08-09 About

Bringing the power of machine reading comprehension to specialized documents - Microsoft Research

Tags:

2019-07-28 About

[1602.01137] A Dual Embedding Space Model for Document Ranking

Tags:

2019-07-17 About

Mining Quality Phrases from Massive Text Corpora (2015)

Tags:

2019-07-15 About

Microsoft Concept Graph and Concept Tagging Release

Tags:

Microsoft Concept Graph

2019-05-28 About

Microsoft Concept Graph: Mining Semantic Concepts for Short Text Understanding | MIT Press Journals (2019)

Tags:

2019-05-28 About

Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification (2017)

Tags:

2019-05-20 About

Robust Language Representation Learning via Multi-task Knowledge Distillation - Microsoft Research

Tags:

2019-05-19 About

List-only Entity Linking - ACL Anthology (2017)

Tags:

2019-04-24 About

A Short Introduction to Learning to Rank (2010)

Tags:

2019-04-18 About

Ranking Measures and Loss Functions in Learning to Rank (2009)

Tags:

2019-04-18 About

Microsoft Academic

Tags:

2019-02-25 About

[1901.11504] Multi-Task Deep Neural Networks for Natural Language Understanding

Tags:

2019-02-17 About

[1810.00438] Parameter-free Sentence Embedding via Orthogonal Basis

Tags:

2018-10-06 About

Automatic Tag Recommendation Algorithms for Social Recommender Systems - Microsoft Research (2009)

Tags:

2018-08-07 About

Application Security With Apache Shiro

Tags:

Apache Shiro

2018-08-05 About

Apache Shiro | Simple. Java. Security.

Tags:

Apache Shiro

2018-08-05 About

Jersey vs. RESTEasy: A JAX-RS Implementation Comparison - Genuitec

Tags:

2018-08-05 About

The problem with programming and how to fix it – Alarming Development

Tags:

2018-08-05 About

Bringing low-resource languages and spoken dialects into play with Semi-Supervised Universal Neural Machine Translation - Microsoft Research

Tags:

2018-05-18 About

Install wget in Mac OS X Without Homebrew or MacPorts

Tags:

2018-05-16 About

Extreme Multi-label Learning with Label Features for Warm-start Tagging, Ranking & Recommendation (2018)

Tags:

2018-03-04 About

Knowledge Graph and Text Jointly Embedding (2014)

Tags:

2018-01-05 About

Learning Deep Structured Semantic Models for Web Search using Clickthrough Data - Microsoft Research (2013)

Tags:

2017-12-30 About

DSSM ("Deep Semantic Similarity Model") - Microsoft Research

Tags:

DSSM (Deep Semantic Similarity Model)

2017-12-30 About

#PageRank-based #Wikidata autocomplete powered by #Apache #Solr

Tags:

2017-10-21 About

Enriching Word Embeddings Using Knowledge Graph for Semantic Tagging in Conversational Dialog Systems - Microsoft Research (2015)

Tags:

> new simple, yet effective approaches to
learn domain specific word embeddings.

## Intro

> Adapting word embeddings, such as jointly capturing
syntactic and semantic information, can further enrich semantic
word representations for several tasks, e.g., sentiment
analysis (Tang et al. 2014), named entity recognition
(Lebret, Legrand, and Collobert 2013), entity-relation extraction
(Weston et al. 2013), etc. (Yu and Dredze 2014)
has introduced a lightly supervised word embedding learning
extending word2vec. They incorporate prior information to the objective
function as a regularization term considering synonymy relations
between words from Wordnet (Fellbaum 1999).

> In this work, we go one step further and investigate if
enriching the word2vec word embeddings trained on unstructured/
unlabeled text with domain specific semantic relations
obtained from knowledge sources (e.g., knowledge
graphs, search query logs, etc.) can help to discover relation
aware word embeddings. Unlike earlier work, **we encode the
information about the relations between phrases, thereby,
entities and relation mentions are all embedded into a low dimensional
vector space**.

## Related work (Learning Word Embeddings with Priors)

- word2vec
- Relational Constrained Model (RTM) (Yu and Dredze 2014)
While CBOW learns lexical word embeddings from provided text, the RTM learns embeddings of words based on their similarity to other words provided by a knowledge resource (eg. wordnet)
- Joint model  (Yu and Dredze 2014)
combines CBOW and RTM through linear combination

2017-10-02 About

Vectorland: Brief Notes from Using Text Embeddings for Search

Tags: