Semanlink - Software AND Deep Learning

> **Part of what makes LoRA so effective is that ... it’s stackable.**
>
> By contrast, training giant models from scratch not only
throws away the pretraining, but also any iterative
improvements that have been made on top.

> LoRA updates are very cheap to produce (~$100) for the
most popular model sizes.

> Many of these projects are saving time by training on
small, highly curated datasets...
> These
datasets are built using synthetic methods (e.g. filtering
the best responses from an existing model) and
scavenging from other projects

> Directly Competing With Open Source
Is a Losing Proposition

> Paradoxically, the one clear winner in all of this is Meta.
Because the leaked model was theirs ([LLaMA](tag:llama)), they have
effectively garnered an entire planet's worth of free labor.
Since most open source innovation is happening on top of
their architecture, there is nothing stopping them from
directly incorporating it into their products.

2023-05-04 A propos

Azure-Samples/azure-search-openai-demo: A sample app for the Retrieval-Augmented Generation pattern running in Azure

Tags:

2023-04-14 A propos

Physics of AI - YouTube

Tags:

- Intelligence has emerged: why? how?
- Let's study this with *controlled experiments* and *toy models*
- Clean and clear insights that peer slightly behind the magic curtain

emergence; rise of scale. Phase transition: fct(compute time for training: nb of params, dataset size)

(12:25) Transformers: the jump when compared to traditional NN: instead on oparating on single input X, operates on a set of inputs. (cf. training to say whether an image contains 2 elements of the same kind). **The aptitude to compare elements in the input sequence enables analogies** - the essence of reasoning.(Classical NN treats one high dimensional vector, tranformers hanlde sets of vectors. Attention layer replaces learned filters W of classical NN by the other elements in the sequence. Attention module compares part of input against other part of input. "Relative machines rather than absolute". Sets are a very powerful level of abstraction)

emergence
> The truth is that nobody has a clue what is going on"
>
> "**Something unknown is doing we don't know what**" (A. Eddington)

being inspired by physics' methodology: controled experiments and toy mathematical models

2023-03-28 A propos

Harrison Chase sur Twitter : "LangChain AIPlugins: A first open source attempt at using AIPlugins (the same ones ChatGPT is using)

Tags:

2023-03-24 A propos

Jimmy Lin sur Twitter : "GPT-4 and its ilk are awesome for rapid prototyping and one-offs, but at the end of the day, enterprises will deploy far smaller distilled models in production. Here's my contrarian take -" / Twitter

Tags:

2023-03-21 A propos

Jim Fan sur Twitter : "Do you know that DeepMind has actually open-sourced the heart of AlphaGo & AlphaZero?... "

Tags:

2023-02-15 A propos

Microsoft launches Teams Premium with features powered by OpenAI - The Verge

Tags:

2023-02-07 A propos

Ramsri Goutham Golla sur Twitter : "The most practical open-source competitor to @OpenAI 's GPT-3 is Google's Flan-T5 Here are 5 Flan-T5 resources to try out easily, deploy, or fine-tune it! 🧵" / Twitter

Tags:

2023-02-04 A propos

BioGPT

Tags:

2023-02-02 A propos

merve sur Twitter : "Do you want to know how models like ChatGPT work? Did you know you could build your own conversational product using open-source alternatives..."

Tags:

2022-12-06 A propos

[2008.12813] HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Tags:

2022-06-30 A propos

Unveiling Transformers with LEGO - YouTube

Tags:

2022-06-30 A propos

Lenka Zdeborova sur Twitter : "Amazing lecture of @SebastienBubeck giving intuition on transformers and how to approach them theoretically..."

Tags:

2022-06-30 A propos

[2004.05119] Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer

Tags:

2022-03-31 A propos

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine

Tags:

2021-12-05 A propos

[2106.13474] Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Tags:

2021-10-21 A propos

[2007.15779] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Tags:

2021-04-11 A propos

[1909.01259] Neural Attentive Bag-of-Entities Model for Text Classification

Tags:

A model that performs **text classification using entities in a knowledge base**.

> Entities provide unambiguous and relevant semantic signals that are beneficial for capturing semantics in texts. We combine **simple high-recall entity detection based on a dictionary** (word->list of entities), to detect entities in a document, with a novel neural **attention mechanism that enables the model to focus on a small number of unambiguous and relevant entities**.

2 steps:

1. Entity detection
2. Classification using the detected entities (+text) as inputs

Regarding entity linking, a local model which uses cosine
similarity between the embedding of the target
entity and the word-based representation of
the document to capture the relevance of an entity
given a document.

Embeddings from the KB: computed using [#Wikipedia2Vec](tag:wikipedia2vec) (similar words and entities
close to one another in a unified vector space)

Model using attention, with 2 features :

- cosine similarity between the
embedding of the entity and the word based
representation of the document
- the probability that the entity
name refers to the entity in KB.

Somewhat [related](doc:2020/01/investigating_entity_knowledge_)

### Conclusion:

>a neural
network model that performs text classification using
entities in Wikipedia. We combined simple
dictionary-based entity detection with a neural attention
mechanism to enable the model to focus
on a small number of unambiguous and relevant
entities in a document.

2020-09-02 A propos

[1911.03814] Scalable Zero-shot Entity Linking with Dense Entity Retrieval

Tags: