Semanlink - Arxiv Doc AND LLM

> we propose to
leverage the factual knowledge from KGs to enhance LLMs,
while still benefiting from circumventing the burdensome
training expenses by using pre-trained LLMs

> Graph Neural Prompting
(GNP), a plug-and-play method to assist pre-trained
LLMs in learning beneficial knowledge from KGs
>
> GNP
encodes the pertinent grounded knowledge and complex
structural information to derive Graph Neural Prompt, an
embedding vector that can be sent into LLMs to provide
guidance and instructions

> - GNP first utilizes
a GNN to capture and encode the
intricate graph knowledge into **entity/node embeddings**. 
> - Then,
a cross-modality pooling module is present to determine
the **most relevant node embeddings in relation to the text
input**, and consolidate these node embeddings into **a holistic
graph-level embedding**.
> - After that, GNP encompasses a
**domain projector** to bridge the inherent disparities between
the graph and text domains.
> - Finally, a **self-supervised link
prediction objective** is introduced to enhance the model
comprehension of relationships between entities and capture
graph knowledge in a self-supervised manner.

2023-09-28 A propos

[2309.12307] LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

Tags:

2023-09-26 A propos

[2306.04640] ModuleFormer: Modularity Emerges from Mixture-of-Experts

Tags:

2023-09-16 A propos

[2307.08621] Retentive Network: A Successor to Transformer for Large Language Models

Tags:

2023-07-20 A propos

[2305.14128] Dr.ICL: Demonstration-Retrieved In-context Learning

Tags:

2023-07-14 A propos

[2307.02486] LongNet: Scaling Transformers to 1,000,000,000 Tokens

Tags:

2023-07-06 A propos

[2305.07185] MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Tags:

2023-07-01 A propos

[2212.14024] Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

Tags:

2023-06-23 A propos

[2306.08302] Unifying Large Language Models and Knowledge Graphs: A Roadmap

Tags:

2023-06-18 A propos

[2305.12517] Retrieving Texts based on Abstract Descriptions

Tags:

2023-06-15 A propos

[2306.07536] TART: A plug-and-play Transformer module for task-agnostic reasoning

Tags:

2023-06-15 A propos

[2202.08904] SGPT: GPT Sentence Embeddings for Semantic Search

Tags:

2023-04-25 A propos

[2304.02711] Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning

Tags:

2023-04-07 A propos

[2303.17651] Self-Refine: Iterative Refinement with Self-Feedback

Tags:

2023-04-03 A propos

[2302.10724] ChatGPT: Jack of all trades, master of none

Tags:

2023-02-22 A propos

[2212.09741] One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Tags:

2023-02-17 A propos

[2302.08091] Do We Still Need Clinical Language Models?

Tags:

2023-02-17 A propos

[2302.04761] Toolformer: Language Models Can Teach Themselves to Use Tools

Tags:

> Toolformer, **a model
trained to decide which APIs to call, when to
call them, what arguments to pass, and how to
best incorporate the results into future token
prediction**.

> fulfills the
following desiderata:
> - The use of tools should be learned in a
self-supervised way without requiring large
amounts of human annotations
>- The LM should be able to decide for itself when
and how to use which tool.

> Approach based
on the recent idea of using large LMs with incontext
learning (Brown et al., 2020) to generate
entire datasets from scratch.
>
> Given just a handful of human-written examples
of how an API can be used, 
> - we let a LM annotate
a huge language modeling dataset with potential
API calls. 
> - We then use a self-supervised loss to
determine which of these API calls actually help
the model in predicting future tokens. 
>- Finally, we
finetune the LM itself on the API calls that it considers
useful.

[Jay Hack @mathemagic1an sur twitter](https://twitter.com/mathemagic1an/status/1624870248221663232):

> from a small seed set of human inputs (essentially demonstrating usage of APIs), the training set for this behavior is generated by the LLM itself.
> 
> So what does this mean? We've found a promising way to tightly integrate arbitrary APIs with our best-performing models.

2023-02-13 A propos

[2302.04870] Offsite-Tuning: Transfer Learning without Full Model

Tags: