Stefano Fancello is talking about #Langchain, an open-source Python-based toolkit for Retrieval Augmentation Generation #RAG. It helps preparing your own data as a context for a question you send to a Large Language Model #LLM. Langchain tools can ingest all kinds of document formats, split documents into Chunks, and create so called #Embeddings and send it to the LLM. #fosdem2024#ai#opensource
not sure if this exists — i wish i could prompt an embedding model.
i.e. provide context for the prompt to be interpreted, but not have the prompt contribute to the embedding value. like, the prompt could have concepts in it that aren’t lit up at all in the embedding, unless the core text references them #LLMs#embeddings#transformers
I've dreamed about this for quite some time now, and now I've finally been able to cobble it together!
What you're seeing, is org-roam node (subtree or file) live Jina AI (fully local) similarity search in the org-roam buffer, along with your backlinks and reflinks. This automatically surfaces other org-roam nodes which are related to the one you're currently reading, or even working on!
This open source setup currently works as follows:
export all of your org-roam nodes as text files using supplied emacs-lisp
use embed.py to calculate embeddings for all of these txt files and store them in a parq file
run serve.py which waits for submission of any text to return the N closest node ids, according to the Jina AI learned embeddings. These are really quite good and fully local, but it would be straight-forward to use a service like OpenAI embeddings for everything
There is more emacs lisp that customizes the org-roam buffer setup to call to serve.py's endpoint and renders the list of similar nodes
@sleslie@clintlalonde That’s logical to want a more purely trained LLM, but isnt amount needed super large? I’m trying to warp my head around it but if I read Simon Wilkinson, we don’t necessarily need to build new LLMs but understand how to deploy embeddings to the source content we want it to draw from? https://simonwillison.net/2023/Aug/27/wordcamp-llms/#embeddings I read all the explanations of them being 1563 dimension vectorized representations of tokens but struggle to grok it
i wish i knew more about comparing #embeddings. anyone have resources? one thing i’ve wondered is how to convert an embedding from a “point” to an “area” or “volume”. e.g. an embedding of a 5 paragraph essay will occupy a single point in embedding space, but if you broke it down (e.g. by paragraph), there would be several points and the whole would presumably be at the center. is there a way to trace the full space a text occupies in #embedding space? #LLMs#LLM#AI#NLP