Back in January, we started looking at AI and how to run a large language model (LLM) locally (instead of just using something like ChatGPT or Gemini). A tool like Ollama is great for building a system that uses AI without dependence on OpenAI. Today, we will look at creating a Retrieval-augmented generation (RAG) application, using Python, LangChain, Chroma DB, and Ollama. Retrieval-augmented generation is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response. If you have a source of truth that isn’t in the training data, it is a good way to get the model to know about it. Let’s get started!
Your RAG will need a model (like llama3 or mistral), an embedding model (like mxbai-embed-large), and a vector database. The vector database contains relevant documentation to help the model answer specific questions better. For this demo, our vector database is going to be Chroma DB. You will need to “chunk” the text you are feeding into the database. Let’s start there.
Chunking
There are many ways of choosing the right chunk size and overlap but for this demo, I am just going to use a chunk size of 7500 characters and an overlap of 100 characters. I am also going to use LangChain‘s CharacterTextSplitter to do the chunking. It means that the last 100 characters in the value will be duplicated in the next database record.
The Vector Database
A vector database is a type of database designed to store, manage, and manipulate vector embeddings. Vector embeddings are representations of data (such as text, images, or sounds) in a high-dimensional space, where each data item is represented as a dense vector of real numbers. When you query a vector database, your query is transformed into a vector of real numbers. The database then uses this vector to perform similarity searches.
You can think of it as being like a two-dimensional chart with points on it. One of those points is your query. The rest are your database records. What are the points that are closest to the query point?
Our main model for this demo is going to be phi3. It is a 3.8B parameters model that was trained by Microsoft.
LangChain
You will notice that today’s demo is heavily using LangChain. LangChain is an open-source framework designed for developing applications that use LLMs. It provides tools and structures that enhance the customization, accuracy, and relevance of the outputs produced by these models. Developers can leverage LangChain to create new prompt chains or modify existing ones. LangChain pretty much has APIs for everything that we need to do in this app.
The Actual App
Before we start, you are going to want to pip install tiktoken langchain langchain-community langchain-core. You are also going to want to ollama pull phi3 and ollama pull nomic-embed-text. This is going to be a CLI app. You can run it from the terminal like python3 app.py "<Question Here>".
You also need a sources.txt file containing the URLs of things that you want to have in your vector database.
The May 7th event is too recent to be in the model’s training data. This makes sure that the model knows about it. You could also feed the model company policy documents, the rules to a board game, or your diary and it will magically know that information. Since you are running the model in Ollama, there is no risk of that information getting out, too. It is pretty awesome.
Have any questions, comments, etc? Feel free to drop a comment, below.
I just issued a data deletion request to #StackOverflow to erase all of the associations between my name and the questions, answers and comments I have on the platform.
One of the key ways in which #RAG works to supplement #LLMs is based on proven associations. Higher ranked Stack Overflow members' answers will carry more weight in any #LLM that is produced.
By asking for my name to be disassociated from the textual data, it removes a semantic relationship that is helpful for determining which tokens of text to use in an #LLM.
If you sell out your user base without consultation, expect a backlash.
I am trying to build a RAG with LLAMA 3 and... getting really crazy with the strange formats I get in the response....
Not only the response, but additional text, XML tags... #Llama3#LLM#RAG
If you have documents with the answers you're looking for, why not search the documents directly? Why are you embedding the documents then using #RAG (Retrieval Augmenter Generation) to make a large language model give you answers? An LLM generates text, it doesn't search a DB to give you results. So just search the damn DB directly, we already have great search algorithms with O(1) retrieval speeds! #LLMs are so stupid.
FreeCodeCamp released today a new course on building RAG from scratch with LangChain. The course, which is by Lance Martin from LangChain, focuses on the foundations of Retrieval Augmented Generation (RAG).
"We were supposed to research #enshittification, not embrace it as a business model!" implored the DVC Research.
The Vice-Chancellor sighed audibly and exhaled.
"We're out of options."
She raised her hands, palms up, reminiscent of prayer.
"The research grants don't cover the research we do, much less the research we want to do.
International students have declined 20% year on year since India, China and Indonesia have on-shore partnerships with Deakin and Monash that still get the grads a permanent residency.
We have PhDs teaching most of the undergrad courses. The endowment took a major hit when the stock market crashed in '25.
Federation's gone bust, Adelaide's half the size it was before the merger, and you've seen CQ merge with SCU and James Cook and Charles Darwin just to be viable."
She took a sharp inhalation of burnt autumn air.
"It's tens of millions a year in recurring revenue. That's a School's worth of people."
#AI#GenerativeAI#LLMs#Chatbots#RAG: "There are two reasons why using a publicly available LLM such as ChatGPT might not be appropriate for processing internal documents. Confidentiality is the first and obvious one. But the second reason, also important, is that the training data of a public LLM did not include your internal company information. Hence that LLM is unlikely to give useful answers when asked about that information.
Enter retrieval-augmented generation, or RAG. RAG is a technique used to augment an LLM with external data, such as your company documents, that provide the model with the knowledge and context it needs to produce accurate and useful output for your specific use case. RAG is a pragmatic and effective approach to using LLMs in the enterprise.
The @thoughtworks#TechRadar for April 2024 is worth a read - covers the movement away from #OpenSource licenses (the @osi gets a mention for their stewardship of licenses to date, and flags challenges), the rise of #AI code generation and the impacts on #CI and #CD workflows (AI code generators change workflow patterns), and #architectures for #LLM creation - we're seeing distinct patterns emerge through #RAG.
I've always enjoyed the Tech Radar format - it simplifies the complexity of a fast-changing landscape and makes it tractable for decision makers.
The RAG From Scratch is a crash course by Lance Martin from LangChain focusing on the foundations of Retrieval Augmented Generation (RAG). This tutorial covers the process of index, retrieval, and generation of a query from scratch 🚀.
@leftpaddotpy Thanks for the talk. Learned a few new things. Wanted to mention nil integration into vim but that's been touched upon close enough in the last question. If someone asks again about how to get true value out of LLMs in this context you might steer people towards #RAG. This might enable contextualized auto completion but wouldn't be trivial to build unfortunately
Stefano Fancello is talking about #Langchain, an open-source Python-based toolkit for Retrieval Augmentation Generation #RAG. It helps preparing your own data as a context for a question you send to a Large Language Model #LLM. Langchain tools can ingest all kinds of document formats, split documents into Chunks, and create so called #Embeddings and send it to the LLM. #fosdem2024#ai#opensource
🙏 A plea for lean software
💽 PocketBase is a backend in 1 file
🏭 Vanna AI framework for text-to-SQL
💭 Henrik Karlsson on what to focus on
🏡 Calvin Wankhede’s offline smart home
🎙 hosted by @jerod