Back in January, we started looking at AI and how to run a large language model (LLM) locally (instead of just using something like ChatGPT or Gemini). A tool like Ollama is great for building a system that uses AI without dependence on OpenAI. Today, we will look at creating a Retrieval-augmented generation (RAG) application, using Python, LangChain, Chroma DB, and Ollama. Retrieval-augmented generation is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response. If you have a source of truth that isn’t in the training data, it is a good way to get the model to know about it. Let’s get started!
Your RAG will need a model (like llama3 or mistral), an embedding model (like mxbai-embed-large), and a vector database. The vector database contains relevant documentation to help the model answer specific questions better. For this demo, our vector database is going to be Chroma DB. You will need to “chunk” the text you are feeding into the database. Let’s start there.
Chunking
There are many ways of choosing the right chunk size and overlap but for this demo, I am just going to use a chunk size of 7500 characters and an overlap of 100 characters. I am also going to use LangChain‘s CharacterTextSplitter to do the chunking. It means that the last 100 characters in the value will be duplicated in the next database record.
The Vector Database
A vector database is a type of database designed to store, manage, and manipulate vector embeddings. Vector embeddings are representations of data (such as text, images, or sounds) in a high-dimensional space, where each data item is represented as a dense vector of real numbers. When you query a vector database, your query is transformed into a vector of real numbers. The database then uses this vector to perform similarity searches.
You can think of it as being like a two-dimensional chart with points on it. One of those points is your query. The rest are your database records. What are the points that are closest to the query point?
Our main model for this demo is going to be phi3. It is a 3.8B parameters model that was trained by Microsoft.
LangChain
You will notice that today’s demo is heavily using LangChain. LangChain is an open-source framework designed for developing applications that use LLMs. It provides tools and structures that enhance the customization, accuracy, and relevance of the outputs produced by these models. Developers can leverage LangChain to create new prompt chains or modify existing ones. LangChain pretty much has APIs for everything that we need to do in this app.
The Actual App
Before we start, you are going to want to pip install tiktoken langchain langchain-community langchain-core. You are also going to want to ollama pull phi3 and ollama pull nomic-embed-text. This is going to be a CLI app. You can run it from the terminal like python3 app.py "<Question Here>".
You also need a sources.txt file containing the URLs of things that you want to have in your vector database.
The May 7th event is too recent to be in the model’s training data. This makes sure that the model knows about it. You could also feed the model company policy documents, the rules to a board game, or your diary and it will magically know that information. Since you are running the model in Ollama, there is no risk of that information getting out, too. It is pretty awesome.
Have any questions, comments, etc? Feel free to drop a comment, below.
Instead of picking winners in AI I prefer tools that allow you to explore and quickly adapt so we’re going to be walking through how I prepared my podcast transcriptions to be used in a RAG application using #LangChain and #OpenSearch.
Recordings for registered folks only so sign up even if you can’t attend live. https://hachyderm.io/
Fun fact about the Langchain's logo. The parrot and a chain link were inspired by people calling LLMs stochastic parrots. Models mimic human writing but don't really understand what they're saying. #Langchain serves to "chain" the "parrots" together 😃 #llm#javascript#webdev
FreeCodeCamp released today a new course on building RAG from scratch with LangChain. The course, which is by Lance Martin from LangChain, focuses on the foundations of Retrieval Augmented Generation (RAG).
Google Gemini aims for a 10 mil tokens context. It's so large that you can put books, docs, videos. They all fit in this context size. Will this replace RAG?
Don't think so because:
-💸 money; you still pay per token
-🐢 slow response time
-🐞 a huge context is hard to debug
The RAG From Scratch is a crash course by Lance Martin from LangChain focusing on the foundations of Retrieval Augmented Generation (RAG). This tutorial covers the process of index, retrieval, and generation of a query from scratch 🚀.
Demo time: WiFi is not working, changing to video. Demo gods still in a bad mood at #fosdem2024
There are a lot of good videos on YouTube about doing RAG with Langchain, also with open-soure models. #Langchain#ai#python#fosdem2024
Stefano Fancello is talking about #Langchain, an open-source Python-based toolkit for Retrieval Augmentation Generation #RAG. It helps preparing your own data as a context for a question you send to a Large Language Model #LLM. Langchain tools can ingest all kinds of document formats, split documents into Chunks, and create so called #Embeddings and send it to the LLM. #fosdem2024#ai#opensource
This short tutorial, by Sam Witteveen, provides a short intro to LangGraph 🦜, the LangChain new 🐍 library. The tutorial focuses on the foundations of LangGraph - StateGraph, nodes and edges, agent executer, and agent supervisor.
A new course for LangChain by Krish Naik and freeCodeCamp. The course focuses on the functionality of LangChain with a practical example of setting up a chatbot using Streamlit with the following LLMs:
✅ OpenAI's GPT-3.5 and GPT-4
✅ Llama2
✅ Google Gemini Pro
✅ Working with Hugingfaces models
The LangGraph series provides a short introduction to the LangGraph 🦜🔗 library. This includes the following topics:
✅ The library core functionality
✅ Agend executor
✅ Dynamicly returning a tool output directly
✅ Managing agent steps
FreeCodeCamp released a crash course for LangChain.js, a JavaScript version of LangChain for running LLMs in applications. The course, by Tom Chant, focuses on integrating LLMs in JS applications using the OpenAI API.
In my two most recent videos, I introduced you to some of the basics of Spring AI. Now, let's take a step back and look at LangChain, one of the projects that inspired Spring AI.
Today I took a first look at LangSmith, a new platform for LLM production pipelines by LangChain.
I can't hook it up to a working pipeline yet because it's in closed beta, but it surely looks ambitious. It should make it easier to do logging, monitoring, debugging and evaluating pipelines (chains) against each other. It's tightly integrated with LangChain but it should support other frameworks/models as well.
i wish there were some generic solution to DAGs. #LangChain is a heavily OOP API. it's great if you want to swap component B for component A, but you're pretty much on your own if you want programs to flow differently. e.g. chat is treated entirely different than question-answer, and creating a new flow means getting neck deep in internal langchain APIs. i wish the components were loosely coupled so i could quickly rearrange the flow #LLM#LLMs
Pezzo is an #opensource platform that provides tools for prompt engineering at scale. That includes features for managing prompts, creating workflows, deploying, and observability.
It supports prompts with tools such as #chatGPT, #LangChain, AI21labs, etc., and is based on node.js and runs on #Docker.
The LangChain LLM framework is only available in Python and JavaScript. But if you're an R user, @jayeung12 shows how to incorporate LangChain Python code into an R script with the {reticulate} 📦. This lets you include #LangChain in an #rstats workflow. https://jason-yeung.netlify.app/posts/2023_5_26/
"If you google around enough, you’ll find people talking about using LangChain to chain together LLM calls and get better outputs. However, chaining calls to an LLM just makes the latency problem with LLMs worse, which is a nonstarter for us. But even if it wasn’t, we have the potential to get bitten by compound probabilities."
➥ @honeycombio
Loved your latest podcast on exponential change and the inability of our #brains to keep up.
Perhaps fight fire with fire - an AI information bodyguard that you trust. Basically Siri with #AgentGPT actually doing what you ask it to do instead of faking it and sharing your data with predatory BigTech.
Daniel Miessler had some ideas but he's too techno optimist for me. #AIEthics #DeepFakes#Security#Identification #technopocalypse#LangChain#LlamaImdex @joshbressers
Revolutionizing NLP Summarization with LangChain: Overcoming Challenges of Large Document Processing and Information Fusion (dev.to)
In the realm of Natural Language Processing (NLP), summarizing extensive or multiple documents...