• View
• Edit
• History
• Print

RAG

One of the interesting applications of transformers is the ability to catalog and search text semantically. In retrieval augmented generation (RAG) a body of text is used to construct a vector store -- chunks of text that are encoded in the abstract vector space of a sentence transformer (or more technically, embedded using a sentence transformer). Similarly embedded queries can then be compared against the vector store to find matching text. The search isn't a literal one, but it captures the semantic similarity of the query and matching vectors. The text retrieved from this search is then passed to an LLM to generate the query output. RAG can be used for such a semantic search or to summarize the retrieved documents. It also provides a way of introducing specific information an LLM can draw from outside of fine tuning a model.

A basic RAG flow is illustrated below. Before running a user's query, a body of text ("documents") is chunked into smaller pieces that fit the context length of the embedding model. These embedded chunks become the vector store. When a query is run, the user's input text is embedded by the same model. The resulting vector is compared to those in the vector store, for instance by calculating the cosine similarity between the vectors. Vector store entries with the highest similarity are returned and are sent to a generating LLM to compose the response.

Here, I will describe building a simple RAG query pipeline using the llamaindex library. The advantage of this library is that it can more or less automatically handle the ingestion of documents and chunking into the vector store. The vector store will be a simple memory-based dictionary. This limits the size and number of documents we can store, but is sufficient for our demonstration. Larger and more sophisticated implementations use a database of some form to store and retrieve the vector store. Importantly, the pipeline will run solely using local models for the embedding and generation -- an approach that is ideal for working with private document collections.

Getting started

There are two main scripts: build.py and query.py. The first will build a vector store using an embedding model. The second will run simple queries against this model. I'm using an Apple MacBook M3 Pro with 18GB of memory. The Apple Silicon GPU has 18 cores and the CPU is a 12-core model, so the machine is a reasonably high-performance consumer laptop. You need a few GB of disk storage available for the models and text.

Before getting started, set up an environment. You'll need:

1. A Python installation
2. Download ollama (the approach should accommodate a similar local model manager like llama.cpp, etc.)

If you haven't tried ollama, see this guide: Attach:Main.LLM2025/llm_2.pdf

Another hint: If you're on a mac like me, you may have had problems with macOS and getting python to work. My solution is to run it through homebrew (link). The executable (or a link to the executable) will be in something like /opt/homebrew/bin/python3.12.

Python virtual environment

We'll use a Python virtual environment (venv) for this example to manage the libraries and their dependencies. Create a working directory and install and activate venv:

$ python3 -m venv .venv
$ source .venv/bin/activate
((.venv) ) $ 

We'll need the following libraries:

Use pip to install them in the venv:

((.venv) ) $ pip install llama-index-core llama-index-readers-file \
 llama-index-llms-ollama llama-index-embeddings-huggingface

My command installed the following:

Download the LLM model

Using ollama, pull the LLM generating model with the command

$ ollama pull command-r7b

Alternatives to Cohere's command-r7b that I've tested include llama3.1:8B, deepseek-r1:8B, and gemma3:1b. All of these generate fairly quickly on my machine (the gemma models are very fast), but command-r7b is fine-tuned for RAG and tends to stick to the returned content better than the others in my tests. Be aware that the command-r7b license is Creative Commons Attribution-NonCommercial 4.0 International Public.

Download the embedding model

The Hugging Face Transformers library has the annoying behavior that it only temporarily caches the embedding model. This leads to a hang if there is no internet connection. Instead, download the model and cache it manually in a local directory with a short script:

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
embed_model = HuggingFaceEmbedding(cache_folder="./models",model_name="BAAI/bge-large-en-v1.5")

Here, we use the BAAI/bge-large-en-v1.5 model. I tested this and all-mpnet-base-v2, and got better performance from the BAAI model. It's a little larger and slower, though.

Importing documents

For this demo, I saved 294 email messages from the university president's emails to the university community as eml files. I first tried converting these using

$ textutil -convert txt *.eml

and moving the resulting .txt files to the ./data directory. The drawback here is that the text includes headers and MIME content. It turns out that this leads to a pretty long embedding run and a large vector store. So, instead I used a script to process the .eml files to strip out the body text:

This generates 294 text files and just under a megabyte of text. The script includes the subject and date and uses the latter to save the filename. The filename and path become metadata in the vector store that's useful for search.

With the text files in ./data, we can build the vector store.

Building the vector store

The script build.py below will chunk the files in ./data and create a simple vector store in ./storage. By default, the routines preserve metadata for each node, which corresponds to the filename and file path. We'll use that information in our query, too.

build.py

Besides the embedding model, the most important parameters are the chunk size and overlap. Chunk size is the number of tokens that will be embedded for each vector store entry. The overlap can help retain context across the vector store entries.

Running build.py takes a minute or so with this number and size of documents.

((.venv) ) $ python build.py 
Parsing nodes: 100%|███████████████████████████████████████████████████████████████████████| 294/294 [00:03<00:00, 79.53it/s]
Generating embeddings: 100%|█████████████████████████████████████████████████████████████| 1098/1098 [01:13<00:00, 15.01it/s]
Index built and saved to ./storage

You should see the files in the ./storage directory:

$ ls -lh storage
total 58608
-rw-r--r--@ 1 furst  staff    25M Aug 28 11:36 default__vector_store.json
-rw-r--r--@ 1 furst  staff   3.5M Aug 28 11:35 docstore.json
-rw-r--r--@ 1 furst  staff    18B Aug 28 11:35 graph_store.json
-rw-r--r--@ 1 furst  staff    72B Aug 28 11:36 image__vector_store.json
-rw-r--r--@ 1 furst  staff    92K Aug 28 11:35 index_store.json

A vector store Side Quest.

Running a query

Now we can run queries against the vector store. This simple example below uses the llamaindex library. It first loads the vector store as index, then converts this to a query engine class. The query method of the class passes the user input and does the rest -- it embeds the user input, calculates similarities against the vector store, and returns the best matching documents in the form of a response generated by the LLM. That final step takes the prompt, returned chunk text, and metadata and sends it to the LLM (command-r7b in this case) through ollama.

query.py

The code creates index, which is a BaseIndex object, from the vector store. Next, the method as_query_engine creates a query engine object (BaseQueryEngine). The query engine is what processes a natural language query by retrieving relevant nodes, optionally post-processing them, and synthesizing a response using the LLM ( about query engines).

There are some commented out settings in the code above that can be interesting to experiment with. See the Exercises below.

Prompt

Llamaindex's default query engine prompt is

You can inspect or update this prompt using the get_prompts and update_prompts methods on the query engine object. In the query.py script, you can see that the LLM generation step uses on a custom prompt that helps direct the output:

Example output

Here's an example of a run. The user's query is "Find documents that highlight the excellence of the university." The query returns a summary and ten files that reference a number of expressions of the university's commitment to excellence.

The output here ends with a summary of the top 15 matching documents in the vector store. This output is what the LLM "sees" as the CONTEXT. It includes the file path, document title, and the similarity score. The search and generation run in just over a minute on battery power, which also accounts for some time to load the generating model in ollama.

The semantic search qualities are on full display in this example:

Remember, all of this is done locally on consumer-grade hardware without passing any information to a cloud service. Such local RAG implementations should be useful for organizing, searching, and summarizing sensitive or proprietary information in private document collections. It's pretty zippy on higher performance hardware, too. On a laboratory machine equipped with a single NVIDIA GeForce RTX 4090 GPU, a query will run in about 5-10 seconds -- roughly 10 times faster than my laptop.

Exercises

1. Repeat the same query and note changes (and similarities) in the output.
2. Try changing the embedding and generating models.
3. Try changing the chunk size and overlap in build.py.
4. Try changing the context length in query.py.
5. Make changes to the prompt. Write a new prompt for different effects, possibly using a different generating model. (Here are two Examples.)
6. The modest similarity scores in the examples above might improve by rewriting the query using strategies like HyDE (Hypothetical Document Embeddings).

Links and resources

• [[https://docs.llamaindex.ai/en/stable/] - llamaindex
• https://ollama.com - ollama
• https://nvlpubs.nist.gov/nistpubs/ir/2025/NIST.IR.8579.ipd.pdf - Developing the NCCoE Chatbot, NIST Internal Report, NIST IR 8579 ipd

License

Copyright © 2025 by Eric M. Furst

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.