From 5ec2b5c258bb819c063fa0bc3e01cee4a4247ce7 Mon Sep 17 00:00:00 2001
From: Eric Furst <furst@udel.edu>
Date: Sun, 20 Sep 2026 12:18:03 -0400
Subject: [PATCH] Clarify retrieval diagram in README

---
 README.md | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/README.md b/README.md
index bb6d158..706694e 100644
--- a/README.md
+++ b/README.md
@@ -5,14 +5,33 @@ Semantic search over a personal journal archive and a collection of clippings. U
 ## How it works
 
 ```
-Query ─┬→ Vector similarity  (BAAI/bge-large-en-v1.5, top-20) ─┐
-       └→ BM25 term matching             (top-20) ─┘
-            merge + dedupe → Cross-encoder re-rank (BAAI/bge-reranker-v2-m3, top-15)
-            → LLM synthesis (Gemma4 via Ollama, or OpenAI API) → Response + sources
+Query
+  |
+  |--> Vector retriever  --> top 20 chunks --+    embeds the query with
+  |                                          |    BAAI/bge-large-en-v1.5
+  '--> BM25 retriever    --> top 20 chunks --+    no model at all; lexical
+                                             |    term-frequency scoring
+                                             v
+             merge + dedupe by node ID  (~35-40 unique chunks:
+             the two lists overlap, so it is fewer than 40)
+                                             |
+                                             v
+             cross-encoder re-rank  -->  top 15 chunks
+             BAAI/bge-reranker-v2-m3 scores every (query, chunk) pair
+                                             |
+                                             v
+             LLM synthesis  gemma4:e4b via Ollama, or gpt-4o-mini via API
+                                             |
+                                             v
+             answer + file citations
 ```
 
 1. **Build**: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates. Journal entries are named `YYYY-MM-DD.txt`, and `build_store.py` parses that into `entry_date`/`year`/`month` metadata (filesystem timestamps are not the entry date).
-2. **Retrieve**: Two retrievers run in parallel. The query is embedded with the same model and matched by cosine similarity (top 20), while BM25 scores the same corpus by term frequency (top 20). The two candidate sets are merged and deduplicated by node ID, typically yielding 35-40 unique chunks.
+2. **Retrieve**: Two independent retrievers run over the same index, each returning its own 20 chunks.
+   - The **vector retriever** embeds the query with `BAAI/bge-large-en-v1.5` and ranks chunks by cosine similarity. This is the only step that uses the embedding model.
+   - The **BM25 retriever** uses no model. It is classical lexical scoring on term frequency, which is what catches exact names, dates, and rare words that embeddings blur together.
+
+   The two lists of 20 are then merged and deduplicated by node ID. Because they usually overlap on a few chunks, the merged pool is around 35-40 unique chunks rather than a full 40.
 3. **Re-rank**: A cross-encoder (`BAAI/bge-reranker-v2-m3`) scores each (query, chunk) pair jointly and keeps the top 15. It does not care which retriever nominated a candidate; BM25's job is only to surface exact names, dates, and terms that embeddings miss.
 4. **Synthesize**: The re-ranked chunks are passed to a local LLM with a custom prompt that encourages multi-source synthesis, producing a grounded answer with file citations.