ssearch

Semantic search over a personal journal archive and a collection of clippings. Uses vector embeddings and a local LLM to find and synthesize information across 2000+ dated text entries spanning 2000-2026, plus a library of PDFs, articles, and web saves.

How it works

Query ─┬→ Vector similarity  (BAAI/bge-large-en-v1.5, top-20) ─┐
       └→ BM25 term matching             (top-20) ─┘
            merge + dedupe → Cross-encoder re-rank (BAAI/bge-reranker-v2-m3, top-15)
            → LLM synthesis (Gemma4 via Ollama, or OpenAI API) → Response + sources

1. Build: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates. Journal entries are named YYYY-MM-DD.txt, and build_store.py parses that into entry_date/year/month metadata (filesystem timestamps are not the entry date).
2. Retrieve: Two retrievers run in parallel. The query is embedded with the same model and matched by cosine similarity (top 20), while BM25 scores the same corpus by term frequency (top 20). The two candidate sets are merged and deduplicated by node ID, typically yielding 35-40 unique chunks.
3. Re-rank: A cross-encoder (BAAI/bge-reranker-v2-m3) scores each (query, chunk) pair jointly and keeps the top 15. It does not care which retriever nominated a candidate; BM25's job is only to surface exact names, dates, and terms that embeddings miss.
4. Synthesize: The re-ranked chunks are passed to a local LLM with a custom prompt that encourages multi-source synthesis, producing a grounded answer with file citations.

Project structure

ssearch/
├── build_store.py              # Build/update journal vector store (incremental)
├── query_hybrid.py             # Hybrid BM25+vector query with LLM synthesis
├── retrieve.py                 # Verbatim hybrid retrieval (no LLM)
├── search_keywords.py          # Keyword search via POS-based term extraction
├── run_query.sh                # Interactive shell wrapper with timing and logging
├── run_retrieve.sh             # Interactive wrapper for verbatim retrieval (no LLM)
├── clippings_search/
│   ├── build_clippings.py      # Build/update clippings vector store (ChromaDB)
│   ├── retrieve_clippings.py   # Verbatim clippings chunk retrieval
│   └── store_clippings/        # Persisted clippings vector store (ChromaDB)
├── data/                       # Symlink to journal .txt files
├── clippings/                  # Symlink to clippings (PDFs, TXT, webarchive, RTF)
├── store/                      # Persisted journal vector store
├── models/                     # Cached HuggingFace models (offline)
├── pyproject.toml              # Project metadata and dependencies (uv)
├── uv.lock                     # Pinned dependency lockfile (uv)
└── .python-version             # Pins Python 3.12 for uv

Setup

Prerequisites: uv, Ollama with gemma4:e4b or similar pulled. uv manages the Python 3.12 toolchain and dependencies.

cd ssearch
uv sync                    # create .venv and install the search pipeline
uv sync --group notebook   # also install Jupyter / analysis deps (optional)

uv sync reads pyproject.toml and the pinned uv.lock, creating .venv automatically (no manual python -m venv or activation needed). Run any script with uv run:

uv run python retrieve.py "your query"

You can still source .venv/bin/activate and call python directly if you prefer.

The data/ symlink should point to the journal archive (plain .txt files). The clippings/ symlink should point to the clippings folder. The embedding model (BAAI/bge-large-en-v1.5) and cross-encoder (BAAI/bge-reranker-v2-m3) are cached in ./models/ for offline use.

Offline model loading

All query scripts set three environment variables to prevent HuggingFace from making network requests:

os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["HF_HUB_CACHE"] = "./models"
os.environ["HF_HUB_OFFLINE"] = "1"

These must appear before any imports that touch HuggingFace libraries. The huggingface_hub library evaluates HF_HUB_OFFLINE once at import time (in huggingface_hub/constants.py). If the env var is set after imports, the library will still attempt network access and fail offline.

HF_HUB_CACHE replaced SENTENCE_TRANSFORMERS_HOME, which sentence-transformers 5.x forwards as a deprecated cache_dir argument.

Alternatively, set the variable in your shell before running Python:

export HF_HUB_OFFLINE=1
uv run python query_hybrid.py "your query"

Usage

Build the vector stores

# Journal index -- incremental update (default)
uv run python build_store.py

# Journal index -- full rebuild
uv run python build_store.py --rebuild

# Clippings index -- incremental update (default)
uv run python clippings_search/build_clippings.py

# Clippings index -- full rebuild
uv run python clippings_search/build_clippings.py --rebuild

The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters, the embedding model, or the metadata policy change.

build_clippings.py handles PDFs, TXT, webarchive, and RTF files. PDFs are validated before indexing — those without extractable text are skipped and written to ocr_needed.txt for later OCR.

Search journals

Semantic search with LLM synthesis

Requires Ollama running with gemma4.

Hybrid BM25 + vector (query_hybrid.py): Retrieves top 20 by vector similarity and top 20 by BM25 term frequency, merges and deduplicates, re-ranks the union to top 15, synthesizes. Catches exact name/term matches that vector-only retrieval misses.

uv run python query_hybrid.py "What does the author say about creativity?"

Interactive wrapper (run_query.sh): Loops for queries, displays timing, and appends queries to query.log.

./run_query.sh

Verbatim chunk retrieval (no LLM)

Same hybrid retrieval and re-ranking pipeline but outputs raw chunk text. Each chunk is annotated with its source: [vector-only], [bm25-only], or [vector+bm25]. No Ollama needed.

uv run python retrieve.py "Kondiaronk and the Wendats"

Keyword search (no vector store, no LLM)

Extracts nouns and adjectives from the query using NLTK POS tagging, then greps journal files for matches with surrounding context.

uv run python search_keywords.py "Discussions of Kondiaronk and the Wendats"

Search clippings

Verbatim chunk retrieval from the clippings index. Same embedding model and cross-encoder re-ranking. Outputs a summary of source files and rankings, then full chunk text. Includes page numbers for PDF sources. No Ollama needed.

uv run python clippings_search/retrieve_clippings.py "creativity and innovation"

Output format

Response:
<LLM-synthesized answer citing specific files>

Source documents:
data/2024-03-15.txt  0.683
data/2023-11-02.txt  0.651
...

Configuration

Key parameters (set in source files):

Parameter	Value	Location
Embedding model	BAAI/bge-large-en-v1.5	all build and query scripts
Chunk size	256 tokens	build_store.py, clippings_search/build_clippings.py
Chunk overlap	25 tokens	build_store.py, clippings_search/build_clippings.py
Paragraph separator	\n\n	build_store.py
Vector retrieval	top 20	query and retrieve scripts
BM25 retrieval	top 20	query and retrieve scripts
Re-rank model	BAAI/bge-reranker-v2-m3	query and retrieve scripts
Re-rank top-n	15	query and retrieve scripts
LLM	gemma4:e4b (Ollama) or gpt-4o-mini (OpenAI API)	query_hybrid.py
Temperature	0.3	query_hybrid.py
Thinking	enabled	query_hybrid.py
Context window	16384 tokens	query_hybrid.py
Request timeout	360 seconds	query_hybrid.py

Metadata reaching the embedder and the LLM is controlled by EXCLUDED_EMBED_KEYS / EXCLUDED_LLM_KEYS in build_store.py. These affect only the text each component sees; the full metadata dict is always stored and returned with every retrieval result. The embed text carries entry_date (BM25 indexes MetadataMode.EMBED, so this is its only source of dates); the LLM text carries file_name; file_path is in neither.

Key dependencies

• llama-index-core (0.14.14) -- RAG framework
• llama-index-embeddings-huggingface -- embedding integration
• llama-index-vector-stores-chroma -- ChromaDB vector store for clippings
• llama-index-llms-ollama -- local LLM via Ollama
• llama-index-llms-openai -- OpenAI API LLM (optional)
• llama-index-retrievers-bm25 -- BM25 sparse retrieval for hybrid search
• chromadb -- persistent vector store for clippings index
• sentence-transformers -- cross-encoder re-ranking
• torch -- ML runtime

Design decisions

• BAAI/bge-large-en-v1.5 over all-mpnet-base-v2: Better semantic matching quality for journal text despite slower embedding.
• 256-token chunks: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches.
• gemma4:e4b over command-r7b: Sticks closer to provided context with less hallucination at comparable speed. Earlier, selected command-r7b over llama3.1:8B for similar reasons.
• Cross-encoder re-ranking: Merge the vector and BM25 candidates, then re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. ms-marco-MiniLM-L-12-v2 was used initially, over stsb-roberta-base (wrong task) and BAAI/bge-reranker-v2-m3 (slower). Reversed in Sept 2026: ms-marco is a 2021 model trained on MS MARCO web queries, which is out of distribution for journal-style questions — across ten test queries its median score over its own top-15 was -5.47, far outside its calibrated range. On paraphrased passage queries the two tie on recall@15 (0.875) but differ on MRR (0.762 vs 0.685): bge does not find more, it ranks better. Costs about 5s more per query. BAAI/bge-reranker-base is the middle option if that matters.
• gemma4:e4b over gemma4:12b: On identical frozen context, 12b matched e4b on citation coverage at roughly 2.3x the latency, with no detectable quality advantage. Separately, thinking=True costs ~2.4x latency for no extra citation coverage — kept on for now, but it is the open question, not the model choice.
• Retrieval, not re-ranking, is the bottleneck: On paraphrased passage queries the gold entry reaches the candidate pool 90% of the time while the final top-15 captures 87.5% — so the re-ranker already recovers 97% of what retrieval offers. Further gains belong in the embedding model, chunking, and query expansion.
• HyDE query rewriting tested and dropped: Did not improve results over direct prompt engineering.
• Hybrid BM25 + vector retrieval: BM25 nominates candidates with exact term matches that embeddings miss; the cross-encoder decides final relevance.
• ChromaDB for clippings: Persistent SQLite-backed store. Chosen over the JSON store for its metadata filtering and direct chunk-level operations for incremental updates.
• PDF validation before indexing: Pre-check each PDF with pypdf — skip if text extraction yields <100 chars or low printable ratio. Skipped files written to ocr_needed.txt.