ssearch/README.md

# ssearch

Semantic search over a personal journal archive and a collection of clippings. Uses vector embeddings and a local LLM to find and synthesize information across 2000+ dated text entries spanning 2000-2026, plus a library of PDFs, articles, and web saves.

## How it works

```
Query
  |
  |--> Vector retriever  --> top 20 chunks --+    embeds the query with
  |                                          |    BAAI/bge-large-en-v1.5
  '--> BM25 retriever    --> top 20 chunks --+    no model at all; lexical
                                             |    term-frequency scoring
                                             v
             merge + dedupe by node ID  (~35-40 unique chunks:
             the two lists overlap, so it is fewer than 40)
                                             |
                                             v
             cross-encoder re-rank  -->  top 15 chunks
             BAAI/bge-reranker-v2-m3 scores every (query, chunk) pair
                                             |
                                             v
             LLM synthesis  gemma4:e4b via Ollama, or gpt-4o-mini via API
                                             |
                                             v
             answer + file citations
```

1. **Build**: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates. Journal entries are named `YYYY-MM-DD.txt`, and `build_store.py` parses that into `entry_date`/`year`/`month` metadata (filesystem timestamps are not the entry date).
2. **Retrieve**: Two independent retrievers run over the same index, each returning its own 20 chunks.
   - The **vector retriever** embeds the query with `BAAI/bge-large-en-v1.5` and ranks chunks by cosine similarity. This is the only step that uses the embedding model.
   - The **BM25 retriever** uses no model. It is classical lexical scoring on term frequency, which is what catches exact names, dates, and rare words that embeddings blur together.

   The two lists of 20 are then merged and deduplicated by node ID. Because they usually overlap on a few chunks, the merged pool is around 35-40 unique chunks rather than a full 40.
3. **Re-rank**: A cross-encoder (`BAAI/bge-reranker-v2-m3`) scores each (query, chunk) pair jointly and keeps the top 15. It does not care which retriever nominated a candidate; BM25's job is only to surface exact names, dates, and terms that embeddings miss.
4. **Synthesize**: The re-ranked chunks are passed to a local LLM with a custom prompt that encourages multi-source synthesis, producing a grounded answer with file citations.

## Project structure

```
ssearch/
├── build_store.py              # Build/update journal vector store (incremental)
├── query_hybrid.py             # Hybrid BM25+vector query with LLM synthesis
├── retrieve.py                 # Verbatim hybrid retrieval (no LLM)
├── search_keywords.py          # Keyword search via POS-based term extraction
├── run_query.sh                # Interactive shell wrapper with timing and logging
├── run_retrieve.sh             # Interactive wrapper for verbatim retrieval (no LLM)
├── clippings_search/
│   ├── build_clippings.py      # Build/update clippings vector store (ChromaDB)
│   ├── retrieve_clippings.py   # Verbatim clippings chunk retrieval
│   └── store_clippings/        # Persisted clippings vector store (ChromaDB)
├── data/                       # Symlink to journal .txt files
├── clippings/                  # Symlink to clippings (PDFs, TXT, Markdown, webarchive, RTF)
├── store/                      # Persisted journal vector store
├── models/                     # Cached HuggingFace models (offline)
├── pyproject.toml              # Project metadata and dependencies (uv)
├── uv.lock                     # Pinned dependency lockfile (uv)
└── .python-version             # Pins Python 3.12 for uv
```

## Setup

**Prerequisites**: [uv](https://docs.astral.sh/uv/), [Ollama](https://ollama.com) with `gemma4:e4b` or similar pulled. uv manages the Python 3.12 toolchain and dependencies.

```bash
cd ssearch
uv sync                    # create .venv and install the search pipeline
uv sync --group notebook   # also install Jupyter / analysis deps (optional)
```

`uv sync` reads `pyproject.toml` and the pinned `uv.lock`, creating `.venv` automatically (no manual `python -m venv` or activation needed). Run any script with `uv run`:

```bash
uv run python retrieve.py "your query"
```

You can still `source .venv/bin/activate` and call `python` directly if you prefer.

The `data/` symlink should point to the journal archive (plain `.txt` files). The `clippings/` symlink should point to the clippings folder. The embedding model (`BAAI/bge-large-en-v1.5`) and cross-encoder (`BAAI/bge-reranker-v2-m3`) are cached in `./models/` for offline use.

### Offline model loading

All query scripts set three environment variables to prevent HuggingFace from making network requests:

```python
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["HF_HUB_CACHE"] = "./models"
os.environ["HF_HUB_OFFLINE"] = "1"
```

**These must appear before any imports that touch HuggingFace libraries.** The `huggingface_hub` library evaluates `HF_HUB_OFFLINE` once at import time (in `huggingface_hub/constants.py`). If the env var is set after imports, the library will still attempt network access and fail offline.

`HF_HUB_CACHE` replaced `SENTENCE_TRANSFORMERS_HOME`, which sentence-transformers 5.x forwards as a deprecated `cache_dir` argument.

Alternatively, set the variable in your shell before running Python:
```bash
export HF_HUB_OFFLINE=1
uv run python query_hybrid.py "your query"
```

## Usage

### Build the vector stores

```bash
# Journal index -- incremental update (default)
uv run python build_store.py

# Journal index -- full rebuild
uv run python build_store.py --rebuild

# Clippings index -- incremental update (default)
uv run python clippings_search/build_clippings.py

# Clippings index -- full rebuild
uv run python clippings_search/build_clippings.py --rebuild
```

The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters, the embedding model, or the metadata policy change.

`build_clippings.py` handles PDFs, TXT, Markdown, webarchive, and RTF files. PDFs are validated before indexing — those without extractable text are skipped and written to `ocr_needed.txt` for later OCR.

### Search journals

#### Semantic search with LLM synthesis

**Requires Ollama running with `gemma4`.**

**Hybrid BM25 + vector** (`query_hybrid.py`): Retrieves top 20 by vector similarity and top 20 by BM25 term frequency, merges and deduplicates, re-ranks the union to top 15, synthesizes. Catches exact name/term matches that vector-only retrieval misses.
```bash
uv run python query_hybrid.py "What does the author say about creativity?"
```

**Interactive wrapper** (`run_query.sh`): Loops for queries, displays timing, and appends queries to `query.log`.
```bash
./run_query.sh
```

#### Verbatim chunk retrieval (no LLM)

Same hybrid retrieval and re-ranking pipeline but outputs raw chunk text. Each chunk is annotated with its source: `[vector-only]`, `[bm25-only]`, or `[vector+bm25]`. **No Ollama needed.**

```bash
uv run python retrieve.py "Kondiaronk and the Wendats"
```

#### Keyword search (no vector store, no LLM)

Extracts nouns and adjectives from the query using NLTK POS tagging, then greps journal files for matches with surrounding context.
```bash
uv run python search_keywords.py "Discussions of Kondiaronk and the Wendats"
```

### Search clippings

Verbatim chunk retrieval from the clippings index. Same embedding model and cross-encoder re-ranking. Outputs a summary of source files and rankings, then full chunk text. Includes page numbers for PDF sources. **No Ollama needed.**

```bash
uv run python clippings_search/retrieve_clippings.py "creativity and innovation"
```

### Output format

```
Response:
<LLM-synthesized answer citing specific files>

Source documents:
data/2024-03-15.txt  0.683
data/2023-11-02.txt  0.651
...
```

## Configuration

Key parameters (set in source files):

| Parameter | Value | Location |
|-----------|-------|----------|
| Embedding model | `BAAI/bge-large-en-v1.5` | all build and query scripts |
| Chunk size | 256 tokens, counted with the embedding model's own tokenizer | `build_store.py`, `clippings_search/build_clippings.py` |
| Chunk overlap | 25 tokens | `build_store.py`, `clippings_search/build_clippings.py` |
| Paragraph separator | `\n\n` | `build_store.py` |
| Vector retrieval | top 20 | query and retrieve scripts |
| BM25 retrieval | top 20 | query and retrieve scripts |
| Re-rank model | `BAAI/bge-reranker-v2-m3` | query and retrieve scripts |
| Re-rank top-n | 15 | query and retrieve scripts |
| LLM | `gemma4:e4b` (Ollama) or `gpt-4o-mini` (OpenAI API) | `query_hybrid.py` |
| Temperature | 0.3 | `query_hybrid.py` |
| Thinking | enabled | `query_hybrid.py` |
| Context window | 16384 tokens | `query_hybrid.py` |
| Request timeout | 360 seconds | `query_hybrid.py` |

Metadata reaching the embedder and the LLM is controlled by `EXCLUDED_EMBED_KEYS` / `EXCLUDED_LLM_KEYS` in `build_store.py`. These affect only the *text* each component sees; the full metadata dict is always stored and returned with every retrieval result. The embed text carries `entry_date` (BM25 indexes `MetadataMode.EMBED`, so this is its only source of dates); the LLM text carries `file_name`; `file_path` is in neither.

## Key dependencies

- **llama-index-core** (0.14.14) -- RAG framework
- **llama-index-embeddings-huggingface** -- embedding integration
- **llama-index-vector-stores-chroma** -- ChromaDB vector store for clippings
- **llama-index-llms-ollama** -- local LLM via Ollama
- **llama-index-llms-openai** -- OpenAI API LLM (optional)
- **llama-index-retrievers-bm25** -- BM25 sparse retrieval for hybrid search
- **chromadb** -- persistent vector store for clippings index
- **sentence-transformers** -- cross-encoder re-ranking
- **torch** -- ML runtime

## Design decisions

- **BAAI/bge-large-en-v1.5 over all-mpnet-base-v2**: Better semantic matching quality for journal text despite slower embedding.
- **256-token chunks**: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches. `SentenceSplitter` is given the embedding model's own tokenizer, because its tiktoken default and bge's BERT WordPiece diverge on unusual text (up to 2.96x here) — which previously let chunks exceed bge-large's 512-token limit and be silently truncated at embed time.
- **gemma4:e4b over command-r7b**: Sticks closer to provided context with less hallucination at comparable speed. Earlier, selected **command-r7b over llama3.1:8B** for similar reasons.
- **Cross-encoder re-ranking**: Merge the vector and BM25 candidates, then re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. `ms-marco-MiniLM-L-12-v2` was used initially, over `stsb-roberta-base` (wrong task) and `BAAI/bge-reranker-v2-m3` (slower). **Reversed in Sept 2026**: ms-marco is a 2021 model trained on MS MARCO web queries, which is out of distribution for journal-style questions — across ten test queries its median score over its own top-15 was -5.47, far outside its calibrated range. On paraphrased passage queries the two tie on recall@15 (0.875) but differ on MRR (0.762 vs 0.685): bge does not find more, it ranks better. Costs about 5s more per query. `BAAI/bge-reranker-base` is the middle option if that matters.
- **gemma4:e4b over gemma4:12b**: On identical frozen context, 12b matched e4b on citation coverage at roughly 2.3x the latency, with no detectable quality advantage. Separately, `thinking=True` costs ~2.4x latency for no extra citation coverage — kept on for now, but it is the open question, not the model choice.
- **Retrieval, not re-ranking, is the bottleneck**: On paraphrased passage queries the gold entry reaches the candidate pool 90% of the time while the final top-15 captures 87.5% — so the re-ranker already recovers 97% of what retrieval offers. Further gains belong in the embedding model, chunking, and query expansion.
- **HyDE query rewriting tested and dropped**: Did not improve results over direct prompt engineering.
- **Hybrid BM25 + vector retrieval**: BM25 nominates candidates with exact term matches that embeddings miss; the cross-encoder decides final relevance.
- **ChromaDB for clippings**: Persistent SQLite-backed store. Chosen over the JSON store for its metadata filtering and direct chunk-level operations for incremental updates.
- **PDF validation before indexing**: Pre-check each PDF with pypdf — skip if text extraction yields <100 chars or low printable ratio. Skipped files written to `ocr_needed.txt`.