Update README: hybrid retrieval, bge-reranker-v2-m3, config
This commit is contained in:
parent
24a38ac43f
commit
5cac92959d
1 changed files with 36 additions and 23 deletions
59
README.md
59
README.md
|
|
@ -1,16 +1,19 @@
|
|||
# ssearch
|
||||
|
||||
Semantic search over a personal journal archive and a collection of clippings. Uses vector embeddings and a local LLM to find and synthesize information across 1800+ dated text entries spanning 2000-2025, plus a library of PDFs, articles, and web saves.
|
||||
Semantic search over a personal journal archive and a collection of clippings. Uses vector embeddings and a local LLM to find and synthesize information across 2000+ dated text entries spanning 2000-2026, plus a library of PDFs, articles, and web saves.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
Query → Embed (BAAI/bge-large-en-v1.5) → Vector similarity (top-30) → Cross-encoder re-rank (top-15) → LLM synthesis (Gemma4 via Ollama, or OpenAI API) → Response + sources
|
||||
Query ─┬→ Vector similarity (BAAI/bge-large-en-v1.5, top-20) ─┐
|
||||
└→ BM25 term matching (top-20) ─┘
|
||||
merge + dedupe → Cross-encoder re-rank (BAAI/bge-reranker-v2-m3, top-15)
|
||||
→ LLM synthesis (Gemma4 via Ollama, or OpenAI API) → Response + sources
|
||||
```
|
||||
|
||||
1. **Build**: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates.
|
||||
2. **Retrieve**: A user query is embedded with the same model and matched against stored vectors by cosine similarity, returning the top 30 candidate chunks.
|
||||
3. **Re-rank**: A cross-encoder (`cross-encoder/ms-marco-MiniLM-L-12-v2`) scores each (query, chunk) pair jointly and keeps the top 15.
|
||||
1. **Build**: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates. Journal entries are named `YYYY-MM-DD.txt`, and `build_store.py` parses that into `entry_date`/`year`/`month` metadata (filesystem timestamps are not the entry date).
|
||||
2. **Retrieve**: Two retrievers run in parallel. The query is embedded with the same model and matched by cosine similarity (top 20), while BM25 scores the same corpus by term frequency (top 20). The two candidate sets are merged and deduplicated by node ID, typically yielding 35-40 unique chunks.
|
||||
3. **Re-rank**: A cross-encoder (`BAAI/bge-reranker-v2-m3`) scores each (query, chunk) pair jointly and keeps the top 15. It does not care which retriever nominated a candidate; BM25's job is only to surface exact names, dates, and terms that embeddings miss.
|
||||
4. **Synthesize**: The re-ranked chunks are passed to a local LLM with a custom prompt that encourages multi-source synthesis, producing a grounded answer with file citations.
|
||||
|
||||
## Project structure
|
||||
|
|
@ -22,6 +25,7 @@ ssearch/
|
|||
├── retrieve.py # Verbatim hybrid retrieval (no LLM)
|
||||
├── search_keywords.py # Keyword search via POS-based term extraction
|
||||
├── run_query.sh # Interactive shell wrapper with timing and logging
|
||||
├── run_retrieve.sh # Interactive wrapper for verbatim retrieval (no LLM)
|
||||
├── clippings_search/
|
||||
│ ├── build_clippings.py # Build/update clippings vector store (ChromaDB)
|
||||
│ ├── retrieve_clippings.py # Verbatim clippings chunk retrieval
|
||||
|
|
@ -32,6 +36,7 @@ ssearch/
|
|||
├── models/ # Cached HuggingFace models (offline)
|
||||
├── pyproject.toml # Project metadata and dependencies (uv)
|
||||
├── uv.lock # Pinned dependency lockfile (uv)
|
||||
└── .python-version # Pins Python 3.12 for uv
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
|
@ -52,7 +57,7 @@ uv run python retrieve.py "your query"
|
|||
|
||||
You can still `source .venv/bin/activate` and call `python` directly if you prefer.
|
||||
|
||||
The `data/` symlink should point to the journal archive (plain `.txt` files). The `clippings/` symlink should point to the clippings folder. The embedding model (`BAAI/bge-large-en-v1.5`) and cross-encoder (`cross-encoder/ms-marco-MiniLM-L-12-v2`) are cached in `./models/` for offline use.
|
||||
The `data/` symlink should point to the journal archive (plain `.txt` files). The `clippings/` symlink should point to the clippings folder. The embedding model (`BAAI/bge-large-en-v1.5`) and cross-encoder (`BAAI/bge-reranker-v2-m3`) are cached in `./models/` for offline use.
|
||||
|
||||
### Offline model loading
|
||||
|
||||
|
|
@ -60,16 +65,18 @@ All query scripts set three environment variables to prevent HuggingFace from ma
|
|||
|
||||
```python
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
os.environ["SENTENCE_TRANSFORMERS_HOME"] = "./models"
|
||||
os.environ["HF_HUB_CACHE"] = "./models"
|
||||
os.environ["HF_HUB_OFFLINE"] = "1"
|
||||
```
|
||||
|
||||
**These must appear before any imports that touch HuggingFace libraries.** The `huggingface_hub` library evaluates `HF_HUB_OFFLINE` once at import time (in `huggingface_hub/constants.py`). If the env var is set after imports, the library will still attempt network access and fail offline.
|
||||
|
||||
`HF_HUB_CACHE` replaced `SENTENCE_TRANSFORMERS_HOME`, which sentence-transformers 5.x forwards as a deprecated `cache_dir` argument.
|
||||
|
||||
Alternatively, set the variable in your shell before running Python:
|
||||
```bash
|
||||
export HF_HUB_OFFLINE=1
|
||||
python query_hybrid.py "your query"
|
||||
uv run python query_hybrid.py "your query"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
|
@ -78,19 +85,19 @@ python query_hybrid.py "your query"
|
|||
|
||||
```bash
|
||||
# Journal index -- incremental update (default)
|
||||
python build_store.py
|
||||
uv run python build_store.py
|
||||
|
||||
# Journal index -- full rebuild
|
||||
python build_store.py --rebuild
|
||||
uv run python build_store.py --rebuild
|
||||
|
||||
# Clippings index -- incremental update (default)
|
||||
python clippings_search/build_clippings.py
|
||||
uv run python clippings_search/build_clippings.py
|
||||
|
||||
# Clippings index -- full rebuild
|
||||
python clippings_search/build_clippings.py --rebuild
|
||||
uv run python clippings_search/build_clippings.py --rebuild
|
||||
```
|
||||
|
||||
The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters or the embedding model change.
|
||||
The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters, the embedding model, or the metadata policy change.
|
||||
|
||||
`build_clippings.py` handles PDFs, TXT, webarchive, and RTF files. PDFs are validated before indexing — those without extractable text are skipped and written to `ocr_needed.txt` for later OCR.
|
||||
|
||||
|
|
@ -102,7 +109,7 @@ The default incremental mode loads the existing index, compares file sizes and m
|
|||
|
||||
**Hybrid BM25 + vector** (`query_hybrid.py`): Retrieves top 20 by vector similarity and top 20 by BM25 term frequency, merges and deduplicates, re-ranks the union to top 15, synthesizes. Catches exact name/term matches that vector-only retrieval misses.
|
||||
```bash
|
||||
python query_hybrid.py "What does the author say about creativity?"
|
||||
uv run python query_hybrid.py "What does the author say about creativity?"
|
||||
```
|
||||
|
||||
**Interactive wrapper** (`run_query.sh`): Loops for queries, displays timing, and appends queries to `query.log`.
|
||||
|
|
@ -115,14 +122,14 @@ python query_hybrid.py "What does the author say about creativity?"
|
|||
Same hybrid retrieval and re-ranking pipeline but outputs raw chunk text. Each chunk is annotated with its source: `[vector-only]`, `[bm25-only]`, or `[vector+bm25]`. **No Ollama needed.**
|
||||
|
||||
```bash
|
||||
python retrieve.py "Kondiaronk and the Wendats"
|
||||
uv run python retrieve.py "Kondiaronk and the Wendats"
|
||||
```
|
||||
|
||||
#### Keyword search (no vector store, no LLM)
|
||||
|
||||
Extracts nouns and adjectives from the query using NLTK POS tagging, then greps journal files for matches with surrounding context.
|
||||
```bash
|
||||
python search_keywords.py "Discussions of Kondiaronk and the Wendats"
|
||||
uv run python search_keywords.py "Discussions of Kondiaronk and the Wendats"
|
||||
```
|
||||
|
||||
### Search clippings
|
||||
|
|
@ -130,7 +137,7 @@ python search_keywords.py "Discussions of Kondiaronk and the Wendats"
|
|||
Verbatim chunk retrieval from the clippings index. Same embedding model and cross-encoder re-ranking. Outputs a summary of source files and rankings, then full chunk text. Includes page numbers for PDF sources. **No Ollama needed.**
|
||||
|
||||
```bash
|
||||
python clippings_search/retrieve_clippings.py "creativity and innovation"
|
||||
uv run python clippings_search/retrieve_clippings.py "creativity and innovation"
|
||||
```
|
||||
|
||||
### Output format
|
||||
|
|
@ -140,8 +147,8 @@ Response:
|
|||
<LLM-synthesized answer citing specific files>
|
||||
|
||||
Source documents:
|
||||
2024-03-15.txt ./data/2024-03-15.txt 0.683
|
||||
2023-11-02.txt ./data/2023-11-02.txt 0.651
|
||||
data/2024-03-15.txt 0.683
|
||||
data/2023-11-02.txt 0.651
|
||||
...
|
||||
```
|
||||
|
||||
|
|
@ -155,14 +162,18 @@ Key parameters (set in source files):
|
|||
| Chunk size | 256 tokens | `build_store.py`, `clippings_search/build_clippings.py` |
|
||||
| Chunk overlap | 25 tokens | `build_store.py`, `clippings_search/build_clippings.py` |
|
||||
| Paragraph separator | `\n\n` | `build_store.py` |
|
||||
| Initial retrieval | 30 chunks | query and retrieve scripts |
|
||||
| Re-rank model | `cross-encoder/ms-marco-MiniLM-L-12-v2` | query and retrieve scripts |
|
||||
| Vector retrieval | top 20 | query and retrieve scripts |
|
||||
| BM25 retrieval | top 20 | query and retrieve scripts |
|
||||
| Re-rank model | `BAAI/bge-reranker-v2-m3` | query and retrieve scripts |
|
||||
| Re-rank top-n | 15 | query and retrieve scripts |
|
||||
| LLM | `gemma4:e4b` (Ollama) or `gpt-4o-mini` (OpenAI API) | `query_hybrid.py` |
|
||||
| Temperature | 0.3 | `query_hybrid.py` |
|
||||
| Context window | 8000 tokens | `query_hybrid.py` |
|
||||
| Thinking | enabled | `query_hybrid.py` |
|
||||
| Context window | 16384 tokens | `query_hybrid.py` |
|
||||
| Request timeout | 360 seconds | `query_hybrid.py` |
|
||||
|
||||
Metadata reaching the embedder and the LLM is controlled by `EXCLUDED_EMBED_KEYS` / `EXCLUDED_LLM_KEYS` in `build_store.py`. These affect only the *text* each component sees; the full metadata dict is always stored and returned with every retrieval result. The embed text carries `entry_date` (BM25 indexes `MetadataMode.EMBED`, so this is its only source of dates); the LLM text carries `file_name`; `file_path` is in neither.
|
||||
|
||||
## Key dependencies
|
||||
|
||||
- **llama-index-core** (0.14.14) -- RAG framework
|
||||
|
|
@ -180,7 +191,9 @@ Key parameters (set in source files):
|
|||
- **BAAI/bge-large-en-v1.5 over all-mpnet-base-v2**: Better semantic matching quality for journal text despite slower embedding.
|
||||
- **256-token chunks**: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches.
|
||||
- **gemma4:e4b over command-r7b**: Sticks closer to provided context with less hallucination at comparable speed. Earlier, selected **command-r7b over llama3.1:8B** for similar reasons.
|
||||
- **Cross-encoder re-ranking**: Retrieve top-30 via bi-encoder, re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. Tested three models; `ms-marco-MiniLM-L-12-v2` selected over `stsb-roberta-base` (wrong task) and `BAAI/bge-reranker-v2-m3` (slower, weak score tail).
|
||||
- **Cross-encoder re-ranking**: Merge the vector and BM25 candidates, then re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. `ms-marco-MiniLM-L-12-v2` was used initially, over `stsb-roberta-base` (wrong task) and `BAAI/bge-reranker-v2-m3` (slower). **Reversed in Sept 2026**: ms-marco is a 2021 model trained on MS MARCO web queries, which is out of distribution for journal-style questions — across ten test queries its median score over its own top-15 was -5.47, far outside its calibrated range. On paraphrased passage queries the two tie on recall@15 (0.875) but differ on MRR (0.762 vs 0.685): bge does not find more, it ranks better. Costs about 5s more per query. `BAAI/bge-reranker-base` is the middle option if that matters.
|
||||
- **gemma4:e4b over gemma4:12b**: On identical frozen context, 12b matched e4b on citation coverage at roughly 2.3x the latency, with no detectable quality advantage. Separately, `thinking=True` costs ~2.4x latency for no extra citation coverage — kept on for now, but it is the open question, not the model choice.
|
||||
- **Retrieval, not re-ranking, is the bottleneck**: On paraphrased passage queries the gold entry reaches the candidate pool 90% of the time while the final top-15 captures 87.5% — so the re-ranker already recovers 97% of what retrieval offers. Further gains belong in the embedding model, chunking, and query expansion.
|
||||
- **HyDE query rewriting tested and dropped**: Did not improve results over direct prompt engineering.
|
||||
- **Hybrid BM25 + vector retrieval**: BM25 nominates candidates with exact term matches that embeddings miss; the cross-encoder decides final relevance.
|
||||
- **ChromaDB for clippings**: Persistent SQLite-backed store. Chosen over the JSON store for its metadata filtering and direct chunk-level operations for incremental updates.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue