furst/ssearch
0
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Upgrade RAG stack; fix chunk_size tokenizer mismatch

Browse source
This commit is contained in:
Eric Furst 2026-09-20 12:53:58 -04:00
commit d8d03b3d0a
5 changed files with 210 additions and 165 deletions
Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -178,7 +178,7 @@ Key parameters (set in source files):
| Parameter | Value | Location |
|-----------|-------|----------|
| Embedding model | `BAAI/bge-large-en-v1.5` | all build and query scripts |
| Chunk size | 256 tokens | `build_store.py`, `clippings_search/build_clippings.py` |
| Chunk size | 256 tokens, counted with the embedding model's own tokenizer | `build_store.py`, `clippings_search/build_clippings.py` |
| Chunk overlap | 25 tokens | `build_store.py`, `clippings_search/build_clippings.py` |
| Paragraph separator | `\n\n` | `build_store.py` |
| Vector retrieval | top 20 | query and retrieve scripts |
@ -208,7 +208,7 @@ Metadata reaching the embedder and the LLM is controlled by `EXCLUDED_EMBED_KEYS
## Design decisions
- **BAAI/bge-large-en-v1.5 over all-mpnet-base-v2**: Better semantic matching quality for journal text despite slower embedding.
- **256-token chunks**: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches.
- **256-token chunks**: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches. `SentenceSplitter` is given the embedding model's own tokenizer, because its tiktoken default and bge's BERT WordPiece diverge on unusual text (up to 2.96x here) — which previously let chunks exceed bge-large's 512-token limit and be silently truncated at embed time.
- **gemma4:e4b over command-r7b**: Sticks closer to provided context with less hallucination at comparable speed. Earlier, selected **command-r7b over llama3.1:8B** for similar reasons.
- **Cross-encoder re-ranking**: Merge the vector and BM25 candidates, then re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. `ms-marco-MiniLM-L-12-v2` was used initially, over `stsb-roberta-base` (wrong task) and `BAAI/bge-reranker-v2-m3` (slower). **Reversed in Sept 2026**: ms-marco is a 2021 model trained on MS MARCO web queries, which is out of distribution for journal-style questions — across ten test queries its median score over its own top-15 was -5.47, far outside its calibrated range. On paraphrased passage queries the two tie on recall@15 (0.875) but differ on MRR (0.762 vs 0.685): bge does not find more, it ranks better. Costs about 5s more per query. `BAAI/bge-reranker-base` is the middle option if that matters.
- **gemma4:e4b over gemma4:12b**: On identical frozen context, 12b matched e4b on citation coverage at roughly 2.3x the latency, with no detectable quality advantage. Separately, `thinking=True` costs ~2.4x latency for no extra citation coverage — kept on for now, but it is the open question, not the model choice.