ssearch/tests

LLM Comparison Tests

Query used for all tests: "Passages that quote Louis Menand." Script: query_hybrid_bm25_v4.py (hybrid BM25 + vector, cross-encoder re-rank to top 15)

Retrieval is identical across all tests (same 15 chunks, same scores). Only the LLM synthesis step differs.

File naming: results_<model>_t<temperature>.txt

Results

File	LLM	Temperature	Files cited	Time	Notes
results_gpt4omini_t0.1.txt	gpt-4o-mini (OpenAI API)	0.1	6	44s	Broader coverage, structured numbered list, drew from chunks ranked as low as #14
results_commandr7b_t0.8.txt	command-r7b (Ollama local)	0.8 (default)	2	78s	Focused on top chunks, reproduced exact quotes verbatim
results_gpt4omini_t0.3.txt	gpt-4o-mini (OpenAI API)	0.3	6	45s	Very similar to 0.1 run -- same 6 files, same structure, slightly more interpretive phrasing
results_commandr7b_t0.3.txt	command-r7b (Ollama local)	0.3	6	94s	Major improvement over 0.8 default: cited 6 files (was 2), drew from lower-ranked chunks including 2024-08-03 (#15)

Observations

• Lowering command-r7b from 0.8 to 0.3 dramatically improved breadth (2 → 6 files cited). At 0.8, the model focused narrowly on the top-scored chunks. At 0.3, it used the full context window much more effectively.
• gpt-4o-mini showed little difference between 0.1 and 0.3. It already used the full context at 0.1. The API model appears less sensitive to temperature for this task.
• command-r7b at 0.3 took longer (94s vs 78s), likely due to generating more text.
• At temperature=0.3, both models converge on similar quality: 6 files cited, good coverage of the context window, mix of direct quotes and paraphrases.