diff --git a/README.md b/README.md index 0499f9a..a9362b4 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ Semantic search over a personal journal archive and a collection of clippings. U ## How it works ``` -Query → Embed (BAAI/bge-large-en-v1.5) → Vector similarity (top-30) → Cross-encoder re-rank (top-15) → LLM synthesis (command-r7b via Ollama, or OpenAI API) → Response + sources +Query → Embed (BAAI/bge-large-en-v1.5) → Vector similarity (top-30) → Cross-encoder re-rank (top-15) → LLM synthesis (Gemma4 via Ollama, or OpenAI API) → Response + sources ``` 1. **Build**: Source files are chunked (256 tokens, 25-token overlap) and embedded into a vector store using LlamaIndex. The journal index uses LlamaIndex's JSON store; the clippings index uses ChromaDB. Both support incremental updates. @@ -35,7 +35,7 @@ ssearch/ ## Setup -**Prerequisites**: Python 3.12, [Ollama](https://ollama.com) with `command-r7b` pulled. +**Prerequisites**: Python 3.12, [Ollama](https://ollama.com) with `gemma4:e4b` or similar pulled. ```bash cd ssearch @@ -90,7 +90,7 @@ The default incremental mode loads the existing index, compares file sizes and m #### Semantic search with LLM synthesis -**Requires Ollama running with `command-r7b`.** +**Requires Ollama running with `gemma4`.** **Hybrid BM25 + vector** (`query_hybrid.py`): Retrieves top 20 by vector similarity and top 20 by BM25 term frequency, merges and deduplicates, re-ranks the union to top 15, synthesizes. Catches exact name/term matches that vector-only retrieval misses. ```bash @@ -150,7 +150,7 @@ Key parameters (set in source files): | Initial retrieval | 30 chunks | query and retrieve scripts | | Re-rank model | `cross-encoder/ms-marco-MiniLM-L-12-v2` | query and retrieve scripts | | Re-rank top-n | 15 | query and retrieve scripts | -| LLM | `command-r7b` (Ollama) or `gpt-4o-mini` (OpenAI API) | `query_hybrid.py` | +| LLM | `gemma4:e4b` (Ollama) or `gpt-4o-mini` (OpenAI API) | `query_hybrid.py` | | Temperature | 0.3 | `query_hybrid.py` | | Context window | 8000 tokens | `query_hybrid.py` | | Request timeout | 360 seconds | `query_hybrid.py` | @@ -171,7 +171,7 @@ Key parameters (set in source files): - **BAAI/bge-large-en-v1.5 over all-mpnet-base-v2**: Better semantic matching quality for journal text despite slower embedding. - **256-token chunks**: Tested 512 and 384; 256 with 25-token overlap produced the highest quality matches. -- **command-r7b over llama3.1:8B**: Sticks closer to provided context with less hallucination at comparable speed. +- **gemma4:e4b over command-r7b**: Sticks closer to provided context with less hallucination at comparable speed. Earlier, selected **command-r7b over llama3.1:8B** for similar reasons. - **Cross-encoder re-ranking**: Retrieve top-30 via bi-encoder, re-rank to top-15 with a cross-encoder that scores each (query, chunk) pair jointly. Tested three models; `ms-marco-MiniLM-L-12-v2` selected over `stsb-roberta-base` (wrong task) and `BAAI/bge-reranker-v2-m3` (slower, weak score tail). - **HyDE query rewriting tested and dropped**: Did not improve results over direct prompt engineering. - **Hybrid BM25 + vector retrieval**: BM25 nominates candidates with exact term matches that embeddings miss; the cross-encoder decides final relevance. diff --git a/query_hybrid.py b/query_hybrid.py index e32d942..62175fa 100644 --- a/query_hybrid.py +++ b/query_hybrid.py @@ -43,8 +43,10 @@ import sys # Embedding model (must match build_store.py) EMBED_MODEL = HuggingFaceEmbedding(cache_folder="./models", model_name="BAAI/bge-large-en-v1.5", local_files_only=True) -# LLM model for generation -LLM_MODEL = "command-r7b" +# LLM model for generation. Use temp 0.3. +#LLM_MODEL = "command-r7b" +# testing gemma4:e4b. Recommendations are to use temp 1.0 +LLM_MODEL = "gemma4:e4b" # Cross-encoder model for re-ranking (cached in ./models/) RERANK_MODEL = "cross-encoder/ms-marco-MiniLM-L-12-v2" @@ -89,7 +91,8 @@ def main(): # Note: Ollama temperature defaults to 0.8 Settings.llm = Ollama( model=LLM_MODEL, - temperature=0.3, + temperature=1.0, + thinking=True, # enable request_timeout=360.0, context_window=8000, ) @@ -153,9 +156,20 @@ def main(): n_bm25_only = len([n for n in bm25_nodes if n.node.node_id not in {v.node.node_id for v in vector_nodes}]) n_both = len(vector_nodes) + len(bm25_nodes) - len(merged) + # Estimate context length (prompt + node text) + context_text = "\n\n".join(n.get_content() for n in reranked) + prompt_text = PROMPT.format(context_str=context_text, query_str=q) + n_context_tokens = len(prompt_text.split()) # rough word count; ~1.3 tokens/word + print(f"\nQuery: {q}") - print(f"Vector: {len(vector_nodes)}, BM25: {len(bm25_nodes)}, " + print(f"Vector: {len(vector_nodes)} ({n_vector_only} unique), " + f"BM25: {len(bm25_nodes)} ({n_bm25_only} unique), " f"overlap: {n_both}, merged: {len(merged)}, re-ranked to: {len(reranked)}") + + # The token estimate uses a ~1.3 tokens/word ratio, which is a rough approximation. + # For an exact count you'd need the model's tokenizer, but this gives a useful ballpark + # for gauging how much of the context window we use. + print(f"Context: ~{n_context_tokens} words (~{int(n_context_tokens * 1.3)} tokens)") # Synthesize response with LLM synthesizer = get_response_synthesizer(text_qa_template=PROMPT) @@ -169,7 +183,7 @@ def main(): for node in response.source_nodes: meta = getattr(node, "metadata", None) or node.node.metadata score = getattr(node, "score", None) - print(f"{meta.get('file_name')} {meta.get('file_path')} {score:.3f}") + print(f"data/{meta.get('file_name')} {score:.3f}") if __name__ == "__main__": diff --git a/run_query.sh b/run_query.sh index 674b79f..7a70b77 100755 --- a/run_query.sh +++ b/run_query.sh @@ -7,6 +7,17 @@ # Usage: ./run_query.sh QUERY_SCRIPT="query_hybrid.py" +VENV_DIR=".venv" + +# Activate the virtual environment +if [ -d "$VENV_DIR" ]; then + source "$VENV_DIR/bin/activate" + echo "Activated virtual environment: $VENV_DIR" +else + echo "Error: Virtual environment not found at '$VENV_DIR'" >&2 + echo "Create one with: python3 -m venv $VENV_DIR" >&2 + exit 1 +fi echo -e "Current query engine is $QUERY_SCRIPT\n" diff --git a/run_retrieve.sh b/run_retrieve.sh index c56d584..70805ad 100755 --- a/run_retrieve.sh +++ b/run_retrieve.sh @@ -7,6 +7,17 @@ # Usage: ./run_query.sh QUERY_SCRIPT="retrieve.py" +VENV_DIR=".venv" + +# Activate the virtual environment +if [ -d "$VENV_DIR" ]; then + source "$VENV_DIR/bin/activate" + echo "Activated virtual environment: $VENV_DIR" +else + echo "Error: Virtual environment not found at '$VENV_DIR'" >&2 + echo "Create one with: python3 -m venv $VENV_DIR" >&2 + exit 1 +fi echo -e "$QUERY_SCRIPT -- retrieve vector store chunks based on similaity + BM25 with reranking.\n"