furst/ssearch
0
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Index Markdown files in clippings

Browse source
This commit is contained in:
Eric Furst 2026-09-21 09:06:14 -04:00
commit 210e88275e
2 changed files with 18 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

4
README.md
View file

@ -50,7 +50,7 @@ ssearch/
│ ├── retrieve_clippings.py # Verbatim clippings chunk retrieval │ ├── retrieve_clippings.py # Verbatim clippings chunk retrieval
│ └── store_clippings/ # Persisted clippings vector store (ChromaDB) │ └── store_clippings/ # Persisted clippings vector store (ChromaDB)
├── data/ # Symlink to journal .txt files ├── data/ # Symlink to journal .txt files
├── clippings/ # Symlink to clippings (PDFs, TXT, webarchive, RTF) ├── clippings/ # Symlink to clippings (PDFs, TXT, Markdown, webarchive, RTF)
├── store/ # Persisted journal vector store ├── store/ # Persisted journal vector store
├── models/ # Cached HuggingFace models (offline) ├── models/ # Cached HuggingFace models (offline)
├── pyproject.toml # Project metadata and dependencies (uv) ├── pyproject.toml # Project metadata and dependencies (uv)
@ -118,7 +118,7 @@ uv run python clippings_search/build_clippings.py --rebuild
The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters, the embedding model, or the metadata policy change. The default incremental mode loads the existing index, compares file sizes and modification dates, and only re-indexes what changed. A full rebuild is only needed when chunk parameters, the embedding model, or the metadata policy change.
`build_clippings.py` handles PDFs, TXT, webarchive, and RTF files. PDFs are validated before indexing — those without extractable text are skipped and written to `ocr_needed.txt` for later OCR. `build_clippings.py` handles PDFs, TXT, Markdown, webarchive, and RTF files. PDFs are validated before indexing — those without extractable text are skipped and written to `ocr_needed.txt` for later OCR.
### Search journals ### Search journals

19
clippings_search/build_clippings.py
View file

@ -5,7 +5,7 @@
# Default mode (incremental): loads the existing index and adds only # Default mode (incremental): loads the existing index and adds only
# new or modified files. Use --rebuild for a full rebuild from scratch. # new or modified files. Use --rebuild for a full rebuild from scratch.
# #
# Handles PDFs, TXT, webarchive, and RTF files. Skips non-extractable PDFs # Handles PDFs, TXT, Markdown, webarchive, and RTF files. Skips non-extractable PDFs
# and writes them to ocr_needed.txt for later OCR processing. # and writes them to ocr_needed.txt for later OCR processing.
# #
# February 2026 # February 2026
@ -50,7 +50,9 @@ CHUNK_SIZE = 256
CHUNK_OVERLAP = 25 CHUNK_OVERLAP = 25
# File types handled by SimpleDirectoryReader (PDF + TXT) # File types handled by SimpleDirectoryReader (PDF + TXT)
READER_EXTS = {".pdf", ".txt"} # .md is read as plain text by SimpleDirectoryReader (one document, full
# text, standard metadata) -- no special Markdown handling needed.
READER_EXTS = {".pdf", ".txt", ".md"}
# File types handled by custom loaders # File types handled by custom loaders
CUSTOM_EXTS = {".webarchive", ".rtf"} CUSTOM_EXTS = {".webarchive", ".rtf"}
# All supported extensions # All supported extensions
@ -241,7 +243,13 @@ def scan_clippings():
continue continue
reader_files.append(fpath) reader_files.append(fpath)
elif ext == ".txt": elif ext in READER_EXTS:
# Everything else in READER_EXTS is plain text that
# SimpleDirectoryReader handles directly (.txt, .md). Keyed off the
# set rather than a literal so adding an extension there is enough;
# previously this was `elif ext == ".txt"`, so .md passed the
# SUPPORTED_EXTS guard and then fell through every branch below,
# silently indexing nothing and not even counting as skipped.
reader_files.append(fpath) reader_files.append(fpath)
elif ext == ".webarchive": elif ext == ".webarchive":
@ -258,6 +266,11 @@ def scan_clippings():
else: else:
skipped.append((fpath, "no extractable text from RTF")) skipped.append((fpath, "no extractable text from RTF"))
else:
# Reachable only if SUPPORTED_EXTS gains a type with no handler.
# Make that loud rather than silently dropping the file.
skipped.append((fpath, f"NO HANDLER for supported type {ext}"))
return reader_files, custom_docs, skipped, ocr_needed return reader_files, custom_docs, skipped, ocr_needed