Initial public release of pdf-tools

Command-line tools for marked-up PDFs and scanned handwritten notes:
pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on
request), scan-notes-index (per-page search metadata), and
scan-notes-stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
Eric Furst 2026-11-03 07:25:51 -05:00
commit e464871b69
29 changed files with 5296 additions and 0 deletions

95
CLAUDE.md Normal file
View file

@ -0,0 +1,95 @@
# CLAUDE.md -- pdf-tools
Small command-line utilities for marked-up PDFs and scanned handwritten
notes. Python 3.11+, managed with `uv`. User-facing documentation is in
`README.md`.
## Layout
```
src/pdf_tools/
scrub.py pdf-scrub
extract_pdf.py pdf-notes-md
extract_scan.py scan-notes-md: CLI, page rendering, Claude backend
ollama.py scan-notes-md Ollama backend: prompt, host selection
companion.py scan-notes-md companion writer, driven by CompanionSpec
scan_config.py scan-notes-md config file and defaults
scan_stats.py per-page stats log and scan-notes-stats summary
scan_index.py scan-notes-index: per-page metadata files from transcripts
common.py shared helpers
scripts/ shell wrappers (Quick Actions, batch-scan.sh, batch-index.sh)
docs/ findings, e.g. ollama-context-and-gpu.md
tests/ pytest; run with `uv run pytest`
```
## scan-notes-md rules
- **No cloud fallback.** `--backend ollama` must never call Claude or any
other cloud service, and `ollama.py` must not import `anthropic`. If no
Ollama host is usable, exit with an error (status 3). The tests fail if
Claude is called on an Ollama run; keep them that way.
- **Nothing personal in the code.** The package is meant to be published.
Folder paths, host names, organization names, title tables, and name
lists belong in the user's config file, never in `src/` or the defaults.
`scan-notes.example.toml` uses placeholder names.
- **Confidential folders.** Anything under a `[[confidential]]` root
defaults to Ollama, and Claude is refused without `--allow-cloud`. This
check is based on location, not on `--companion`. Keep the backend
decision in Python, not in the shell wrappers. Reject unknown config keys,
so a typo cannot silently drop a protection.
- **No stray copies.** Page images are rendered in memory. Nothing is
written outside the scan's folder except to the system temp directory, and
temp files are removed. Do not log transcript text or images.
- **Companion edits are narrow.** In an existing companion, change only
the `### Draft transcript (unverified)` block and the `transcribed_by:`
field. Layouts come from `[confidential.companion]` tables; add new
layout features as `CompanionSpec` options, not as named special cases.
- **One prompt.** The Ollama instruction is `HANDWRITING_PROMPT` in
`ollama.py`. `build_prompt` appends configured `prompt_extra` text (top
level, then per folder) and the name list. Keep domain guidance, such as
the drawing and equation rules, in config, not in the base prompt. The
Claude backend keeps its own `TRANSCRIBE_PROMPT` in `extract_scan.py`.
- **Record what we learn.** Context size, GPU memory, speed, and model
failures go in `docs/ollama-context-and-gpu.md` with a date, model,
GPU, and Ollama version. The stats log holds numbers and page labels
only, never transcript text.
- **Looping pages.** A `RepetitionError` page is retried with
`REPEAT_RETRIES` and then replaced by `REPEAT_PLACEHOLDER`; first
attempts stay at temperature 0.
- **Batch runs.** `scripts/batch-scan.sh` skips PDFs that have a `.md`.
`extract_scan` checkpoints after every page to a hidden
`.<stem>.scan-partial.json` next to the scan, keyed on inputs, model,
prompt, and dpi, and removes it once the output is written.
- **Config, not code.** Hosts, models, timeouts, folders, layouts, and
per-folder names live in `~/.config/pdf-tools/scan-notes.toml`; generic
defaults are in `scan_config.py`. Keep `scan-notes.example.toml` loading
(a test checks it).
Command-line flags override the file.
- **HTTP is standard library.** Ollama calls go through `ollama._http_json`
(`urllib`), which the tests replace. Do not add an HTTP or Ollama SDK.
## scan-notes-index rules
- **Ollama only.** It has no Claude backend and uses the same host
selection and no-fallback rule as `--backend ollama`.
- **Generated text is labeled.** Summaries and key points are interpretation,
unlike the transcript; page files mark them machine-generated and
unverified. The prompt tells the model never to add reference details
that are not on the page; keep that.
- **Prompt and schema.** `INDEX_PROMPT` and `schema()` in `scan_index.py`
are the base; names, `prompt_extra`, and `kinds` come from `[index]` in
the config. Answers are constrained with Ollama's `format` (JSON schema)
through `ollama.chat_json`, which retries loops, output cut off at the
cap, and invalid JSON.
- **Resume by file.** A page whose file exists is skipped unless `--force`.
A failed page is reported, the others are written, and the exit status is
4. The stats log gets `"task": "index"` records with counts only.
## Conventions
- Add dependencies with `uv add` (or `uv add --dev`); keep them few.
- Status and progress go to stderr; document output goes to stdout or files.
- US English spelling. No emojis in code, output, or documentation.
- Tests must not touch the network or real confidential files: fake
`_http_json`, use `tmp_path`, and declare temporary confidential folders
in a config file passed through `PDF_TOOLS_SCAN_CONFIG`.