Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
95 lines
5.1 KiB
Markdown
95 lines
5.1 KiB
Markdown
# CLAUDE.md -- pdf-tools
|
|
|
|
Small command-line utilities for marked-up PDFs and scanned handwritten
|
|
notes. Python 3.11+, managed with `uv`. User-facing documentation is in
|
|
`README.md`.
|
|
|
|
## Layout
|
|
|
|
```
|
|
src/pdf_tools/
|
|
scrub.py pdf-scrub
|
|
extract_pdf.py pdf-notes-md
|
|
extract_scan.py scan-notes-md: CLI, page rendering, Claude backend
|
|
ollama.py scan-notes-md Ollama backend: prompt, host selection
|
|
companion.py scan-notes-md companion writer, driven by CompanionSpec
|
|
scan_config.py scan-notes-md config file and defaults
|
|
scan_stats.py per-page stats log and scan-notes-stats summary
|
|
scan_index.py scan-notes-index: per-page metadata files from transcripts
|
|
common.py shared helpers
|
|
scripts/ shell wrappers (Quick Actions, batch-scan.sh, batch-index.sh)
|
|
docs/ findings, e.g. ollama-context-and-gpu.md
|
|
tests/ pytest; run with `uv run pytest`
|
|
```
|
|
|
|
## scan-notes-md rules
|
|
|
|
- **No cloud fallback.** `--backend ollama` must never call Claude or any
|
|
other cloud service, and `ollama.py` must not import `anthropic`. If no
|
|
Ollama host is usable, exit with an error (status 3). The tests fail if
|
|
Claude is called on an Ollama run; keep them that way.
|
|
- **Nothing personal in the code.** The package is meant to be published.
|
|
Folder paths, host names, organization names, title tables, and name
|
|
lists belong in the user's config file, never in `src/` or the defaults.
|
|
`scan-notes.example.toml` uses placeholder names.
|
|
- **Confidential folders.** Anything under a `[[confidential]]` root
|
|
defaults to Ollama, and Claude is refused without `--allow-cloud`. This
|
|
check is based on location, not on `--companion`. Keep the backend
|
|
decision in Python, not in the shell wrappers. Reject unknown config keys,
|
|
so a typo cannot silently drop a protection.
|
|
- **No stray copies.** Page images are rendered in memory. Nothing is
|
|
written outside the scan's folder except to the system temp directory, and
|
|
temp files are removed. Do not log transcript text or images.
|
|
- **Companion edits are narrow.** In an existing companion, change only
|
|
the `### Draft transcript (unverified)` block and the `transcribed_by:`
|
|
field. Layouts come from `[confidential.companion]` tables; add new
|
|
layout features as `CompanionSpec` options, not as named special cases.
|
|
- **One prompt.** The Ollama instruction is `HANDWRITING_PROMPT` in
|
|
`ollama.py`. `build_prompt` appends configured `prompt_extra` text (top
|
|
level, then per folder) and the name list. Keep domain guidance, such as
|
|
the drawing and equation rules, in config, not in the base prompt. The
|
|
Claude backend keeps its own `TRANSCRIBE_PROMPT` in `extract_scan.py`.
|
|
- **Record what we learn.** Context size, GPU memory, speed, and model
|
|
failures go in `docs/ollama-context-and-gpu.md` with a date, model,
|
|
GPU, and Ollama version. The stats log holds numbers and page labels
|
|
only, never transcript text.
|
|
- **Looping pages.** A `RepetitionError` page is retried with
|
|
`REPEAT_RETRIES` and then replaced by `REPEAT_PLACEHOLDER`; first
|
|
attempts stay at temperature 0.
|
|
- **Batch runs.** `scripts/batch-scan.sh` skips PDFs that have a `.md`.
|
|
`extract_scan` checkpoints after every page to a hidden
|
|
`.<stem>.scan-partial.json` next to the scan, keyed on inputs, model,
|
|
prompt, and dpi, and removes it once the output is written.
|
|
- **Config, not code.** Hosts, models, timeouts, folders, layouts, and
|
|
per-folder names live in `~/.config/pdf-tools/scan-notes.toml`; generic
|
|
defaults are in `scan_config.py`. Keep `scan-notes.example.toml` loading
|
|
(a test checks it).
|
|
Command-line flags override the file.
|
|
- **HTTP is standard library.** Ollama calls go through `ollama._http_json`
|
|
(`urllib`), which the tests replace. Do not add an HTTP or Ollama SDK.
|
|
|
|
## scan-notes-index rules
|
|
|
|
- **Ollama only.** It has no Claude backend and uses the same host
|
|
selection and no-fallback rule as `--backend ollama`.
|
|
- **Generated text is labeled.** Summaries and key points are interpretation,
|
|
unlike the transcript; page files mark them machine-generated and
|
|
unverified. The prompt tells the model never to add reference details
|
|
that are not on the page; keep that.
|
|
- **Prompt and schema.** `INDEX_PROMPT` and `schema()` in `scan_index.py`
|
|
are the base; names, `prompt_extra`, and `kinds` come from `[index]` in
|
|
the config. Answers are constrained with Ollama's `format` (JSON schema)
|
|
through `ollama.chat_json`, which retries loops, output cut off at the
|
|
cap, and invalid JSON.
|
|
- **Resume by file.** A page whose file exists is skipped unless `--force`.
|
|
A failed page is reported, the others are written, and the exit status is
|
|
4. The stats log gets `"task": "index"` records with counts only.
|
|
|
|
## Conventions
|
|
|
|
- Add dependencies with `uv add` (or `uv add --dev`); keep them few.
|
|
- Status and progress go to stderr; document output goes to stdout or files.
|
|
- US English spelling. No emojis in code, output, or documentation.
|
|
- Tests must not touch the network or real confidential files: fake
|
|
`_http_json`, use `tmp_path`, and declare temporary confidential folders
|
|
in a config file passed through `PDF_TOOLS_SCAN_CONFIG`.
|