Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
commit
e464871b69
29 changed files with 5296 additions and 0 deletions
95
CLAUDE.md
Normal file
95
CLAUDE.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
# CLAUDE.md -- pdf-tools
|
||||
|
||||
Small command-line utilities for marked-up PDFs and scanned handwritten
|
||||
notes. Python 3.11+, managed with `uv`. User-facing documentation is in
|
||||
`README.md`.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
src/pdf_tools/
|
||||
scrub.py pdf-scrub
|
||||
extract_pdf.py pdf-notes-md
|
||||
extract_scan.py scan-notes-md: CLI, page rendering, Claude backend
|
||||
ollama.py scan-notes-md Ollama backend: prompt, host selection
|
||||
companion.py scan-notes-md companion writer, driven by CompanionSpec
|
||||
scan_config.py scan-notes-md config file and defaults
|
||||
scan_stats.py per-page stats log and scan-notes-stats summary
|
||||
scan_index.py scan-notes-index: per-page metadata files from transcripts
|
||||
common.py shared helpers
|
||||
scripts/ shell wrappers (Quick Actions, batch-scan.sh, batch-index.sh)
|
||||
docs/ findings, e.g. ollama-context-and-gpu.md
|
||||
tests/ pytest; run with `uv run pytest`
|
||||
```
|
||||
|
||||
## scan-notes-md rules
|
||||
|
||||
- **No cloud fallback.** `--backend ollama` must never call Claude or any
|
||||
other cloud service, and `ollama.py` must not import `anthropic`. If no
|
||||
Ollama host is usable, exit with an error (status 3). The tests fail if
|
||||
Claude is called on an Ollama run; keep them that way.
|
||||
- **Nothing personal in the code.** The package is meant to be published.
|
||||
Folder paths, host names, organization names, title tables, and name
|
||||
lists belong in the user's config file, never in `src/` or the defaults.
|
||||
`scan-notes.example.toml` uses placeholder names.
|
||||
- **Confidential folders.** Anything under a `[[confidential]]` root
|
||||
defaults to Ollama, and Claude is refused without `--allow-cloud`. This
|
||||
check is based on location, not on `--companion`. Keep the backend
|
||||
decision in Python, not in the shell wrappers. Reject unknown config keys,
|
||||
so a typo cannot silently drop a protection.
|
||||
- **No stray copies.** Page images are rendered in memory. Nothing is
|
||||
written outside the scan's folder except to the system temp directory, and
|
||||
temp files are removed. Do not log transcript text or images.
|
||||
- **Companion edits are narrow.** In an existing companion, change only
|
||||
the `### Draft transcript (unverified)` block and the `transcribed_by:`
|
||||
field. Layouts come from `[confidential.companion]` tables; add new
|
||||
layout features as `CompanionSpec` options, not as named special cases.
|
||||
- **One prompt.** The Ollama instruction is `HANDWRITING_PROMPT` in
|
||||
`ollama.py`. `build_prompt` appends configured `prompt_extra` text (top
|
||||
level, then per folder) and the name list. Keep domain guidance, such as
|
||||
the drawing and equation rules, in config, not in the base prompt. The
|
||||
Claude backend keeps its own `TRANSCRIBE_PROMPT` in `extract_scan.py`.
|
||||
- **Record what we learn.** Context size, GPU memory, speed, and model
|
||||
failures go in `docs/ollama-context-and-gpu.md` with a date, model,
|
||||
GPU, and Ollama version. The stats log holds numbers and page labels
|
||||
only, never transcript text.
|
||||
- **Looping pages.** A `RepetitionError` page is retried with
|
||||
`REPEAT_RETRIES` and then replaced by `REPEAT_PLACEHOLDER`; first
|
||||
attempts stay at temperature 0.
|
||||
- **Batch runs.** `scripts/batch-scan.sh` skips PDFs that have a `.md`.
|
||||
`extract_scan` checkpoints after every page to a hidden
|
||||
`.<stem>.scan-partial.json` next to the scan, keyed on inputs, model,
|
||||
prompt, and dpi, and removes it once the output is written.
|
||||
- **Config, not code.** Hosts, models, timeouts, folders, layouts, and
|
||||
per-folder names live in `~/.config/pdf-tools/scan-notes.toml`; generic
|
||||
defaults are in `scan_config.py`. Keep `scan-notes.example.toml` loading
|
||||
(a test checks it).
|
||||
Command-line flags override the file.
|
||||
- **HTTP is standard library.** Ollama calls go through `ollama._http_json`
|
||||
(`urllib`), which the tests replace. Do not add an HTTP or Ollama SDK.
|
||||
|
||||
## scan-notes-index rules
|
||||
|
||||
- **Ollama only.** It has no Claude backend and uses the same host
|
||||
selection and no-fallback rule as `--backend ollama`.
|
||||
- **Generated text is labeled.** Summaries and key points are interpretation,
|
||||
unlike the transcript; page files mark them machine-generated and
|
||||
unverified. The prompt tells the model never to add reference details
|
||||
that are not on the page; keep that.
|
||||
- **Prompt and schema.** `INDEX_PROMPT` and `schema()` in `scan_index.py`
|
||||
are the base; names, `prompt_extra`, and `kinds` come from `[index]` in
|
||||
the config. Answers are constrained with Ollama's `format` (JSON schema)
|
||||
through `ollama.chat_json`, which retries loops, output cut off at the
|
||||
cap, and invalid JSON.
|
||||
- **Resume by file.** A page whose file exists is skipped unless `--force`.
|
||||
A failed page is reported, the others are written, and the exit status is
|
||||
4. The stats log gets `"task": "index"` records with counts only.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Add dependencies with `uv add` (or `uv add --dev`); keep them few.
|
||||
- Status and progress go to stderr; document output goes to stdout or files.
|
||||
- US English spelling. No emojis in code, output, or documentation.
|
||||
- Tests must not touch the network or real confidential files: fake
|
||||
`_http_json`, use `tmp_path`, and declare temporary confidential folders
|
||||
in a config file passed through `PDF_TOOLS_SCAN_CONFIG`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue