Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
commit
e464871b69
29 changed files with 5296 additions and 0 deletions
293
README.md
Normal file
293
README.md
Normal file
|
|
@ -0,0 +1,293 @@
|
|||
# pdf-tools
|
||||
|
||||
Three small command-line utilities for working with marked-up PDFs and
|
||||
scanned notebook pages on macOS.
|
||||
|
||||
1. **`pdf-scrub`** — strip annotations from a PDF (highlights, sticky
|
||||
notes, freehand, text boxes) so you can send a clean copy.
|
||||
2. **`pdf-notes-md`** — extract your notes plus the highlighted passages
|
||||
they refer to, write as Markdown for downstream pipelines.
|
||||
3. **`scan-notes-md`** — transcribe scanned handwritten notebook pages
|
||||
to Markdown via a vision-language model: a local model through Ollama
|
||||
by default, or Claude through the Anthropic API when asked. Scans in
|
||||
confidential folders always stay on local machines.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
git clone <repo> pdf-tools
|
||||
cd pdf-tools
|
||||
uv sync
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
After this, `pdf-scrub`, `pdf-notes-md`, and `scan-notes-md` are on
|
||||
your `PATH` inside the project's venv. To use them from anywhere, either
|
||||
activate the venv or use the shell wrappers in `scripts/`.
|
||||
|
||||
`scan-notes-md` needs `ANTHROPIC_API_KEY` in the environment for the
|
||||
Claude backend only. The Ollama backend needs no key and no extra packages.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Tool 1
|
||||
pdf-scrub paper.pdf # -> paper_clean.pdf
|
||||
pdf-scrub paper.pdf -o out.pdf
|
||||
pdf-scrub paper.pdf --include-forms # also drop form widgets
|
||||
|
||||
# Tool 2
|
||||
pdf-notes-md paper.pdf # -> paper.md
|
||||
pdf-notes-md paper.pdf --stdout | less
|
||||
|
||||
# Tool 3
|
||||
scan-notes-md page1.jpg page2.jpg # -> page1.md
|
||||
scan-notes-md notebook.pdf -o out.md
|
||||
scan-notes-md --backend ollama notebook.pdf # local model
|
||||
scan-notes-md --backend ollama --host local scan.pdf # laptop only
|
||||
scan-notes-md ~/confidential/.../2026-10-29-notes.pdf
|
||||
# confidential folder: ollama + companion
|
||||
```
|
||||
|
||||
## scan-notes-md backends
|
||||
|
||||
**Claude** (`--backend claude`, or `backend = "claude"` in the config) calls
|
||||
the Anthropic API, one request per page, with `ANTHROPIC_API_KEY` and model
|
||||
`claude-opus-5-5` unless `--model` says otherwise. Output is Markdown with one
|
||||
`##` section per page. It is never used for scans in confidential folders
|
||||
unless `--allow-cloud` is given.
|
||||
|
||||
**Ollama** (`--backend ollama`, the default) sends each page, rendered to PNG in memory,
|
||||
to a vision model on a machine you own. Hosts are tried in the order listed
|
||||
in the config file. Each is checked first with a 2.5 s request to
|
||||
`/api/tags`. A host that is unreachable or lacks its model is skipped with
|
||||
one line of output. `--host remote` or `--host local` uses only that host
|
||||
and fails if it is unavailable. If no host works, the tool exits with
|
||||
status 3. It never falls back to Claude or any other cloud service.
|
||||
|
||||
Each request uses `stream: false`, `think: false`, and temperature 0. Any
|
||||
thinking text that leaks through is stripped. The instruction is
|
||||
`HANDWRITING_PROMPT` in `src/pdf_tools/ollama.py`. Pages are joined with
|
||||
`<!-- page N -->` markers, each page in a `text` code fence so that line
|
||||
breaks and indentation survive Markdown rendering.
|
||||
|
||||
Hosts and models come from the config file. Without one, the only host is
|
||||
`http://localhost:11434` with `gemma4:e4b`. In testing on handwritten board
|
||||
notes, `gemma4:31b` on a GPU workstation gave the most reliable drafts;
|
||||
`gemma4:e4b` on a laptop gives a rough draft to correct.
|
||||
|
||||
A remote Ollama host must listen on a reachable interface (for example
|
||||
`OLLAMA_HOST=0.0.0.0` in its service environment, reached over Tailscale),
|
||||
or the health check fails and every run falls back to the next host.
|
||||
|
||||
### Config file
|
||||
|
||||
`~/.config/pdf-tools/scan-notes.toml` (or `$PDF_TOOLS_SCAN_CONFIG`, or
|
||||
`--config`) sets the default backend, `dpi`, Ollama timeouts, the host list,
|
||||
and the confidential folders. Flags override it. Start from
|
||||
`scan-notes.example.toml`, which is commented throughout. Unknown keys are
|
||||
reported as errors, so a typo cannot silently disable a setting. In TOML, a
|
||||
key written after a `[table]` header belongs to that table, so top-level
|
||||
keys must come before the first one; a misplaced key is reported.
|
||||
|
||||
`prompt_extra` adds text to the Ollama prompt after the base instruction.
|
||||
The example file uses it for notes with drawings, arrows, and equations:
|
||||
drawings become one `[FIGURE: description]` line, arrows become `->` or an
|
||||
`[ARROW: from -> to]` line, and equations are copied in plain text with
|
||||
Unicode symbols. A confidential folder can add its own `prompt_extra`,
|
||||
which follows the top-level one. Changing the prompt starts interrupted
|
||||
runs over rather than resuming them, so pages are not mixed.
|
||||
|
||||
### Confidential folders and companion files
|
||||
|
||||
No folder is confidential unless the config file says so. Each folder gets
|
||||
one `[[confidential]]` block:
|
||||
|
||||
```toml
|
||||
[[confidential]]
|
||||
root = "~/Documents/personnel"
|
||||
names = ["Pat Example", "Facilities Office"]
|
||||
```
|
||||
|
||||
When folders are nested, the deepest one applies. For a scan inside any
|
||||
confidential folder:
|
||||
|
||||
- The backend defaults to `ollama`. `--backend claude` is refused unless
|
||||
`--allow-cloud` is also given, even with `--no-companion`.
|
||||
- The folder's `prompt_extra` text, if any, is added to the Ollama prompt.
|
||||
- The folder's `names` are added to the Ollama prompt: if a word plausibly
|
||||
matches one of them, the model uses that spelling, and otherwise writes
|
||||
`[illegible]`. Names are per folder, so a long list for one folder does
|
||||
not lengthen the prompt or steer the reading for others. A top-level
|
||||
`names` list applies to every Ollama run. Names are never sent to Claude.
|
||||
- Output goes to the companion `<stem>.md` next to the scan.
|
||||
|
||||
**Companion layout.** Without a `[confidential.companion]` table, a new
|
||||
companion has `date`, `type: notes`, `scan:`, `subjects: []`, and
|
||||
`transcribed_by:` in its front matter, a title from the filename, and one
|
||||
Notes section. A `[confidential.companion]` table describes any other
|
||||
layout:
|
||||
|
||||
| Key | Meaning |
|
||||
|---|---|
|
||||
| `type` | value of the `type:` field |
|
||||
| `sections` | `##` sections, in order |
|
||||
| `transcript_section` | section that holds the draft transcript (default: the first) |
|
||||
| `section_text` | starting text for named sections |
|
||||
| `fields` | front-matter fields read from the path, after `date` |
|
||||
| `paths` | rules that read those fields from folder names (below) |
|
||||
| `allowed` | permitted values for a captured field |
|
||||
| `title` | title template: `{date}`, `{stem}`, `{field}`, `{field.short}` |
|
||||
| `acronyms` | words kept in capitals when folder names become title words |
|
||||
|
||||
Each `[[confidential.companion.paths]]` rule has a `match` pattern over the
|
||||
folders below the root. A pattern segment is a literal folder name,
|
||||
`{field}` to capture one folder, or a trailing `**` for any remaining
|
||||
folders. Rules are tried in order, and the first match wins. A rule can add
|
||||
fixed values with `set` and a different `title`. In titles, `{field}` is
|
||||
replaced by the `[confidential.titles]` entry for the captured folder name,
|
||||
or by the name turned into words. `{field.short}` uses the
|
||||
`"<name>.short"` entry if there is one. `scan-notes.example.toml` shows a
|
||||
complete meeting-notes layout for board-service folders laid out as
|
||||
`<BODY>/committees/<committee>/<meeting>/`.
|
||||
|
||||
**Transcript placement.**
|
||||
|
||||
- The transcript goes in the transcript section under
|
||||
`### Draft transcript (unverified)`, ending with
|
||||
`<!-- end draft transcript -->`. It replaces `Transcript pending; see
|
||||
scan.` if present.
|
||||
- The date comes from the filename, or else from the nearest dated folder.
|
||||
- In an existing companion, only that block and the `transcribed_by:`
|
||||
front-matter field are changed. A rerun replaces the block, including any
|
||||
edits made inside it, so move corrections out of the block first.
|
||||
- `transcribed_by:` records the source, such as
|
||||
`ollama/localhost/gemma4:e4b`.
|
||||
- `--companion` or `--no-companion` overrides the automatic choice.
|
||||
Companion mode takes one PDF and cannot be combined with `-o` or
|
||||
`--stdout`. A run cannot mix confidential scans with other files.
|
||||
|
||||
Page images and transcripts are never written anywhere except the
|
||||
companion file. The companion is written through a temporary file in the
|
||||
same folder, which is renamed into place.
|
||||
|
||||
### Context size, GPU memory, and run statistics
|
||||
|
||||
Each page transcribed through Ollama appends one line of statistics to
|
||||
`~/.local/state/pdf-tools/scan-stats.jsonl` (config key `stats_log`; `""`
|
||||
turns it off): page label, image size, prompt and output tokens, stop
|
||||
reason, timings, and how much of the model sat in GPU memory. No transcript
|
||||
text is logged. Page labels include file names, so turn the log off if file
|
||||
names are themselves sensitive. `scan-notes-stats` summarizes the log,
|
||||
including how many pages would exceed a given `num_ctx`.
|
||||
|
||||
Some pages send a model into a loop. Ollama either aborts the request with
|
||||
"token repeat limit reached" or the output runs on until
|
||||
`max_output_tokens` (`[ollama]` table, default 2048). Either way the page is
|
||||
retried twice with light sampling and a repeat penalty. A page that still
|
||||
trips the repeat check is written as
|
||||
`[page not transcribed: the model kept repeating itself; see scan]`; one
|
||||
that still hits the output cap keeps its text, marked
|
||||
`[transcript cut off at the output limit; see scan]`.
|
||||
|
||||
What we have measured so far, and how to choose `num_ctx`, is in
|
||||
`docs/ollama-context-and-gpu.md`.
|
||||
|
||||
### Batch runs
|
||||
|
||||
`scripts/batch-scan.sh <folder>` transcribes every PDF in a folder with the
|
||||
Ollama backend, skipping PDFs that already have a `.md`. Progress is saved
|
||||
after each page, so rerunning the script resumes an interrupted PDF.
|
||||
|
||||
## scan-notes-index: per-page search metadata
|
||||
|
||||
A transcript is searchable for words you wrote, but not for what a page is
|
||||
about: a derivation written mostly in symbols, a sketch, or a meeting page
|
||||
whose topic is never named. `scan-notes-index` reads a transcribed scan and
|
||||
writes one Markdown file per page with a machine-generated description:
|
||||
|
||||
```bash
|
||||
scan-notes-index "2009 notebook.pdf" # reads "2009 notebook.md"
|
||||
# -> "2009 notebook pages/2009 notebook p001.md", p002.md, ...
|
||||
scripts/batch-index.sh ~/notebooks # every transcribed PDF, newest first
|
||||
```
|
||||
|
||||
Each page file has front matter (`kind`, `topics`, `references`, `event`,
|
||||
`event_date`, `people`, `continues_from`, `continues_to`, `indexed_by`), a
|
||||
summary paragraph and a list of key points marked as machine-generated and
|
||||
unverified, the references written on the page, and the page's transcript.
|
||||
One file per page means Spotlight or DEVONthink returns the page itself
|
||||
rather than the whole notebook, and the files are ready to embed for vector
|
||||
search later.
|
||||
|
||||
The model sees the page image, its transcript, and the transcripts of the
|
||||
previous and next pages, so calculations and notes that run across pages are
|
||||
described as such. It runs only on the configured Ollama hosts, like
|
||||
`--backend ollama`; there is no cloud option. Existing page files are kept,
|
||||
so an interrupted run resumes; `--force` rewrites them. Settings are in the
|
||||
`[index]` table of the config file: extra `names` for the prompt (added to
|
||||
the transcription names), `prompt_extra`, the list of `kinds`,
|
||||
`max_output_tokens` (default 1536), and `context_chars`, how much of each
|
||||
neighboring page to include (default 2000). Statistics go to the same stats
|
||||
log with `"task": "index"`, and `scan-notes-stats` reports them separately.
|
||||
|
||||
On gemma4:31b with `num_ctx = 8192` on an RTX 4090, a page takes about 11 s
|
||||
and uses at most about 3,800 tokens of context.
|
||||
|
||||
### Tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
The tests fake all HTTP calls and fail if Claude is called.
|
||||
|
||||
## macOS integration
|
||||
|
||||
### "Scrub & Mail" Quick Action
|
||||
|
||||
The `scripts/scrub-and-mail.sh` wrapper runs the scrubber, then opens a
|
||||
new Mail.app compose window with the cleaned PDF attached. To wire it
|
||||
up as a right-click Finder action:
|
||||
|
||||
1. Open **Automator** → New Document → **Quick Action**.
|
||||
2. "Workflow receives current" → **PDF files** in **Finder**.
|
||||
3. Add a **Run Shell Script** action.
|
||||
4. "Pass input:" **as arguments**.
|
||||
5. Paste:
|
||||
|
||||
```bash
|
||||
export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
|
||||
/path/to/pdf-tools/scripts/scrub-and-mail.sh "$1"
|
||||
```
|
||||
|
||||
6. Save as "Scrub & Mail".
|
||||
|
||||
It will appear under Finder right-click → Quick Actions.
|
||||
|
||||
You can do the same for the other two tools (`pdf-notes-to-md.sh`,
|
||||
`scan-to-md.sh`); they just save Markdown next to the input rather than
|
||||
opening another app. For `scan-to-md.sh`, `scan-notes-md` chooses the
|
||||
backend itself: Ollama for confidential folders, Claude elsewhere.
|
||||
|
||||
### Shortcuts (alternative)
|
||||
|
||||
If you prefer Shortcuts.app over Automator, create a new shortcut:
|
||||
- Receive **PDF files** from **Quick Actions**.
|
||||
- "Run Shell Script" with the same command.
|
||||
|
||||
## Notes
|
||||
|
||||
- The scrubber removes annotations from the `/Annots` arrays and clears
|
||||
some metadata fields (`dc:creator`, `xmp:CreatorTool`, `pdf:Producer`)
|
||||
that often leak the reviewing app or user identity. Form field
|
||||
widgets are kept unless `--include-forms` is passed.
|
||||
- The scrubber will not remove markings that have been *flattened* into
|
||||
the page content stream by a previous export step. If your PDF
|
||||
workflow ever does that, retain the unflattened original.
|
||||
- The Claude call lives in `_transcribe_image` in `extract_scan.py`; the
|
||||
Ollama call lives in `transcribe_png` in `ollama.py`.
|
||||
|
||||
## License
|
||||
|
||||
MIT; see `LICENSE`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue