pdf-tools/README.md
Eric Furst e464871b69 Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes:
pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on
request), scan-notes-index (per-page search metadata), and
scan-notes-stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
2026-11-03 07:25:51 -05:00

13 KiB

pdf-tools

Three small command-line utilities for working with marked-up PDFs and scanned notebook pages on macOS.

  1. pdf-scrub — strip annotations from a PDF (highlights, sticky notes, freehand, text boxes) so you can send a clean copy.
  2. pdf-notes-md — extract your notes plus the highlighted passages they refer to, write as Markdown for downstream pipelines.
  3. scan-notes-md — transcribe scanned handwritten notebook pages to Markdown via a vision-language model: a local model through Ollama by default, or Claude through the Anthropic API when asked. Scans in confidential folders always stay on local machines.

Install

git clone <repo> pdf-tools
cd pdf-tools
uv sync
uv pip install -e .

After this, pdf-scrub, pdf-notes-md, and scan-notes-md are on your PATH inside the project's venv. To use them from anywhere, either activate the venv or use the shell wrappers in scripts/.

scan-notes-md needs ANTHROPIC_API_KEY in the environment for the Claude backend only. The Ollama backend needs no key and no extra packages.

Usage

# Tool 1
pdf-scrub paper.pdf                   # -> paper_clean.pdf
pdf-scrub paper.pdf -o out.pdf
pdf-scrub paper.pdf --include-forms   # also drop form widgets

# Tool 2
pdf-notes-md paper.pdf                # -> paper.md
pdf-notes-md paper.pdf --stdout | less

# Tool 3
scan-notes-md page1.jpg page2.jpg     # -> page1.md
scan-notes-md notebook.pdf -o out.md
scan-notes-md --backend ollama notebook.pdf          # local model
scan-notes-md --backend ollama --host local scan.pdf # laptop only
scan-notes-md ~/confidential/.../2026-10-29-notes.pdf
                                      # confidential folder: ollama + companion

scan-notes-md backends

Claude (--backend claude, or backend = "claude" in the config) calls the Anthropic API, one request per page, with ANTHROPIC_API_KEY and model claude-opus-5-5 unless --model says otherwise. Output is Markdown with one ## section per page. It is never used for scans in confidential folders unless --allow-cloud is given.

Ollama (--backend ollama, the default) sends each page, rendered to PNG in memory, to a vision model on a machine you own. Hosts are tried in the order listed in the config file. Each is checked first with a 2.5 s request to /api/tags. A host that is unreachable or lacks its model is skipped with one line of output. --host remote or --host local uses only that host and fails if it is unavailable. If no host works, the tool exits with status 3. It never falls back to Claude or any other cloud service.

Each request uses stream: false, think: false, and temperature 0. Any thinking text that leaks through is stripped. The instruction is HANDWRITING_PROMPT in src/pdf_tools/ollama.py. Pages are joined with <!-- page N --> markers, each page in a text code fence so that line breaks and indentation survive Markdown rendering.

Hosts and models come from the config file. Without one, the only host is http://localhost:11434 with gemma4:e4b. In testing on handwritten board notes, gemma4:31b on a GPU workstation gave the most reliable drafts; gemma4:e4b on a laptop gives a rough draft to correct.

A remote Ollama host must listen on a reachable interface (for example OLLAMA_HOST=0.0.0.0 in its service environment, reached over Tailscale), or the health check fails and every run falls back to the next host.

Config file

~/.config/pdf-tools/scan-notes.toml (or $PDF_TOOLS_SCAN_CONFIG, or --config) sets the default backend, dpi, Ollama timeouts, the host list, and the confidential folders. Flags override it. Start from scan-notes.example.toml, which is commented throughout. Unknown keys are reported as errors, so a typo cannot silently disable a setting. In TOML, a key written after a [table] header belongs to that table, so top-level keys must come before the first one; a misplaced key is reported.

prompt_extra adds text to the Ollama prompt after the base instruction. The example file uses it for notes with drawings, arrows, and equations: drawings become one [FIGURE: description] line, arrows become -> or an [ARROW: from -> to] line, and equations are copied in plain text with Unicode symbols. A confidential folder can add its own prompt_extra, which follows the top-level one. Changing the prompt starts interrupted runs over rather than resuming them, so pages are not mixed.

Confidential folders and companion files

No folder is confidential unless the config file says so. Each folder gets one [[confidential]] block:

[[confidential]]
root = "~/Documents/personnel"
names = ["Pat Example", "Facilities Office"]

When folders are nested, the deepest one applies. For a scan inside any confidential folder:

  • The backend defaults to ollama. --backend claude is refused unless --allow-cloud is also given, even with --no-companion.
  • The folder's prompt_extra text, if any, is added to the Ollama prompt.
  • The folder's names are added to the Ollama prompt: if a word plausibly matches one of them, the model uses that spelling, and otherwise writes [illegible]. Names are per folder, so a long list for one folder does not lengthen the prompt or steer the reading for others. A top-level names list applies to every Ollama run. Names are never sent to Claude.
  • Output goes to the companion <stem>.md next to the scan.

Companion layout. Without a [confidential.companion] table, a new companion has date, type: notes, scan:, subjects: [], and transcribed_by: in its front matter, a title from the filename, and one Notes section. A [confidential.companion] table describes any other layout:

Key Meaning
type value of the type: field
sections ## sections, in order
transcript_section section that holds the draft transcript (default: the first)
section_text starting text for named sections
fields front-matter fields read from the path, after date
paths rules that read those fields from folder names (below)
allowed permitted values for a captured field
title title template: {date}, {stem}, {field}, {field.short}
acronyms words kept in capitals when folder names become title words

Each [[confidential.companion.paths]] rule has a match pattern over the folders below the root. A pattern segment is a literal folder name, {field} to capture one folder, or a trailing ** for any remaining folders. Rules are tried in order, and the first match wins. A rule can add fixed values with set and a different title. In titles, {field} is replaced by the [confidential.titles] entry for the captured folder name, or by the name turned into words. {field.short} uses the "<name>.short" entry if there is one. scan-notes.example.toml shows a complete meeting-notes layout for board-service folders laid out as <BODY>/committees/<committee>/<meeting>/.

Transcript placement.

  • The transcript goes in the transcript section under ### Draft transcript (unverified), ending with <!-- end draft transcript -->. It replaces Transcript pending; see scan. if present.
  • The date comes from the filename, or else from the nearest dated folder.
  • In an existing companion, only that block and the transcribed_by: front-matter field are changed. A rerun replaces the block, including any edits made inside it, so move corrections out of the block first.
  • transcribed_by: records the source, such as ollama/localhost/gemma4:e4b.
  • --companion or --no-companion overrides the automatic choice. Companion mode takes one PDF and cannot be combined with -o or --stdout. A run cannot mix confidential scans with other files.

Page images and transcripts are never written anywhere except the companion file. The companion is written through a temporary file in the same folder, which is renamed into place.

Context size, GPU memory, and run statistics

Each page transcribed through Ollama appends one line of statistics to ~/.local/state/pdf-tools/scan-stats.jsonl (config key stats_log; "" turns it off): page label, image size, prompt and output tokens, stop reason, timings, and how much of the model sat in GPU memory. No transcript text is logged. Page labels include file names, so turn the log off if file names are themselves sensitive. scan-notes-stats summarizes the log, including how many pages would exceed a given num_ctx.

Some pages send a model into a loop. Ollama either aborts the request with "token repeat limit reached" or the output runs on until max_output_tokens ([ollama] table, default 2048). Either way the page is retried twice with light sampling and a repeat penalty. A page that still trips the repeat check is written as [page not transcribed: the model kept repeating itself; see scan]; one that still hits the output cap keeps its text, marked [transcript cut off at the output limit; see scan].

What we have measured so far, and how to choose num_ctx, is in docs/ollama-context-and-gpu.md.

Batch runs

scripts/batch-scan.sh <folder> transcribes every PDF in a folder with the Ollama backend, skipping PDFs that already have a .md. Progress is saved after each page, so rerunning the script resumes an interrupted PDF.

scan-notes-index: per-page search metadata

A transcript is searchable for words you wrote, but not for what a page is about: a derivation written mostly in symbols, a sketch, or a meeting page whose topic is never named. scan-notes-index reads a transcribed scan and writes one Markdown file per page with a machine-generated description:

scan-notes-index "2009 notebook.pdf"     # reads "2009 notebook.md"
# -> "2009 notebook pages/2009 notebook p001.md", p002.md, ...
scripts/batch-index.sh ~/notebooks        # every transcribed PDF, newest first

Each page file has front matter (kind, topics, references, event, event_date, people, continues_from, continues_to, indexed_by), a summary paragraph and a list of key points marked as machine-generated and unverified, the references written on the page, and the page's transcript. One file per page means Spotlight or DEVONthink returns the page itself rather than the whole notebook, and the files are ready to embed for vector search later.

The model sees the page image, its transcript, and the transcripts of the previous and next pages, so calculations and notes that run across pages are described as such. It runs only on the configured Ollama hosts, like --backend ollama; there is no cloud option. Existing page files are kept, so an interrupted run resumes; --force rewrites them. Settings are in the [index] table of the config file: extra names for the prompt (added to the transcription names), prompt_extra, the list of kinds, max_output_tokens (default 1536), and context_chars, how much of each neighboring page to include (default 2000). Statistics go to the same stats log with "task": "index", and scan-notes-stats reports them separately.

On gemma4:31b with num_ctx = 8192 on an RTX 4090, a page takes about 11 s and uses at most about 3,800 tokens of context.

Tests

uv run pytest

The tests fake all HTTP calls and fail if Claude is called.

macOS integration

"Scrub & Mail" Quick Action

The scripts/scrub-and-mail.sh wrapper runs the scrubber, then opens a new Mail.app compose window with the cleaned PDF attached. To wire it up as a right-click Finder action:

  1. Open Automator → New Document → Quick Action.

  2. "Workflow receives current" → PDF files in Finder.

  3. Add a Run Shell Script action.

  4. "Pass input:" as arguments.

  5. Paste:

    export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
    /path/to/pdf-tools/scripts/scrub-and-mail.sh "$1"
    
  6. Save as "Scrub & Mail".

It will appear under Finder right-click → Quick Actions.

You can do the same for the other two tools (pdf-notes-to-md.sh, scan-to-md.sh); they just save Markdown next to the input rather than opening another app. For scan-to-md.sh, scan-notes-md chooses the backend itself: Ollama for confidential folders, Claude elsewhere.

Shortcuts (alternative)

If you prefer Shortcuts.app over Automator, create a new shortcut:

  • Receive PDF files from Quick Actions.
  • "Run Shell Script" with the same command.

Notes

  • The scrubber removes annotations from the /Annots arrays and clears some metadata fields (dc:creator, xmp:CreatorTool, pdf:Producer) that often leak the reviewing app or user identity. Form field widgets are kept unless --include-forms is passed.
  • The scrubber will not remove markings that have been flattened into the page content stream by a previous export step. If your PDF workflow ever does that, retain the unflattened original.
  • The Claude call lives in _transcribe_image in extract_scan.py; the Ollama call lives in transcribe_png in ollama.py.

License

MIT; see LICENSE.