Initial public release of pdf-tools

Command-line tools for marked-up PDFs and scanned handwritten notes:
pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on
request), scan-notes-index (per-page search metadata), and
scan-notes-stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
Eric Furst 2026-11-03 07:25:51 -05:00
commit e464871b69
29 changed files with 5296 additions and 0 deletions

10
.gitignore vendored Normal file
View file

@ -0,0 +1,10 @@
.venv/
__pycache__/
*.pyc
.pytest_cache/
*.egg-info/
build/
dist/
.DS_Store
# Scan outputs and progress files, in case a scan is processed in the repo.
.*.scan-partial.json

95
CLAUDE.md Normal file
View file

@ -0,0 +1,95 @@
# CLAUDE.md -- pdf-tools
Small command-line utilities for marked-up PDFs and scanned handwritten
notes. Python 3.11+, managed with `uv`. User-facing documentation is in
`README.md`.
## Layout
```
src/pdf_tools/
scrub.py pdf-scrub
extract_pdf.py pdf-notes-md
extract_scan.py scan-notes-md: CLI, page rendering, Claude backend
ollama.py scan-notes-md Ollama backend: prompt, host selection
companion.py scan-notes-md companion writer, driven by CompanionSpec
scan_config.py scan-notes-md config file and defaults
scan_stats.py per-page stats log and scan-notes-stats summary
scan_index.py scan-notes-index: per-page metadata files from transcripts
common.py shared helpers
scripts/ shell wrappers (Quick Actions, batch-scan.sh, batch-index.sh)
docs/ findings, e.g. ollama-context-and-gpu.md
tests/ pytest; run with `uv run pytest`
```
## scan-notes-md rules
- **No cloud fallback.** `--backend ollama` must never call Claude or any
other cloud service, and `ollama.py` must not import `anthropic`. If no
Ollama host is usable, exit with an error (status 3). The tests fail if
Claude is called on an Ollama run; keep them that way.
- **Nothing personal in the code.** The package is meant to be published.
Folder paths, host names, organization names, title tables, and name
lists belong in the user's config file, never in `src/` or the defaults.
`scan-notes.example.toml` uses placeholder names.
- **Confidential folders.** Anything under a `[[confidential]]` root
defaults to Ollama, and Claude is refused without `--allow-cloud`. This
check is based on location, not on `--companion`. Keep the backend
decision in Python, not in the shell wrappers. Reject unknown config keys,
so a typo cannot silently drop a protection.
- **No stray copies.** Page images are rendered in memory. Nothing is
written outside the scan's folder except to the system temp directory, and
temp files are removed. Do not log transcript text or images.
- **Companion edits are narrow.** In an existing companion, change only
the `### Draft transcript (unverified)` block and the `transcribed_by:`
field. Layouts come from `[confidential.companion]` tables; add new
layout features as `CompanionSpec` options, not as named special cases.
- **One prompt.** The Ollama instruction is `HANDWRITING_PROMPT` in
`ollama.py`. `build_prompt` appends configured `prompt_extra` text (top
level, then per folder) and the name list. Keep domain guidance, such as
the drawing and equation rules, in config, not in the base prompt. The
Claude backend keeps its own `TRANSCRIBE_PROMPT` in `extract_scan.py`.
- **Record what we learn.** Context size, GPU memory, speed, and model
failures go in `docs/ollama-context-and-gpu.md` with a date, model,
GPU, and Ollama version. The stats log holds numbers and page labels
only, never transcript text.
- **Looping pages.** A `RepetitionError` page is retried with
`REPEAT_RETRIES` and then replaced by `REPEAT_PLACEHOLDER`; first
attempts stay at temperature 0.
- **Batch runs.** `scripts/batch-scan.sh` skips PDFs that have a `.md`.
`extract_scan` checkpoints after every page to a hidden
`.<stem>.scan-partial.json` next to the scan, keyed on inputs, model,
prompt, and dpi, and removes it once the output is written.
- **Config, not code.** Hosts, models, timeouts, folders, layouts, and
per-folder names live in `~/.config/pdf-tools/scan-notes.toml`; generic
defaults are in `scan_config.py`. Keep `scan-notes.example.toml` loading
(a test checks it).
Command-line flags override the file.
- **HTTP is standard library.** Ollama calls go through `ollama._http_json`
(`urllib`), which the tests replace. Do not add an HTTP or Ollama SDK.
## scan-notes-index rules
- **Ollama only.** It has no Claude backend and uses the same host
selection and no-fallback rule as `--backend ollama`.
- **Generated text is labeled.** Summaries and key points are interpretation,
unlike the transcript; page files mark them machine-generated and
unverified. The prompt tells the model never to add reference details
that are not on the page; keep that.
- **Prompt and schema.** `INDEX_PROMPT` and `schema()` in `scan_index.py`
are the base; names, `prompt_extra`, and `kinds` come from `[index]` in
the config. Answers are constrained with Ollama's `format` (JSON schema)
through `ollama.chat_json`, which retries loops, output cut off at the
cap, and invalid JSON.
- **Resume by file.** A page whose file exists is skipped unless `--force`.
A failed page is reported, the others are written, and the exit status is
4. The stats log gets `"task": "index"` records with counts only.
## Conventions
- Add dependencies with `uv add` (or `uv add --dev`); keep them few.
- Status and progress go to stderr; document output goes to stdout or files.
- US English spelling. No emojis in code, output, or documentation.
- Tests must not touch the network or real confidential files: fake
`_http_json`, use `tmp_path`, and declare temporary confidential folders
in a config file passed through `PDF_TOOLS_SCAN_CONFIG`.

21
LICENSE Normal file
View file

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Eric Furst
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

293
README.md Normal file
View file

@ -0,0 +1,293 @@
# pdf-tools
Three small command-line utilities for working with marked-up PDFs and
scanned notebook pages on macOS.
1. **`pdf-scrub`** — strip annotations from a PDF (highlights, sticky
notes, freehand, text boxes) so you can send a clean copy.
2. **`pdf-notes-md`** — extract your notes plus the highlighted passages
they refer to, write as Markdown for downstream pipelines.
3. **`scan-notes-md`** — transcribe scanned handwritten notebook pages
to Markdown via a vision-language model: a local model through Ollama
by default, or Claude through the Anthropic API when asked. Scans in
confidential folders always stay on local machines.
## Install
```bash
git clone <repo> pdf-tools
cd pdf-tools
uv sync
uv pip install -e .
```
After this, `pdf-scrub`, `pdf-notes-md`, and `scan-notes-md` are on
your `PATH` inside the project's venv. To use them from anywhere, either
activate the venv or use the shell wrappers in `scripts/`.
`scan-notes-md` needs `ANTHROPIC_API_KEY` in the environment for the
Claude backend only. The Ollama backend needs no key and no extra packages.
## Usage
```bash
# Tool 1
pdf-scrub paper.pdf # -> paper_clean.pdf
pdf-scrub paper.pdf -o out.pdf
pdf-scrub paper.pdf --include-forms # also drop form widgets
# Tool 2
pdf-notes-md paper.pdf # -> paper.md
pdf-notes-md paper.pdf --stdout | less
# Tool 3
scan-notes-md page1.jpg page2.jpg # -> page1.md
scan-notes-md notebook.pdf -o out.md
scan-notes-md --backend ollama notebook.pdf # local model
scan-notes-md --backend ollama --host local scan.pdf # laptop only
scan-notes-md ~/confidential/.../2026-10-29-notes.pdf
# confidential folder: ollama + companion
```
## scan-notes-md backends
**Claude** (`--backend claude`, or `backend = "claude"` in the config) calls
the Anthropic API, one request per page, with `ANTHROPIC_API_KEY` and model
`claude-opus-5-5` unless `--model` says otherwise. Output is Markdown with one
`##` section per page. It is never used for scans in confidential folders
unless `--allow-cloud` is given.
**Ollama** (`--backend ollama`, the default) sends each page, rendered to PNG in memory,
to a vision model on a machine you own. Hosts are tried in the order listed
in the config file. Each is checked first with a 2.5 s request to
`/api/tags`. A host that is unreachable or lacks its model is skipped with
one line of output. `--host remote` or `--host local` uses only that host
and fails if it is unavailable. If no host works, the tool exits with
status 3. It never falls back to Claude or any other cloud service.
Each request uses `stream: false`, `think: false`, and temperature 0. Any
thinking text that leaks through is stripped. The instruction is
`HANDWRITING_PROMPT` in `src/pdf_tools/ollama.py`. Pages are joined with
`<!-- page N -->` markers, each page in a `text` code fence so that line
breaks and indentation survive Markdown rendering.
Hosts and models come from the config file. Without one, the only host is
`http://localhost:11434` with `gemma4:e4b`. In testing on handwritten board
notes, `gemma4:31b` on a GPU workstation gave the most reliable drafts;
`gemma4:e4b` on a laptop gives a rough draft to correct.
A remote Ollama host must listen on a reachable interface (for example
`OLLAMA_HOST=0.0.0.0` in its service environment, reached over Tailscale),
or the health check fails and every run falls back to the next host.
### Config file
`~/.config/pdf-tools/scan-notes.toml` (or `$PDF_TOOLS_SCAN_CONFIG`, or
`--config`) sets the default backend, `dpi`, Ollama timeouts, the host list,
and the confidential folders. Flags override it. Start from
`scan-notes.example.toml`, which is commented throughout. Unknown keys are
reported as errors, so a typo cannot silently disable a setting. In TOML, a
key written after a `[table]` header belongs to that table, so top-level
keys must come before the first one; a misplaced key is reported.
`prompt_extra` adds text to the Ollama prompt after the base instruction.
The example file uses it for notes with drawings, arrows, and equations:
drawings become one `[FIGURE: description]` line, arrows become `->` or an
`[ARROW: from -> to]` line, and equations are copied in plain text with
Unicode symbols. A confidential folder can add its own `prompt_extra`,
which follows the top-level one. Changing the prompt starts interrupted
runs over rather than resuming them, so pages are not mixed.
### Confidential folders and companion files
No folder is confidential unless the config file says so. Each folder gets
one `[[confidential]]` block:
```toml
[[confidential]]
root = "~/Documents/personnel"
names = ["Pat Example", "Facilities Office"]
```
When folders are nested, the deepest one applies. For a scan inside any
confidential folder:
- The backend defaults to `ollama`. `--backend claude` is refused unless
`--allow-cloud` is also given, even with `--no-companion`.
- The folder's `prompt_extra` text, if any, is added to the Ollama prompt.
- The folder's `names` are added to the Ollama prompt: if a word plausibly
matches one of them, the model uses that spelling, and otherwise writes
`[illegible]`. Names are per folder, so a long list for one folder does
not lengthen the prompt or steer the reading for others. A top-level
`names` list applies to every Ollama run. Names are never sent to Claude.
- Output goes to the companion `<stem>.md` next to the scan.
**Companion layout.** Without a `[confidential.companion]` table, a new
companion has `date`, `type: notes`, `scan:`, `subjects: []`, and
`transcribed_by:` in its front matter, a title from the filename, and one
Notes section. A `[confidential.companion]` table describes any other
layout:
| Key | Meaning |
|---|---|
| `type` | value of the `type:` field |
| `sections` | `##` sections, in order |
| `transcript_section` | section that holds the draft transcript (default: the first) |
| `section_text` | starting text for named sections |
| `fields` | front-matter fields read from the path, after `date` |
| `paths` | rules that read those fields from folder names (below) |
| `allowed` | permitted values for a captured field |
| `title` | title template: `{date}`, `{stem}`, `{field}`, `{field.short}` |
| `acronyms` | words kept in capitals when folder names become title words |
Each `[[confidential.companion.paths]]` rule has a `match` pattern over the
folders below the root. A pattern segment is a literal folder name,
`{field}` to capture one folder, or a trailing `**` for any remaining
folders. Rules are tried in order, and the first match wins. A rule can add
fixed values with `set` and a different `title`. In titles, `{field}` is
replaced by the `[confidential.titles]` entry for the captured folder name,
or by the name turned into words. `{field.short}` uses the
`"<name>.short"` entry if there is one. `scan-notes.example.toml` shows a
complete meeting-notes layout for board-service folders laid out as
`<BODY>/committees/<committee>/<meeting>/`.
**Transcript placement.**
- The transcript goes in the transcript section under
`### Draft transcript (unverified)`, ending with
`<!-- end draft transcript -->`. It replaces `Transcript pending; see
scan.` if present.
- The date comes from the filename, or else from the nearest dated folder.
- In an existing companion, only that block and the `transcribed_by:`
front-matter field are changed. A rerun replaces the block, including any
edits made inside it, so move corrections out of the block first.
- `transcribed_by:` records the source, such as
`ollama/localhost/gemma4:e4b`.
- `--companion` or `--no-companion` overrides the automatic choice.
Companion mode takes one PDF and cannot be combined with `-o` or
`--stdout`. A run cannot mix confidential scans with other files.
Page images and transcripts are never written anywhere except the
companion file. The companion is written through a temporary file in the
same folder, which is renamed into place.
### Context size, GPU memory, and run statistics
Each page transcribed through Ollama appends one line of statistics to
`~/.local/state/pdf-tools/scan-stats.jsonl` (config key `stats_log`; `""`
turns it off): page label, image size, prompt and output tokens, stop
reason, timings, and how much of the model sat in GPU memory. No transcript
text is logged. Page labels include file names, so turn the log off if file
names are themselves sensitive. `scan-notes-stats` summarizes the log,
including how many pages would exceed a given `num_ctx`.
Some pages send a model into a loop. Ollama either aborts the request with
"token repeat limit reached" or the output runs on until
`max_output_tokens` (`[ollama]` table, default 2048). Either way the page is
retried twice with light sampling and a repeat penalty. A page that still
trips the repeat check is written as
`[page not transcribed: the model kept repeating itself; see scan]`; one
that still hits the output cap keeps its text, marked
`[transcript cut off at the output limit; see scan]`.
What we have measured so far, and how to choose `num_ctx`, is in
`docs/ollama-context-and-gpu.md`.
### Batch runs
`scripts/batch-scan.sh <folder>` transcribes every PDF in a folder with the
Ollama backend, skipping PDFs that already have a `.md`. Progress is saved
after each page, so rerunning the script resumes an interrupted PDF.
## scan-notes-index: per-page search metadata
A transcript is searchable for words you wrote, but not for what a page is
about: a derivation written mostly in symbols, a sketch, or a meeting page
whose topic is never named. `scan-notes-index` reads a transcribed scan and
writes one Markdown file per page with a machine-generated description:
```bash
scan-notes-index "2009 notebook.pdf" # reads "2009 notebook.md"
# -> "2009 notebook pages/2009 notebook p001.md", p002.md, ...
scripts/batch-index.sh ~/notebooks # every transcribed PDF, newest first
```
Each page file has front matter (`kind`, `topics`, `references`, `event`,
`event_date`, `people`, `continues_from`, `continues_to`, `indexed_by`), a
summary paragraph and a list of key points marked as machine-generated and
unverified, the references written on the page, and the page's transcript.
One file per page means Spotlight or DEVONthink returns the page itself
rather than the whole notebook, and the files are ready to embed for vector
search later.
The model sees the page image, its transcript, and the transcripts of the
previous and next pages, so calculations and notes that run across pages are
described as such. It runs only on the configured Ollama hosts, like
`--backend ollama`; there is no cloud option. Existing page files are kept,
so an interrupted run resumes; `--force` rewrites them. Settings are in the
`[index]` table of the config file: extra `names` for the prompt (added to
the transcription names), `prompt_extra`, the list of `kinds`,
`max_output_tokens` (default 1536), and `context_chars`, how much of each
neighboring page to include (default 2000). Statistics go to the same stats
log with `"task": "index"`, and `scan-notes-stats` reports them separately.
On gemma4:31b with `num_ctx = 8192` on an RTX 4090, a page takes about 11 s
and uses at most about 3,800 tokens of context.
### Tests
```bash
uv run pytest
```
The tests fake all HTTP calls and fail if Claude is called.
## macOS integration
### "Scrub & Mail" Quick Action
The `scripts/scrub-and-mail.sh` wrapper runs the scrubber, then opens a
new Mail.app compose window with the cleaned PDF attached. To wire it
up as a right-click Finder action:
1. Open **Automator** → New Document → **Quick Action**.
2. "Workflow receives current" → **PDF files** in **Finder**.
3. Add a **Run Shell Script** action.
4. "Pass input:" **as arguments**.
5. Paste:
```bash
export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
/path/to/pdf-tools/scripts/scrub-and-mail.sh "$1"
```
6. Save as "Scrub & Mail".
It will appear under Finder right-click → Quick Actions.
You can do the same for the other two tools (`pdf-notes-to-md.sh`,
`scan-to-md.sh`); they just save Markdown next to the input rather than
opening another app. For `scan-to-md.sh`, `scan-notes-md` chooses the
backend itself: Ollama for confidential folders, Claude elsewhere.
### Shortcuts (alternative)
If you prefer Shortcuts.app over Automator, create a new shortcut:
- Receive **PDF files** from **Quick Actions**.
- "Run Shell Script" with the same command.
## Notes
- The scrubber removes annotations from the `/Annots` arrays and clears
some metadata fields (`dc:creator`, `xmp:CreatorTool`, `pdf:Producer`)
that often leak the reviewing app or user identity. Form field
widgets are kept unless `--include-forms` is passed.
- The scrubber will not remove markings that have been *flattened* into
the page content stream by a previous export step. If your PDF
workflow ever does that, retain the unflattened original.
- The Claude call lives in `_transcribe_image` in `extract_scan.py`; the
Ollama call lives in `transcribe_png` in `ollama.py`.
## License
MIT; see `LICENSE`.

View file

@ -0,0 +1,142 @@
# Ollama context size and GPU memory
What we have learned about how much context a handwritten page needs, how
the context size decides whether a model fits in GPU memory, and how to
measure both. Update the findings below as new runs add data.
## Why it matters
Ollama reserves memory for the model's context window (the KV cache) when it
loads the model. A larger `num_ctx` reserves more memory. If weights plus
cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every
token gets slower. Too small a context has the opposite risk: the prompt,
which includes the page image, is truncated, or the output stops early.
One page uses: prompt text (about 300 tokens with `prompt_extra`) + image
tokens + output tokens. The context only needs to hold one page at a time,
because each page is a separate request.
## How to measure
Every page transcribed with the Ollama backend appends one line to the stats
log (`stats_log` in the config; default
`~/.local/state/pdf-tools/scan-stats.jsonl`). A line records the page label,
image size in pixels, prompt and output tokens, the stop reason, timings,
and how the model was loaded: total size, bytes in VRAM, and the context
length (from Ollama's `/api/ps`). It never records transcript text.
```bash
scan-notes-stats # summary of everything logged
scan-notes-stats --run last # most recent run only
scan-notes-stats --threshold 6144 --threshold 8192 --top 10
```
The summary reports, per model and loaded context size, token percentiles,
how many pages would need more than each threshold, pages whose output hit a
length limit, pages where the model looped, the share of the model in GPU
memory, and seconds per page.
During a run, `scan-notes-md` warns when a page's output stops at a length
limit or uses more than 90% of the loaded context.
To try a smaller context, set `num_ctx` on the host in the config:
```toml
[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:31b"
num_ctx = 8192
```
Changing `num_ctx` does not invalidate saved progress, so an interrupted
notebook resumes. Check `ollama ps` afterwards: the PROCESSOR column should
read `100% GPU`.
## Findings
### 2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13
- **Default context.** With no `num_ctx` set, Ollama loaded the model with a
32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, so `ollama ps`
showed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB).
- **First pages.** On the first 10 logged pages (2200 x 1700 to 2339 x 1653
px at 200 dpi), prompts were 768 to 787 tokens (base prompt,
`prompt_extra`, and image) and outputs 351 to 462 tokens; the largest page
used 1,230 tokens, under 4% of the loaded context. The image accounts for
most of the prompt and varies little with page size. Pages averaged 15 s,
so a 2,562-page batch takes about 11 hours at this setting.
- **Looping.** Some pages make the model repeat itself until Ollama aborts
the request with `prediction aborted, token repeat limit reached`. At
temperature 0 the same page usually fails the same way: one page failed
in two separate runs, though it went through on a third, because GPU
inference at temperature 0 is not perfectly deterministic. It happened on
2 of the first 70 pages. `scan-notes-md` now retries
such a page with a little sampling (temperature 0.3, then 0.6, with a
repeat penalty) and, if it still loops, writes a visible placeholder:
`[page not transcribed: the model kept repeating itself; see scan]`.
`scan-notes-stats` counts retried pages and placeholders.
- **Runaway output.** A model can also loop without tripping the repeat
check. Ollama's default output limit is unlimited, so such a page ran
past six minutes and would have failed at the 600 s timeout on every
rerun. `max_output_tokens` (default 2048, passed as `num_predict`) now
stops it early. A page that hits the cap is retried like a looping page;
if it hits the cap every time, its text is kept and marked
`[transcript cut off at the output limit; see scan]`. The output cap also
bounds the context a page can use: prompt plus at most 2,048 output
tokens.
### 2026-11-02: full run, 40 notebooks, same setup
The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages)
was transcribed before logging was added.
| Measure | mean | p50 | p90 | p99 | max |
|---|---|---|---|---|---|
| Prompt tokens (incl. image) | 781 | 787 | 787 | 787 | 797 |
| Output tokens | 278 | 278 | 453 | 674 | 1,125 |
| Context used | 1,059 | 1,058 | 1,232 | 1,459 | 1,912 |
- **Context.** No page needed more than 1,912 tokens, under 6% of the
loaded 32,768 and under half of 4,096. No page reached the 2,048-token
output cap on its final attempt. The prompt is nearly constant (768 to 797
tokens), which suggests the model encodes each image into a fixed number of tokens; page size
and resolution barely matter; context use is set by how much is written
on the page. Even a page that hits the output cap needs only about 2,850
tokens, so 4,096 would hold every page with this prompt.
- **GPU fit.** The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM)
for the whole run; the default context is what keeps it off the GPU
entirely.
- **Speed.** 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of
model time for the logged pages. Wall-clock time for the batch was about
8 h. Blank and nearly blank pages take 2 to 4 s.
- **Retries.** 65 pages (2.6%) were retried: 53 after Ollama's repeat
limit, 7 after reaching the output cap, and 5 before reasons were logged.
Every one finished normally on its first retry (temperature 0.3, repeat
penalty 1.15), with 67 to 888 output tokens, so these were loops rather
than long pages. No page was left as a placeholder or marked cut off. The
repeat penalty can also discourage repetition that is really on the page
(the same value down a column), so retried pages are the ones most worth
checking against the scan; they are the log lines with `"retries": 1`.
### 2026-11-02: 8192 context compared with 32768, one notebook
The same 85-page notebook, same prompt and settings except
`num_ctx = 8192`:
| | 32,768 (default) | 8,192 |
|---|---|---|
| Loaded size, GPU share | 21.2 GB, 94% GPU | 20 GB, 100% GPU |
| Seconds per page (mean) | 11.6 | 8.3 |
| Total for the notebook | 16.4 min | 11.8 min |
| Prompt time per page | 0.84 s | 0.69 s |
| Output speed | 28.9 tokens/s | 42.7 tokens/s |
| Output tokens (all pages) | 24,777 | 25,092 |
| Pages retried | 2 | 3 |
At 8192 the whole model fits on the GPU, and output runs about 48% faster,
so pages take about 28% less time. The two runs wrote nearly the same amount
of text, so the gain is not from shorter output. The largest page in the full
run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set
`num_ctx = 8192` for gemma4:31b on a 24 GB card. At this speed the 40-notebook
batch would take about 6 hours instead of 8.

42
pyproject.toml Normal file
View file

@ -0,0 +1,42 @@
[project]
name = "pdf-tools"
version = "0.1.0"
description = "Local PDF utilities: annotation scrubbing, note extraction, and handwritten-page OCR."
readme = "README.md"
requires-python = ">=3.11"
authors = [{ name = "Eric Furst" }]
license = { text = "MIT" }
dependencies = [
"pikepdf>=9.0", # robust PDF rewriting for scrubbing
"pypdf>=5.0", # text extraction under highlight rects
"pdfminer.six>=20240706", # used by pdfannots
"pdfannots>=0.4", # annotation extraction with underlying text
"Pillow>=10.4", # image handling for scan tool
"anthropic>=0.39", # Tool 3 OCR
"click>=8.1", # CLI ergonomics
"pypdfium2>=4.30", # PDF page rendering for scan tool
]
[project.scripts]
pdf-scrub = "pdf_tools.scrub:main"
pdf-notes-md = "pdf_tools.extract_pdf:main"
scan-notes-md = "pdf_tools.extract_scan:main"
scan-notes-stats = "pdf_tools.scan_stats:main"
scan-notes-index = "pdf_tools.scan_index:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/pdf_tools"]
[dependency-groups]
dev = [
"pytest>=9.2.0",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-p no:anyio"

133
scan-notes.example.toml Normal file
View file

@ -0,0 +1,133 @@
# Example settings for scan-notes-md. Copy to
# ~/.config/pdf-tools/scan-notes.toml and edit. Command-line flags override
# these, and anything left out falls back to the built-in defaults.
# Backend for scans outside confidential folders: "ollama" (default) or
# "claude" (the Anthropic API; needs ANTHROPIC_API_KEY).
backend = "ollama"
# PDF render resolution.
dpi = 200
# Names added to the Ollama prompt for every scan. Prefer per-folder lists
# (below), which keep the prompt short and avoid steering unrelated scans.
names = []
# Added to the Ollama prompt after the base instruction, for every scan.
# Confidential folders can add their own prompt_extra as well.
prompt_extra = """
The page may also contain drawings, arrows, and equations. Handle them this way:
- In place of a drawing, sketch, plot, or diagram, write one line:
[FIGURE: short description]. Name what is drawn, and include any axis
labels, units, and words written on it. Do not interpret what it shows.
- Write an arrow between words as -> (or <-, <->), keeping it on the line
where it appears. If an arrow links items on different lines or points to
something away from the text, write [ARROW: from -> to] on its own line.
- Copy equations exactly as written, in plain text. Use Unicode for Greek
letters and symbols (η, φ, ∂, ∝, ≈), ^ for superscripts, _ for
subscripts, and / for fractions. Do not solve, simplify, or complete them.
"""
# Per-page token counts, timings, and GPU fit for Ollama runs (no transcript
# text). Summarize with scan-notes-stats. Set to "" to turn off.
stats_log = "~/.local/state/pdf-tools/scan-stats.jsonl"
[ollama]
timeout_s = 300 # per page
probe_timeout_s = 2.5 # health check against /api/tags
max_output_tokens = 2048 # per page; stops a looping model early (0 = no cap)
# Hosts are tried in this order. A host is skipped (with one line of output)
# if it is unreachable or does not have its model.
[[ollama.hosts]]
name = "remote"
url = "http://gpu-workstation:11434"
model = "gemma4:31b"
# num_ctx = 8192 # optional; set if the model needs it to fit on the GPU
[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:e4b"
# scan-notes-index: per-page search metadata (summary, key points, topics,
# references, event, people) written from a scan and its transcript.
[index]
names = [] # added to the names above, for indexing only
prompt_extra = ""
kinds = ["calculation", "idea", "sketch", "talk notes", "meeting notes",
"reading notes", "to-do", "other", "blank"]
max_output_tokens = 1536
context_chars = 2000 # characters of the previous and next pages sent as context
# ---------------------------------------------------------------------------
# Confidential folders. None are built in. Scans under a listed root stay on
# the Ollama hosts (Claude is refused without --allow-cloud) and are written
# to a companion <stem>.md next to the scan. When folders are nested, the
# deepest one applies.
# ---------------------------------------------------------------------------
# Simplest form: the basic companion (date, scan, subjects, one Notes section).
[[confidential]]
root = "~/Documents/personnel"
names = []
# Full form: a board-service folder laid out as
# <BODY>/committees/<committee>/<meeting>/
# <BODY>/meetings/<meeting>/
# chair/<project>/
[[confidential]]
root = "~/Documents/board-service"
# Board members, staff, and organizations; used only for scans in this folder.
names = []
[confidential.companion]
type = "meeting-notes"
# Front-matter fields read from the path, in this order after `date`.
fields = ["body", "committee"]
sections = ["Attendees", "Discussion", "Decisions", "Action items"]
transcript_section = "Discussion"
# Title placeholders: {date} (long form), {stem} (filename without its date),
# {field} (display name from [confidential.titles], else the folder name in
# words), {field.short} (the "<value>.short" title, if any).
title = "{body}, {date}"
# Words kept in capitals when a folder name is turned into words.
acronyms = ["ceo", "cfo", "llc"]
[confidential.companion.section_text]
"Action items" = "- [ ] Owner: task (due date)"
# Optional: values a captured field may take. A rule whose capture is not
# listed does not match, so the next rule is tried.
[confidential.companion.allowed]
body = ["BOD", "BOM"]
# Rules are tried in order; the first match sets the fields. "{name}" captures
# one folder, a literal must match exactly, and a trailing "**" matches any
# remaining folders. `set` gives fixed values; `title` overrides the title.
[[confidential.companion.paths]]
match = "{body}/committees/{committee}/**"
title = "{body.short} {committee}, {date}"
[[confidential.companion.paths]]
match = "{body}/meetings/**"
set = { committee = "board" }
[[confidential.companion.paths]]
match = "chair/{committee}/**"
set = { body = "chair" }
title = "{committee}, {date}"
[[confidential.companion.paths]]
match = "{body}/{committee}/**"
title = "{body.short} {committee}, {date}"
[[confidential.companion.paths]]
match = "{body}/**"
[confidential.titles]
"BOD" = "Board of Directors"
"BOD.short" = "Board"
"BOM" = "Board of Managers"
"BOM.short" = "Managers"
"finance-audit" = "Finance and Audit Committee"

52
scripts/batch-index.sh Executable file
View file

@ -0,0 +1,52 @@
#!/usr/bin/env bash
# batch-index.sh — run scan-notes-index on every transcribed PDF in a folder.
#
# usage: batch-index.sh <folder> [scan-notes-index options...]
#
# PDFs are taken newest first (reverse filename order, which is date order
# for date-prefixed names). A PDF without a <stem>.md transcript is skipped.
# Pages already indexed are kept, so the script can simply be rerun after an
# interruption. A failed PDF is reported and the batch moves on.
#
# Long runs belong in tmux: tmux new -s index 'batch-index.sh ~/notebooks'
set -uo pipefail
if [[ $# -lt 1 || ! -d "$1" ]]; then
echo "usage: $0 <folder> [scan-notes-index options...]" >&2
exit 64
fi
dir="$1"; shift
if command -v scan-notes-index >/dev/null 2>&1; then
run() { scan-notes-index "$@"; }
else
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-index}"
run() { (cd "$PDF_TOOLS_DIR" && uv run --quiet scan-notes-index "$@"); }
fi
pdfs=()
while IFS= read -r -d '' f; do pdfs+=("$f"); done \
< <(find "$dir" -type f -iname '*.pdf' -not -name '.*' -print0 | sort -rz)
total=${#pdfs[@]}
done_n=0; skipped=0; failed=()
start=$(date +%s)
for i in "${!pdfs[@]}"; do
pdf="${pdfs[$i]}"
echo "=== [$((i + 1))/$total] ${pdf#"$dir"/} ($(date '+%H:%M'))" >&2
if [[ ! -e "${pdf%.*}.md" ]]; then
echo "no transcript; skipping" >&2
skipped=$((skipped + 1))
continue
fi
if run "$@" "$pdf"; then
done_n=$((done_n + 1))
else
failed+=("${pdf#"$dir"/}")
fi
done
mins=$(( ($(date +%s) - start) / 60 ))
echo "=== finished in ${mins} min: $done_n indexed, $skipped skipped, ${#failed[@]} failed" >&2
for f in ${failed[@]+"${failed[@]}"}; do echo "failed: $f" >&2; done
[[ ${#failed[@]} -eq 0 ]]

56
scripts/batch-scan.sh Executable file
View file

@ -0,0 +1,56 @@
#!/usr/bin/env bash
# batch-scan.sh — transcribe every PDF in a folder with scan-notes-md.
#
# usage: batch-scan.sh <folder> [scan-notes-md options...]
#
# PDFs that already have a <stem>.md are skipped, and an interrupted PDF
# resumes from its last saved page, so the script can simply be rerun. A
# failed PDF is reported and the batch moves on to the next one. Extra
# options are passed to scan-notes-md; the default is --backend ollama, so
# nothing is sent to a cloud service unless you ask for it.
#
# Long runs belong in tmux: tmux new -s scan 'batch-scan.sh ~/notebooks'
set -uo pipefail
if [[ $# -lt 1 || ! -d "$1" ]]; then
echo "usage: $0 <folder> [scan-notes-md options...]" >&2
exit 64
fi
dir="$1"; shift
opts=("$@")
[[ ${#opts[@]} -eq 0 ]] && opts=(--backend ollama)
if command -v scan-notes-md >/dev/null 2>&1; then
run() { scan-notes-md "$@"; }
else
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-md}"
run() { (cd "$PDF_TOOLS_DIR" && uv run --quiet scan-notes-md "$@"); }
fi
pdfs=()
while IFS= read -r -d '' f; do pdfs+=("$f"); done \
< <(find "$dir" -type f -iname '*.pdf' -not -name '.*' -print0 | sort -z)
total=${#pdfs[@]}
done_n=0; skipped=0; failed=()
start=$(date +%s)
for i in "${!pdfs[@]}"; do
pdf="${pdfs[$i]}"
md="${pdf%.*}.md"
echo "=== [$((i + 1))/$total] ${pdf#"$dir"/} ($(date '+%H:%M'))" >&2
if [[ -e "$md" ]]; then
echo "already transcribed; skipping" >&2
skipped=$((skipped + 1))
continue
fi
if run "${opts[@]}" "$pdf"; then
done_n=$((done_n + 1))
else
failed+=("${pdf#"$dir"/}")
fi
done
mins=$(( ($(date +%s) - start) / 60 ))
echo "=== finished in ${mins} min: $done_n transcribed, $skipped skipped, ${#failed[@]} failed" >&2
for f in ${failed[@]+"${failed[@]}"}; do echo "failed: $f" >&2; done
[[ ${#failed[@]} -eq 0 ]]

18
scripts/pdf-notes-to-md.sh Executable file
View file

@ -0,0 +1,18 @@
#!/usr/bin/env bash
# pdf-notes-to-md.sh — extract PDF annotations to Markdown next to the PDF.
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "usage: $0 <pdf> [output.md]" >&2
exit 64
fi
IN="$1"
OUT="${2:-${IN%.pdf}.md}"
if command -v pdf-notes-md >/dev/null 2>&1; then
pdf-notes-md "$IN" -o "$OUT"
else
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install pdf-notes-md}"
(cd "$PDF_TOOLS_DIR" && uv run pdf-notes-md "$IN" -o "$OUT")
fi

19
scripts/scan-to-md.sh Executable file
View file

@ -0,0 +1,19 @@
#!/usr/bin/env bash
# scan-to-md.sh — transcribe scanned pages to Markdown.
#
# scan-notes-md itself decides the backend: scans in the confidential
# folders listed in ~/.config/pdf-tools/scan-notes.toml go to Ollama and
# their companion <stem>.md; everything else uses Claude.
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "usage: $0 <image-or-pdf> [more inputs...]" >&2
exit 64
fi
if command -v scan-notes-md >/dev/null 2>&1; then
scan-notes-md "$@"
else
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-md}"
(cd "$PDF_TOOLS_DIR" && uv run scan-notes-md "$@")
fi

49
scripts/scrub-and-mail.sh Executable file
View file

@ -0,0 +1,49 @@
#!/usr/bin/env bash
# scrub-and-mail.sh — scrub a PDF and open a new Mail.app message with it attached.
#
# Usage:
# scrub-and-mail.sh /path/to/file.pdf
#
# Designed to be called from a macOS Quick Action / Service / Shortcut
# with the selected PDF passed as $1.
set -euo pipefail
if [[ $# -lt 1 ]]; then
echo "usage: $0 <pdf>" >&2
exit 64
fi
IN="$1"
if [[ ! -f "$IN" ]]; then
echo "error: not a file: $IN" >&2
exit 1
fi
# Write the scrubbed copy to a temp dir so we don't pollute Dropbox.
TMPDIR_OUT="$(mktemp -d -t pdfscrub)"
BASENAME="$(basename "$IN" .pdf)"
OUT="$TMPDIR_OUT/${BASENAME}_clean.pdf"
# Resolve the CLI. Prefer the installed entry point; fall back to
# `uv run` from the project directory if you keep it ephemeral.
if command -v pdf-scrub >/dev/null 2>&1; then
pdf-scrub "$IN" -o "$OUT" >/dev/null
else
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR to the project path or install pdf-scrub}"
(cd "$PDF_TOOLS_DIR" && uv run pdf-scrub "$IN" -o "$OUT" >/dev/null)
fi
# Hand off to Mail.app. POSIX path -> HFS path for AppleScript's `file`.
osascript <<APPLESCRIPT
set theAttachment to POSIX file "$OUT"
tell application "Mail"
set theMessage to make new outgoing message with properties {visible:true, subject:"$BASENAME"}
tell theMessage
make new attachment with properties {file name:theAttachment} at after last paragraph
end tell
activate
end tell
APPLESCRIPT
echo "opened Mail with: $OUT" >&2

View file

@ -0,0 +1,2 @@
"""pdf-tools: scrub, extract, and OCR utilities."""
__version__ = "0.1.0"

21
src/pdf_tools/common.py Normal file
View file

@ -0,0 +1,21 @@
"""Shared helpers across the three tools."""
from __future__ import annotations
import sys
from pathlib import Path
def err(msg: str) -> None:
print(msg, file=sys.stderr)
def must_exist(p: Path) -> Path:
if not p.exists():
err(f"error: not found: {p}")
sys.exit(2)
return p
def with_suffix_before_ext(p: Path, marker: str) -> Path:
"""foo.pdf + '_clean' -> foo_clean.pdf"""
return p.with_name(f"{p.stem}{marker}{p.suffix}")

327
src/pdf_tools/companion.py Normal file
View file

@ -0,0 +1,327 @@
"""Companion mode: write the Markdown companion for a scan in a confidential folder.
A scan `<stem>.pdf` is paired with `<stem>.md` in the same folder. This module
fills the front matter from the path and filename, and puts the transcript
under DRAFT_HEADING in the folder's transcript section.
Nothing here is specific to any one organization. The shape of a companion
(front-matter fields read from the path, sections, title) comes from a
CompanionSpec, built from the folder's [confidential.companion] table in the
config file. BASIC is used when a folder does not define one.
Existing companions are edited in two places only: the DRAFT_HEADING block
(replaced, or added if missing) and the transcribed_by: front-matter field.
Everything else in the file is kept as is.
"""
from __future__ import annotations
import datetime as _dt
import os
import re
import tempfile
from dataclasses import dataclass, field
from pathlib import Path
DRAFT_HEADING = "### Draft transcript (unverified)"
END_MARKER = "<!-- end draft transcript -->"
PENDING_LINE = "Transcript pending; see scan."
_DATE_PREFIX = re.compile(r"^(\d{4}-\d{2}-(?:\d{2}|xx))")
_PLACEHOLDER = re.compile(r"\{(\w+)(?:\.(short))?\}")
@dataclass
class PathRule:
"""One way to read front-matter fields from the folders under the root.
`match` is a "/"-separated pattern over folder names: a literal name,
"{field}" to capture one folder, or a trailing "**" for any remaining
folders. `set` gives fixed values; `title` overrides the spec's title.
"""
match: str
set: dict[str, str] = field(default_factory=dict)
title: str | None = None
@dataclass
class CompanionSpec:
type: str = "notes"
sections: list[str] = field(default_factory=lambda: ["Notes"])
transcript_section: str = "Notes"
section_text: dict[str, str] = field(default_factory=dict)
title: str = "{stem}, {date}"
fields: list[str] = field(default_factory=list)
paths: list[PathRule] = field(default_factory=list)
allowed: dict[str, list[str]] = field(default_factory=dict)
acronyms: set[str] = field(default_factory=set)
BASIC = CompanionSpec()
@dataclass
class NoteMeta:
date: str # ISO date, YYYY-MM-xx, or "" if unknown
fields: dict[str, str] # spec.fields -> value ("" if unknown)
scan: str # PDF filename
title: str # title template chosen for this scan
warnings: list[str]
def _match(pattern: str, folders: tuple[str, ...],
allowed: dict[str, list[str]]) -> dict[str, str] | None:
segs = [s for s in pattern.strip("/").split("/") if s]
tail = bool(segs) and segs[-1] == "**"
if tail:
segs = segs[:-1]
if len(folders) < len(segs) or (not tail and len(folders) != len(segs)):
return None
got: dict[str, str] = {}
for seg, name in zip(segs, folders):
m = re.fullmatch(r"\{(\w+)\}", seg)
if m:
key = m.group(1)
if key in allowed and name not in allowed[key]:
return None
got[key] = name
elif seg != name:
return None
return got
def meta_from_path(pdf: Path, root: Path, spec: CompanionSpec = BASIC) -> NoteMeta:
"""Derive the date and the spec's fields from where the scan sits."""
rel = pdf.resolve().relative_to(root.resolve())
folders = rel.parts[:-1]
warnings: list[str] = []
values = {f: "" for f in spec.fields}
title = spec.title
if spec.fields:
for rule in spec.paths:
got = _match(rule.match, folders, spec.allowed)
if got is not None:
values.update({k: v for k, v in {**got, **rule.set}.items()
if k in values})
title = rule.title or title
break
missing = [f for f, v in values.items() if not v]
if missing:
warnings.append(f"cannot tell {', '.join(missing)} from the folder "
f"{'/'.join(folders) or '.'}; left empty")
date = ""
m = _DATE_PREFIX.match(pdf.name)
if m:
date = m.group(1)
else:
for folder in reversed(folders):
m = _DATE_PREFIX.match(folder)
if m:
date = m.group(1)
break
if not date:
warnings.append("no date in the filename or folder; left date empty")
elif date.endswith("-xx"):
warnings.append(f"date {date} has no day; check it")
return NoteMeta(date=date, fields=values, scan=pdf.name, title=title,
warnings=warnings)
def _words(slug: str, acronyms: set[str]) -> str:
return " ".join(w.upper() if w.lower() in acronyms else w.capitalize()
for w in re.split(r"[-_ ]+", slug) if w)
def title_for(meta: NoteMeta, spec: CompanionSpec = BASIC,
titles: dict[str, str] | None = None) -> str:
"""Fill the title template.
{date} is the long date; {stem} is the filename without its date prefix;
{field} is the field's display name from `titles` (or the folder name
turned into words); {field.short} uses the `<value>.short` entry if any.
"""
titles = titles or {}
acr = {a.lower() for a in spec.acronyms}
def display(value: str, short: bool) -> str:
if short and f"{value}.short" in titles:
return titles[f"{value}.short"]
return titles.get(value) or _words(value, acr)
try:
d = _dt.date.fromisoformat(meta.date)
long_date = f"{d.strftime('%B')} {d.day}, {d.year}"
except ValueError:
long_date = ""
def sub(m: re.Match) -> str:
key, short = m.group(1), bool(m.group(2))
if key == "date":
return long_date
if key == "stem":
return _words(_DATE_PREFIX.sub("", Path(meta.scan).stem), acr) or "Notes"
return display(meta.fields.get(key, ""), short) if meta.fields.get(key) else ""
out = _PLACEHOLDER.sub(sub, meta.title)
out = re.sub(r"\s{2,}", " ", out)
return out.strip(" ,") or Path(meta.scan).stem
def render_transcript(pages: list[str]) -> str:
"""Page markers plus one fenced block per page.
Fencing keeps line breaks and indentation exactly as transcribed, and
stops a line such as "# 3" on the page from turning into a heading.
"""
out = []
for n, text in enumerate(pages, start=1):
fence = "```"
while fence in text:
fence += "`"
out += [f"<!-- page {n} -->", "", f"{fence}text", text, fence, ""]
return "\n".join(out).rstrip("\n")
def draft_block(pages: list[str]) -> str:
return f"{DRAFT_HEADING}\n\n{render_transcript(pages)}\n\n{END_MARKER}"
def new_companion(meta: NoteMeta, pages: list[str], transcribed_by: str,
spec: CompanionSpec = BASIC,
titles: dict[str, str] | None = None) -> str:
out = ["---", f"date: {meta.date}"]
out += [f"{k}: {v}" for k, v in meta.fields.items()]
out += [f"type: {spec.type}", f"scan: {meta.scan}", "subjects: []",
f"transcribed_by: {transcribed_by}", "---", "",
f"# {title_for(meta, spec, titles)}", ""]
sections = list(spec.sections)
if spec.transcript_section not in sections:
sections.append(spec.transcript_section)
for s in sections:
out += [f"## {s}", ""]
if s == spec.transcript_section:
out += [draft_block(pages), ""]
elif spec.section_text.get(s):
out += [spec.section_text[s].rstrip("\n"), ""]
return "\n".join(out)
def _set_transcribed_by(text: str, value: str) -> str:
line = f"transcribed_by: {value}"
if not text.startswith("---\n"):
return f"---\n{line}\n---\n\n{text}"
end = text.find("\n---", 4)
if end == -1:
return f"---\n{line}\n---\n\n{text}"
head, rest = text[:end], text[end:]
if re.search(r"^transcribed_by:.*$", head, flags=re.MULTILINE):
head = re.sub(r"^transcribed_by:.*$", line, head, count=1, flags=re.MULTILINE)
else:
head = f"{head}\n{line}"
return head + rest
_HEADING = re.compile(r"^(#{1,6})\s")
_FENCE = re.compile(r"^(`{3,}|~{3,})")
def _heading_level(line: str) -> int:
m = _HEADING.match(line)
return len(m.group(1)) if m else 0
def _outside_fence_lines(lines: list[str]):
"""Yield (index, line) for lines that are not inside a fenced code block."""
fence = None
for i, line in enumerate(lines):
m = _FENCE.match(line)
if fence is None:
if m:
fence = m.group(1)
continue
yield i, line
elif (m and m.group(1)[0] == fence[0] and len(m.group(1)) >= len(fence)
and line.strip() == m.group(1)):
fence = None
def merge_companion(text: str, pages: list[str], transcribed_by: str,
section: str = "Notes") -> str:
"""Put a fresh draft block into an existing companion, keeping all else."""
text = _set_transcribed_by(text, transcribed_by)
lines = text.split("\n")
block = draft_block(pages).split("\n")
visible = list(_outside_fence_lines(lines))
# 1. Replace an existing draft block.
start = next((i for i, l in visible if l.rstrip() == DRAFT_HEADING), None)
if start is not None:
end = None
for i, l in visible:
if i <= start:
continue
if l.strip() == END_MARKER:
end = i + 1
break
if 0 < _heading_level(l) <= 3:
end = i
# Keep the blank line that separated the block from the heading.
while end - 1 > start and lines[end - 1].strip() == "":
end -= 1
break
if end is None:
end = len(lines)
while end - 1 > start and lines[end - 1].strip() == "":
end -= 1
return "\n".join(lines[:start] + block + lines[end:])
# 2. Replace the pending line, wherever it is.
pending = next((i for i, l in visible if l.strip() == PENDING_LINE), None)
if pending is not None:
return "\n".join(lines[:pending] + block + lines[pending + 1:])
# 3. Append to the end of the transcript section.
disc = next((i for i, l in visible if l.rstrip() == f"## {section}"), None)
if disc is not None:
nxt = next((i for i, l in visible if i > disc and 0 < _heading_level(l) <= 2),
len(lines))
ins = nxt
while ins - 1 > disc and lines[ins - 1].strip() == "":
ins -= 1
rest = lines[nxt:]
new = lines[:ins] + [""] + block + [""]
return "\n".join(new + rest if rest else new)
# 4. No transcript section: add the block at the end.
return f"{text.rstrip(chr(10))}\n\n{draft_block(pages)}\n"
def write_atomic(path: Path, text: str) -> None:
"""Write via a temp file in the same folder, so a crash leaves no partial file."""
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.stem}.", suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(text)
os.replace(tmp, path)
except BaseException:
try:
os.unlink(tmp)
except FileNotFoundError:
pass
raise
def write_companion(pdf: Path, root: Path, pages: list[str], transcribed_by: str,
spec: CompanionSpec = BASIC,
titles: dict[str, str] | None = None) -> tuple[Path, bool, list[str]]:
"""Create or update `<stem>.md`. Returns (path, created, warnings)."""
md = pdf.with_suffix(".md")
if md.exists():
text = md.read_text(encoding="utf-8")
write_atomic(md, merge_companion(text, pages, transcribed_by,
spec.transcript_section))
return md, False, []
meta = meta_from_path(pdf, root, spec)
write_atomic(md, new_companion(meta, pages, transcribed_by, spec, titles))
return md, True, meta.warnings

View file

@ -0,0 +1,126 @@
"""Tool 2: extract annotations + underlying highlighted text to Markdown.
For each annotation:
- The note/popup text you wrote
- The text under the highlight rectangle (for Highlight / Underline /
Squiggly / StrikeOut)
- Page number
Output is grouped by page. Each entry uses the highlighted passage as a
blockquote, followed by the note as ordinary prose. YAML frontmatter at
the top carries source filename and counts so downstream pipelines can
parse it cleanly.
"""
from __future__ import annotations
import datetime as _dt
import sys
from dataclasses import dataclass
from pathlib import Path
import click
# pdfannots gives us highlight text + note text together, which is what
# we want. Its API has shifted across versions; we use the documented
# process_file entrypoint.
import pdfannots
@dataclass
class Note:
page: int # 1-indexed
kind: str # 'Highlight', 'Text', etc.
highlighted: str # underlying text (may be empty for sticky notes)
note: str # the comment the user typed (may be empty for bare highlights)
author: str | None
created: str | None
def extract(pdf_path: Path) -> list[Note]:
notes: list[Note] = []
with pdf_path.open("rb") as fh:
doc = pdfannots.process_file(fh, emit_progress_to=None)
for annot in doc.iter_annots():
page_num = annot.pos.page.pageno + 1 if annot.pos else 0
notes.append(Note(
page=page_num,
kind=annot.subtype.name if hasattr(annot.subtype, "name") else str(annot.subtype),
highlighted=(annot.gettext() or "").strip(),
note=(annot.contents or "").strip(),
author=getattr(annot, "author", None),
created=str(annot.created) if getattr(annot, "created", None) else None,
))
notes.sort(key=lambda n: (n.page, n.kind))
return notes
def to_markdown(pdf_path: Path, notes: list[Note]) -> str:
today = _dt.date.today().isoformat()
lines: list[str] = []
lines.append("---")
lines.append(f'source: "{pdf_path.name}"')
lines.append(f"extracted: {today}")
lines.append(f"annotations: {len(notes)}")
lines.append("---")
lines.append("")
lines.append(f"# Notes from {pdf_path.stem}")
lines.append("")
if not notes:
lines.append("_No annotations found._")
return "\n".join(lines) + "\n"
current_page = None
for n in notes:
if n.page != current_page:
lines.append(f"## Page {n.page}")
lines.append("")
current_page = n.page
if n.highlighted:
# Blockquote the highlighted passage. Collapse internal newlines
# so the quote reads as one passage.
passage = " ".join(n.highlighted.split())
lines.append(f"> {passage}")
lines.append("")
if n.note:
# Sticky notes have no highlighted passage; tag them so the
# reader knows the note is freestanding, not a comment on
# the preceding quote.
if not n.highlighted:
lines.append(f"**Note ({n.kind}):** {n.note}")
else:
lines.append(n.note)
lines.append("")
elif not n.highlighted:
lines.append(f"_({n.kind} with no text)_")
lines.append("")
return "\n".join(lines) + "\n"
@click.command()
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.option(
"-o", "--output",
type=click.Path(dir_okay=False, path_type=Path),
help="Output .md path. Default: <input>.md next to the PDF.",
)
@click.option("--stdout", is_flag=True, help="Write markdown to stdout instead of a file.")
def main(pdf: Path, output: Path | None, stdout: bool) -> None:
"""Extract PDF annotations to Markdown."""
notes = extract(pdf)
md = to_markdown(pdf, notes)
if stdout:
sys.stdout.write(md)
return
out_path = output or pdf.with_suffix(".md")
out_path.write_text(md, encoding="utf-8")
print(f"wrote {len(notes)} annotations -> {out_path}", file=sys.stderr)
if __name__ == "__main__":
main()

View file

@ -0,0 +1,439 @@
"""Tool 3: extract notes from scanned notebook pages to Markdown.
Accepts either:
- One or more image files (jpg/png/heic/tiff)
- A PDF (each page is rasterized and processed)
Two backends:
- ollama (default): a vision model on a machine you own, through the Ollama HTTP
API (see ollama.py). Hosts are tried in the order given in the config
file (see scan_config.py). There is no fallback to any cloud service.
- claude: the Anthropic API, used only when asked for with --backend
claude or the config. Set ANTHROPIC_API_KEY. Model defaults to Claude
Opus; override with --model.
Scans under a confidential folder (declared in the config file; none are
built in) default to the ollama backend, refuse the claude backend unless
--allow-cloud is given, and are written to a companion Markdown file shaped
by that folder's settings (see companion.py).
"""
from __future__ import annotations
import base64
import dataclasses
import hashlib
import io
import json
import os
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable
from urllib.parse import urlparse
import click
from PIL import Image
from . import companion
from .ollama import (REPEAT_PLACEHOLDER, REPEAT_RETRIES, NoHostAvailable,
OllamaError, PageOutput, RepetitionError, build_prompt,
loaded_model, provenance, select_host, transcribe_png)
from .scan_config import load_config
from .scan_stats import StatsLog
# Anthropic is imported lazily inside _transcribe_image so the rest of
# the module is usable without the API key set (e.g. for --help), and so
# the ollama backend never loads it.
# Reasonable defaults. The model is one of the few things you might
# want to tweak per-run as new versions ship. Opus is the default
# because handwriting transcription is the one place accuracy
# clearly justifies the cost.
DEFAULT_MODEL = "claude-opus-5-5"
SUPPORTED_IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".tif", ".tiff", ".heic"}
# Exit codes beyond click's usual 1 and 2.
EXIT_NO_HOST = 3
EXIT_TRANSCRIBE_FAILED = 4
TRANSCRIBE_PROMPT = """\
You are transcribing a scanned page from a handwritten research notebook.
Output requirements:
- Reproduce the page content as clean Markdown.
- Preserve structure: headings, bullet points, indentation, numbered lists.
- Render equations and chemical formulas in LaTeX. Inline math uses $...$, display math uses $$...$$.
- Preserve units, subscripts, and Greek letters faithfully.
- For figures, diagrams, or sketches, insert a single line like:
`[FIGURE: brief description of what is drawn]`
- For text you cannot read with confidence, use `[?]` for a single illegible word
or `[??: best guess]` if you have a tentative reading.
- Do NOT add commentary, summaries, or explanations. Transcribe only.
- Do NOT wrap your answer in code fences. Output raw Markdown directly.
If the page is blank or contains no notebook content, output exactly:
[BLANK PAGE]
"""
@dataclass
class PageResult:
index: int # 1-indexed page/image number
source: str # filename or "page N of source.pdf"
markdown: str
def _image_to_jpeg_bytes(img: Image.Image, max_side: int = 2000, quality: int = 85) -> bytes:
"""Downscale + re-encode to keep API payloads reasonable."""
img = img.convert("RGB")
w, h = img.size
longest = max(w, h)
if longest > max_side:
scale = max_side / longest
img = img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=quality, optimize=True)
return buf.getvalue()
def _image_to_png_bytes(img: Image.Image) -> bytes:
"""Lossless, full resolution, in memory only (nothing is written to disk)."""
buf = io.BytesIO()
img.convert("RGB").save(buf, format="PNG")
return buf.getvalue()
def _transcribe_image(jpeg_bytes: bytes, model: str) -> str:
"""One Claude call. The ollama backend lives in ollama.transcribe_png."""
import anthropic # imported here so --help works without the SDK
client = anthropic.Anthropic() # picks up ANTHROPIC_API_KEY
b64 = base64.standard_b64encode(jpeg_bytes).decode("ascii")
resp = client.messages.create(
model=model,
max_tokens=4096,
messages=[{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": b64,
},
},
{"type": "text", "text": TRANSCRIBE_PROMPT},
],
}],
)
# Concatenate text blocks (usually just one).
return "".join(b.text for b in resp.content if getattr(b, "type", None) == "text").strip()
def _iter_pages(paths: list[Path], dpi: int = 200) -> list[tuple[str, Callable[[], Image.Image]]]:
"""Return (label, loader) for every page across all inputs.
Pages are rendered only when their loader is called, so a long notebook
never holds more than one page image in memory.
"""
out: list[tuple[str, Callable[[], Image.Image]]] = []
for p in paths:
ext = p.suffix.lower()
if ext == ".pdf":
import pypdfium2 as pdfium
doc = pdfium.PdfDocument(p)
for i in range(len(doc)):
# 200 DPI is plenty for handwriting OCR.
out.append((f"{p.name} p.{i + 1}",
lambda doc=doc, i=i: doc[i].render(scale=dpi / 72).to_pil()))
elif ext in SUPPORTED_IMAGE_EXT:
out.append((p.name, lambda p=p: Image.open(p)))
else:
print(f"warning: skipping unsupported file: {p}", file=sys.stderr)
return out
def _checkpoint_path(first_input: Path) -> Path:
"""Progress file kept next to the scan while a run is incomplete."""
return first_input.parent / f".{first_input.stem}.scan-partial.json"
def _checkpoint_key(paths: list[Path], transcribed_by: str, prompt: str, dpi: int) -> dict:
return {
"inputs": [[p.name, p.stat().st_size] for p in paths],
"transcribed_by": transcribed_by,
"prompt_sha1": hashlib.sha1(prompt.encode()).hexdigest(),
"dpi": dpi,
}
def _load_checkpoint(path: Path, key: dict) -> list[dict]:
"""Pages saved by an earlier, interrupted run with the same settings."""
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError):
return []
if data.get("key") != key:
print("note: ignoring saved progress from a run with different "
"inputs, model, prompt, or dpi", file=sys.stderr)
return []
return list(data.get("pages", []))
def _save_checkpoint(path: Path, key: dict, done: list[dict]) -> None:
companion.write_atomic(path, json.dumps({"key": key, "pages": done}))
def to_markdown(results: list[PageResult], title: str) -> str:
import datetime as _dt
lines = [
"---",
f'title: "{title}"',
f"extracted: {_dt.date.today().isoformat()}",
f"pages: {len(results)}",
"---",
"",
f"# {title}",
"",
]
for r in results:
lines.append(f"## {r.source}")
lines.append("")
lines.append(r.markdown if r.markdown else "_(empty)_")
lines.append("")
return "\n".join(lines) + "\n"
def to_plain_markdown(pages: list[str], title: str, transcribed_by: str) -> str:
"""Output for ollama runs outside companion mode: <!-- page N --> markers."""
import datetime as _dt
return "\n".join([
"---",
f'title: "{title}"',
f"extracted: {_dt.date.today().isoformat()}",
f"pages: {len(pages)}",
f"transcribed_by: {transcribed_by}",
"---",
"",
f"# {title}",
"",
companion.render_transcript(pages),
"",
])
def _fail(msg: str, code: int = 2) -> None:
print(f"error: {msg}", file=sys.stderr)
sys.exit(code)
@click.command()
@click.argument(
"inputs", nargs=-1, required=True,
type=click.Path(exists=True, dir_okay=False, path_type=Path),
)
@click.option(
"-o", "--output",
type=click.Path(dir_okay=False, path_type=Path),
help="Output .md path. Default: <first input stem>.md",
)
@click.option(
"--backend", type=click.Choice(["claude", "ollama"]), default=None,
help="Transcription backend. Default: ollama for scans in confidential "
"folders, else the "
"config's backend (ollama unless set).",
)
@click.option(
"--host", default=None,
help="Ollama only: use just this host from the config (e.g. remote, local) "
"and fail if it is unavailable.",
)
@click.option("--model", default=None,
help=f"Model name. Default: {DEFAULT_MODEL} for claude, the "
"config's per-host model for ollama.")
@click.option("--companion/--no-companion", "companion_flag", default=None,
help="Write the companion <stem>.md next to the scan. Default: on "
"for scans in confidential folders.")
@click.option("--allow-cloud", is_flag=True,
help="Permit --backend claude for scans in confidential folders.")
@click.option("--config", "config_path",
type=click.Path(dir_okay=False, path_type=Path), default=None,
help="Config file. Default: ~/.config/pdf-tools/scan-notes.toml")
@click.option("--dpi", type=int, default=None, help="PDF render resolution (config default 200).")
@click.option("--timeout", type=float, default=None,
help="Ollama per-page timeout in seconds (config default 300).")
@click.option(
"--title", default=None,
help="Title for the markdown document. Default: first input stem.",
)
@click.option("--stdout", is_flag=True, help="Write markdown to stdout.")
def main(inputs: tuple[Path, ...], output: Path | None, backend: str | None,
host: str | None, model: str | None, companion_flag: bool | None,
allow_cloud: bool, config_path: Path | None, dpi: int | None,
timeout: float | None, title: str | None, stdout: bool) -> None:
"""Transcribe scanned notebook pages to Markdown."""
cfg = load_config(config_path)
paths = list(inputs)
folders = [cfg.confidential_folder(p) for p in paths]
confidential = any(folders)
if confidential and not all(folders):
_fail("do not mix scans from confidential folders with other files in one run")
folder = folders[0]
companion_mode = confidential if companion_flag is None else companion_flag
if companion_mode and not confidential:
roots = ", ".join(str(c.root) for c in cfg.confidential) or "none configured"
_fail(f"--companion needs a scan in a confidential folder ({roots})")
if companion_mode:
if len(paths) != 1 or paths[0].suffix.lower() != ".pdf":
_fail("companion mode takes exactly one PDF")
if output or stdout:
_fail("companion mode writes <stem>.md next to the scan; "
"-o and --stdout need --no-companion")
backend = backend or ("ollama" if confidential else cfg.backend)
if backend not in ("claude", "ollama"):
_fail(f"unknown backend '{backend}' in config")
if backend == "claude" and confidential and not allow_cloud:
_fail(f"scans under the confidential folder {folder.root} are not "
"sent to Claude unless "
"--allow-cloud is given; use --backend ollama")
if host and backend != "ollama":
_fail("--host applies only to --backend ollama")
if backend == "claude":
if not os.environ.get("ANTHROPIC_API_KEY"):
_fail("ANTHROPIC_API_KEY not set")
claude_model = model or DEFAULT_MODEL
transcribed_by = f"claude/{claude_model}"
else:
hosts = cfg.hosts
if model:
hosts = [dataclasses.replace(h, model=model) for h in hosts]
try:
chosen = select_host(hosts, cfg.probe_timeout_s, forced=host)
except NoHostAvailable as e:
_fail(f"{e}. Not falling back to a cloud service.", EXIT_NO_HOST)
print(f"using Ollama host {chosen.name} "
f"({chosen.url}, {chosen.model})", file=sys.stderr)
transcribed_by = provenance(chosen)
# Global names and prompt_extra text, plus those of every folder
# the inputs come from.
names = dict.fromkeys(n for f in folders for n in cfg.names_for(f))
extras = dict.fromkeys(e for f in folders for e in cfg.extras_for(f))
prompt = build_prompt(list(names), list(extras))
page_timeout = timeout or cfg.timeout_s
dpi = dpi or cfg.dpi
pages = _iter_pages(paths, dpi=dpi)
if not pages:
_fail("no usable pages found", 1)
# Progress is saved after every page, so an interrupted run resumes
# where it stopped. The file sits next to the scan and is removed once
# the output is written.
ckpt = _checkpoint_path(paths[0])
key = _checkpoint_key(paths, transcribed_by,
TRANSCRIBE_PROMPT if backend == "claude" else prompt, dpi)
done = _load_checkpoint(ckpt, key)[:len(pages)]
if done:
print(f"resuming after page {len(done)} of {len(pages)}", file=sys.stderr)
stats_log = StatsLog(cfg.stats_log if backend == "ollama" else None)
run_id = f"{time.strftime('%Y%m%dT%H%M%S')}-{os.getpid()}"
loaded: dict | None = None # /api/ps snapshot, taken once the model is loaded
for i, (label, load) in enumerate(pages, start=1):
if i <= len(done):
continue
if backend == "claude":
print(f"[{i}/{len(pages)}] {label} ...", file=sys.stderr)
jpeg = _image_to_jpeg_bytes(load())
md = _transcribe_image(jpeg, model=claude_model)
else:
print(f"[{i}/{len(pages)}] {label} on {chosen.name} ...",
end="", file=sys.stderr, flush=True)
t0 = time.monotonic()
img = load()
try:
page = transcribe_png(_image_to_png_bytes(img), chosen, prompt,
page_timeout, cfg.max_output_tokens)
except RepetitionError:
print("", file=sys.stderr)
print(f"warning: page {i} kept looping after {len(REPEAT_RETRIES)} "
"retries; writing a placeholder", file=sys.stderr)
page = PageOutput(REPEAT_PLACEHOLDER, {
"prompt_tokens": 0, "output_tokens": 0, "context_tokens": 0,
"done_reason": "repeat_abort", "retries": len(REPEAT_RETRIES),
"total_s": round(time.monotonic() - t0, 2)})
except OllamaError as e:
print("", file=sys.stderr)
saved = (f"pages 1-{len(done)} are saved; rerun to resume"
if done else "nothing was saved")
_fail(f"page {i}: {e}. {saved}.", EXIT_TRANSCRIBE_FAILED)
md, st = page.text, page.stats
if loaded is None:
loaded = loaded_model(chosen, cfg.probe_timeout_s)
retried = f", {st['retries']} retries" if st.get("retries") else ""
print(f" {time.monotonic() - t0:.0f} s, {st['context_tokens']} tokens{retried}",
file=sys.stderr)
nctx = loaded.get("loaded_num_ctx") or chosen.num_ctx
if st["done_reason"] == "length":
print(f"warning: page {i} output stopped at {st['output_tokens']} "
"tokens after retries; marked as cut off", file=sys.stderr)
elif nctx and st["context_tokens"] >= 0.9 * nctx:
print(f"warning: page {i} used {st['context_tokens']} of "
f"{nctx} context tokens", file=sys.stderr)
stats_log.write({
"time": time.strftime("%Y-%m-%dT%H:%M:%S"),
"run": run_id,
"host": urlparse(chosen.url).hostname,
"model": chosen.model,
"num_ctx_setting": chosen.num_ctx,
"dpi": dpi,
"label": label,
"page": i,
"image_px": list(img.size),
**st,
**loaded,
})
done.append({"label": label, "text": md})
_save_checkpoint(ckpt, key, done)
texts = [d["text"] for d in done]
results = [PageResult(index=i, source=d["label"], markdown=d["text"])
for i, d in enumerate(done, start=1)]
if companion_mode:
path, created, warnings = companion.write_companion(
paths[0], folder.root, texts, transcribed_by, folder.companion,
folder.titles)
for w in warnings:
print(f"warning: {w}", file=sys.stderr)
verb = "created" if created else "updated draft transcript in"
print(f"{verb} {path}", file=sys.stderr)
ckpt.unlink(missing_ok=True)
return
doc_title = title or paths[0].stem
if backend == "claude":
md_out = to_markdown(results, doc_title)
else:
md_out = to_plain_markdown(texts, doc_title, transcribed_by)
if stdout:
sys.stdout.write(md_out)
ckpt.unlink(missing_ok=True)
return
out_path = output or paths[0].with_suffix(".md")
companion.write_atomic(out_path, md_out)
ckpt.unlink(missing_ok=True)
print(f"wrote {len(results)} pages -> {out_path}", file=sys.stderr)
if __name__ == "__main__":
main()

319
src/pdf_tools/ollama.py Normal file
View file

@ -0,0 +1,319 @@
"""Local transcription through the Ollama HTTP API.
Nothing in this module talks to a cloud service, and it never imports the
Anthropic SDK. If no configured Ollama host is usable, select_host() raises
NoHostAvailable and the caller exits; there is deliberately no fallback to
Claude, because this backend exists for confidential material.
HTTP goes through _http_json(), which tests replace with a fake.
"""
from __future__ import annotations
import base64
import json
import re
import socket
import sys
import urllib.error
import urllib.request
from dataclasses import dataclass, field
from urllib.parse import urlparse
from .scan_config import OllamaHost
# The instruction sent with every page. Edit here to change it.
HANDWRITING_PROMPT = """\
Transcribe the handwriting in this image exactly as written. Preserve line
breaks, indentation, and bullets. Do not correct spelling, expand
abbreviations, or complete partial words. If you cannot read a word with
confidence, write [illegible] or [?word] instead of guessing; this applies
especially to names. Output only the transcript, in plain text, with no
commentary and no emojis.
"""
# Added only when the config lists names.
NAMES_PROMPT = """\
The notes may mention the following people and organizations. If a word
plausibly matches one of these names, use that spelling; otherwise write
[illegible].
"""
@dataclass
class PageOutput:
text: str
# Counts and timings Ollama reports with each response (see page_stats).
stats: dict = field(default_factory=dict)
class OllamaError(RuntimeError):
pass
class NoHostAvailable(OllamaError):
pass
class RepetitionError(OllamaError):
"""Ollama stopped the model because it kept repeating the same tokens."""
# At temperature 0 a page that sends the model into a loop fails the same
# way every time. These retries add a little randomness and a repeat
# penalty; they are used only after that specific failure.
REPEAT_RETRIES = (
{"temperature": 0.3, "repeat_penalty": 1.15},
{"temperature": 0.6, "repeat_penalty": 1.3},
)
# Written in place of a page that still loops after the retries.
REPEAT_PLACEHOLDER = "[page not transcribed: the model kept repeating itself; see scan]"
# Appended to a page whose output still hits max_output_tokens after the retries.
CUT_OFF_MARKER = "[transcript cut off at the output limit; see scan]"
def build_prompt(names: list[str], extras: list[str] | None = None) -> str:
"""HANDWRITING_PROMPT, then any configured prompt_extra text, then names."""
parts = [HANDWRITING_PROMPT]
parts += [e.strip("\n") + "\n" for e in (extras or []) if e.strip()]
if names:
listed = "\n".join(f"- {n}" for n in names)
parts.append(f"{NAMES_PROMPT}{listed}\n")
return "\n".join(parts)
def provenance(host: OllamaHost) -> str:
"""Value for the transcribed_by: field, e.g. ollama/localhost/gemma4:e4b."""
return f"ollama/{urlparse(host.url).hostname}/{host.model}"
def _http_json(method: str, url: str, payload: dict | None, timeout: float) -> dict:
data = json.dumps(payload).encode() if payload is not None else None
req = urllib.request.Request(
url, data=data, method=method,
headers={"Content-Type": "application/json"} if data else {},
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return json.loads(resp.read().decode("utf-8"))
def _model_present(wanted: str, available: list[str]) -> bool:
# Ollama reports untagged models as "name:latest".
if ":" not in wanted:
wanted = f"{wanted}:latest"
return wanted in available
def check_host(host: OllamaHost, timeout: float) -> str | None:
"""Return None if the host is up and has its model, else a short reason."""
try:
tags = _http_json("GET", f"{host.url}/api/tags", None, timeout)
except (urllib.error.URLError, socket.timeout, TimeoutError, OSError, ValueError):
return "unreachable"
available = [m.get("name", "") for m in tags.get("models", [])]
if not _model_present(host.model, available):
return f"does not have {host.model}"
return None
def _describe(host: OllamaHost) -> str:
return f"{host.name} ({urlparse(host.url).hostname}, {host.model})"
def select_host(hosts: list[OllamaHost], timeout: float,
forced: str | None = None) -> OllamaHost:
"""Return the first usable host, printing one line per host skipped.
With forced set, only that host is considered and failure is an error.
"""
if forced is not None:
match = [h for h in hosts if h.name == forced]
if not match:
names = ", ".join(h.name for h in hosts) or "none"
raise NoHostAvailable(f"no Ollama host named '{forced}' in config (have: {names})")
reason = check_host(match[0], timeout)
if reason:
raise NoHostAvailable(f"Ollama host {_describe(match[0])} {reason}")
return match[0]
for i, host in enumerate(hosts):
reason = check_host(host, timeout)
if reason is None:
return host
nxt = hosts[i + 1] if i + 1 < len(hosts) else None
tail = f"; falling back to {_describe(nxt)}" if nxt else ""
print(f"Ollama host {_describe(host)} {reason}{tail}", file=sys.stderr)
raise NoHostAvailable("no Ollama host is available")
_THINK_BLOCK = re.compile(r"<(think|thinking)>.*?</\1>", re.DOTALL | re.IGNORECASE)
_THINK_CLOSE = re.compile(r"^.*?</(think|thinking)>", re.DOTALL | re.IGNORECASE)
def strip_thinking(text: str) -> str:
"""Remove reasoning text that some models emit despite think: false."""
text = _THINK_BLOCK.sub("", text)
# An unmatched closing tag means the opening tag was never sent.
text = _THINK_CLOSE.sub("", text, count=1)
return text.strip("\n").rstrip()
def page_stats(resp: dict) -> dict:
"""Token counts, stop reason, and timings from an /api/chat response.
prompt_tokens includes the image; context_tokens is what the request
occupied in the model's context window (prompt plus output).
"""
ns = 1e-9
prompt = int(resp.get("prompt_eval_count") or 0)
output = int(resp.get("eval_count") or 0)
return {
"prompt_tokens": prompt,
"output_tokens": output,
"context_tokens": prompt + output,
"done_reason": resp.get("done_reason"),
"total_s": round((resp.get("total_duration") or 0) * ns, 2),
"load_s": round((resp.get("load_duration") or 0) * ns, 2),
"prompt_s": round((resp.get("prompt_eval_duration") or 0) * ns, 2),
"output_s": round((resp.get("eval_duration") or 0) * ns, 2),
}
def loaded_model(host: OllamaHost, timeout: float) -> dict:
"""How the host has the model loaded, from /api/ps: total size, the part
in GPU memory, and the context length it was loaded with. Empty if the
host does not report it."""
try:
ps = _http_json("GET", f"{host.url}/api/ps", None, timeout)
except (urllib.error.URLError, socket.timeout, TimeoutError, OSError, ValueError):
return {}
wanted = host.model if ":" in host.model else f"{host.model}:latest"
for m in ps.get("models", []):
if m.get("name") == wanted or m.get("model") == wanted:
size, vram = int(m.get("size") or 0), int(m.get("size_vram") or 0)
return {
"loaded_bytes": size,
"vram_bytes": vram,
"gpu_fraction": round(vram / size, 3) if size else None,
"loaded_num_ctx": m.get("context_length"),
}
return {}
def _chat(url: str, payload: dict, timeout: float) -> dict:
try:
return _http_json("POST", url, payload, timeout)
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", "replace")
if e.code == 400 and "think" in body.lower() and "think" in payload:
# Some models reject the think field outright; retry without it.
payload.pop("think")
return _chat(url, payload, timeout)
if "repeat limit" in body.lower():
raise RepetitionError(f"{url}: {body[:200]}") from None
raise OllamaError(f"{url} returned HTTP {e.code}: {body[:200]}") from None
except (socket.timeout, TimeoutError):
raise OllamaError(f"no response from {url} within {timeout:.0f} s") from None
except (urllib.error.URLError, OSError) as e:
raise OllamaError(f"cannot reach {url}: {e}") from None
def _page_request(png_bytes: bytes, host: OllamaHost, prompt: str,
max_output: int | None) -> tuple[dict, dict]:
"""First-attempt options and the /api/chat payload for one page image."""
options: dict = {"temperature": 0}
if host.num_ctx:
options["num_ctx"] = host.num_ctx
if max_output:
options["num_predict"] = max_output
payload = {
"model": host.model,
"messages": [{
"role": "user",
"content": prompt,
"images": [base64.b64encode(png_bytes).decode("ascii")],
}],
"stream": False,
"think": False,
"options": options,
}
return options, payload
def transcribe_png(png_bytes: bytes, host: OllamaHost, prompt: str,
timeout: float, max_output: int | None = None) -> PageOutput:
"""One page, one request. Returns the transcript text and its stats.
A model that loops either trips Ollama's repeat check (RepetitionError)
or runs on until max_output tokens (done_reason "length"). Either way the
page is retried with the REPEAT_RETRIES options; stats["retries"] records
how many were needed. If every retry loops, RepetitionError is raised
again, or the last output is returned with CUT_OFF_MARKER appended.
"""
options, payload = _page_request(png_bytes, host, prompt, max_output)
url = f"{host.url}/api/chat"
last = len(REPEAT_RETRIES)
reasons: list[str] = [] # why each earlier attempt was retried
for attempt, extra in enumerate(((), *REPEAT_RETRIES)):
payload["options"] = {**options, **dict(extra)}
try:
resp = _chat(url, payload, timeout)
except RepetitionError:
if attempt == last:
raise
reasons.append("repeat_limit")
continue
if resp.get("done_reason") == "length" and attempt < last:
reasons.append("output_cap")
continue
stats = page_stats(resp)
stats["retries"] = attempt
if attempt:
stats["retry_options"] = dict(extra)
stats["retry_reasons"] = reasons
text = strip_thinking((resp.get("message") or {}).get("content", ""))
if stats["done_reason"] == "length":
text = f"{text}\n{CUT_OFF_MARKER}"
return PageOutput(text, stats)
raise AssertionError("unreachable")
def chat_json(png_bytes: bytes, host: OllamaHost, prompt: str, schema: dict,
timeout: float, max_output: int | None = None) -> tuple[dict, dict]:
"""One page, one request, with the answer constrained to a JSON schema.
Returns (answer, stats). Retried with REPEAT_RETRIES when the model loops,
stops at max_output (the JSON would be incomplete), or returns something
that is not a JSON object with the schema's required keys. Raises
OllamaError if no attempt gives a usable answer.
"""
options, payload = _page_request(png_bytes, host, prompt, max_output)
payload["format"] = schema
url = f"{host.url}/api/chat"
required = schema.get("required", [])
reasons: list[str] = []
for attempt, extra in enumerate(((), *REPEAT_RETRIES)):
payload["options"] = {**options, **dict(extra)}
try:
resp = _chat(url, payload, timeout)
except RepetitionError:
reasons.append("repeat_limit")
continue
if resp.get("done_reason") == "length":
reasons.append("output_cap")
continue
try:
answer = json.loads(strip_thinking((resp.get("message") or {}).get("content", "")))
except ValueError:
answer = None
if not isinstance(answer, dict) or any(k not in answer for k in required):
reasons.append("bad_json")
continue
stats = page_stats(resp)
stats["retries"] = attempt
if attempt:
stats["retry_options"] = dict(extra)
stats["retry_reasons"] = reasons
return answer, stats
raise OllamaError(f"no usable answer after {len(REPEAT_RETRIES)} retries "
f"({', '.join(reasons)})")

View file

@ -0,0 +1,254 @@
"""Configuration for scan-notes-md.
Settings come from three layers, later ones winning:
1. DEFAULTS below
2. ~/.config/pdf-tools/scan-notes.toml (or $PDF_TOOLS_SCAN_CONFIG, or --config)
3. command-line flags (applied in extract_scan.main)
The file is optional. Only the keys you set are overridden; a [[ollama.hosts]]
list in the file replaces the default host list as a whole.
Confidential folders exist only in the config file; none are built in. Scans
under any listed root stay on Ollama hosts (Claude is refused without
--allow-cloud) and go to a companion <stem>.md shaped by the folder's
[confidential.companion] table (see companion.CompanionSpec). Each folder can
also carry its own `names` list and `prompt_extra` text for the Ollama
prompt, added to the top-level ones. The [index] table holds the settings
for scan-notes-index (see scan_index.py). See
scan-notes.example.toml for a worked example.
"""
from __future__ import annotations
import copy
import os
import sys
import tomllib
from dataclasses import dataclass, field, fields
from pathlib import Path
from .companion import BASIC, CompanionSpec, PathRule
from .scan_stats import DEFAULT_STATS_LOG
CONFIG_ENV = "PDF_TOOLS_SCAN_CONFIG"
DEFAULT_CONFIG_PATH = Path.home() / ".config" / "pdf-tools" / "scan-notes.toml"
DEFAULTS: dict = {
"backend": "ollama",
"dpi": 200,
"names": [],
"prompt_extra": "",
# Per-page context and GPU statistics (no transcript text); "" disables.
"stats_log": DEFAULT_STATS_LOG,
"ollama": {
"timeout_s": 300,
"probe_timeout_s": 2.5,
# Output cap per page; a model that loops stops here instead of
# running until the timeout. 0 means no cap.
"max_output_tokens": 2048,
"hosts": [
{"name": "local", "url": "http://localhost:11434", "model": "gemma4:e4b"},
],
},
"confidential": [],
# scan-notes-index: per-page search metadata from transcripts.
"index": {
# Added to the index prompt, on top of the transcription names.
"names": [],
"prompt_extra": "",
"kinds": ["calculation", "idea", "sketch", "talk notes", "meeting notes",
"reading notes", "to-do", "other", "blank"],
"max_output_tokens": 1536,
# Characters of the previous and next page transcripts sent as context.
"context_chars": 2000,
},
}
_FOLDER_KEYS = {"root", "names", "prompt_extra", "titles", "companion"}
_SPEC_KEYS = {f.name for f in fields(CompanionSpec)}
_RULE_KEYS = {"match", "set", "title"}
_HOST_KEYS = {"name", "url", "model", "num_ctx"}
@dataclass
class OllamaHost:
name: str
url: str
model: str
num_ctx: int | None = None
@dataclass
class ConfidentialFolder:
root: Path
companion: CompanionSpec = field(default_factory=lambda: BASIC)
# Display names for the "# Title" line of a new companion, keyed by
# folder name (or "<name>.short"). Others are turned into words.
titles: dict[str, str] = field(default_factory=dict)
# Added to the Ollama prompt only for scans in this folder.
names: list[str] = field(default_factory=list)
prompt_extra: str = ""
@dataclass
class IndexSettings:
names: list[str] = field(default_factory=list)
prompt_extra: str = ""
kinds: list[str] = field(default_factory=list)
max_output_tokens: int = 1536
context_chars: int = 2000
@dataclass
class ScanConfig:
backend: str
dpi: int
names: list[str]
prompt_extra: str
timeout_s: float
probe_timeout_s: float
max_output_tokens: int
hosts: list[OllamaHost]
confidential: list[ConfidentialFolder]
stats_log: Path | None = None
index: IndexSettings = field(default_factory=IndexSettings)
source: Path | None = None # file the settings were read from, if any
def confidential_folder(self, path: Path) -> ConfidentialFolder | None:
"""The deepest confidential folder containing path, if any."""
p = path.expanduser().resolve()
best = None
for c in self.confidential:
root = c.root.resolve()
if p == root or root in p.parents:
if best is None or len(root.parts) > len(best.root.resolve().parts):
best = c
return best
def names_for(self, folder: ConfidentialFolder | None) -> list[str]:
"""Global names plus the folder's own, without duplicates."""
extra = folder.names if folder else []
return list(dict.fromkeys([*self.names, *extra]))
def extras_for(self, folder: ConfidentialFolder | None) -> list[str]:
"""Top-level prompt_extra plus the folder's own, skipping empty ones."""
texts = [self.prompt_extra, folder.prompt_extra if folder else ""]
return [t for t in dict.fromkeys(texts) if t.strip()]
def _merge(base: dict, over: dict) -> dict:
out = copy.deepcopy(base)
for k, v in over.items():
if isinstance(v, dict) and isinstance(out.get(k), dict):
out[k] = _merge(out[k], v)
else:
out[k] = v
return out
def _die(path: Path, msg: str) -> None:
print(f"error: {path}: {msg}", file=sys.stderr)
sys.exit(2)
def _unknown(path: Path, where: str, keys, allowed: set[str]) -> None:
extra = sorted(set(keys) - allowed)
if extra:
_die(path, f"unknown key(s) in {where}: {', '.join(extra)}")
def _spec(path: Path, raw) -> CompanionSpec:
if raw is None:
return BASIC
if not isinstance(raw, dict):
_die(path, "companion must be a table ([confidential.companion]); "
"leave it out for the basic layout")
_unknown(path, "[confidential.companion]", raw, _SPEC_KEYS)
rules = []
for r in raw.get("paths", []):
_unknown(path, "[[confidential.companion.paths]]", r, _RULE_KEYS)
if "match" not in r:
_die(path, "each [[confidential.companion.paths]] entry needs match")
rules.append(PathRule(match=str(r["match"]),
set={k: str(v) for k, v in r.get("set", {}).items()},
title=r.get("title")))
kw = {k: v for k, v in raw.items() if k != "paths"}
if "acronyms" in kw:
kw["acronyms"] = set(kw["acronyms"])
spec = CompanionSpec(**{**BASIC.__dict__, **kw, "paths": rules})
if "transcript_section" not in raw and spec.transcript_section not in spec.sections:
spec.transcript_section = spec.sections[0] if spec.sections else "Notes"
return spec
def load_config(path: Path | None = None) -> ScanConfig:
"""Read the config file (if present) on top of DEFAULTS."""
explicit = path is not None or bool(os.environ.get(CONFIG_ENV))
path = path or Path(os.environ.get(CONFIG_ENV) or DEFAULT_CONFIG_PATH)
raw = DEFAULTS
source = None
if path.exists():
try:
with path.open("rb") as f:
raw = _merge(DEFAULTS, tomllib.load(f))
except tomllib.TOMLDecodeError as e:
_die(path, f"cannot parse: {e}")
source = path
elif explicit:
print(f"error: config file not found: {path}", file=sys.stderr)
sys.exit(2)
_unknown(path, "the top level", raw, set(DEFAULTS))
o = raw["ollama"]
_unknown(path, "[ollama] (top-level keys must come before the first "
"[table])", o, set(DEFAULTS["ollama"]))
hosts = []
for h in o["hosts"]:
_unknown(path, "[[ollama.hosts]] (top-level keys must come before "
"the first [table])", h, _HOST_KEYS)
try:
hosts.append(OllamaHost(
name=str(h["name"]), url=str(h["url"]).rstrip("/"),
model=str(h["model"]), num_ctx=h.get("num_ctx"),
))
except KeyError as e:
_die(path, f"each [[ollama.hosts]] entry needs {e}")
confidential = []
for c in raw["confidential"]:
_unknown(path, "[[confidential]]", c, _FOLDER_KEYS)
if "root" not in c:
_die(path, "each [[confidential]] entry needs a root")
confidential.append(ConfidentialFolder(
root=Path(c["root"]).expanduser(),
companion=_spec(path, c.get("companion")),
titles={str(k): str(v) for k, v in c.get("titles", {}).items()},
names=[str(n) for n in c.get("names", [])],
prompt_extra=str(c.get("prompt_extra", "")),
))
ix = raw["index"]
_unknown(path, "[index]", ix, set(DEFAULTS["index"]))
kinds = [str(k) for k in ix["kinds"]]
if not kinds:
_die(path, "[index] kinds must not be empty")
index = IndexSettings(
names=[str(n) for n in ix["names"]],
prompt_extra=str(ix["prompt_extra"]),
kinds=kinds,
max_output_tokens=int(ix["max_output_tokens"]),
context_chars=int(ix["context_chars"]),
)
return ScanConfig(
backend=raw["backend"],
dpi=int(raw["dpi"]),
names=[str(n) for n in raw.get("names", [])],
prompt_extra=str(raw.get("prompt_extra", "")),
timeout_s=float(o["timeout_s"]),
probe_timeout_s=float(o["probe_timeout_s"]),
max_output_tokens=int(o["max_output_tokens"]),
hosts=hosts,
confidential=confidential,
stats_log=Path(raw["stats_log"]).expanduser() if raw.get("stats_log") else None,
index=index,
source=source,
)

316
src/pdf_tools/scan_index.py Normal file
View file

@ -0,0 +1,316 @@
"""scan-notes-index: per-page search metadata for a transcribed scan.
Reads a scan (PDF) and its transcript (the <stem>.md written by
scan-notes-md, plain or companion; any file with <!-- page N --> markers and
fenced page text). For each page it sends the page image, the page's
transcript, and the neighboring pages' transcripts to an Ollama host and asks
for a summary, key points, a kind of page, topics, references, the event the
page records, and the people named. Each page becomes one Markdown file,
"<stem> pNNN.md", in "<stem> pages/" next to the transcript, so Spotlight,
DEVONthink, or a later vector index finds the page itself rather than the
whole notebook.
Like the Ollama backend of scan-notes-md, this never calls a cloud service.
Existing page files are kept, so an interrupted run resumes where it stopped;
--force rewrites them. Per-page stats go to the stats log with task "index".
"""
from __future__ import annotations
import dataclasses
import datetime as dt
import json
import os
import re
import sys
import time
from pathlib import Path
from urllib.parse import urlparse
import click
from .companion import write_atomic
from .extract_scan import EXIT_NO_HOST, EXIT_TRANSCRIBE_FAILED, _image_to_png_bytes, _iter_pages
from .ollama import (NAMES_PROMPT, NoHostAvailable, OllamaError, chat_json,
loaded_model, provenance, select_host)
from .scan_config import load_config
from .scan_stats import StatsLog
INDEX_PROMPT = """\
You are indexing one page of a handwritten notebook so that it can be found
later by text search. A notebook may mix calculations, research ideas, notes
taken during talks and seminars, reading notes, to-do lists, and notes from
meetings.
You are given the page image and a draft transcript of it. The transcripts of
the previous and next pages are included only as context: describe THIS page
only.
Fill in the fields as follows.
- summary: a short paragraph (three to six sentences) on what this page is
about, in plain words the writer would use when searching. Say what the
page is (a calculation, notes from a meeting or talk, an idea), what it
covers, and what it concludes, decides, or leaves open. For meeting notes,
name the discussion items; for a talk, the speaker's main points; for a
calculation, what is computed, from what, and the result if written. If
the page names a law, model, method, or quantity, use that name. If an
equation is unmistakably a standard one, you may name it; otherwise
describe what it relates. Do not guess at a purpose that is not on the
page.
- key_points: up to eight short items, each one specific thing on the page:
a discussion item, decision, action, result, number with units, idea, or
question. Use the page's own terms. Empty for a blank page.
- kind: the one word or phrase from the list that fits best.{kind_note}
- topics: three to eight short search phrases (subjects, methods,
materials, projects, organizations). Empty for a blank page.
- references: every citation, paper, book, DOI, arXiv number, or URL written
on the page, copied as written and tidied into one line (authors, journal or
book, volume, pages, year, as far as written). Never add a title, year, or
author that is not on the page. Empty if there are none.
- event: the talk, seminar, meeting, or call the page records, with speaker,
organization, board, or committee if written or shown in a page heading.
Empty if the page does not record an event.
- event_date: a date written on the page, as YYYY-MM-DD if complete,
otherwise as written. Empty if none.
- people: names of people written on the page, spelled as written.
- continues_from_previous / continues_to_next: true if this page plainly
continues a calculation, list, or set of notes from the previous page, or
onto the next.
"""
SUMMARY_HEADING = "## Summary (machine-generated, unverified)"
_PAGE = re.compile(r"^<!-- page (\d+) -->\s*\n\s*(`{3,})text\n(.*?)\n?\2[ \t]*$",
re.MULTILINE | re.DOTALL)
_STR_FIELDS = ("summary", "event", "event_date")
_LIST_FIELDS = ("key_points", "topics", "references", "people")
def schema(kinds: list[str]) -> dict:
"""JSON schema the model's answer must follow."""
strings = {"type": "array", "items": {"type": "string"}}
props = {
"summary": {"type": "string"},
"key_points": strings,
"kind": {"type": "string", "enum": kinds},
"topics": strings,
"references": strings,
"event": {"type": "string"},
"event_date": {"type": "string"},
"people": strings,
"continues_from_previous": {"type": "boolean"},
"continues_to_next": {"type": "boolean"},
}
return {"type": "object", "properties": props, "required": list(props)}
def build_index_prompt(kinds: list[str], names: list[str], extras: list[str]) -> str:
"""INDEX_PROMPT, then any [index] prompt_extra text, then names."""
note = ('\n Use "calculation" for derivations and estimates as well.'
if "calculation" in kinds else "")
parts = [INDEX_PROMPT.format(kind_note=note)]
parts += [e.strip("\n") + "\n" for e in extras if e.strip()]
if names:
parts.append(NAMES_PROMPT + "\n".join(f"- {n}" for n in names) + "\n")
return "\n".join(parts)
def read_transcript(path: Path) -> tuple[dict[int, str], str | None]:
"""Page texts by page number, and the file's transcribed_by value."""
text = path.read_text(encoding="utf-8")
pages = {int(m.group(1)): m.group(3).strip("\n") for m in _PAGE.finditer(text)}
by = re.search(r"^transcribed_by:\s*(.+?)\s*$", text, re.MULTILINE)
return pages, by.group(1) if by else None
def page_prompt(base: str, pages: dict[int, str], n: int, chars: int) -> str:
def block(label: str, text: str | None, limit: int) -> str:
if text is None:
return f"{label}: (none)\n"
return f"{label}:\n<<<\n{text[:limit]}\n>>>\n"
return "\n".join([base,
block("Previous page transcript", pages.get(n - 1), chars),
block("THIS page's draft transcript", pages.get(n, ""), 4 * chars),
block("Next page transcript", pages.get(n + 1), chars)])
def clean(answer: dict, kinds: list[str]) -> dict:
"""Coerce the model's answer to the expected types."""
out = {k: str(answer.get(k) or "").strip() for k in _STR_FIELDS}
for k in _LIST_FIELDS:
v = answer.get(k) or []
out[k] = [str(x).strip() for x in (v if isinstance(v, list) else [v]) if str(x).strip()]
kind = str(answer.get("kind") or "")
out["kind"] = kind if kind in kinds else ("other" if "other" in kinds else kinds[-1])
for k in ("continues_from_previous", "continues_to_next"):
out[k] = answer.get(k) is True
return out
def _q(s: str) -> str:
return json.dumps(s, ensure_ascii=False)
def _qlist(items: list[str]) -> str:
return "[" + ", ".join(_q(x) for x in items) + "]"
def render_page(stem: str, scan: str, n: int, total: int, meta: dict, transcript: str,
indexed_by: str, transcribed_by: str | None, today: str) -> str:
fence = "```"
while fence in transcript:
fence += "`"
prev = n - 1 if meta["continues_from_previous"] and n > 1 else None
nxt = n + 1 if meta["continues_to_next"] and n < total else None
lines = [
"---",
f"notebook: {_q(stem)}",
f"scan: {_q(scan)}",
f"page: {n}",
f"kind: {_q(meta['kind'])}",
f"topics: {_qlist(meta['topics'])}",
f"references: {_qlist(meta['references'])}",
f"event: {_q(meta['event'])}",
f"event_date: {_q(meta['event_date'])}",
f"people: {_qlist(meta['people'])}",
f"continues_from: {prev if prev else 'null'}",
f"continues_to: {nxt if nxt else 'null'}",
f"indexed_by: {indexed_by}",
f"indexed: {today}",
]
if transcribed_by:
lines.append(f"transcribed_by: {transcribed_by}")
lines += ["---", "", f"# {stem}, page {n}", "", SUMMARY_HEADING, "",
meta["summary"] or "(no summary)", ""]
if meta["key_points"]:
lines += ["## Key points", "", *[f"- {k}" for k in meta["key_points"]], ""]
if meta["references"]:
lines += ["## References", "", *[f"- {r}" for r in meta["references"]], ""]
lines += ["## Transcript (unverified)", "", f"{fence}text", transcript, fence, ""]
return "\n".join(lines)
def page_path(out_dir: Path, stem: str, n: int) -> Path:
return out_dir / f"{stem} p{n:03d}.md"
def _fail(msg: str, code: int = 2) -> None:
print(f"error: {msg}", file=sys.stderr)
sys.exit(code)
@click.command()
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.option("--transcript", "transcript_path",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help="Transcript with <!-- page N --> markers. Default: <stem>.md next to the scan.")
@click.option("-o", "--output-dir", type=click.Path(file_okay=False, path_type=Path),
help='Folder for the page files. Default: "<stem> pages" next to the transcript.')
@click.option("--host", default=None,
help="Use just this host from the config and fail if it is unavailable.")
@click.option("--model", default=None, help="Model name. Default: the config's per-host model.")
@click.option("--config", "config_path", type=click.Path(dir_okay=False, path_type=Path),
default=None, help="Config file. Default: ~/.config/pdf-tools/scan-notes.toml")
@click.option("--dpi", type=int, default=None, help="PDF render resolution (config default 200).")
@click.option("--timeout", type=float, default=None,
help="Per-page timeout in seconds (config default 300).")
@click.option("--force", is_flag=True, help="Rewrite page files that already exist.")
def main(pdf: Path, transcript_path: Path | None, output_dir: Path | None,
host: str | None, model: str | None, config_path: Path | None,
dpi: int | None, timeout: float | None, force: bool) -> None:
"""Write one searchable Markdown file of metadata per page of a transcribed scan."""
cfg = load_config(config_path)
ix = cfg.index
transcript_path = transcript_path or pdf.with_suffix(".md")
if not transcript_path.exists():
_fail(f"no transcript at {transcript_path}; run scan-notes-md first")
texts, transcribed_by = read_transcript(transcript_path)
if not texts:
_fail(f"{transcript_path} has no <!-- page N --> transcript blocks")
pages = _iter_pages([pdf], dpi=dpi or cfg.dpi)
total = len(pages)
if set(texts) != set(range(1, total + 1)):
_fail(f"{transcript_path} has {len(texts)} transcribed pages; {pdf.name} has {total}")
out_dir = output_dir or transcript_path.parent / f"{pdf.stem} pages"
todo = [n for n in range(1, total + 1)
if force or not page_path(out_dir, pdf.stem, n).exists()]
if not todo:
print(f"all {total} pages already indexed in {out_dir}", file=sys.stderr)
return
hosts = cfg.hosts
if model:
hosts = [dataclasses.replace(h, model=model) for h in hosts]
try:
chosen = select_host(hosts, cfg.probe_timeout_s, forced=host)
except NoHostAvailable as e:
_fail(f"{e}. Not falling back to a cloud service.", EXIT_NO_HOST)
print(f"using Ollama host {chosen.name} ({chosen.url}, {chosen.model})", file=sys.stderr)
if len(todo) < total:
print(f"{total - len(todo)} of {total} pages already indexed; skipping them",
file=sys.stderr)
folder = cfg.confidential_folder(pdf)
names = list(dict.fromkeys([*cfg.names_for(folder), *ix.names]))
base = build_index_prompt(ix.kinds, names, [ix.prompt_extra])
fmt = schema(ix.kinds)
indexed_by = provenance(chosen)
today = dt.date.today().isoformat()
page_timeout = timeout or cfg.timeout_s
out_dir.mkdir(parents=True, exist_ok=True)
stats_log = StatsLog(cfg.stats_log)
run_id = f"{time.strftime('%Y%m%dT%H%M%S')}-{os.getpid()}"
loaded: dict | None = None
failed: list[int] = []
t_start = time.monotonic()
for n in todo:
label, load = pages[n - 1]
print(f"[{n}/{total}] {label} on {chosen.name} ...", end="", file=sys.stderr, flush=True)
t0 = time.monotonic()
try:
answer, st = chat_json(_image_to_png_bytes(load()), chosen,
page_prompt(base, texts, n, ix.context_chars),
fmt, page_timeout, ix.max_output_tokens)
except OllamaError as e:
print(f" failed: {e}", file=sys.stderr)
failed.append(n)
continue
meta = clean(answer, ix.kinds)
write_atomic(page_path(out_dir, pdf.stem, n),
render_page(pdf.stem, pdf.name, n, total, meta, texts[n],
indexed_by, transcribed_by, today))
if loaded is None:
loaded = loaded_model(chosen, cfg.probe_timeout_s)
retried = f", {st['retries']} retries" if st.get("retries") else ""
print(f" {time.monotonic() - t0:.0f} s, {st['context_tokens']} tokens, "
f"{meta['kind']}{retried}", file=sys.stderr)
nctx = loaded.get("loaded_num_ctx") or chosen.num_ctx
if nctx and st["context_tokens"] >= 0.9 * nctx:
print(f"warning: page {n} used {st['context_tokens']} of {nctx} context tokens",
file=sys.stderr)
stats_log.write({
"time": time.strftime("%Y-%m-%dT%H:%M:%S"),
"task": "index",
"run": run_id,
"host": urlparse(chosen.url).hostname,
"model": chosen.model,
"num_ctx_setting": chosen.num_ctx,
"label": label,
"page": n,
"kind": meta["kind"],
"n_references": len(meta["references"]),
**st,
**loaded,
})
mins = (time.monotonic() - t_start) / 60
done = len(todo) - len(failed)
print(f"indexed {done} page(s) in {mins:.1f} min -> {out_dir}", file=sys.stderr)
if failed:
_fail(f"{len(failed)} page(s) failed ({', '.join(map(str, failed))}); "
"rerun to retry them", EXIT_TRANSCRIBE_FAILED)
if __name__ == "__main__":
main()

146
src/pdf_tools/scan_stats.py Normal file
View file

@ -0,0 +1,146 @@
"""Per-page context and GPU statistics for scan-notes-md Ollama runs.
Each transcribed page appends one JSON line to the stats log (config key
`stats_log`; default ~/.local/state/pdf-tools/scan-stats.jsonl). A line
holds the page label, image size, token counts, stop reason, timings, and
how the model was loaded (total size, share in GPU memory, context length).
It never holds transcript text or images.
scan-notes-index writes the same kind of line with "task": "index".
`scan-notes-stats` summarizes the log, one block per task: token percentiles, how many pages
would exceed a given context size, truncated pages, GPU fit, and speed.
"""
from __future__ import annotations
import json
import math
import sys
from collections import defaultdict
from pathlib import Path
import click
DEFAULT_STATS_LOG = "~/.local/state/pdf-tools/scan-stats.jsonl"
THRESHOLDS = (4096, 8192, 16384, 32768)
class StatsLog:
"""Append-only JSON-lines log; a no-op when path is None."""
def __init__(self, path: Path | None):
self.path = path
if path is not None:
path.parent.mkdir(parents=True, exist_ok=True)
def write(self, record: dict) -> None:
if self.path is None:
return
with self.path.open("a", encoding="utf-8") as f:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
def read_log(path: Path) -> list[dict]:
out = []
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if line:
try:
out.append(json.loads(line))
except ValueError:
pass
return out
def _pct(values: list[float], q: float) -> float:
s = sorted(values)
if not s:
return 0
k = max(0, min(len(s) - 1, math.ceil(q * len(s)) - 1))
return s[k]
def summarize(records: list[dict], thresholds=THRESHOLDS, top: int = 5) -> str:
"""Plain-text report, one block per (task, host, model, loaded context).
The task is "transcribe" (scan-notes-md) or "index" (scan-notes-index);
records written before the field existed are transcriptions.
"""
groups: dict[tuple, list[dict]] = defaultdict(list)
for r in records:
groups[(r.get("task", "transcribe"), r.get("host"), r.get("model"),
r.get("loaded_num_ctx"))].append(r)
if not groups:
return "no pages logged yet"
out = []
for (task, host, model, nctx), rs in sorted(groups.items(), key=lambda kv: str(kv[0])):
n = len(rs)
out.append(f"{task}: {model} on {host}, loaded context {nctx or 'unknown'}: {n} pages, "
f"{len({r.get('run') for r in rs})} run(s)")
for key, name in [("prompt_tokens", "prompt (incl. image)"),
("output_tokens", "output"),
("context_tokens", "context used")]:
v = [r.get(key, 0) for r in rs]
out.append(f" {name:21s} mean {sum(v) / n:7.0f} p50 {_pct(v, .5):6.0f}"
f" p90 {_pct(v, .9):6.0f} p99 {_pct(v, .99):6.0f} max {max(v):6.0f}")
ctx = [r.get("context_tokens", 0) for r in rs]
over = ", ".join(f"> {t}: {sum(c > t for c in ctx)} ({100 * sum(c > t for c in ctx) / n:.1f}%)"
for t in thresholds)
out.append(f" pages needing more context than {over}")
cut = [r for r in rs if r.get("done_reason") == "length"]
out.append(f" output stopped at a length limit: {len(cut)} page(s)")
retried = [r for r in rs if r.get("retries")]
aborted = [r for r in rs if r.get("done_reason") == "repeat_abort"]
why = defaultdict(int)
for r in retried:
for reason in r.get("retry_reasons", ["unrecorded"]):
why[reason] += 1
detail = ", ".join(f"{k} {v}" for k, v in sorted(why.items()))
out.append(f" model looped: {len(retried)} page(s) retried"
f"{f' ({detail})' if detail else ''}, "
f"{len(aborted)} left as placeholders")
gpu = [r["gpu_fraction"] for r in rs if r.get("gpu_fraction") is not None]
vram = [r["vram_bytes"] for r in rs if r.get("vram_bytes")]
size = [r["loaded_bytes"] for r in rs if r.get("loaded_bytes")]
if gpu:
out.append(f" model in GPU memory: {100 * min(gpu):.0f}-{100 * max(gpu):.0f}% "
f"(loaded {max(size) / 1e9:.1f} GB, in VRAM {max(vram) / 1e9:.1f} GB)")
secs = [r.get("total_s", 0) for r in rs]
out.append(f" seconds per page: mean {sum(secs) / n:.1f}, p90 {_pct(secs, .9):.1f}, "
f"max {max(secs):.1f}; total {sum(secs) / 3600:.1f} h")
out.append(f" largest pages by context used:")
for r in sorted(rs, key=lambda r: -r.get("context_tokens", 0))[:top]:
px = "x".join(str(x) for x in r.get("image_px", [])) or "?"
out.append(f" {r.get('context_tokens', 0):6d} ({r.get('prompt_tokens', 0)} in, "
f"{r.get('output_tokens', 0)} out, {px} px) {r.get('label')}")
out.append("")
return "\n".join(out).rstrip() + "\n"
@click.command()
@click.argument("log", required=False, type=click.Path(dir_okay=False, path_type=Path))
@click.option("--run", "run_id", default=None,
help="Only this run id; 'last' for the most recent run.")
@click.option("--threshold", "thresholds", type=int, multiple=True,
help="Context sizes to test against (repeatable). "
f"Default: {', '.join(map(str, THRESHOLDS))}.")
@click.option("--top", default=5, show_default=True, help="Largest pages to list.")
def main(log: Path | None, run_id: str | None, thresholds: tuple[int, ...],
top: int) -> None:
"""Summarize context use and GPU fit from the scan-notes-md stats log."""
if log is None:
from .scan_config import load_config
log = load_config().stats_log
if log is None or not log.exists():
print(f"error: no stats log at {log}", file=sys.stderr)
sys.exit(1)
records = read_log(log)
if run_id == "last" and records:
run_id = records[-1].get("run")
if run_id:
records = [r for r in records if r.get("run") == run_id]
sys.stdout.write(summarize(records, thresholds or THRESHOLDS, top))
if __name__ == "__main__":
main()

120
src/pdf_tools/scrub.py Normal file
View file

@ -0,0 +1,120 @@
"""Tool 1: strip annotations from a PDF.
Removes the /Annots array from every page. This catches Preview/Skim/
PDF Expert highlights, sticky notes, freehand drawings, text boxes,
strikethroughs, underlines — anything stored in the annotation layer.
What it does NOT remove:
- Markings that were "flattened" into the page content stream
(some apps offer this as an export option). Those become part of
the page itself and can only be removed by going back to an
unmarked source.
- Form field values. We deliberately leave AcroForm /Widget annots
alone unless --include-forms is given.
- Embedded files / attachments. Use a dedicated tool if you need
those gone too.
"""
from __future__ import annotations
import sys
from pathlib import Path
import click
import pikepdf
from .common import err, must_exist, with_suffix_before_ext
# Subtypes that represent user markup. Widget = form field, which we
# usually want to KEEP.
MARKUP_SUBTYPES = {
"/Text", "/FreeText", "/Line", "/Square", "/Circle", "/Polygon",
"/PolyLine", "/Highlight", "/Underline", "/Squiggly", "/StrikeOut",
"/Stamp", "/Caret", "/Ink", "/Popup", "/FileAttachment", "/Sound",
"/Movie", "/Redact", "/Watermark",
}
def scrub(in_path: Path, out_path: Path, include_forms: bool = False) -> dict:
"""Scrub annotations. Returns a count dict for reporting."""
counts = {"pages": 0, "removed": 0, "kept": 0}
with pikepdf.open(in_path) as pdf:
for page in pdf.pages:
counts["pages"] += 1
if "/Annots" not in page:
continue
annots = page["/Annots"]
kept = []
for a in annots:
try:
subtype = str(a.get("/Subtype", ""))
except Exception:
subtype = ""
is_widget = subtype == "/Widget"
is_markup = subtype in MARKUP_SUBTYPES
if is_widget and not include_forms:
kept.append(a)
counts["kept"] += 1
elif is_markup or (include_forms and is_widget):
counts["removed"] += 1
else:
# Unknown subtype — keep it to be safe.
kept.append(a)
counts["kept"] += 1
if kept:
page["/Annots"] = pikepdf.Array(kept)
else:
del page["/Annots"]
# Linearize + strip metadata that often leaks reviewer identity.
# We keep /Title and /Subject; drop /Author and /Producer hints
# from the editing app.
try:
with pdf.open_metadata() as meta:
for k in ("dc:creator", "xmp:CreatorTool", "pdf:Producer"):
if k in meta:
del meta[k]
except Exception:
pass
pdf.save(out_path, linearize=True)
return counts
@click.command()
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.option(
"-o", "--output",
type=click.Path(dir_okay=False, path_type=Path),
help="Output path. Default: <input>_clean.pdf",
)
@click.option(
"--include-forms", is_flag=True,
help="Also strip form field widgets (default: keep them).",
)
@click.option(
"--print-path", is_flag=True,
help="Print only the output path on stdout (for shell pipelines).",
)
def main(pdf: Path, output: Path | None, include_forms: bool, print_path: bool) -> None:
"""Strip annotations from a PDF for sending."""
in_path = must_exist(pdf)
out_path = output or with_suffix_before_ext(in_path, "_clean")
counts = scrub(in_path, out_path, include_forms=include_forms)
if print_path:
print(str(out_path))
else:
err(
f"scrubbed {counts['removed']} annotations across "
f"{counts['pages']} pages (kept {counts['kept']}) -> {out_path}"
)
if __name__ == "__main__":
main()

308
tests/test_cli.py Normal file
View file

@ -0,0 +1,308 @@
"""End-to-end CLI behavior with Ollama faked and Claude forbidden."""
from __future__ import annotations
import io
import json
import urllib.error
from pathlib import Path
import pypdfium2 as pdfium
import pytest
from click.testing import CliRunner
from PIL import Image
from pdf_tools import extract_scan, ollama
@pytest.fixture
def env(tmp_path, monkeypatch):
root = tmp_path / "ACME"
cfg = tmp_path / "scan-notes.toml"
cfg.write_text(f"""
prompt_extra = "Mark drawings."
stats_log = "{tmp_path / "stats.jsonl"}"
[[ollama.hosts]]
name = "remote"
url = "http://gpu-box:11434"
model = "gemma4:31b"
[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:e4b"
[[confidential]]
root = "{root}"
names = ["Board Person"]
prompt_extra = "Board-only instruction."
[confidential.companion]
type = "meeting-notes"
fields = ["body", "committee"]
sections = ["Attendees", "Discussion", "Decisions", "Action items"]
transcript_section = "Discussion"
[[confidential.companion.paths]]
match = "{{body}}/committees/{{committee}}/**"
[[confidential]]
root = "{tmp_path / "lab"}"
names = ["Lab Person"]
""")
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(cfg))
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
def no_claude(*a, **k):
raise AssertionError("Claude must not be called")
monkeypatch.setattr(extract_scan, "_transcribe_image", no_claude)
folder = root / "ACMEP-BOM" / "committees" / "finance-audit" / "2026-10-29-fa"
folder.mkdir(parents=True)
pdf = folder / "2026-10-29-acmep-fa-notes.pdf"
doc = pdfium.PdfDocument.new()
doc.new_page(200, 260)
doc.save(pdf)
return {"root": root, "pdf": pdf, "tmp": tmp_path}
def fake_ollama(monkeypatch, up: dict[str, list[str]]):
def _fake(method, url, payload, timeout):
base, _, path = url.partition("/api/")
if path == "tags":
if base not in up:
raise OSError("unreachable")
return {"models": [{"name": m} for m in up[base]]}
if path == "ps":
return {"models": [{"name": m, "size": 100, "size_vram": 100,
"context_length": 4096} for m in up.get(base, [])]}
_fake.prompts.append(payload["messages"][0]["content"])
return {"message": {"content": f"transcript via {payload['model']}"},
"prompt_eval_count": 1000, "eval_count": 200, "done_reason": "stop",
"total_duration": 2_000_000_000}
_fake.prompts = []
monkeypatch.setattr(ollama, "_http_json", _fake)
return _fake
def run(*args):
return CliRunner().invoke(extract_scan.main, [str(a) for a in args])
def test_confidential_scan_defaults_to_ollama_and_writes_companion(env, monkeypatch):
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
r = run(env["pdf"])
assert r.exit_code == 0, r.output
md = env["pdf"].with_suffix(".md").read_text()
assert "body: ACMEP-BOM\ncommittee: finance-audit\ntype: meeting-notes" in md
assert "## Discussion\n\n### Draft transcript (unverified)" in md
assert "transcribed_by: ollama/localhost/gemma4:e4b" in md
assert "transcript via gemma4:e4b" in md
assert "falling back to local" in r.stderr
def test_no_host_exits_without_cloud(env, monkeypatch):
fake_ollama(monkeypatch, {})
monkeypatch.setenv("ANTHROPIC_API_KEY", "set-but-must-not-be-used")
r = run(env["pdf"])
assert r.exit_code == extract_scan.EXIT_NO_HOST
assert "Not falling back to a cloud service" in r.stderr
assert not env["pdf"].with_suffix(".md").exists()
def test_claude_refused_for_confidential_scan(env, monkeypatch):
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
r = run("--backend", "claude", env["pdf"])
assert r.exit_code == 2 and "--allow-cloud" in r.stderr
def test_claude_refused_even_with_no_companion(env, monkeypatch):
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
r = run("--backend", "claude", "--no-companion", "--stdout", env["pdf"])
assert r.exit_code == 2 and "--allow-cloud" in r.stderr
def test_forced_remote_fails_clearly(env, monkeypatch):
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
r = run("--host", "remote", env["pdf"])
assert r.exit_code == extract_scan.EXIT_NO_HOST
assert "remote" in r.stderr and "falling back to local" not in r.stderr
def test_model_flag_overrides_config(env, monkeypatch):
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["qwen3-vl:8b"]})
r = run("--model", "qwen3-vl:8b", "--host", "remote", env["pdf"])
assert r.exit_code == 0, r.output
assert "ollama/gpu-box/qwen3-vl:8b" in env["pdf"].with_suffix(".md").read_text()
def test_non_companion_ollama_output_uses_page_markers(env, monkeypatch):
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
img = env["tmp"] / "page.png"
Image.new("RGB", (50, 50), "white").save(img)
r = run("--backend", "ollama", "--stdout", img)
assert r.exit_code == 0, r.output
assert "<!-- page 1 -->" in r.stdout
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in r.stdout
def test_companion_mode_rejects_stdout(env):
r = run("--stdout", env["pdf"])
assert r.exit_code == 2 and "--no-companion" in r.stderr
def test_second_confidential_folder_uses_its_layout(env, monkeypatch):
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
lab = env["tmp"] / "lab" / "reviews"
lab.mkdir(parents=True)
pdf = lab / "2026-11-02-review-notes.pdf"
doc = pdfium.PdfDocument.new()
doc.new_page(200, 260)
doc.save(pdf)
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
assert run("--backend", "claude", pdf).exit_code == 2
r = run(pdf)
assert r.exit_code == 0, r.output
md = pdf.with_suffix(".md").read_text()
assert "type: notes" in md and "## Notes" in md and "body:" not in md
def test_mixing_confidential_and_other_files_refused(env, monkeypatch):
img = env["tmp"] / "page.png"
Image.new("RGB", (50, 50), "white").save(img)
r = run("--backend", "ollama", "--no-companion", "--stdout", env["pdf"], img)
assert r.exit_code == 2 and "do not mix" in r.stderr
def test_names_come_only_from_the_scans_folder(env, monkeypatch):
fake = fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
assert run(env["pdf"]).exit_code == 0
assert "- Board Person" in fake.prompts[0]
assert "Lab Person" not in fake.prompts[0]
assert "Mark drawings." in fake.prompts[0]
assert "Board-only instruction." in fake.prompts[0]
def test_only_top_level_settings_outside_confidential_folders(env, monkeypatch):
fake = fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
img = env["tmp"] / "page.png"
Image.new("RGB", (50, 50), "white").save(img)
assert run("--backend", "ollama", "--stdout", img).exit_code == 0
assert fake.prompts[0] == ollama.build_prompt([], ["Mark drawings."])
assert "Board" not in fake.prompts[0]
def test_no_confidential_folders_without_config(tmp_path, monkeypatch):
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(tmp_path / "empty.toml"))
(tmp_path / "empty.toml").write_text("")
from pdf_tools.scan_config import load_config
assert load_config().confidential == []
def test_interrupted_run_resumes_and_cleans_up(env, monkeypatch):
nb = env["tmp"] / "notebooks"
nb.mkdir()
pdf = nb / "2024-04 notebook.pdf"
doc = pdfium.PdfDocument.new()
for _ in range(3):
doc.new_page(200, 260)
doc.save(pdf)
calls = []
def _fake(method, url, payload, timeout):
if url.endswith("/api/tags"):
return {"models": [{"name": "gemma4:e4b"}]}
if url.endswith("/api/ps"):
return {"models": []}
calls.append(1)
if len(calls) == 2 and not getattr(_fake, "healthy", False):
raise OSError("connection reset")
return {"message": {"content": f"page text {len(calls)}"}}
monkeypatch.setattr(ollama, "_http_json", _fake)
r = run("--backend", "ollama", "--host", "local", pdf)
assert r.exit_code == extract_scan.EXIT_TRANSCRIBE_FAILED
assert "pages 1-1 are saved; rerun to resume" in r.stderr
ckpt = nb / ".2024-04 notebook.scan-partial.json"
assert ckpt.exists() and not pdf.with_suffix(".md").exists()
_fake.healthy = True
r = run("--backend", "ollama", "--host", "local", pdf)
assert r.exit_code == 0, r.output
assert "resuming after page 1 of 3" in r.stderr
md = pdf.with_suffix(".md").read_text()
assert "page text 1" in md and "<!-- page 3 -->" in md
assert len(calls) == 4 # page 1 was not sent again
assert not ckpt.exists()
assert {p.name for p in nb.iterdir()} == {pdf.name, pdf.with_suffix(".md").name}
def test_checkpoint_from_other_model_is_ignored(env, monkeypatch):
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
img = env["tmp"] / "page.png"
Image.new("RGB", (50, 50), "white").save(img)
ckpt = env["tmp"] / ".page.scan-partial.json"
ckpt.write_text('{"key": {"transcribed_by": "other"}, "pages": [{"label": "x", "text": "stale"}]}')
r = run("--backend", "ollama", "--stdout", img)
assert r.exit_code == 0 and "stale" not in r.stdout
assert "ignoring saved progress" in r.stderr and not ckpt.exists()
def test_stats_logged_per_page_without_text(env, monkeypatch):
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
r = run(env["pdf"])
assert r.exit_code == 0, r.output
assert "1200 tokens" in r.stderr
lines = (env["tmp"] / "stats.jsonl").read_text().splitlines()
assert len(lines) == 1
rec = json.loads(lines[0])
assert rec["model"] == "gemma4:e4b" and rec["host"] == "localhost"
assert (rec["prompt_tokens"], rec["output_tokens"], rec["context_tokens"]) == (1000, 200, 1200)
assert rec["gpu_fraction"] == 1.0 and rec["loaded_num_ctx"] == 4096
assert rec["image_px"][0] > 0 and rec["page"] == 1
assert "transcript via" not in lines[0]
def test_warns_when_output_hits_length_limit(env, monkeypatch):
def _fake(method, url, payload, timeout):
if url.endswith("/api/tags"):
return {"models": [{"name": "gemma4:e4b"}]}
if url.endswith("/api/ps"):
return {"models": [{"name": "gemma4:e4b", "size": 1, "size_vram": 1,
"context_length": 4096}]}
return {"message": {"content": "partial"}, "prompt_eval_count": 3900,
"eval_count": 196, "done_reason": "length"}
monkeypatch.setattr(ollama, "_http_json", _fake)
r = run(env["pdf"])
assert r.exit_code == 0
assert "marked as cut off" in r.stderr
assert "cut off at the output limit" in env["pdf"].with_suffix(".md").read_text()
def test_looping_page_becomes_placeholder_and_run_finishes(env, monkeypatch):
def _fake(method, url, payload, timeout):
if url.endswith("/api/tags"):
return {"models": [{"name": "gemma4:e4b"}]}
if url.endswith("/api/ps"):
return {"models": []}
raise urllib.error.HTTPError(
url, 500, "err", {},
io.BytesIO(b'{"error":"prediction aborted, token repeat limit reached"}'))
monkeypatch.setattr(ollama, "_http_json", _fake)
r = run(env["pdf"])
assert r.exit_code == 0, r.output
assert "writing a placeholder" in r.stderr
assert ollama.REPEAT_PLACEHOLDER in env["pdf"].with_suffix(".md").read_text()
rec = json.loads((env["tmp"] / "stats.jsonl").read_text().splitlines()[0])
assert rec["done_reason"] == "repeat_abort" and rec["retries"] == 2
def test_ollama_is_the_default_backend_outside_confidential_folders(env, monkeypatch):
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
monkeypatch.setenv("ANTHROPIC_API_KEY", "set-but-must-not-be-used")
img = env["tmp"] / "page.png"
Image.new("RGB", (50, 50), "white").save(img)
r = run("--stdout", img)
assert r.exit_code == 0, r.output
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in r.stdout

291
tests/test_companion.py Normal file
View file

@ -0,0 +1,291 @@
"""Companion files: front matter from paths, and preserving existing text."""
from __future__ import annotations
from pathlib import Path
import pytest
from pdf_tools import companion as comp
from pdf_tools.companion import CompanionSpec, PathRule
ROOT = Path("/tmp/acme-test-root")
# A meetings layout like the one in the user's config (test data only; the
# code knows nothing about any organization).
SPEC = CompanionSpec(
type="meeting-notes",
fields=["body", "committee"],
sections=["Attendees", "Discussion", "Decisions", "Action items"],
transcript_section="Discussion",
section_text={"Action items": "- [ ] Owner: task (due date)"},
title="{body}, {date}",
allowed={"body": ["ACME-BOD", "ACMEP-BOM", "ACMEF-BOT"]},
acronyms={"acmep", "ceo"},
paths=[
PathRule("{body}/committees/{committee}/**", title="{body.short} {committee}, {date}"),
PathRule("{body}/meetings/**", set={"committee": "board"}),
PathRule("chair/{committee}/**", set={"body": "chair"}, title="{committee}, {date}"),
PathRule("{body}/{committee}/**", title="{body.short} {committee}, {date}"),
PathRule("{body}/**"),
],
)
TITLES = {
"ACME-BOD": "ACME Board of Directors", "ACME-BOD.short": "ACME",
"ACMEP-BOM": "ACMEP Board of Managers", "ACMEP-BOM.short": "ACMEP",
"finance-audit": "Finance and Audit Committee",
}
def meta(rel: str) -> comp.NoteMeta:
return comp.meta_from_path(ROOT / rel, ROOT, SPEC)
@pytest.mark.parametrize("rel, date, body, committee", [
("ACMEP-BOM/committees/finance-audit/2026-10-29-fa/2026-10-29-acmep-fa-notes.pdf",
"2026-10-29", "ACMEP-BOM", "finance-audit"),
("ACME-BOD/meetings/2026-11-12/2026-11-12-acme-bod-notes.pdf",
"2026-11-12", "ACME-BOD", "board"),
("chair/acmep-ceo-search/2026-10-30-acmep-ceo-search-notes.pdf",
"2026-10-30", "chair", "acmep-ceo-search"),
("ACME-BOD/committees/iac/2026-06-xx/notes.pdf",
"2026-06-xx", "ACME-BOD", "iac"),
("ACMEP-BOM/meetings/2026-06-08/scan.pdf",
"2026-06-08", "ACMEP-BOM", "board"),
])
def test_meta_from_path(rel, date, body, committee):
m = meta(rel)
assert (m.date, m.fields["body"], m.fields["committee"]) == (date, body, committee)
assert m.scan == Path(rel).name
def test_filename_date_wins_over_folder():
m = meta("ACMEP-BOM/meetings/2026-06-08/2026-06-09-acmep-notes.pdf")
assert m.date == "2026-06-09"
def test_unknown_location_warns():
m = meta("inbox/scan.pdf")
assert m.fields["body"] == "" and m.date == ""
assert any("body" in w for w in m.warnings)
assert any("date" in w for w in m.warnings)
def test_partial_date_warns():
assert any("no day" in w for w in meta("ACME-BOD/meetings/2026-06-xx/n.pdf").warnings)
def test_titles():
def t(rel, titles=TITLES):
return comp.title_for(meta(rel), SPEC, titles)
assert t("ACMEP-BOM/committees/finance-audit/x/2026-10-29-n.pdf") == \
"ACMEP Finance and Audit Committee, October 29, 2026"
assert t("ACME-BOD/meetings/2026-11-12/n.pdf") == \
"ACME Board of Directors, November 12, 2026"
assert t("chair/acmep-ceo-search/2026-10-30-n.pdf") == \
"ACMEP CEO Search, October 30, 2026"
assert t("chair/acmep-ceo-search/2026-10-30-n.pdf",
{"acmep-ceo-search": "ACMEP CEO search"}).startswith("ACMEP CEO search,")
assert t("ACME-BOD/meetings/x/n.pdf") == "ACME Board of Directors"
def test_allowed_values_and_rule_order():
m = meta("inbox/2026-01-02-n.pdf") # "inbox" is not an allowed body
assert m.fields == {"body": "", "committee": ""}
m = meta("ACME-BOD/2026-01-02-n.pdf") # matched by the last rule
assert m.fields == {"body": "ACME-BOD", "committee": ""}
assert any("committee" in w for w in m.warnings)
@pytest.mark.parametrize("pattern, folders, expected", [
("a/{x}", ("a", "b"), {"x": "b"}),
("a/{x}", ("a", "b", "c"), None),
("a/{x}/**", ("a", "b", "c"), {"x": "b"}),
("a/**", ("a",), {}),
("{x}/b", ("a", "c"), None),
])
def test_match(pattern, folders, expected):
assert comp._match(pattern, folders, {}) == expected
def test_new_companion_matches_template():
m = meta("ACMEP-BOM/committees/finance-audit/2026-10-29-fa/2026-10-29-acmep-fa-notes.pdf")
text = comp.new_companion(m, ["page one", "page two"], "ollama/gpu-box/gemma4:31b",
SPEC, TITLES)
assert text.startswith(
"---\ndate: 2026-10-29\nbody: ACMEP-BOM\ncommittee: finance-audit\n"
"type: meeting-notes\nscan: 2026-10-29-acmep-fa-notes.pdf\nsubjects: []\n"
"transcribed_by: ollama/gpu-box/gemma4:31b\n---\n")
for h in ("## Attendees", "## Discussion", comp.DRAFT_HEADING,
"## Decisions", "## Action items"):
assert h in text
assert text.index("## Discussion") < text.index(comp.DRAFT_HEADING) < text.index("## Decisions")
assert "<!-- page 1 -->" in text and "<!-- page 2 -->" in text
assert comp.PENDING_LINE not in text
def test_transcript_keeps_layout_and_fences_headings():
page = "# 3 items\n - indented\n\nafter blank"
r = comp.render_transcript([page])
assert "```text\n# 3 items\n - indented\n\nafter blank\n```" in r
r2 = comp.render_transcript(["has ``` inside"])
assert "````text\nhas ``` inside\n````" in r2
# ---- preserving existing companion text -----------------------------------
EXISTING = """\
---
date: 2026-10-29
body: ACMEP-BOM
committee: finance-audit
type: meeting-notes
scan: 2026-10-29-acmep-fa-notes.pdf
subjects: [ACMEP performance, audit timeline]
---
# ACMEP Finance and Audit Committee, October 29, 2026
## Attendees
Pat, the committee, staff.
## Discussion
My own summary paragraph.
With an indented line.
{middle}
## Decisions
- Decision I wrote.
## Action items
- [ ] Owner: task (due date)
"""
def _mine(text: str) -> list[str]:
"""Every line Pat wrote, which must survive any merge."""
return [l for l in EXISTING.format(middle="").splitlines() if l.strip()]
def assert_preserved(before: str, after: str):
after_lines = after.splitlines()
pos = 0
for line in _mine(before):
assert line in after_lines[pos:], f"lost or reordered: {line!r}"
pos = after_lines.index(line, pos) + 1
def test_pending_line_replaced():
before = EXISTING.format(middle=comp.PENDING_LINE + "\n")
after = comp.merge_companion(before, ["draft"], "ollama/localhost/gemma4:e4b", "Discussion")
assert comp.PENDING_LINE not in after
assert after.count(comp.DRAFT_HEADING) == 1
assert_preserved(before, after)
assert after.index("My own summary") < after.index(comp.DRAFT_HEADING) \
< after.index("## Decisions")
def test_block_added_when_missing():
before = EXISTING.format(middle="")
after = comp.merge_companion(before, ["draft"], "ollama/localhost/gemma4:e4b", "Discussion")
assert after.count(comp.DRAFT_HEADING) == 1
assert_preserved(before, after)
assert after.index("With an indented line.") < after.index(comp.DRAFT_HEADING) \
< after.index("## Decisions")
assert "\n\n## Decisions" in after
def test_existing_block_replaced_not_duplicated():
before = EXISTING.format(middle="")
once = comp.merge_companion(before, ["first draft"], "ollama/localhost/gemma4:e4b", "Discussion")
twice = comp.merge_companion(once, ["second draft"], "ollama/gpu-box/gemma4:31b", "Discussion")
assert twice.count(comp.DRAFT_HEADING) == 1
assert "second draft" in twice and "first draft" not in twice
assert_preserved(before, twice)
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in twice
assert twice.count("transcribed_by:") == 1
def test_rerun_is_stable():
before = EXISTING.format(middle="")
once = comp.merge_companion(before, ["same"], "x", "Discussion")
assert comp.merge_companion(once, ["same"], "x", "Discussion") == once
def test_old_block_without_end_marker_stops_at_next_heading():
before = EXISTING.format(middle=(
f"{comp.DRAFT_HEADING}\n\nold draft text\n\n"
"### My corrections\n\nKeep this.\n"))
after = comp.merge_companion(before, ["new"], "x", "Discussion")
assert "old draft text" not in after
assert "### My corrections\n\nKeep this." in after
assert_preserved(before, after)
def test_hash_line_inside_transcript_does_not_end_block():
before = EXISTING.format(middle="")
once = comp.merge_companion(before, ["# not a heading\nmore"], "x", "Discussion")
twice = comp.merge_companion(once, ["replacement"], "x", "Discussion")
assert "not a heading" not in twice and "more\n" not in twice.split("## Decisions")[0]
def test_front_matter_only_gains_transcribed_by():
before = EXISTING.format(middle="")
after = comp.merge_companion(before, ["d"], "ollama/localhost/gemma4:e4b", "Discussion")
fm_before = before.split("\n---\n", 1)[0]
fm_after = after.split("\n---\n", 1)[0]
assert fm_after == fm_before + "\ntranscribed_by: ollama/localhost/gemma4:e4b"
def test_no_discussion_section_appends_at_end():
before = "---\ndate: 2026-10-29\n---\n\n# Notes\n\nFree text.\n"
after = comp.merge_companion(before, ["d"], "x")
assert after.startswith("---\ndate: 2026-10-29\ntranscribed_by: x\n---\n\n# Notes\n\nFree text.\n\n")
assert after.rstrip().endswith(comp.END_MARKER)
def test_write_companion_creates_then_preserves(tmp_path):
folder = tmp_path / "ACMEP-BOM" / "committees" / "finance-audit" / "2026-10-29-fa"
folder.mkdir(parents=True)
pdf = folder / "2026-10-29-acmep-fa-notes.pdf"
pdf.write_bytes(b"%PDF-1.4")
path, created, _ = comp.write_companion(pdf, tmp_path, ["one"], "x", SPEC)
assert created and path == pdf.with_suffix(".md")
edited = path.read_text().replace("subjects: []", "subjects: [audit]") \
.replace("## Attendees\n", "## Attendees\n\nSomeone.\n")
path.write_text(edited)
_, created, _ = comp.write_companion(pdf, tmp_path, ["two"], "y", SPEC)
final = path.read_text()
assert not created
assert "subjects: [audit]" in final and "Someone." in final
assert "two" in final and "\none\n" not in final
assert {p.name for p in folder.iterdir()} == {pdf.name, path.name}
# ---- basic layout (no companion table) -----------------------------------
def test_basic_layout_meta_has_no_body_or_warnings():
m = comp.meta_from_path(ROOT / "projects" / "2026-11-02-lab-review-notes.pdf", ROOT)
assert (m.date, m.fields, m.warnings) == ("2026-11-02", {}, [])
def test_basic_layout_template():
m = comp.meta_from_path(ROOT / "x" / "2026-11-02-lab-review-notes.pdf", ROOT)
text = comp.new_companion(m, ["p"], "ollama/localhost/gemma4:e4b")
assert text.startswith(
"---\ndate: 2026-11-02\ntype: notes\nscan: 2026-11-02-lab-review-notes.pdf\n"
"subjects: []\ntranscribed_by: ollama/localhost/gemma4:e4b\n---\n\n"
"# Lab Review Notes, November 2, 2026\n\n## Notes\n\n" + comp.DRAFT_HEADING)
assert "body:" not in text and "## Attendees" not in text
def test_basic_layout_merge_keeps_text():
before = ("---\ndate: 2026-11-02\n---\n\n# Notes\n\n## Notes\n\nMine.\n\n"
"## Later\n\nAlso mine.\n")
after = comp.merge_companion(before, ["d"], "x")
assert after.index("Mine.") < after.index(comp.DRAFT_HEADING) < after.index("## Later")
assert "Also mine." in after

103
tests/test_config.py Normal file
View file

@ -0,0 +1,103 @@
"""Confidential-folder declarations in the config file."""
from __future__ import annotations
from pathlib import Path
import pytest
from pdf_tools.companion import BASIC
from pdf_tools.scan_config import load_config
EXAMPLE = Path(__file__).resolve().parent.parent / "scan-notes.example.toml"
def cfg_from(tmp_path, text: str):
p = tmp_path / "scan-notes.toml"
p.write_text(text)
return load_config(p)
def test_nothing_built_in(tmp_path, monkeypatch):
monkeypatch.delenv("PDF_TOOLS_SCAN_CONFIG", raising=False)
monkeypatch.setattr("pdf_tools.scan_config.DEFAULT_CONFIG_PATH", tmp_path / "none.toml")
cfg = load_config()
assert cfg.confidential == []
assert [h.url for h in cfg.hosts] == ["http://localhost:11434"]
def test_example_file_loads():
cfg = load_config(EXAMPLE)
assert len(cfg.confidential) == 2
basic, meetings = cfg.confidential
assert basic.companion is BASIC
assert meetings.companion.transcript_section == "Discussion"
assert meetings.companion.fields == ["body", "committee"]
assert [r.match for r in meetings.companion.paths][0] == "{body}/committees/{committee}/**"
assert meetings.titles["BOD.short"] == "Board"
def test_folder_without_companion_is_basic(tmp_path):
cfg = cfg_from(tmp_path, f'[[confidential]]\nroot = "{tmp_path / "x"}"\n')
assert cfg.confidential[0].companion is BASIC
def test_transcript_section_defaults_to_first_section(tmp_path):
cfg = cfg_from(tmp_path, f'[[confidential]]\nroot = "{tmp_path}"\n'
'[confidential.companion]\nsections = ["Summary", "Log"]\n')
assert cfg.confidential[0].companion.transcript_section == "Summary"
def test_tilde_expanded_and_deepest_root_wins(tmp_path):
outer, inner = tmp_path / "a", tmp_path / "a" / "b"
cfg = cfg_from(tmp_path,
f'[[confidential]]\nroot = "{outer}"\n\n'
f'[[confidential]]\nroot = "{inner}"\n')
assert cfg.confidential_folder(inner / "c" / "s.pdf").root == inner
assert cfg.confidential_folder(outer / "s.pdf").root == outer
assert cfg.confidential_folder(tmp_path / "other.pdf") is None
# A sibling whose name merely starts with the root's name is not inside it.
assert cfg.confidential_folder(tmp_path / "ab" / "s.pdf") is None
cfg = cfg_from(tmp_path, '[[confidential]]\nroot = "~/somewhere"\n')
assert cfg.confidential[0].root == Path.home() / "somewhere"
def test_names_are_per_folder(tmp_path):
a, b = tmp_path / "a", tmp_path / "b"
cfg = cfg_from(tmp_path, f'names = ["Everyone"]\n'
f'[[confidential]]\nroot = "{a}"\nnames = ["Alpha", "Everyone"]\n'
f'[[confidential]]\nroot = "{b}"\nnames = ["Beta"]\n')
fa, fb = cfg.confidential
assert cfg.names_for(fa) == ["Everyone", "Alpha"]
assert cfg.names_for(fb) == ["Everyone", "Beta"]
assert cfg.names_for(None) == ["Everyone"]
@pytest.mark.parametrize("text", [
'[[confidential]]\nnames = []\n', # no root
'[[confidential]]\nroot = "/x"\ncompanion = "acme"\n', # old string form
'[[confidential]]\nroot = "/x"\nnmaes = []\n', # typo
'[[confidential]]\nroot = "/x"\n[confidential.companion]\nsectoins = []\n',
'[[confidential]]\nroot = "/x"\n[[confidential.companion.paths]]\ntitle = "t"\n',
# A top-level key written after a host table lands inside that table.
'[[ollama.hosts]]\nname = "l"\nurl = "http://x"\nmodel = "m"\nprompt_extra = "x"\n',
'[ollama]\ntimeout_s = 5\nnames = []\n',
'prompt_extar = "x"\n',
])
def test_bad_entries_rejected(tmp_path, text):
with pytest.raises(SystemExit):
cfg_from(tmp_path, text)
def test_prompt_extra_top_level_and_per_folder(tmp_path):
a, b = tmp_path / "a", tmp_path / "b"
cfg = cfg_from(tmp_path, 'prompt_extra = "Figures."\n'
f'[[confidential]]\nroot = "{a}"\nprompt_extra = "Board terms."\n'
f'[[confidential]]\nroot = "{b}"\n')
fa, fb = cfg.confidential
assert cfg.extras_for(fa) == ["Figures.", "Board terms."]
assert cfg.extras_for(fb) == ["Figures."]
assert cfg.extras_for(None) == ["Figures."]
def test_example_prompt_extra_mentions_figures():
assert "[FIGURE:" in load_config(EXAMPLE).prompt_extra

240
tests/test_index.py Normal file
View file

@ -0,0 +1,240 @@
"""scan-notes-index: transcript parsing, page files, and the CLI with Ollama faked."""
from __future__ import annotations
import io
import json
import urllib.error
import pypdfium2 as pdfium
import pytest
from click.testing import CliRunner
from pdf_tools import companion, ollama, scan_index
from pdf_tools.extract_scan import to_plain_markdown
from pdf_tools.scan_config import OllamaHost, load_config
from pdf_tools.scan_stats import read_log, summarize
HOST = OllamaHost("local", "http://localhost:11434", "gemma4:31b", num_ctx=8192)
KINDS = ["calculation", "idea", "meeting notes", "other", "blank"]
def answer(**over):
a = {"summary": "Estimate of a drag force.", "key_points": ["F = 6 pi eta a U"],
"kind": "calculation", "topics": ["Stokes drag"], "references": [],
"event": "", "event_date": "", "people": [],
"continues_from_previous": False, "continues_to_next": True}
return {**a, **over}
def make_scan(folder, pages=("first page", "second page", "third page")):
folder.mkdir(parents=True, exist_ok=True)
pdf = folder / "2009 notebook.pdf"
doc = pdfium.PdfDocument.new()
for _ in pages:
doc.new_page(200, 260)
doc.save(pdf)
md = pdf.with_suffix(".md")
md.write_text(to_plain_markdown(list(pages), pdf.stem, "ollama/localhost/gemma4:31b"))
return pdf
@pytest.fixture
def env(tmp_path, monkeypatch):
cfg = tmp_path / "scan-notes.toml"
cfg.write_text(f"""
names = ["Global Person"]
stats_log = "{tmp_path / "stats.jsonl"}"
[ollama]
max_output_tokens = 2048
[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:31b"
num_ctx = 8192
[index]
names = ["Board Member"]
prompt_extra = "Index-only instruction."
""")
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(cfg))
return tmp_path
def fake(monkeypatch, replies=None):
"""Ollama with gemma4:31b; chat replies come from the list, then answer()."""
replies = list(replies or [])
def _fake(method, url, payload, timeout):
base, _, path = url.partition("/api/")
if path == "tags":
return {"models": [{"name": "gemma4:31b"}]}
if path == "ps":
return {"models": [{"name": "gemma4:31b", "size": 20, "size_vram": 20,
"context_length": 8192}]}
_fake.payloads.append(payload)
r = replies.pop(0) if replies else answer()
if isinstance(r, Exception):
raise r
if isinstance(r, dict) and "message" in r:
return r
return {"message": {"content": r if isinstance(r, str) else json.dumps(r)},
"prompt_eval_count": 2000, "eval_count": 300, "done_reason": "stop",
"total_duration": 9_000_000_000}
_fake.payloads = []
monkeypatch.setattr(ollama, "_http_json", _fake)
return _fake
def run(*args):
return CliRunner().invoke(scan_index.main, [str(a) for a in args])
def test_reads_plain_and_companion_transcripts(tmp_path):
plain = tmp_path / "a.md"
plain.write_text(to_plain_markdown(["one", "has ``` inside", ""], "a", "ollama/x/m"))
pages, by = scan_index.read_transcript(plain)
assert pages == {1: "one", 2: "has ``` inside", 3: ""} and by == "ollama/x/m"
comp = tmp_path / "b.md"
comp.write_text("---\ntranscribed_by: ollama/y/m\n---\n\n## Notes\n\n"
+ companion.draft_block(["alpha", "beta"]) + "\n")
pages, by = scan_index.read_transcript(comp)
assert pages == {1: "alpha", 2: "beta"} and by == "ollama/y/m"
def test_prompt_has_kinds_note_extras_and_names():
p = scan_index.build_index_prompt(KINDS, ["Pat Doe"], ["Extra rule."])
assert '"calculation" for derivations' in p
assert p.index("Extra rule.") < p.index(ollama.NAMES_PROMPT) < p.index("- Pat Doe")
assert '"calculation"' not in scan_index.build_index_prompt(["idea", "other"], [], [])
def test_page_prompt_marks_this_page_and_neighbors():
pages = {1: "p1", 2: "p2" * 50, 3: "p3"}
p = scan_index.page_prompt("BASE", pages, 1, chars=10)
assert "Previous page transcript: (none)" in p
assert "THIS page's draft transcript:\n<<<\np1\n>>>" in p
p = scan_index.page_prompt("BASE", pages, 3, chars=10)
assert "<<<\n" + "p2" * 5 + "\n>>>" in p and "Next page transcript: (none)" in p
def test_clean_coerces_types_and_unknown_kind():
m = scan_index.clean({"summary": None, "kind": "derivation", "topics": "one",
"people": ["", " A "], "continues_to_next": "yes"}, KINDS)
assert m["summary"] == "" and m["kind"] == "other"
assert m["topics"] == ["one"] and m["people"] == ["A"]
assert m["continues_to_next"] is False and m["references"] == []
def test_render_page_front_matter_and_sections():
meta = scan_index.clean(answer(references=['Smith, "x", J. Fluid Mech. 1 (2001)'],
continues_from_previous=True), KINDS)
text = scan_index.render_page("nb", "nb.pdf", 1, 3, meta, "a ``` b", "ollama/h/m",
"ollama/h/t", "2026-11-02")
assert text.startswith('---\nnotebook: "nb"\nscan: "nb.pdf"\npage: 1\nkind: "calculation"')
assert 'references: ["Smith, \\"x\\", J. Fluid Mech. 1 (2001)"]' in text
assert "continues_from: null\ncontinues_to: 2\n" in text # page 1 has no previous
assert "transcribed_by: ollama/h/t" in text
assert scan_index.SUMMARY_HEADING + "\n\nEstimate of a drag force." in text
assert "## Key points\n\n- F = 6 pi eta a U" in text
assert "````text\na ``` b\n````" in text
def test_cli_writes_one_file_per_page(env, monkeypatch):
f = fake(monkeypatch)
pdf = make_scan(env / "nb")
r = run(pdf)
assert r.exit_code == 0, r.output
out = env / "nb" / "2009 notebook pages"
files = sorted(p.name for p in out.iterdir())
assert files == [f"2009 notebook p00{n}.md" for n in (1, 2, 3)]
p2 = (out / "2009 notebook p002.md").read_text()
assert "page: 2\n" in p2 and "continues_to: 3" in p2 and "second page" in p2
assert "indexed_by: ollama/localhost/gemma4:31b" in p2
assert "continues_to: null" in (out / "2009 notebook p003.md").read_text()
req = f.payloads[1]
assert req["format"]["properties"]["kind"]["enum"][0] == "calculation"
assert req["options"] == {"temperature": 0, "num_ctx": 8192, "num_predict": 1536}
prompt = req["messages"][0]["content"]
assert "- Global Person" in prompt and "- Board Member" in prompt
assert "Index-only instruction." in prompt
assert "<<<\nsecond page\n>>>" in prompt and "<<<\nfirst page\n>>>" in prompt
log = read_log(env / "stats.jsonl")
assert [x["task"] for x in log] == ["index"] * 3 and log[0]["kind"] == "calculation"
assert "summary" not in json.dumps(log) # no generated text in the log
assert "index: gemma4:31b on localhost" in summarize(log)
def test_existing_pages_are_skipped_unless_forced(env, monkeypatch):
f = fake(monkeypatch)
pdf = make_scan(env / "nb")
out = env / "nb" / "2009 notebook pages"
out.mkdir()
(out / "2009 notebook p001.md").write_text("kept")
assert run(pdf).exit_code == 0
assert len(f.payloads) == 2 and (out / "2009 notebook p001.md").read_text() == "kept"
r = run(pdf)
assert r.exit_code == 0 and "already indexed" in r.stderr and len(f.payloads) == 2
assert run("--force", pdf).exit_code == 0 and len(f.payloads) == 5
def test_bad_json_is_retried(env, monkeypatch):
f = fake(monkeypatch, ["not json", {"summary": "missing the other keys"}])
pdf = make_scan(env / "nb", pages=("only",))
assert run(pdf).exit_code == 0
assert len(f.payloads) == 3
log = read_log(env / "stats.jsonl")
assert log[0]["retries"] == 2 and log[0]["retry_reasons"] == ["bad_json", "bad_json"]
def test_failed_page_reported_and_others_written(env, monkeypatch):
def err():
return urllib.error.HTTPError("u", 500, "x", {}, io.BytesIO(b'{"error":"token repeat limit"}'))
fake(monkeypatch, [err(), err(), err()])
pdf = make_scan(env / "nb", pages=("a", "b"))
r = run(pdf)
assert r.exit_code == scan_index.EXIT_TRANSCRIBE_FAILED
assert "1 page(s) failed (1)" in r.stderr
out = env / "nb" / "2009 notebook pages"
assert [p.name for p in out.iterdir()] == ["2009 notebook p002.md"]
def test_page_count_mismatch_refused(env, monkeypatch):
fake(monkeypatch)
pdf = make_scan(env / "nb")
pdf.with_suffix(".md").write_text(to_plain_markdown(["a"], "x", "y"))
r = run(pdf)
assert r.exit_code == 2 and "1 transcribed pages" in r.stderr
def test_missing_transcript_refused(env, monkeypatch):
fake(monkeypatch)
pdf = make_scan(env / "nb")
pdf.with_suffix(".md").unlink()
r = run(pdf)
assert r.exit_code == 2 and "run scan-notes-md first" in r.stderr
def test_no_host_exits_without_cloud(env, monkeypatch):
def down(*a):
raise urllib.error.URLError("refused")
monkeypatch.setattr(ollama, "_http_json", down)
pdf = make_scan(env / "nb")
r = run(pdf)
assert r.exit_code == scan_index.EXIT_NO_HOST
assert "Not falling back to a cloud service" in r.stderr
def test_index_config_rejects_unknown_keys(tmp_path, capsys):
cfg = tmp_path / "c.toml"
cfg.write_text("[index]\nkind = ['x']\n")
with pytest.raises(SystemExit):
load_config(cfg)
assert "unknown key(s) in [index]: kind" in capsys.readouterr().err
cfg.write_text("[index]\nkinds = []\n")
with pytest.raises(SystemExit):
load_config(cfg)

247
tests/test_ollama_hosts.py Normal file
View file

@ -0,0 +1,247 @@
"""Host selection, fallback, and response cleanup. All HTTP is faked."""
from __future__ import annotations
import io
import json
import urllib.error
import pytest
from pdf_tools import ollama
from pdf_tools.scan_config import OllamaHost
REMOTE = OllamaHost("remote", "http://gpu-box:11434", "gemma4:31b")
LOCAL = OllamaHost("local", "http://localhost:11434", "gemma4:e4b")
def fake_http(tags: dict[str, list[str] | None], chat=None):
"""tags maps base URL -> model names, or None for an unreachable host."""
calls = []
def _fake(method, url, payload, timeout):
calls.append((method, url, payload, timeout))
base, _, path = url.partition("/api/")
if path == "tags":
models = tags.get(base)
if models is None:
raise urllib.error.URLError("connection refused")
return {"models": [{"name": m} for m in models]}
if path == "chat":
return chat(payload) if chat else {"message": {"content": "ok"}}
raise AssertionError(url)
_fake.calls = calls
return _fake
def test_remote_used_when_available(monkeypatch, capsys):
monkeypatch.setattr(ollama, "_http_json", fake_http({
REMOTE.url: ["gemma4:31b"], LOCAL.url: ["gemma4:e4b"]}))
assert ollama.select_host([REMOTE, LOCAL], 2.5) is REMOTE
assert capsys.readouterr().err == ""
def test_falls_back_when_remote_unreachable(monkeypatch, capsys):
monkeypatch.setattr(ollama, "_http_json", fake_http({
REMOTE.url: None, LOCAL.url: ["gemma4:e4b"]}))
assert ollama.select_host([REMOTE, LOCAL], 2.5) is LOCAL
err = capsys.readouterr().err.strip().splitlines()
assert len(err) == 1
assert "unreachable" in err[0] and "falling back to local" in err[0]
def test_falls_back_when_remote_lacks_model(monkeypatch, capsys):
monkeypatch.setattr(ollama, "_http_json", fake_http({
REMOTE.url: ["qwen3-vl:8b"], LOCAL.url: ["gemma4:e4b"]}))
assert ollama.select_host([REMOTE, LOCAL], 2.5) is LOCAL
assert "does not have gemma4:31b" in capsys.readouterr().err
def test_probe_uses_short_timeout(monkeypatch):
fake = fake_http({REMOTE.url: ["gemma4:31b"]})
monkeypatch.setattr(ollama, "_http_json", fake)
ollama.select_host([REMOTE, LOCAL], 2.5)
assert fake.calls == [("GET", f"{REMOTE.url}/api/tags", None, 2.5)]
def test_no_host_raises(monkeypatch):
monkeypatch.setattr(ollama, "_http_json", fake_http({}))
with pytest.raises(ollama.NoHostAvailable):
ollama.select_host([REMOTE, LOCAL], 2.5)
def test_forced_host_skips_others(monkeypatch):
fake = fake_http({REMOTE.url: ["gemma4:31b"], LOCAL.url: ["gemma4:e4b"]})
monkeypatch.setattr(ollama, "_http_json", fake)
assert ollama.select_host([REMOTE, LOCAL], 2.5, forced="local") is LOCAL
assert [c[1] for c in fake.calls] == [f"{LOCAL.url}/api/tags"]
def test_forced_host_unavailable_is_error_not_fallback(monkeypatch):
monkeypatch.setattr(ollama, "_http_json", fake_http({
REMOTE.url: None, LOCAL.url: ["gemma4:e4b"]}))
with pytest.raises(ollama.NoHostAvailable, match="remote"):
ollama.select_host([REMOTE, LOCAL], 2.5, forced="remote")
def test_forced_host_unknown_name(monkeypatch):
monkeypatch.setattr(ollama, "_http_json", fake_http({}))
with pytest.raises(ollama.NoHostAvailable, match="no Ollama host named"):
ollama.select_host([REMOTE, LOCAL], 2.5, forced="laptop")
def test_untagged_model_matches_latest(monkeypatch):
host = OllamaHost("x", "http://x:11434", "llama3.2")
monkeypatch.setattr(ollama, "_http_json", fake_http({host.url: ["llama3.2:latest"]}))
assert ollama.select_host([host], 1) is host
def test_chat_request_shape(monkeypatch):
seen = {}
def chat(payload):
seen.update(payload)
return {"message": {"content": "line one\n line two"}}
monkeypatch.setattr(ollama, "_http_json", fake_http({}, chat))
out = ollama.transcribe_png(b"\x89PNG", REMOTE, "PROMPT", 300)
assert out.text == "line one\n line two"
assert seen["stream"] is False and seen["think"] is False
assert seen["options"]["temperature"] == 0
assert seen["model"] == "gemma4:31b"
assert len(seen["messages"]) == 1 and seen["messages"][0]["images"]
def test_retries_without_think_when_rejected(monkeypatch):
payloads = []
def _fake(method, url, payload, timeout):
payloads.append(dict(payload))
if "think" in payload:
raise urllib.error.HTTPError(
url, 400, "Bad Request", {},
io.BytesIO(json.dumps({"error": "model does not support thinking"}).encode()))
return {"message": {"content": "text"}}
monkeypatch.setattr(ollama, "_http_json", _fake)
assert ollama.transcribe_png(b"x", LOCAL, "P", 10).text == "text"
assert "think" in payloads[0] and "think" not in payloads[1]
@pytest.mark.parametrize("raw, clean", [
("<think>hmm, looks like a list</think>\nA\nB", "A\nB"),
("<thinking>x</thinking>A", "A"),
("reasoning without an opening tag</think>\nA", "A"),
("A\n - indented\nB\n", "A\n - indented\nB"),
])
def test_strip_thinking(raw, clean):
assert ollama.strip_thinking(raw) == clean
def test_names_added_only_when_configured():
assert ollama.build_prompt([]) == ollama.HANDWRITING_PROMPT
p = ollama.build_prompt(["Pat Example", "ACME Foundation"])
assert p.startswith(ollama.HANDWRITING_PROMPT)
assert "- Pat Example" in p and "otherwise write\n[illegible]" in p
def test_provenance():
assert ollama.provenance(REMOTE) == "ollama/gpu-box/gemma4:31b"
assert ollama.provenance(LOCAL) == "ollama/localhost/gemma4:e4b"
def test_extras_go_between_base_prompt_and_names():
p = ollama.build_prompt(["Pat"], ["Draw figures as [FIGURE: x].", " "])
assert p.startswith(ollama.HANDWRITING_PROMPT + "\nDraw figures as [FIGURE: x].\n\n")
assert p.index("[FIGURE") < p.index(ollama.NAMES_PROMPT) < p.index("- Pat")
assert ollama.build_prompt([], ["", "\n"]) == ollama.HANDWRITING_PROMPT
def test_page_stats_from_response():
st = ollama.page_stats({"prompt_eval_count": 1200, "eval_count": 300,
"done_reason": "stop", "total_duration": 9_500_000_000,
"load_duration": 0, "prompt_eval_duration": 1_000_000_000,
"eval_duration": 8_000_000_000})
assert st["context_tokens"] == 1500 and st["done_reason"] == "stop"
assert (st["total_s"], st["prompt_s"], st["output_s"]) == (9.5, 1.0, 8.0)
assert ollama.page_stats({})["context_tokens"] == 0
def test_loaded_model_reports_gpu_fit(monkeypatch):
def _fake(method, url, payload, timeout):
assert url.endswith("/api/ps")
return {"models": [{"name": "gemma4:31b", "size": 21_000_000_000,
"size_vram": 19_740_000_000, "context_length": 32768}]}
monkeypatch.setattr(ollama, "_http_json", _fake)
got = ollama.loaded_model(REMOTE, 2)
assert got == {"loaded_bytes": 21_000_000_000, "vram_bytes": 19_740_000_000,
"gpu_fraction": 0.94, "loaded_num_ctx": 32768}
def test_loaded_model_unreachable_is_empty(monkeypatch):
def _down(*a):
raise urllib.error.URLError("connection refused")
monkeypatch.setattr(ollama, "_http_json", _down)
assert ollama.loaded_model(REMOTE, 2) == {}
def _repeat_error(url):
return urllib.error.HTTPError(
url, 500, "Internal Server Error", {},
io.BytesIO(b'{"error":"prediction aborted, token repeat limit reached"}'))
def test_looping_page_retried_with_sampling(monkeypatch):
seen = []
def _fake(method, url, payload, timeout):
seen.append(dict(payload["options"]))
if len(seen) == 1:
raise _repeat_error(url)
return {"message": {"content": "ok"}, "eval_count": 5}
monkeypatch.setattr(ollama, "_http_json", _fake)
out = ollama.transcribe_png(b"x", REMOTE, "P", 10)
assert out.text == "ok" and out.stats["retries"] == 1
assert out.stats["retry_reasons"] == ["repeat_limit"]
assert seen[0] == {"temperature": 0}
assert seen[1] == {"temperature": 0.3, "repeat_penalty": 1.15}
assert out.stats["retry_options"] == ollama.REPEAT_RETRIES[0]
def test_first_try_keeps_temperature_zero(monkeypatch):
seen = []
monkeypatch.setattr(ollama, "_http_json", lambda m, u, p, t: seen.append(p["options"])
or {"message": {"content": "ok"}})
out = ollama.transcribe_png(b"x", REMOTE, "P", 10)
assert seen == [{"temperature": 0}] and out.stats["retries"] == 0
def test_repetition_error_after_all_retries(monkeypatch):
def _fake(method, url, payload, timeout):
raise _repeat_error(url)
monkeypatch.setattr(ollama, "_http_json", _fake)
with pytest.raises(ollama.RepetitionError):
ollama.transcribe_png(b"x", REMOTE, "P", 10)
def test_runaway_output_capped_and_retried(monkeypatch):
seen = []
def _fake(method, url, payload, timeout):
seen.append(dict(payload["options"]))
if len(seen) == 1:
return {"message": {"content": "x x x"}, "done_reason": "length", "eval_count": 2048}
return {"message": {"content": "fine"}, "done_reason": "stop", "eval_count": 300}
monkeypatch.setattr(ollama, "_http_json", _fake)
out = ollama.transcribe_png(b"x", REMOTE, "P", 10, max_output=2048)
assert seen[0] == {"temperature": 0, "num_predict": 2048}
assert out.text == "fine" and out.stats["retries"] == 1
assert out.stats["retry_reasons"] == ["output_cap"]
def test_runaway_on_every_attempt_is_marked_cut_off(monkeypatch):
monkeypatch.setattr(ollama, "_http_json", lambda m, u, p, t: {
"message": {"content": "long"}, "done_reason": "length", "eval_count": 2048})
out = ollama.transcribe_png(b"x", REMOTE, "P", 10, max_output=2048)
assert out.text == "long\n" + ollama.CUT_OFF_MARKER
assert out.stats["retries"] == 2 and out.stats["done_reason"] == "length"

36
tests/test_stats.py Normal file
View file

@ -0,0 +1,36 @@
"""The scan-notes-stats summary."""
from __future__ import annotations
from pdf_tools.scan_stats import StatsLog, read_log, summarize
def rec(ctx, run="r1", reason="stop", label="nb.pdf p.1"):
return {"run": run, "host": "gpu-box", "model": "gemma4:31b",
"loaded_num_ctx": 32768, "label": label, "image_px": [2000, 1500],
"prompt_tokens": ctx - 300, "output_tokens": 300, "context_tokens": ctx,
"done_reason": reason, "total_s": 10.0, "gpu_fraction": 0.94,
"vram_bytes": 19.7e9, "loaded_bytes": 21e9}
def test_threshold_counts_and_extremes():
rs = [rec(1500), rec(3000), rec(5000), rec(9000, reason="length", label="big p.9")]
out = summarize(rs, thresholds=(4096, 8192))
assert "4 pages, 1 run(s)" in out
assert "> 4096: 2 (50.0%)" in out and "> 8192: 1 (25.0%)" in out
assert "length limit: 1 page(s)" in out
assert "model looped: 0 page(s) retried, 0 left as placeholders" in out
assert "model in GPU memory: 94-94%" in out
assert out.index("big p.9") < out.index("nb.pdf p.1")
def test_log_roundtrip(tmp_path):
log = StatsLog(tmp_path / "sub" / "s.jsonl")
log.write(rec(100))
log.write(rec(200))
(tmp_path / "sub" / "s.jsonl").open("a").write("not json\n")
assert [r["context_tokens"] for r in read_log(tmp_path / "sub" / "s.jsonl")] == [100, 200]
StatsLog(None).write(rec(1)) # disabled log is a no-op
def test_empty():
assert summarize([]) == "no pages logged yet"

1071
uv.lock generated Normal file

File diff suppressed because it is too large Load diff