Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
commit
e464871b69
29 changed files with 5296 additions and 0 deletions
10
.gitignore
vendored
Normal file
10
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,10 @@
|
||||||
|
.venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
.pytest_cache/
|
||||||
|
*.egg-info/
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
.DS_Store
|
||||||
|
# Scan outputs and progress files, in case a scan is processed in the repo.
|
||||||
|
.*.scan-partial.json
|
||||||
95
CLAUDE.md
Normal file
95
CLAUDE.md
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
# CLAUDE.md -- pdf-tools
|
||||||
|
|
||||||
|
Small command-line utilities for marked-up PDFs and scanned handwritten
|
||||||
|
notes. Python 3.11+, managed with `uv`. User-facing documentation is in
|
||||||
|
`README.md`.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
src/pdf_tools/
|
||||||
|
scrub.py pdf-scrub
|
||||||
|
extract_pdf.py pdf-notes-md
|
||||||
|
extract_scan.py scan-notes-md: CLI, page rendering, Claude backend
|
||||||
|
ollama.py scan-notes-md Ollama backend: prompt, host selection
|
||||||
|
companion.py scan-notes-md companion writer, driven by CompanionSpec
|
||||||
|
scan_config.py scan-notes-md config file and defaults
|
||||||
|
scan_stats.py per-page stats log and scan-notes-stats summary
|
||||||
|
scan_index.py scan-notes-index: per-page metadata files from transcripts
|
||||||
|
common.py shared helpers
|
||||||
|
scripts/ shell wrappers (Quick Actions, batch-scan.sh, batch-index.sh)
|
||||||
|
docs/ findings, e.g. ollama-context-and-gpu.md
|
||||||
|
tests/ pytest; run with `uv run pytest`
|
||||||
|
```
|
||||||
|
|
||||||
|
## scan-notes-md rules
|
||||||
|
|
||||||
|
- **No cloud fallback.** `--backend ollama` must never call Claude or any
|
||||||
|
other cloud service, and `ollama.py` must not import `anthropic`. If no
|
||||||
|
Ollama host is usable, exit with an error (status 3). The tests fail if
|
||||||
|
Claude is called on an Ollama run; keep them that way.
|
||||||
|
- **Nothing personal in the code.** The package is meant to be published.
|
||||||
|
Folder paths, host names, organization names, title tables, and name
|
||||||
|
lists belong in the user's config file, never in `src/` or the defaults.
|
||||||
|
`scan-notes.example.toml` uses placeholder names.
|
||||||
|
- **Confidential folders.** Anything under a `[[confidential]]` root
|
||||||
|
defaults to Ollama, and Claude is refused without `--allow-cloud`. This
|
||||||
|
check is based on location, not on `--companion`. Keep the backend
|
||||||
|
decision in Python, not in the shell wrappers. Reject unknown config keys,
|
||||||
|
so a typo cannot silently drop a protection.
|
||||||
|
- **No stray copies.** Page images are rendered in memory. Nothing is
|
||||||
|
written outside the scan's folder except to the system temp directory, and
|
||||||
|
temp files are removed. Do not log transcript text or images.
|
||||||
|
- **Companion edits are narrow.** In an existing companion, change only
|
||||||
|
the `### Draft transcript (unverified)` block and the `transcribed_by:`
|
||||||
|
field. Layouts come from `[confidential.companion]` tables; add new
|
||||||
|
layout features as `CompanionSpec` options, not as named special cases.
|
||||||
|
- **One prompt.** The Ollama instruction is `HANDWRITING_PROMPT` in
|
||||||
|
`ollama.py`. `build_prompt` appends configured `prompt_extra` text (top
|
||||||
|
level, then per folder) and the name list. Keep domain guidance, such as
|
||||||
|
the drawing and equation rules, in config, not in the base prompt. The
|
||||||
|
Claude backend keeps its own `TRANSCRIBE_PROMPT` in `extract_scan.py`.
|
||||||
|
- **Record what we learn.** Context size, GPU memory, speed, and model
|
||||||
|
failures go in `docs/ollama-context-and-gpu.md` with a date, model,
|
||||||
|
GPU, and Ollama version. The stats log holds numbers and page labels
|
||||||
|
only, never transcript text.
|
||||||
|
- **Looping pages.** A `RepetitionError` page is retried with
|
||||||
|
`REPEAT_RETRIES` and then replaced by `REPEAT_PLACEHOLDER`; first
|
||||||
|
attempts stay at temperature 0.
|
||||||
|
- **Batch runs.** `scripts/batch-scan.sh` skips PDFs that have a `.md`.
|
||||||
|
`extract_scan` checkpoints after every page to a hidden
|
||||||
|
`.<stem>.scan-partial.json` next to the scan, keyed on inputs, model,
|
||||||
|
prompt, and dpi, and removes it once the output is written.
|
||||||
|
- **Config, not code.** Hosts, models, timeouts, folders, layouts, and
|
||||||
|
per-folder names live in `~/.config/pdf-tools/scan-notes.toml`; generic
|
||||||
|
defaults are in `scan_config.py`. Keep `scan-notes.example.toml` loading
|
||||||
|
(a test checks it).
|
||||||
|
Command-line flags override the file.
|
||||||
|
- **HTTP is standard library.** Ollama calls go through `ollama._http_json`
|
||||||
|
(`urllib`), which the tests replace. Do not add an HTTP or Ollama SDK.
|
||||||
|
|
||||||
|
## scan-notes-index rules
|
||||||
|
|
||||||
|
- **Ollama only.** It has no Claude backend and uses the same host
|
||||||
|
selection and no-fallback rule as `--backend ollama`.
|
||||||
|
- **Generated text is labeled.** Summaries and key points are interpretation,
|
||||||
|
unlike the transcript; page files mark them machine-generated and
|
||||||
|
unverified. The prompt tells the model never to add reference details
|
||||||
|
that are not on the page; keep that.
|
||||||
|
- **Prompt and schema.** `INDEX_PROMPT` and `schema()` in `scan_index.py`
|
||||||
|
are the base; names, `prompt_extra`, and `kinds` come from `[index]` in
|
||||||
|
the config. Answers are constrained with Ollama's `format` (JSON schema)
|
||||||
|
through `ollama.chat_json`, which retries loops, output cut off at the
|
||||||
|
cap, and invalid JSON.
|
||||||
|
- **Resume by file.** A page whose file exists is skipped unless `--force`.
|
||||||
|
A failed page is reported, the others are written, and the exit status is
|
||||||
|
4. The stats log gets `"task": "index"` records with counts only.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Add dependencies with `uv add` (or `uv add --dev`); keep them few.
|
||||||
|
- Status and progress go to stderr; document output goes to stdout or files.
|
||||||
|
- US English spelling. No emojis in code, output, or documentation.
|
||||||
|
- Tests must not touch the network or real confidential files: fake
|
||||||
|
`_http_json`, use `tmp_path`, and declare temporary confidential folders
|
||||||
|
in a config file passed through `PDF_TOOLS_SCAN_CONFIG`.
|
||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Eric Furst
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
293
README.md
Normal file
293
README.md
Normal file
|
|
@ -0,0 +1,293 @@
|
||||||
|
# pdf-tools
|
||||||
|
|
||||||
|
Three small command-line utilities for working with marked-up PDFs and
|
||||||
|
scanned notebook pages on macOS.
|
||||||
|
|
||||||
|
1. **`pdf-scrub`** — strip annotations from a PDF (highlights, sticky
|
||||||
|
notes, freehand, text boxes) so you can send a clean copy.
|
||||||
|
2. **`pdf-notes-md`** — extract your notes plus the highlighted passages
|
||||||
|
they refer to, write as Markdown for downstream pipelines.
|
||||||
|
3. **`scan-notes-md`** — transcribe scanned handwritten notebook pages
|
||||||
|
to Markdown via a vision-language model: a local model through Ollama
|
||||||
|
by default, or Claude through the Anthropic API when asked. Scans in
|
||||||
|
confidential folders always stay on local machines.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <repo> pdf-tools
|
||||||
|
cd pdf-tools
|
||||||
|
uv sync
|
||||||
|
uv pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
After this, `pdf-scrub`, `pdf-notes-md`, and `scan-notes-md` are on
|
||||||
|
your `PATH` inside the project's venv. To use them from anywhere, either
|
||||||
|
activate the venv or use the shell wrappers in `scripts/`.
|
||||||
|
|
||||||
|
`scan-notes-md` needs `ANTHROPIC_API_KEY` in the environment for the
|
||||||
|
Claude backend only. The Ollama backend needs no key and no extra packages.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tool 1
|
||||||
|
pdf-scrub paper.pdf # -> paper_clean.pdf
|
||||||
|
pdf-scrub paper.pdf -o out.pdf
|
||||||
|
pdf-scrub paper.pdf --include-forms # also drop form widgets
|
||||||
|
|
||||||
|
# Tool 2
|
||||||
|
pdf-notes-md paper.pdf # -> paper.md
|
||||||
|
pdf-notes-md paper.pdf --stdout | less
|
||||||
|
|
||||||
|
# Tool 3
|
||||||
|
scan-notes-md page1.jpg page2.jpg # -> page1.md
|
||||||
|
scan-notes-md notebook.pdf -o out.md
|
||||||
|
scan-notes-md --backend ollama notebook.pdf # local model
|
||||||
|
scan-notes-md --backend ollama --host local scan.pdf # laptop only
|
||||||
|
scan-notes-md ~/confidential/.../2026-10-29-notes.pdf
|
||||||
|
# confidential folder: ollama + companion
|
||||||
|
```
|
||||||
|
|
||||||
|
## scan-notes-md backends
|
||||||
|
|
||||||
|
**Claude** (`--backend claude`, or `backend = "claude"` in the config) calls
|
||||||
|
the Anthropic API, one request per page, with `ANTHROPIC_API_KEY` and model
|
||||||
|
`claude-opus-5-5` unless `--model` says otherwise. Output is Markdown with one
|
||||||
|
`##` section per page. It is never used for scans in confidential folders
|
||||||
|
unless `--allow-cloud` is given.
|
||||||
|
|
||||||
|
**Ollama** (`--backend ollama`, the default) sends each page, rendered to PNG in memory,
|
||||||
|
to a vision model on a machine you own. Hosts are tried in the order listed
|
||||||
|
in the config file. Each is checked first with a 2.5 s request to
|
||||||
|
`/api/tags`. A host that is unreachable or lacks its model is skipped with
|
||||||
|
one line of output. `--host remote` or `--host local` uses only that host
|
||||||
|
and fails if it is unavailable. If no host works, the tool exits with
|
||||||
|
status 3. It never falls back to Claude or any other cloud service.
|
||||||
|
|
||||||
|
Each request uses `stream: false`, `think: false`, and temperature 0. Any
|
||||||
|
thinking text that leaks through is stripped. The instruction is
|
||||||
|
`HANDWRITING_PROMPT` in `src/pdf_tools/ollama.py`. Pages are joined with
|
||||||
|
`<!-- page N -->` markers, each page in a `text` code fence so that line
|
||||||
|
breaks and indentation survive Markdown rendering.
|
||||||
|
|
||||||
|
Hosts and models come from the config file. Without one, the only host is
|
||||||
|
`http://localhost:11434` with `gemma4:e4b`. In testing on handwritten board
|
||||||
|
notes, `gemma4:31b` on a GPU workstation gave the most reliable drafts;
|
||||||
|
`gemma4:e4b` on a laptop gives a rough draft to correct.
|
||||||
|
|
||||||
|
A remote Ollama host must listen on a reachable interface (for example
|
||||||
|
`OLLAMA_HOST=0.0.0.0` in its service environment, reached over Tailscale),
|
||||||
|
or the health check fails and every run falls back to the next host.
|
||||||
|
|
||||||
|
### Config file
|
||||||
|
|
||||||
|
`~/.config/pdf-tools/scan-notes.toml` (or `$PDF_TOOLS_SCAN_CONFIG`, or
|
||||||
|
`--config`) sets the default backend, `dpi`, Ollama timeouts, the host list,
|
||||||
|
and the confidential folders. Flags override it. Start from
|
||||||
|
`scan-notes.example.toml`, which is commented throughout. Unknown keys are
|
||||||
|
reported as errors, so a typo cannot silently disable a setting. In TOML, a
|
||||||
|
key written after a `[table]` header belongs to that table, so top-level
|
||||||
|
keys must come before the first one; a misplaced key is reported.
|
||||||
|
|
||||||
|
`prompt_extra` adds text to the Ollama prompt after the base instruction.
|
||||||
|
The example file uses it for notes with drawings, arrows, and equations:
|
||||||
|
drawings become one `[FIGURE: description]` line, arrows become `->` or an
|
||||||
|
`[ARROW: from -> to]` line, and equations are copied in plain text with
|
||||||
|
Unicode symbols. A confidential folder can add its own `prompt_extra`,
|
||||||
|
which follows the top-level one. Changing the prompt starts interrupted
|
||||||
|
runs over rather than resuming them, so pages are not mixed.
|
||||||
|
|
||||||
|
### Confidential folders and companion files
|
||||||
|
|
||||||
|
No folder is confidential unless the config file says so. Each folder gets
|
||||||
|
one `[[confidential]]` block:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[[confidential]]
|
||||||
|
root = "~/Documents/personnel"
|
||||||
|
names = ["Pat Example", "Facilities Office"]
|
||||||
|
```
|
||||||
|
|
||||||
|
When folders are nested, the deepest one applies. For a scan inside any
|
||||||
|
confidential folder:
|
||||||
|
|
||||||
|
- The backend defaults to `ollama`. `--backend claude` is refused unless
|
||||||
|
`--allow-cloud` is also given, even with `--no-companion`.
|
||||||
|
- The folder's `prompt_extra` text, if any, is added to the Ollama prompt.
|
||||||
|
- The folder's `names` are added to the Ollama prompt: if a word plausibly
|
||||||
|
matches one of them, the model uses that spelling, and otherwise writes
|
||||||
|
`[illegible]`. Names are per folder, so a long list for one folder does
|
||||||
|
not lengthen the prompt or steer the reading for others. A top-level
|
||||||
|
`names` list applies to every Ollama run. Names are never sent to Claude.
|
||||||
|
- Output goes to the companion `<stem>.md` next to the scan.
|
||||||
|
|
||||||
|
**Companion layout.** Without a `[confidential.companion]` table, a new
|
||||||
|
companion has `date`, `type: notes`, `scan:`, `subjects: []`, and
|
||||||
|
`transcribed_by:` in its front matter, a title from the filename, and one
|
||||||
|
Notes section. A `[confidential.companion]` table describes any other
|
||||||
|
layout:
|
||||||
|
|
||||||
|
| Key | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `type` | value of the `type:` field |
|
||||||
|
| `sections` | `##` sections, in order |
|
||||||
|
| `transcript_section` | section that holds the draft transcript (default: the first) |
|
||||||
|
| `section_text` | starting text for named sections |
|
||||||
|
| `fields` | front-matter fields read from the path, after `date` |
|
||||||
|
| `paths` | rules that read those fields from folder names (below) |
|
||||||
|
| `allowed` | permitted values for a captured field |
|
||||||
|
| `title` | title template: `{date}`, `{stem}`, `{field}`, `{field.short}` |
|
||||||
|
| `acronyms` | words kept in capitals when folder names become title words |
|
||||||
|
|
||||||
|
Each `[[confidential.companion.paths]]` rule has a `match` pattern over the
|
||||||
|
folders below the root. A pattern segment is a literal folder name,
|
||||||
|
`{field}` to capture one folder, or a trailing `**` for any remaining
|
||||||
|
folders. Rules are tried in order, and the first match wins. A rule can add
|
||||||
|
fixed values with `set` and a different `title`. In titles, `{field}` is
|
||||||
|
replaced by the `[confidential.titles]` entry for the captured folder name,
|
||||||
|
or by the name turned into words. `{field.short}` uses the
|
||||||
|
`"<name>.short"` entry if there is one. `scan-notes.example.toml` shows a
|
||||||
|
complete meeting-notes layout for board-service folders laid out as
|
||||||
|
`<BODY>/committees/<committee>/<meeting>/`.
|
||||||
|
|
||||||
|
**Transcript placement.**
|
||||||
|
|
||||||
|
- The transcript goes in the transcript section under
|
||||||
|
`### Draft transcript (unverified)`, ending with
|
||||||
|
`<!-- end draft transcript -->`. It replaces `Transcript pending; see
|
||||||
|
scan.` if present.
|
||||||
|
- The date comes from the filename, or else from the nearest dated folder.
|
||||||
|
- In an existing companion, only that block and the `transcribed_by:`
|
||||||
|
front-matter field are changed. A rerun replaces the block, including any
|
||||||
|
edits made inside it, so move corrections out of the block first.
|
||||||
|
- `transcribed_by:` records the source, such as
|
||||||
|
`ollama/localhost/gemma4:e4b`.
|
||||||
|
- `--companion` or `--no-companion` overrides the automatic choice.
|
||||||
|
Companion mode takes one PDF and cannot be combined with `-o` or
|
||||||
|
`--stdout`. A run cannot mix confidential scans with other files.
|
||||||
|
|
||||||
|
Page images and transcripts are never written anywhere except the
|
||||||
|
companion file. The companion is written through a temporary file in the
|
||||||
|
same folder, which is renamed into place.
|
||||||
|
|
||||||
|
### Context size, GPU memory, and run statistics
|
||||||
|
|
||||||
|
Each page transcribed through Ollama appends one line of statistics to
|
||||||
|
`~/.local/state/pdf-tools/scan-stats.jsonl` (config key `stats_log`; `""`
|
||||||
|
turns it off): page label, image size, prompt and output tokens, stop
|
||||||
|
reason, timings, and how much of the model sat in GPU memory. No transcript
|
||||||
|
text is logged. Page labels include file names, so turn the log off if file
|
||||||
|
names are themselves sensitive. `scan-notes-stats` summarizes the log,
|
||||||
|
including how many pages would exceed a given `num_ctx`.
|
||||||
|
|
||||||
|
Some pages send a model into a loop. Ollama either aborts the request with
|
||||||
|
"token repeat limit reached" or the output runs on until
|
||||||
|
`max_output_tokens` (`[ollama]` table, default 2048). Either way the page is
|
||||||
|
retried twice with light sampling and a repeat penalty. A page that still
|
||||||
|
trips the repeat check is written as
|
||||||
|
`[page not transcribed: the model kept repeating itself; see scan]`; one
|
||||||
|
that still hits the output cap keeps its text, marked
|
||||||
|
`[transcript cut off at the output limit; see scan]`.
|
||||||
|
|
||||||
|
What we have measured so far, and how to choose `num_ctx`, is in
|
||||||
|
`docs/ollama-context-and-gpu.md`.
|
||||||
|
|
||||||
|
### Batch runs
|
||||||
|
|
||||||
|
`scripts/batch-scan.sh <folder>` transcribes every PDF in a folder with the
|
||||||
|
Ollama backend, skipping PDFs that already have a `.md`. Progress is saved
|
||||||
|
after each page, so rerunning the script resumes an interrupted PDF.
|
||||||
|
|
||||||
|
## scan-notes-index: per-page search metadata
|
||||||
|
|
||||||
|
A transcript is searchable for words you wrote, but not for what a page is
|
||||||
|
about: a derivation written mostly in symbols, a sketch, or a meeting page
|
||||||
|
whose topic is never named. `scan-notes-index` reads a transcribed scan and
|
||||||
|
writes one Markdown file per page with a machine-generated description:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scan-notes-index "2009 notebook.pdf" # reads "2009 notebook.md"
|
||||||
|
# -> "2009 notebook pages/2009 notebook p001.md", p002.md, ...
|
||||||
|
scripts/batch-index.sh ~/notebooks # every transcribed PDF, newest first
|
||||||
|
```
|
||||||
|
|
||||||
|
Each page file has front matter (`kind`, `topics`, `references`, `event`,
|
||||||
|
`event_date`, `people`, `continues_from`, `continues_to`, `indexed_by`), a
|
||||||
|
summary paragraph and a list of key points marked as machine-generated and
|
||||||
|
unverified, the references written on the page, and the page's transcript.
|
||||||
|
One file per page means Spotlight or DEVONthink returns the page itself
|
||||||
|
rather than the whole notebook, and the files are ready to embed for vector
|
||||||
|
search later.
|
||||||
|
|
||||||
|
The model sees the page image, its transcript, and the transcripts of the
|
||||||
|
previous and next pages, so calculations and notes that run across pages are
|
||||||
|
described as such. It runs only on the configured Ollama hosts, like
|
||||||
|
`--backend ollama`; there is no cloud option. Existing page files are kept,
|
||||||
|
so an interrupted run resumes; `--force` rewrites them. Settings are in the
|
||||||
|
`[index]` table of the config file: extra `names` for the prompt (added to
|
||||||
|
the transcription names), `prompt_extra`, the list of `kinds`,
|
||||||
|
`max_output_tokens` (default 1536), and `context_chars`, how much of each
|
||||||
|
neighboring page to include (default 2000). Statistics go to the same stats
|
||||||
|
log with `"task": "index"`, and `scan-notes-stats` reports them separately.
|
||||||
|
|
||||||
|
On gemma4:31b with `num_ctx = 8192` on an RTX 4090, a page takes about 11 s
|
||||||
|
and uses at most about 3,800 tokens of context.
|
||||||
|
|
||||||
|
### Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
The tests fake all HTTP calls and fail if Claude is called.
|
||||||
|
|
||||||
|
## macOS integration
|
||||||
|
|
||||||
|
### "Scrub & Mail" Quick Action
|
||||||
|
|
||||||
|
The `scripts/scrub-and-mail.sh` wrapper runs the scrubber, then opens a
|
||||||
|
new Mail.app compose window with the cleaned PDF attached. To wire it
|
||||||
|
up as a right-click Finder action:
|
||||||
|
|
||||||
|
1. Open **Automator** → New Document → **Quick Action**.
|
||||||
|
2. "Workflow receives current" → **PDF files** in **Finder**.
|
||||||
|
3. Add a **Run Shell Script** action.
|
||||||
|
4. "Pass input:" **as arguments**.
|
||||||
|
5. Paste:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PATH="/opt/homebrew/bin:/usr/local/bin:$PATH"
|
||||||
|
/path/to/pdf-tools/scripts/scrub-and-mail.sh "$1"
|
||||||
|
```
|
||||||
|
|
||||||
|
6. Save as "Scrub & Mail".
|
||||||
|
|
||||||
|
It will appear under Finder right-click → Quick Actions.
|
||||||
|
|
||||||
|
You can do the same for the other two tools (`pdf-notes-to-md.sh`,
|
||||||
|
`scan-to-md.sh`); they just save Markdown next to the input rather than
|
||||||
|
opening another app. For `scan-to-md.sh`, `scan-notes-md` chooses the
|
||||||
|
backend itself: Ollama for confidential folders, Claude elsewhere.
|
||||||
|
|
||||||
|
### Shortcuts (alternative)
|
||||||
|
|
||||||
|
If you prefer Shortcuts.app over Automator, create a new shortcut:
|
||||||
|
- Receive **PDF files** from **Quick Actions**.
|
||||||
|
- "Run Shell Script" with the same command.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- The scrubber removes annotations from the `/Annots` arrays and clears
|
||||||
|
some metadata fields (`dc:creator`, `xmp:CreatorTool`, `pdf:Producer`)
|
||||||
|
that often leak the reviewing app or user identity. Form field
|
||||||
|
widgets are kept unless `--include-forms` is passed.
|
||||||
|
- The scrubber will not remove markings that have been *flattened* into
|
||||||
|
the page content stream by a previous export step. If your PDF
|
||||||
|
workflow ever does that, retain the unflattened original.
|
||||||
|
- The Claude call lives in `_transcribe_image` in `extract_scan.py`; the
|
||||||
|
Ollama call lives in `transcribe_png` in `ollama.py`.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT; see `LICENSE`.
|
||||||
142
docs/ollama-context-and-gpu.md
Normal file
142
docs/ollama-context-and-gpu.md
Normal file
|
|
@ -0,0 +1,142 @@
|
||||||
|
# Ollama context size and GPU memory
|
||||||
|
|
||||||
|
What we have learned about how much context a handwritten page needs, how
|
||||||
|
the context size decides whether a model fits in GPU memory, and how to
|
||||||
|
measure both. Update the findings below as new runs add data.
|
||||||
|
|
||||||
|
## Why it matters
|
||||||
|
|
||||||
|
Ollama reserves memory for the model's context window (the KV cache) when it
|
||||||
|
loads the model. A larger `num_ctx` reserves more memory. If weights plus
|
||||||
|
cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every
|
||||||
|
token gets slower. Too small a context has the opposite risk: the prompt,
|
||||||
|
which includes the page image, is truncated, or the output stops early.
|
||||||
|
|
||||||
|
One page uses: prompt text (about 300 tokens with `prompt_extra`) + image
|
||||||
|
tokens + output tokens. The context only needs to hold one page at a time,
|
||||||
|
because each page is a separate request.
|
||||||
|
|
||||||
|
## How to measure
|
||||||
|
|
||||||
|
Every page transcribed with the Ollama backend appends one line to the stats
|
||||||
|
log (`stats_log` in the config; default
|
||||||
|
`~/.local/state/pdf-tools/scan-stats.jsonl`). A line records the page label,
|
||||||
|
image size in pixels, prompt and output tokens, the stop reason, timings,
|
||||||
|
and how the model was loaded: total size, bytes in VRAM, and the context
|
||||||
|
length (from Ollama's `/api/ps`). It never records transcript text.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
scan-notes-stats # summary of everything logged
|
||||||
|
scan-notes-stats --run last # most recent run only
|
||||||
|
scan-notes-stats --threshold 6144 --threshold 8192 --top 10
|
||||||
|
```
|
||||||
|
|
||||||
|
The summary reports, per model and loaded context size, token percentiles,
|
||||||
|
how many pages would need more than each threshold, pages whose output hit a
|
||||||
|
length limit, pages where the model looped, the share of the model in GPU
|
||||||
|
memory, and seconds per page.
|
||||||
|
|
||||||
|
During a run, `scan-notes-md` warns when a page's output stops at a length
|
||||||
|
limit or uses more than 90% of the loaded context.
|
||||||
|
|
||||||
|
To try a smaller context, set `num_ctx` on the host in the config:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "local"
|
||||||
|
url = "http://localhost:11434"
|
||||||
|
model = "gemma4:31b"
|
||||||
|
num_ctx = 8192
|
||||||
|
```
|
||||||
|
|
||||||
|
Changing `num_ctx` does not invalidate saved progress, so an interrupted
|
||||||
|
notebook resumes. Check `ollama ps` afterwards: the PROCESSOR column should
|
||||||
|
read `100% GPU`.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### 2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13
|
||||||
|
|
||||||
|
- **Default context.** With no `num_ctx` set, Ollama loaded the model with a
|
||||||
|
32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, so `ollama ps`
|
||||||
|
showed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB).
|
||||||
|
- **First pages.** On the first 10 logged pages (2200 x 1700 to 2339 x 1653
|
||||||
|
px at 200 dpi), prompts were 768 to 787 tokens (base prompt,
|
||||||
|
`prompt_extra`, and image) and outputs 351 to 462 tokens; the largest page
|
||||||
|
used 1,230 tokens, under 4% of the loaded context. The image accounts for
|
||||||
|
most of the prompt and varies little with page size. Pages averaged 15 s,
|
||||||
|
so a 2,562-page batch takes about 11 hours at this setting.
|
||||||
|
- **Looping.** Some pages make the model repeat itself until Ollama aborts
|
||||||
|
the request with `prediction aborted, token repeat limit reached`. At
|
||||||
|
temperature 0 the same page usually fails the same way: one page failed
|
||||||
|
in two separate runs, though it went through on a third, because GPU
|
||||||
|
inference at temperature 0 is not perfectly deterministic. It happened on
|
||||||
|
2 of the first 70 pages. `scan-notes-md` now retries
|
||||||
|
such a page with a little sampling (temperature 0.3, then 0.6, with a
|
||||||
|
repeat penalty) and, if it still loops, writes a visible placeholder:
|
||||||
|
`[page not transcribed: the model kept repeating itself; see scan]`.
|
||||||
|
`scan-notes-stats` counts retried pages and placeholders.
|
||||||
|
- **Runaway output.** A model can also loop without tripping the repeat
|
||||||
|
check. Ollama's default output limit is unlimited, so such a page ran
|
||||||
|
past six minutes and would have failed at the 600 s timeout on every
|
||||||
|
rerun. `max_output_tokens` (default 2048, passed as `num_predict`) now
|
||||||
|
stops it early. A page that hits the cap is retried like a looping page;
|
||||||
|
if it hits the cap every time, its text is kept and marked
|
||||||
|
`[transcript cut off at the output limit; see scan]`. The output cap also
|
||||||
|
bounds the context a page can use: prompt plus at most 2,048 output
|
||||||
|
tokens.
|
||||||
|
|
||||||
|
### 2026-11-02: full run, 40 notebooks, same setup
|
||||||
|
|
||||||
|
The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages)
|
||||||
|
was transcribed before logging was added.
|
||||||
|
|
||||||
|
| Measure | mean | p50 | p90 | p99 | max |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| Prompt tokens (incl. image) | 781 | 787 | 787 | 787 | 797 |
|
||||||
|
| Output tokens | 278 | 278 | 453 | 674 | 1,125 |
|
||||||
|
| Context used | 1,059 | 1,058 | 1,232 | 1,459 | 1,912 |
|
||||||
|
|
||||||
|
- **Context.** No page needed more than 1,912 tokens, under 6% of the
|
||||||
|
loaded 32,768 and under half of 4,096. No page reached the 2,048-token
|
||||||
|
output cap on its final attempt. The prompt is nearly constant (768 to 797
|
||||||
|
tokens), which suggests the model encodes each image into a fixed number of tokens; page size
|
||||||
|
and resolution barely matter; context use is set by how much is written
|
||||||
|
on the page. Even a page that hits the output cap needs only about 2,850
|
||||||
|
tokens, so 4,096 would hold every page with this prompt.
|
||||||
|
- **GPU fit.** The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM)
|
||||||
|
for the whole run; the default context is what keeps it off the GPU
|
||||||
|
entirely.
|
||||||
|
- **Speed.** 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of
|
||||||
|
model time for the logged pages. Wall-clock time for the batch was about
|
||||||
|
8 h. Blank and nearly blank pages take 2 to 4 s.
|
||||||
|
- **Retries.** 65 pages (2.6%) were retried: 53 after Ollama's repeat
|
||||||
|
limit, 7 after reaching the output cap, and 5 before reasons were logged.
|
||||||
|
Every one finished normally on its first retry (temperature 0.3, repeat
|
||||||
|
penalty 1.15), with 67 to 888 output tokens, so these were loops rather
|
||||||
|
than long pages. No page was left as a placeholder or marked cut off. The
|
||||||
|
repeat penalty can also discourage repetition that is really on the page
|
||||||
|
(the same value down a column), so retried pages are the ones most worth
|
||||||
|
checking against the scan; they are the log lines with `"retries": 1`.
|
||||||
|
|
||||||
|
### 2026-11-02: 8192 context compared with 32768, one notebook
|
||||||
|
|
||||||
|
The same 85-page notebook, same prompt and settings except
|
||||||
|
`num_ctx = 8192`:
|
||||||
|
|
||||||
|
| | 32,768 (default) | 8,192 |
|
||||||
|
|---|---|---|
|
||||||
|
| Loaded size, GPU share | 21.2 GB, 94% GPU | 20 GB, 100% GPU |
|
||||||
|
| Seconds per page (mean) | 11.6 | 8.3 |
|
||||||
|
| Total for the notebook | 16.4 min | 11.8 min |
|
||||||
|
| Prompt time per page | 0.84 s | 0.69 s |
|
||||||
|
| Output speed | 28.9 tokens/s | 42.7 tokens/s |
|
||||||
|
| Output tokens (all pages) | 24,777 | 25,092 |
|
||||||
|
| Pages retried | 2 | 3 |
|
||||||
|
|
||||||
|
At 8192 the whole model fits on the GPU, and output runs about 48% faster,
|
||||||
|
so pages take about 28% less time. The two runs wrote nearly the same amount
|
||||||
|
of text, so the gain is not from shorter output. The largest page in the full
|
||||||
|
run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set
|
||||||
|
`num_ctx = 8192` for gemma4:31b on a 24 GB card. At this speed the 40-notebook
|
||||||
|
batch would take about 6 hours instead of 8.
|
||||||
42
pyproject.toml
Normal file
42
pyproject.toml
Normal file
|
|
@ -0,0 +1,42 @@
|
||||||
|
[project]
|
||||||
|
name = "pdf-tools"
|
||||||
|
version = "0.1.0"
|
||||||
|
description = "Local PDF utilities: annotation scrubbing, note extraction, and handwritten-page OCR."
|
||||||
|
readme = "README.md"
|
||||||
|
requires-python = ">=3.11"
|
||||||
|
authors = [{ name = "Eric Furst" }]
|
||||||
|
license = { text = "MIT" }
|
||||||
|
|
||||||
|
dependencies = [
|
||||||
|
"pikepdf>=9.0", # robust PDF rewriting for scrubbing
|
||||||
|
"pypdf>=5.0", # text extraction under highlight rects
|
||||||
|
"pdfminer.six>=20240706", # used by pdfannots
|
||||||
|
"pdfannots>=0.4", # annotation extraction with underlying text
|
||||||
|
"Pillow>=10.4", # image handling for scan tool
|
||||||
|
"anthropic>=0.39", # Tool 3 OCR
|
||||||
|
"click>=8.1", # CLI ergonomics
|
||||||
|
"pypdfium2>=4.30", # PDF page rendering for scan tool
|
||||||
|
]
|
||||||
|
|
||||||
|
[project.scripts]
|
||||||
|
pdf-scrub = "pdf_tools.scrub:main"
|
||||||
|
pdf-notes-md = "pdf_tools.extract_pdf:main"
|
||||||
|
scan-notes-md = "pdf_tools.extract_scan:main"
|
||||||
|
scan-notes-stats = "pdf_tools.scan_stats:main"
|
||||||
|
scan-notes-index = "pdf_tools.scan_index:main"
|
||||||
|
|
||||||
|
[build-system]
|
||||||
|
requires = ["hatchling"]
|
||||||
|
build-backend = "hatchling.build"
|
||||||
|
|
||||||
|
[tool.hatch.build.targets.wheel]
|
||||||
|
packages = ["src/pdf_tools"]
|
||||||
|
|
||||||
|
[dependency-groups]
|
||||||
|
dev = [
|
||||||
|
"pytest>=9.2.0",
|
||||||
|
]
|
||||||
|
|
||||||
|
[tool.pytest.ini_options]
|
||||||
|
testpaths = ["tests"]
|
||||||
|
addopts = "-p no:anyio"
|
||||||
133
scan-notes.example.toml
Normal file
133
scan-notes.example.toml
Normal file
|
|
@ -0,0 +1,133 @@
|
||||||
|
# Example settings for scan-notes-md. Copy to
|
||||||
|
# ~/.config/pdf-tools/scan-notes.toml and edit. Command-line flags override
|
||||||
|
# these, and anything left out falls back to the built-in defaults.
|
||||||
|
|
||||||
|
# Backend for scans outside confidential folders: "ollama" (default) or
|
||||||
|
# "claude" (the Anthropic API; needs ANTHROPIC_API_KEY).
|
||||||
|
backend = "ollama"
|
||||||
|
|
||||||
|
# PDF render resolution.
|
||||||
|
dpi = 200
|
||||||
|
|
||||||
|
# Names added to the Ollama prompt for every scan. Prefer per-folder lists
|
||||||
|
# (below), which keep the prompt short and avoid steering unrelated scans.
|
||||||
|
names = []
|
||||||
|
|
||||||
|
# Added to the Ollama prompt after the base instruction, for every scan.
|
||||||
|
# Confidential folders can add their own prompt_extra as well.
|
||||||
|
prompt_extra = """
|
||||||
|
The page may also contain drawings, arrows, and equations. Handle them this way:
|
||||||
|
- In place of a drawing, sketch, plot, or diagram, write one line:
|
||||||
|
[FIGURE: short description]. Name what is drawn, and include any axis
|
||||||
|
labels, units, and words written on it. Do not interpret what it shows.
|
||||||
|
- Write an arrow between words as -> (or <-, <->), keeping it on the line
|
||||||
|
where it appears. If an arrow links items on different lines or points to
|
||||||
|
something away from the text, write [ARROW: from -> to] on its own line.
|
||||||
|
- Copy equations exactly as written, in plain text. Use Unicode for Greek
|
||||||
|
letters and symbols (η, φ, ∂, ∝, ≈), ^ for superscripts, _ for
|
||||||
|
subscripts, and / for fractions. Do not solve, simplify, or complete them.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Per-page token counts, timings, and GPU fit for Ollama runs (no transcript
|
||||||
|
# text). Summarize with scan-notes-stats. Set to "" to turn off.
|
||||||
|
stats_log = "~/.local/state/pdf-tools/scan-stats.jsonl"
|
||||||
|
|
||||||
|
[ollama]
|
||||||
|
timeout_s = 300 # per page
|
||||||
|
probe_timeout_s = 2.5 # health check against /api/tags
|
||||||
|
max_output_tokens = 2048 # per page; stops a looping model early (0 = no cap)
|
||||||
|
|
||||||
|
# Hosts are tried in this order. A host is skipped (with one line of output)
|
||||||
|
# if it is unreachable or does not have its model.
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "remote"
|
||||||
|
url = "http://gpu-workstation:11434"
|
||||||
|
model = "gemma4:31b"
|
||||||
|
# num_ctx = 8192 # optional; set if the model needs it to fit on the GPU
|
||||||
|
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "local"
|
||||||
|
url = "http://localhost:11434"
|
||||||
|
model = "gemma4:e4b"
|
||||||
|
|
||||||
|
# scan-notes-index: per-page search metadata (summary, key points, topics,
|
||||||
|
# references, event, people) written from a scan and its transcript.
|
||||||
|
[index]
|
||||||
|
names = [] # added to the names above, for indexing only
|
||||||
|
prompt_extra = ""
|
||||||
|
kinds = ["calculation", "idea", "sketch", "talk notes", "meeting notes",
|
||||||
|
"reading notes", "to-do", "other", "blank"]
|
||||||
|
max_output_tokens = 1536
|
||||||
|
context_chars = 2000 # characters of the previous and next pages sent as context
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Confidential folders. None are built in. Scans under a listed root stay on
|
||||||
|
# the Ollama hosts (Claude is refused without --allow-cloud) and are written
|
||||||
|
# to a companion <stem>.md next to the scan. When folders are nested, the
|
||||||
|
# deepest one applies.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Simplest form: the basic companion (date, scan, subjects, one Notes section).
|
||||||
|
[[confidential]]
|
||||||
|
root = "~/Documents/personnel"
|
||||||
|
names = []
|
||||||
|
|
||||||
|
# Full form: a board-service folder laid out as
|
||||||
|
# <BODY>/committees/<committee>/<meeting>/
|
||||||
|
# <BODY>/meetings/<meeting>/
|
||||||
|
# chair/<project>/
|
||||||
|
[[confidential]]
|
||||||
|
root = "~/Documents/board-service"
|
||||||
|
# Board members, staff, and organizations; used only for scans in this folder.
|
||||||
|
names = []
|
||||||
|
|
||||||
|
[confidential.companion]
|
||||||
|
type = "meeting-notes"
|
||||||
|
# Front-matter fields read from the path, in this order after `date`.
|
||||||
|
fields = ["body", "committee"]
|
||||||
|
sections = ["Attendees", "Discussion", "Decisions", "Action items"]
|
||||||
|
transcript_section = "Discussion"
|
||||||
|
# Title placeholders: {date} (long form), {stem} (filename without its date),
|
||||||
|
# {field} (display name from [confidential.titles], else the folder name in
|
||||||
|
# words), {field.short} (the "<value>.short" title, if any).
|
||||||
|
title = "{body}, {date}"
|
||||||
|
# Words kept in capitals when a folder name is turned into words.
|
||||||
|
acronyms = ["ceo", "cfo", "llc"]
|
||||||
|
|
||||||
|
[confidential.companion.section_text]
|
||||||
|
"Action items" = "- [ ] Owner: task (due date)"
|
||||||
|
|
||||||
|
# Optional: values a captured field may take. A rule whose capture is not
|
||||||
|
# listed does not match, so the next rule is tried.
|
||||||
|
[confidential.companion.allowed]
|
||||||
|
body = ["BOD", "BOM"]
|
||||||
|
|
||||||
|
# Rules are tried in order; the first match sets the fields. "{name}" captures
|
||||||
|
# one folder, a literal must match exactly, and a trailing "**" matches any
|
||||||
|
# remaining folders. `set` gives fixed values; `title` overrides the title.
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "{body}/committees/{committee}/**"
|
||||||
|
title = "{body.short} {committee}, {date}"
|
||||||
|
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "{body}/meetings/**"
|
||||||
|
set = { committee = "board" }
|
||||||
|
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "chair/{committee}/**"
|
||||||
|
set = { body = "chair" }
|
||||||
|
title = "{committee}, {date}"
|
||||||
|
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "{body}/{committee}/**"
|
||||||
|
title = "{body.short} {committee}, {date}"
|
||||||
|
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "{body}/**"
|
||||||
|
|
||||||
|
[confidential.titles]
|
||||||
|
"BOD" = "Board of Directors"
|
||||||
|
"BOD.short" = "Board"
|
||||||
|
"BOM" = "Board of Managers"
|
||||||
|
"BOM.short" = "Managers"
|
||||||
|
"finance-audit" = "Finance and Audit Committee"
|
||||||
52
scripts/batch-index.sh
Executable file
52
scripts/batch-index.sh
Executable file
|
|
@ -0,0 +1,52 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# batch-index.sh — run scan-notes-index on every transcribed PDF in a folder.
|
||||||
|
#
|
||||||
|
# usage: batch-index.sh <folder> [scan-notes-index options...]
|
||||||
|
#
|
||||||
|
# PDFs are taken newest first (reverse filename order, which is date order
|
||||||
|
# for date-prefixed names). A PDF without a <stem>.md transcript is skipped.
|
||||||
|
# Pages already indexed are kept, so the script can simply be rerun after an
|
||||||
|
# interruption. A failed PDF is reported and the batch moves on.
|
||||||
|
#
|
||||||
|
# Long runs belong in tmux: tmux new -s index 'batch-index.sh ~/notebooks'
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 || ! -d "$1" ]]; then
|
||||||
|
echo "usage: $0 <folder> [scan-notes-index options...]" >&2
|
||||||
|
exit 64
|
||||||
|
fi
|
||||||
|
dir="$1"; shift
|
||||||
|
|
||||||
|
if command -v scan-notes-index >/dev/null 2>&1; then
|
||||||
|
run() { scan-notes-index "$@"; }
|
||||||
|
else
|
||||||
|
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-index}"
|
||||||
|
run() { (cd "$PDF_TOOLS_DIR" && uv run --quiet scan-notes-index "$@"); }
|
||||||
|
fi
|
||||||
|
|
||||||
|
pdfs=()
|
||||||
|
while IFS= read -r -d '' f; do pdfs+=("$f"); done \
|
||||||
|
< <(find "$dir" -type f -iname '*.pdf' -not -name '.*' -print0 | sort -rz)
|
||||||
|
total=${#pdfs[@]}
|
||||||
|
done_n=0; skipped=0; failed=()
|
||||||
|
start=$(date +%s)
|
||||||
|
|
||||||
|
for i in "${!pdfs[@]}"; do
|
||||||
|
pdf="${pdfs[$i]}"
|
||||||
|
echo "=== [$((i + 1))/$total] ${pdf#"$dir"/} ($(date '+%H:%M'))" >&2
|
||||||
|
if [[ ! -e "${pdf%.*}.md" ]]; then
|
||||||
|
echo "no transcript; skipping" >&2
|
||||||
|
skipped=$((skipped + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if run "$@" "$pdf"; then
|
||||||
|
done_n=$((done_n + 1))
|
||||||
|
else
|
||||||
|
failed+=("${pdf#"$dir"/}")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
mins=$(( ($(date +%s) - start) / 60 ))
|
||||||
|
echo "=== finished in ${mins} min: $done_n indexed, $skipped skipped, ${#failed[@]} failed" >&2
|
||||||
|
for f in ${failed[@]+"${failed[@]}"}; do echo "failed: $f" >&2; done
|
||||||
|
[[ ${#failed[@]} -eq 0 ]]
|
||||||
56
scripts/batch-scan.sh
Executable file
56
scripts/batch-scan.sh
Executable file
|
|
@ -0,0 +1,56 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# batch-scan.sh — transcribe every PDF in a folder with scan-notes-md.
|
||||||
|
#
|
||||||
|
# usage: batch-scan.sh <folder> [scan-notes-md options...]
|
||||||
|
#
|
||||||
|
# PDFs that already have a <stem>.md are skipped, and an interrupted PDF
|
||||||
|
# resumes from its last saved page, so the script can simply be rerun. A
|
||||||
|
# failed PDF is reported and the batch moves on to the next one. Extra
|
||||||
|
# options are passed to scan-notes-md; the default is --backend ollama, so
|
||||||
|
# nothing is sent to a cloud service unless you ask for it.
|
||||||
|
#
|
||||||
|
# Long runs belong in tmux: tmux new -s scan 'batch-scan.sh ~/notebooks'
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 || ! -d "$1" ]]; then
|
||||||
|
echo "usage: $0 <folder> [scan-notes-md options...]" >&2
|
||||||
|
exit 64
|
||||||
|
fi
|
||||||
|
dir="$1"; shift
|
||||||
|
opts=("$@")
|
||||||
|
[[ ${#opts[@]} -eq 0 ]] && opts=(--backend ollama)
|
||||||
|
|
||||||
|
if command -v scan-notes-md >/dev/null 2>&1; then
|
||||||
|
run() { scan-notes-md "$@"; }
|
||||||
|
else
|
||||||
|
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-md}"
|
||||||
|
run() { (cd "$PDF_TOOLS_DIR" && uv run --quiet scan-notes-md "$@"); }
|
||||||
|
fi
|
||||||
|
|
||||||
|
pdfs=()
|
||||||
|
while IFS= read -r -d '' f; do pdfs+=("$f"); done \
|
||||||
|
< <(find "$dir" -type f -iname '*.pdf' -not -name '.*' -print0 | sort -z)
|
||||||
|
total=${#pdfs[@]}
|
||||||
|
done_n=0; skipped=0; failed=()
|
||||||
|
start=$(date +%s)
|
||||||
|
|
||||||
|
for i in "${!pdfs[@]}"; do
|
||||||
|
pdf="${pdfs[$i]}"
|
||||||
|
md="${pdf%.*}.md"
|
||||||
|
echo "=== [$((i + 1))/$total] ${pdf#"$dir"/} ($(date '+%H:%M'))" >&2
|
||||||
|
if [[ -e "$md" ]]; then
|
||||||
|
echo "already transcribed; skipping" >&2
|
||||||
|
skipped=$((skipped + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if run "${opts[@]}" "$pdf"; then
|
||||||
|
done_n=$((done_n + 1))
|
||||||
|
else
|
||||||
|
failed+=("${pdf#"$dir"/}")
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
mins=$(( ($(date +%s) - start) / 60 ))
|
||||||
|
echo "=== finished in ${mins} min: $done_n transcribed, $skipped skipped, ${#failed[@]} failed" >&2
|
||||||
|
for f in ${failed[@]+"${failed[@]}"}; do echo "failed: $f" >&2; done
|
||||||
|
[[ ${#failed[@]} -eq 0 ]]
|
||||||
18
scripts/pdf-notes-to-md.sh
Executable file
18
scripts/pdf-notes-to-md.sh
Executable file
|
|
@ -0,0 +1,18 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# pdf-notes-to-md.sh — extract PDF annotations to Markdown next to the PDF.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "usage: $0 <pdf> [output.md]" >&2
|
||||||
|
exit 64
|
||||||
|
fi
|
||||||
|
|
||||||
|
IN="$1"
|
||||||
|
OUT="${2:-${IN%.pdf}.md}"
|
||||||
|
|
||||||
|
if command -v pdf-notes-md >/dev/null 2>&1; then
|
||||||
|
pdf-notes-md "$IN" -o "$OUT"
|
||||||
|
else
|
||||||
|
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install pdf-notes-md}"
|
||||||
|
(cd "$PDF_TOOLS_DIR" && uv run pdf-notes-md "$IN" -o "$OUT")
|
||||||
|
fi
|
||||||
19
scripts/scan-to-md.sh
Executable file
19
scripts/scan-to-md.sh
Executable file
|
|
@ -0,0 +1,19 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scan-to-md.sh — transcribe scanned pages to Markdown.
|
||||||
|
#
|
||||||
|
# scan-notes-md itself decides the backend: scans in the confidential
|
||||||
|
# folders listed in ~/.config/pdf-tools/scan-notes.toml go to Ollama and
|
||||||
|
# their companion <stem>.md; everything else uses Claude.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "usage: $0 <image-or-pdf> [more inputs...]" >&2
|
||||||
|
exit 64
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v scan-notes-md >/dev/null 2>&1; then
|
||||||
|
scan-notes-md "$@"
|
||||||
|
else
|
||||||
|
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR or install scan-notes-md}"
|
||||||
|
(cd "$PDF_TOOLS_DIR" && uv run scan-notes-md "$@")
|
||||||
|
fi
|
||||||
49
scripts/scrub-and-mail.sh
Executable file
49
scripts/scrub-and-mail.sh
Executable file
|
|
@ -0,0 +1,49 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# scrub-and-mail.sh — scrub a PDF and open a new Mail.app message with it attached.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scrub-and-mail.sh /path/to/file.pdf
|
||||||
|
#
|
||||||
|
# Designed to be called from a macOS Quick Action / Service / Shortcut
|
||||||
|
# with the selected PDF passed as $1.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [[ $# -lt 1 ]]; then
|
||||||
|
echo "usage: $0 <pdf>" >&2
|
||||||
|
exit 64
|
||||||
|
fi
|
||||||
|
|
||||||
|
IN="$1"
|
||||||
|
if [[ ! -f "$IN" ]]; then
|
||||||
|
echo "error: not a file: $IN" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Write the scrubbed copy to a temp dir so we don't pollute Dropbox.
|
||||||
|
TMPDIR_OUT="$(mktemp -d -t pdfscrub)"
|
||||||
|
BASENAME="$(basename "$IN" .pdf)"
|
||||||
|
OUT="$TMPDIR_OUT/${BASENAME}_clean.pdf"
|
||||||
|
|
||||||
|
# Resolve the CLI. Prefer the installed entry point; fall back to
|
||||||
|
# `uv run` from the project directory if you keep it ephemeral.
|
||||||
|
if command -v pdf-scrub >/dev/null 2>&1; then
|
||||||
|
pdf-scrub "$IN" -o "$OUT" >/dev/null
|
||||||
|
else
|
||||||
|
: "${PDF_TOOLS_DIR:?set PDF_TOOLS_DIR to the project path or install pdf-scrub}"
|
||||||
|
(cd "$PDF_TOOLS_DIR" && uv run pdf-scrub "$IN" -o "$OUT" >/dev/null)
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Hand off to Mail.app. POSIX path -> HFS path for AppleScript's `file`.
|
||||||
|
osascript <<APPLESCRIPT
|
||||||
|
set theAttachment to POSIX file "$OUT"
|
||||||
|
tell application "Mail"
|
||||||
|
set theMessage to make new outgoing message with properties {visible:true, subject:"$BASENAME"}
|
||||||
|
tell theMessage
|
||||||
|
make new attachment with properties {file name:theAttachment} at after last paragraph
|
||||||
|
end tell
|
||||||
|
activate
|
||||||
|
end tell
|
||||||
|
APPLESCRIPT
|
||||||
|
|
||||||
|
echo "opened Mail with: $OUT" >&2
|
||||||
2
src/pdf_tools/__init__.py
Normal file
2
src/pdf_tools/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
"""pdf-tools: scrub, extract, and OCR utilities."""
|
||||||
|
__version__ = "0.1.0"
|
||||||
21
src/pdf_tools/common.py
Normal file
21
src/pdf_tools/common.py
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
"""Shared helpers across the three tools."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def err(msg: str) -> None:
|
||||||
|
print(msg, file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def must_exist(p: Path) -> Path:
|
||||||
|
if not p.exists():
|
||||||
|
err(f"error: not found: {p}")
|
||||||
|
sys.exit(2)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def with_suffix_before_ext(p: Path, marker: str) -> Path:
|
||||||
|
"""foo.pdf + '_clean' -> foo_clean.pdf"""
|
||||||
|
return p.with_name(f"{p.stem}{marker}{p.suffix}")
|
||||||
327
src/pdf_tools/companion.py
Normal file
327
src/pdf_tools/companion.py
Normal file
|
|
@ -0,0 +1,327 @@
|
||||||
|
"""Companion mode: write the Markdown companion for a scan in a confidential folder.
|
||||||
|
|
||||||
|
A scan `<stem>.pdf` is paired with `<stem>.md` in the same folder. This module
|
||||||
|
fills the front matter from the path and filename, and puts the transcript
|
||||||
|
under DRAFT_HEADING in the folder's transcript section.
|
||||||
|
|
||||||
|
Nothing here is specific to any one organization. The shape of a companion
|
||||||
|
(front-matter fields read from the path, sections, title) comes from a
|
||||||
|
CompanionSpec, built from the folder's [confidential.companion] table in the
|
||||||
|
config file. BASIC is used when a folder does not define one.
|
||||||
|
|
||||||
|
Existing companions are edited in two places only: the DRAFT_HEADING block
|
||||||
|
(replaced, or added if missing) and the transcribed_by: front-matter field.
|
||||||
|
Everything else in the file is kept as is.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime as _dt
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import tempfile
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
DRAFT_HEADING = "### Draft transcript (unverified)"
|
||||||
|
END_MARKER = "<!-- end draft transcript -->"
|
||||||
|
PENDING_LINE = "Transcript pending; see scan."
|
||||||
|
_DATE_PREFIX = re.compile(r"^(\d{4}-\d{2}-(?:\d{2}|xx))")
|
||||||
|
_PLACEHOLDER = re.compile(r"\{(\w+)(?:\.(short))?\}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PathRule:
|
||||||
|
"""One way to read front-matter fields from the folders under the root.
|
||||||
|
|
||||||
|
`match` is a "/"-separated pattern over folder names: a literal name,
|
||||||
|
"{field}" to capture one folder, or a trailing "**" for any remaining
|
||||||
|
folders. `set` gives fixed values; `title` overrides the spec's title.
|
||||||
|
"""
|
||||||
|
match: str
|
||||||
|
set: dict[str, str] = field(default_factory=dict)
|
||||||
|
title: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class CompanionSpec:
|
||||||
|
type: str = "notes"
|
||||||
|
sections: list[str] = field(default_factory=lambda: ["Notes"])
|
||||||
|
transcript_section: str = "Notes"
|
||||||
|
section_text: dict[str, str] = field(default_factory=dict)
|
||||||
|
title: str = "{stem}, {date}"
|
||||||
|
fields: list[str] = field(default_factory=list)
|
||||||
|
paths: list[PathRule] = field(default_factory=list)
|
||||||
|
allowed: dict[str, list[str]] = field(default_factory=dict)
|
||||||
|
acronyms: set[str] = field(default_factory=set)
|
||||||
|
|
||||||
|
|
||||||
|
BASIC = CompanionSpec()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class NoteMeta:
|
||||||
|
date: str # ISO date, YYYY-MM-xx, or "" if unknown
|
||||||
|
fields: dict[str, str] # spec.fields -> value ("" if unknown)
|
||||||
|
scan: str # PDF filename
|
||||||
|
title: str # title template chosen for this scan
|
||||||
|
warnings: list[str]
|
||||||
|
|
||||||
|
|
||||||
|
def _match(pattern: str, folders: tuple[str, ...],
|
||||||
|
allowed: dict[str, list[str]]) -> dict[str, str] | None:
|
||||||
|
segs = [s for s in pattern.strip("/").split("/") if s]
|
||||||
|
tail = bool(segs) and segs[-1] == "**"
|
||||||
|
if tail:
|
||||||
|
segs = segs[:-1]
|
||||||
|
if len(folders) < len(segs) or (not tail and len(folders) != len(segs)):
|
||||||
|
return None
|
||||||
|
got: dict[str, str] = {}
|
||||||
|
for seg, name in zip(segs, folders):
|
||||||
|
m = re.fullmatch(r"\{(\w+)\}", seg)
|
||||||
|
if m:
|
||||||
|
key = m.group(1)
|
||||||
|
if key in allowed and name not in allowed[key]:
|
||||||
|
return None
|
||||||
|
got[key] = name
|
||||||
|
elif seg != name:
|
||||||
|
return None
|
||||||
|
return got
|
||||||
|
|
||||||
|
|
||||||
|
def meta_from_path(pdf: Path, root: Path, spec: CompanionSpec = BASIC) -> NoteMeta:
|
||||||
|
"""Derive the date and the spec's fields from where the scan sits."""
|
||||||
|
rel = pdf.resolve().relative_to(root.resolve())
|
||||||
|
folders = rel.parts[:-1]
|
||||||
|
warnings: list[str] = []
|
||||||
|
|
||||||
|
values = {f: "" for f in spec.fields}
|
||||||
|
title = spec.title
|
||||||
|
if spec.fields:
|
||||||
|
for rule in spec.paths:
|
||||||
|
got = _match(rule.match, folders, spec.allowed)
|
||||||
|
if got is not None:
|
||||||
|
values.update({k: v for k, v in {**got, **rule.set}.items()
|
||||||
|
if k in values})
|
||||||
|
title = rule.title or title
|
||||||
|
break
|
||||||
|
missing = [f for f, v in values.items() if not v]
|
||||||
|
if missing:
|
||||||
|
warnings.append(f"cannot tell {', '.join(missing)} from the folder "
|
||||||
|
f"{'/'.join(folders) or '.'}; left empty")
|
||||||
|
|
||||||
|
date = ""
|
||||||
|
m = _DATE_PREFIX.match(pdf.name)
|
||||||
|
if m:
|
||||||
|
date = m.group(1)
|
||||||
|
else:
|
||||||
|
for folder in reversed(folders):
|
||||||
|
m = _DATE_PREFIX.match(folder)
|
||||||
|
if m:
|
||||||
|
date = m.group(1)
|
||||||
|
break
|
||||||
|
if not date:
|
||||||
|
warnings.append("no date in the filename or folder; left date empty")
|
||||||
|
elif date.endswith("-xx"):
|
||||||
|
warnings.append(f"date {date} has no day; check it")
|
||||||
|
|
||||||
|
return NoteMeta(date=date, fields=values, scan=pdf.name, title=title,
|
||||||
|
warnings=warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def _words(slug: str, acronyms: set[str]) -> str:
|
||||||
|
return " ".join(w.upper() if w.lower() in acronyms else w.capitalize()
|
||||||
|
for w in re.split(r"[-_ ]+", slug) if w)
|
||||||
|
|
||||||
|
|
||||||
|
def title_for(meta: NoteMeta, spec: CompanionSpec = BASIC,
|
||||||
|
titles: dict[str, str] | None = None) -> str:
|
||||||
|
"""Fill the title template.
|
||||||
|
|
||||||
|
{date} is the long date; {stem} is the filename without its date prefix;
|
||||||
|
{field} is the field's display name from `titles` (or the folder name
|
||||||
|
turned into words); {field.short} uses the `<value>.short` entry if any.
|
||||||
|
"""
|
||||||
|
titles = titles or {}
|
||||||
|
acr = {a.lower() for a in spec.acronyms}
|
||||||
|
|
||||||
|
def display(value: str, short: bool) -> str:
|
||||||
|
if short and f"{value}.short" in titles:
|
||||||
|
return titles[f"{value}.short"]
|
||||||
|
return titles.get(value) or _words(value, acr)
|
||||||
|
|
||||||
|
try:
|
||||||
|
d = _dt.date.fromisoformat(meta.date)
|
||||||
|
long_date = f"{d.strftime('%B')} {d.day}, {d.year}"
|
||||||
|
except ValueError:
|
||||||
|
long_date = ""
|
||||||
|
|
||||||
|
def sub(m: re.Match) -> str:
|
||||||
|
key, short = m.group(1), bool(m.group(2))
|
||||||
|
if key == "date":
|
||||||
|
return long_date
|
||||||
|
if key == "stem":
|
||||||
|
return _words(_DATE_PREFIX.sub("", Path(meta.scan).stem), acr) or "Notes"
|
||||||
|
return display(meta.fields.get(key, ""), short) if meta.fields.get(key) else ""
|
||||||
|
|
||||||
|
out = _PLACEHOLDER.sub(sub, meta.title)
|
||||||
|
out = re.sub(r"\s{2,}", " ", out)
|
||||||
|
return out.strip(" ,") or Path(meta.scan).stem
|
||||||
|
|
||||||
|
|
||||||
|
def render_transcript(pages: list[str]) -> str:
|
||||||
|
"""Page markers plus one fenced block per page.
|
||||||
|
|
||||||
|
Fencing keeps line breaks and indentation exactly as transcribed, and
|
||||||
|
stops a line such as "# 3" on the page from turning into a heading.
|
||||||
|
"""
|
||||||
|
out = []
|
||||||
|
for n, text in enumerate(pages, start=1):
|
||||||
|
fence = "```"
|
||||||
|
while fence in text:
|
||||||
|
fence += "`"
|
||||||
|
out += [f"<!-- page {n} -->", "", f"{fence}text", text, fence, ""]
|
||||||
|
return "\n".join(out).rstrip("\n")
|
||||||
|
|
||||||
|
|
||||||
|
def draft_block(pages: list[str]) -> str:
|
||||||
|
return f"{DRAFT_HEADING}\n\n{render_transcript(pages)}\n\n{END_MARKER}"
|
||||||
|
|
||||||
|
|
||||||
|
def new_companion(meta: NoteMeta, pages: list[str], transcribed_by: str,
|
||||||
|
spec: CompanionSpec = BASIC,
|
||||||
|
titles: dict[str, str] | None = None) -> str:
|
||||||
|
out = ["---", f"date: {meta.date}"]
|
||||||
|
out += [f"{k}: {v}" for k, v in meta.fields.items()]
|
||||||
|
out += [f"type: {spec.type}", f"scan: {meta.scan}", "subjects: []",
|
||||||
|
f"transcribed_by: {transcribed_by}", "---", "",
|
||||||
|
f"# {title_for(meta, spec, titles)}", ""]
|
||||||
|
sections = list(spec.sections)
|
||||||
|
if spec.transcript_section not in sections:
|
||||||
|
sections.append(spec.transcript_section)
|
||||||
|
for s in sections:
|
||||||
|
out += [f"## {s}", ""]
|
||||||
|
if s == spec.transcript_section:
|
||||||
|
out += [draft_block(pages), ""]
|
||||||
|
elif spec.section_text.get(s):
|
||||||
|
out += [spec.section_text[s].rstrip("\n"), ""]
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _set_transcribed_by(text: str, value: str) -> str:
|
||||||
|
line = f"transcribed_by: {value}"
|
||||||
|
if not text.startswith("---\n"):
|
||||||
|
return f"---\n{line}\n---\n\n{text}"
|
||||||
|
end = text.find("\n---", 4)
|
||||||
|
if end == -1:
|
||||||
|
return f"---\n{line}\n---\n\n{text}"
|
||||||
|
head, rest = text[:end], text[end:]
|
||||||
|
if re.search(r"^transcribed_by:.*$", head, flags=re.MULTILINE):
|
||||||
|
head = re.sub(r"^transcribed_by:.*$", line, head, count=1, flags=re.MULTILINE)
|
||||||
|
else:
|
||||||
|
head = f"{head}\n{line}"
|
||||||
|
return head + rest
|
||||||
|
|
||||||
|
|
||||||
|
_HEADING = re.compile(r"^(#{1,6})\s")
|
||||||
|
_FENCE = re.compile(r"^(`{3,}|~{3,})")
|
||||||
|
|
||||||
|
|
||||||
|
def _heading_level(line: str) -> int:
|
||||||
|
m = _HEADING.match(line)
|
||||||
|
return len(m.group(1)) if m else 0
|
||||||
|
|
||||||
|
|
||||||
|
def _outside_fence_lines(lines: list[str]):
|
||||||
|
"""Yield (index, line) for lines that are not inside a fenced code block."""
|
||||||
|
fence = None
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
m = _FENCE.match(line)
|
||||||
|
if fence is None:
|
||||||
|
if m:
|
||||||
|
fence = m.group(1)
|
||||||
|
continue
|
||||||
|
yield i, line
|
||||||
|
elif (m and m.group(1)[0] == fence[0] and len(m.group(1)) >= len(fence)
|
||||||
|
and line.strip() == m.group(1)):
|
||||||
|
fence = None
|
||||||
|
|
||||||
|
|
||||||
|
def merge_companion(text: str, pages: list[str], transcribed_by: str,
|
||||||
|
section: str = "Notes") -> str:
|
||||||
|
"""Put a fresh draft block into an existing companion, keeping all else."""
|
||||||
|
text = _set_transcribed_by(text, transcribed_by)
|
||||||
|
lines = text.split("\n")
|
||||||
|
block = draft_block(pages).split("\n")
|
||||||
|
visible = list(_outside_fence_lines(lines))
|
||||||
|
|
||||||
|
# 1. Replace an existing draft block.
|
||||||
|
start = next((i for i, l in visible if l.rstrip() == DRAFT_HEADING), None)
|
||||||
|
if start is not None:
|
||||||
|
end = None
|
||||||
|
for i, l in visible:
|
||||||
|
if i <= start:
|
||||||
|
continue
|
||||||
|
if l.strip() == END_MARKER:
|
||||||
|
end = i + 1
|
||||||
|
break
|
||||||
|
if 0 < _heading_level(l) <= 3:
|
||||||
|
end = i
|
||||||
|
# Keep the blank line that separated the block from the heading.
|
||||||
|
while end - 1 > start and lines[end - 1].strip() == "":
|
||||||
|
end -= 1
|
||||||
|
break
|
||||||
|
if end is None:
|
||||||
|
end = len(lines)
|
||||||
|
while end - 1 > start and lines[end - 1].strip() == "":
|
||||||
|
end -= 1
|
||||||
|
return "\n".join(lines[:start] + block + lines[end:])
|
||||||
|
|
||||||
|
# 2. Replace the pending line, wherever it is.
|
||||||
|
pending = next((i for i, l in visible if l.strip() == PENDING_LINE), None)
|
||||||
|
if pending is not None:
|
||||||
|
return "\n".join(lines[:pending] + block + lines[pending + 1:])
|
||||||
|
|
||||||
|
# 3. Append to the end of the transcript section.
|
||||||
|
disc = next((i for i, l in visible if l.rstrip() == f"## {section}"), None)
|
||||||
|
if disc is not None:
|
||||||
|
nxt = next((i for i, l in visible if i > disc and 0 < _heading_level(l) <= 2),
|
||||||
|
len(lines))
|
||||||
|
ins = nxt
|
||||||
|
while ins - 1 > disc and lines[ins - 1].strip() == "":
|
||||||
|
ins -= 1
|
||||||
|
rest = lines[nxt:]
|
||||||
|
new = lines[:ins] + [""] + block + [""]
|
||||||
|
return "\n".join(new + rest if rest else new)
|
||||||
|
|
||||||
|
# 4. No transcript section: add the block at the end.
|
||||||
|
return f"{text.rstrip(chr(10))}\n\n{draft_block(pages)}\n"
|
||||||
|
|
||||||
|
|
||||||
|
def write_atomic(path: Path, text: str) -> None:
|
||||||
|
"""Write via a temp file in the same folder, so a crash leaves no partial file."""
|
||||||
|
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.stem}.", suffix=".tmp")
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||||
|
f.write(text)
|
||||||
|
os.replace(tmp, path)
|
||||||
|
except BaseException:
|
||||||
|
try:
|
||||||
|
os.unlink(tmp)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def write_companion(pdf: Path, root: Path, pages: list[str], transcribed_by: str,
|
||||||
|
spec: CompanionSpec = BASIC,
|
||||||
|
titles: dict[str, str] | None = None) -> tuple[Path, bool, list[str]]:
|
||||||
|
"""Create or update `<stem>.md`. Returns (path, created, warnings)."""
|
||||||
|
md = pdf.with_suffix(".md")
|
||||||
|
if md.exists():
|
||||||
|
text = md.read_text(encoding="utf-8")
|
||||||
|
write_atomic(md, merge_companion(text, pages, transcribed_by,
|
||||||
|
spec.transcript_section))
|
||||||
|
return md, False, []
|
||||||
|
meta = meta_from_path(pdf, root, spec)
|
||||||
|
write_atomic(md, new_companion(meta, pages, transcribed_by, spec, titles))
|
||||||
|
return md, True, meta.warnings
|
||||||
126
src/pdf_tools/extract_pdf.py
Normal file
126
src/pdf_tools/extract_pdf.py
Normal file
|
|
@ -0,0 +1,126 @@
|
||||||
|
"""Tool 2: extract annotations + underlying highlighted text to Markdown.
|
||||||
|
|
||||||
|
For each annotation:
|
||||||
|
- The note/popup text you wrote
|
||||||
|
- The text under the highlight rectangle (for Highlight / Underline /
|
||||||
|
Squiggly / StrikeOut)
|
||||||
|
- Page number
|
||||||
|
|
||||||
|
Output is grouped by page. Each entry uses the highlighted passage as a
|
||||||
|
blockquote, followed by the note as ordinary prose. YAML frontmatter at
|
||||||
|
the top carries source filename and counts so downstream pipelines can
|
||||||
|
parse it cleanly.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime as _dt
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import click
|
||||||
|
|
||||||
|
# pdfannots gives us highlight text + note text together, which is what
|
||||||
|
# we want. Its API has shifted across versions; we use the documented
|
||||||
|
# process_file entrypoint.
|
||||||
|
import pdfannots
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Note:
|
||||||
|
page: int # 1-indexed
|
||||||
|
kind: str # 'Highlight', 'Text', etc.
|
||||||
|
highlighted: str # underlying text (may be empty for sticky notes)
|
||||||
|
note: str # the comment the user typed (may be empty for bare highlights)
|
||||||
|
author: str | None
|
||||||
|
created: str | None
|
||||||
|
|
||||||
|
|
||||||
|
def extract(pdf_path: Path) -> list[Note]:
|
||||||
|
notes: list[Note] = []
|
||||||
|
with pdf_path.open("rb") as fh:
|
||||||
|
doc = pdfannots.process_file(fh, emit_progress_to=None)
|
||||||
|
for annot in doc.iter_annots():
|
||||||
|
page_num = annot.pos.page.pageno + 1 if annot.pos else 0
|
||||||
|
notes.append(Note(
|
||||||
|
page=page_num,
|
||||||
|
kind=annot.subtype.name if hasattr(annot.subtype, "name") else str(annot.subtype),
|
||||||
|
highlighted=(annot.gettext() or "").strip(),
|
||||||
|
note=(annot.contents or "").strip(),
|
||||||
|
author=getattr(annot, "author", None),
|
||||||
|
created=str(annot.created) if getattr(annot, "created", None) else None,
|
||||||
|
))
|
||||||
|
notes.sort(key=lambda n: (n.page, n.kind))
|
||||||
|
return notes
|
||||||
|
|
||||||
|
|
||||||
|
def to_markdown(pdf_path: Path, notes: list[Note]) -> str:
|
||||||
|
today = _dt.date.today().isoformat()
|
||||||
|
lines: list[str] = []
|
||||||
|
lines.append("---")
|
||||||
|
lines.append(f'source: "{pdf_path.name}"')
|
||||||
|
lines.append(f"extracted: {today}")
|
||||||
|
lines.append(f"annotations: {len(notes)}")
|
||||||
|
lines.append("---")
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"# Notes from {pdf_path.stem}")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
if not notes:
|
||||||
|
lines.append("_No annotations found._")
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
current_page = None
|
||||||
|
for n in notes:
|
||||||
|
if n.page != current_page:
|
||||||
|
lines.append(f"## Page {n.page}")
|
||||||
|
lines.append("")
|
||||||
|
current_page = n.page
|
||||||
|
|
||||||
|
if n.highlighted:
|
||||||
|
# Blockquote the highlighted passage. Collapse internal newlines
|
||||||
|
# so the quote reads as one passage.
|
||||||
|
passage = " ".join(n.highlighted.split())
|
||||||
|
lines.append(f"> {passage}")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
if n.note:
|
||||||
|
# Sticky notes have no highlighted passage; tag them so the
|
||||||
|
# reader knows the note is freestanding, not a comment on
|
||||||
|
# the preceding quote.
|
||||||
|
if not n.highlighted:
|
||||||
|
lines.append(f"**Note ({n.kind}):** {n.note}")
|
||||||
|
else:
|
||||||
|
lines.append(n.note)
|
||||||
|
lines.append("")
|
||||||
|
elif not n.highlighted:
|
||||||
|
lines.append(f"_({n.kind} with no text)_")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||||
|
@click.option(
|
||||||
|
"-o", "--output",
|
||||||
|
type=click.Path(dir_okay=False, path_type=Path),
|
||||||
|
help="Output .md path. Default: <input>.md next to the PDF.",
|
||||||
|
)
|
||||||
|
@click.option("--stdout", is_flag=True, help="Write markdown to stdout instead of a file.")
|
||||||
|
def main(pdf: Path, output: Path | None, stdout: bool) -> None:
|
||||||
|
"""Extract PDF annotations to Markdown."""
|
||||||
|
notes = extract(pdf)
|
||||||
|
md = to_markdown(pdf, notes)
|
||||||
|
|
||||||
|
if stdout:
|
||||||
|
sys.stdout.write(md)
|
||||||
|
return
|
||||||
|
|
||||||
|
out_path = output or pdf.with_suffix(".md")
|
||||||
|
out_path.write_text(md, encoding="utf-8")
|
||||||
|
print(f"wrote {len(notes)} annotations -> {out_path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
439
src/pdf_tools/extract_scan.py
Normal file
439
src/pdf_tools/extract_scan.py
Normal file
|
|
@ -0,0 +1,439 @@
|
||||||
|
"""Tool 3: extract notes from scanned notebook pages to Markdown.
|
||||||
|
|
||||||
|
Accepts either:
|
||||||
|
- One or more image files (jpg/png/heic/tiff)
|
||||||
|
- A PDF (each page is rasterized and processed)
|
||||||
|
|
||||||
|
Two backends:
|
||||||
|
- ollama (default): a vision model on a machine you own, through the Ollama HTTP
|
||||||
|
API (see ollama.py). Hosts are tried in the order given in the config
|
||||||
|
file (see scan_config.py). There is no fallback to any cloud service.
|
||||||
|
- claude: the Anthropic API, used only when asked for with --backend
|
||||||
|
claude or the config. Set ANTHROPIC_API_KEY. Model defaults to Claude
|
||||||
|
Opus; override with --model.
|
||||||
|
|
||||||
|
Scans under a confidential folder (declared in the config file; none are
|
||||||
|
built in) default to the ollama backend, refuse the claude backend unless
|
||||||
|
--allow-cloud is given, and are written to a companion Markdown file shaped
|
||||||
|
by that folder's settings (see companion.py).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import dataclasses
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Callable
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
import click
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from . import companion
|
||||||
|
from .ollama import (REPEAT_PLACEHOLDER, REPEAT_RETRIES, NoHostAvailable,
|
||||||
|
OllamaError, PageOutput, RepetitionError, build_prompt,
|
||||||
|
loaded_model, provenance, select_host, transcribe_png)
|
||||||
|
from .scan_config import load_config
|
||||||
|
from .scan_stats import StatsLog
|
||||||
|
|
||||||
|
# Anthropic is imported lazily inside _transcribe_image so the rest of
|
||||||
|
# the module is usable without the API key set (e.g. for --help), and so
|
||||||
|
# the ollama backend never loads it.
|
||||||
|
|
||||||
|
# Reasonable defaults. The model is one of the few things you might
|
||||||
|
# want to tweak per-run as new versions ship. Opus is the default
|
||||||
|
# because handwriting transcription is the one place accuracy
|
||||||
|
# clearly justifies the cost.
|
||||||
|
DEFAULT_MODEL = "claude-opus-5-5"
|
||||||
|
SUPPORTED_IMAGE_EXT = {".jpg", ".jpeg", ".png", ".webp", ".gif", ".tif", ".tiff", ".heic"}
|
||||||
|
|
||||||
|
# Exit codes beyond click's usual 1 and 2.
|
||||||
|
EXIT_NO_HOST = 3
|
||||||
|
EXIT_TRANSCRIBE_FAILED = 4
|
||||||
|
|
||||||
|
TRANSCRIBE_PROMPT = """\
|
||||||
|
You are transcribing a scanned page from a handwritten research notebook.
|
||||||
|
|
||||||
|
Output requirements:
|
||||||
|
- Reproduce the page content as clean Markdown.
|
||||||
|
- Preserve structure: headings, bullet points, indentation, numbered lists.
|
||||||
|
- Render equations and chemical formulas in LaTeX. Inline math uses $...$, display math uses $$...$$.
|
||||||
|
- Preserve units, subscripts, and Greek letters faithfully.
|
||||||
|
- For figures, diagrams, or sketches, insert a single line like:
|
||||||
|
`[FIGURE: brief description of what is drawn]`
|
||||||
|
- For text you cannot read with confidence, use `[?]` for a single illegible word
|
||||||
|
or `[??: best guess]` if you have a tentative reading.
|
||||||
|
- Do NOT add commentary, summaries, or explanations. Transcribe only.
|
||||||
|
- Do NOT wrap your answer in code fences. Output raw Markdown directly.
|
||||||
|
|
||||||
|
If the page is blank or contains no notebook content, output exactly:
|
||||||
|
[BLANK PAGE]
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PageResult:
|
||||||
|
index: int # 1-indexed page/image number
|
||||||
|
source: str # filename or "page N of source.pdf"
|
||||||
|
markdown: str
|
||||||
|
|
||||||
|
|
||||||
|
def _image_to_jpeg_bytes(img: Image.Image, max_side: int = 2000, quality: int = 85) -> bytes:
|
||||||
|
"""Downscale + re-encode to keep API payloads reasonable."""
|
||||||
|
img = img.convert("RGB")
|
||||||
|
w, h = img.size
|
||||||
|
longest = max(w, h)
|
||||||
|
if longest > max_side:
|
||||||
|
scale = max_side / longest
|
||||||
|
img = img.resize((int(w * scale), int(h * scale)), Image.LANCZOS)
|
||||||
|
buf = io.BytesIO()
|
||||||
|
img.save(buf, format="JPEG", quality=quality, optimize=True)
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def _image_to_png_bytes(img: Image.Image) -> bytes:
|
||||||
|
"""Lossless, full resolution, in memory only (nothing is written to disk)."""
|
||||||
|
buf = io.BytesIO()
|
||||||
|
img.convert("RGB").save(buf, format="PNG")
|
||||||
|
return buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def _transcribe_image(jpeg_bytes: bytes, model: str) -> str:
|
||||||
|
"""One Claude call. The ollama backend lives in ollama.transcribe_png."""
|
||||||
|
import anthropic # imported here so --help works without the SDK
|
||||||
|
|
||||||
|
client = anthropic.Anthropic() # picks up ANTHROPIC_API_KEY
|
||||||
|
b64 = base64.standard_b64encode(jpeg_bytes).decode("ascii")
|
||||||
|
resp = client.messages.create(
|
||||||
|
model=model,
|
||||||
|
max_tokens=4096,
|
||||||
|
messages=[{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "image",
|
||||||
|
"source": {
|
||||||
|
"type": "base64",
|
||||||
|
"media_type": "image/jpeg",
|
||||||
|
"data": b64,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{"type": "text", "text": TRANSCRIBE_PROMPT},
|
||||||
|
],
|
||||||
|
}],
|
||||||
|
)
|
||||||
|
# Concatenate text blocks (usually just one).
|
||||||
|
return "".join(b.text for b in resp.content if getattr(b, "type", None) == "text").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _iter_pages(paths: list[Path], dpi: int = 200) -> list[tuple[str, Callable[[], Image.Image]]]:
|
||||||
|
"""Return (label, loader) for every page across all inputs.
|
||||||
|
|
||||||
|
Pages are rendered only when their loader is called, so a long notebook
|
||||||
|
never holds more than one page image in memory.
|
||||||
|
"""
|
||||||
|
out: list[tuple[str, Callable[[], Image.Image]]] = []
|
||||||
|
for p in paths:
|
||||||
|
ext = p.suffix.lower()
|
||||||
|
if ext == ".pdf":
|
||||||
|
import pypdfium2 as pdfium
|
||||||
|
|
||||||
|
doc = pdfium.PdfDocument(p)
|
||||||
|
for i in range(len(doc)):
|
||||||
|
# 200 DPI is plenty for handwriting OCR.
|
||||||
|
out.append((f"{p.name} p.{i + 1}",
|
||||||
|
lambda doc=doc, i=i: doc[i].render(scale=dpi / 72).to_pil()))
|
||||||
|
elif ext in SUPPORTED_IMAGE_EXT:
|
||||||
|
out.append((p.name, lambda p=p: Image.open(p)))
|
||||||
|
else:
|
||||||
|
print(f"warning: skipping unsupported file: {p}", file=sys.stderr)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _checkpoint_path(first_input: Path) -> Path:
|
||||||
|
"""Progress file kept next to the scan while a run is incomplete."""
|
||||||
|
return first_input.parent / f".{first_input.stem}.scan-partial.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _checkpoint_key(paths: list[Path], transcribed_by: str, prompt: str, dpi: int) -> dict:
|
||||||
|
return {
|
||||||
|
"inputs": [[p.name, p.stat().st_size] for p in paths],
|
||||||
|
"transcribed_by": transcribed_by,
|
||||||
|
"prompt_sha1": hashlib.sha1(prompt.encode()).hexdigest(),
|
||||||
|
"dpi": dpi,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_checkpoint(path: Path, key: dict) -> list[dict]:
|
||||||
|
"""Pages saved by an earlier, interrupted run with the same settings."""
|
||||||
|
try:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return []
|
||||||
|
if data.get("key") != key:
|
||||||
|
print("note: ignoring saved progress from a run with different "
|
||||||
|
"inputs, model, prompt, or dpi", file=sys.stderr)
|
||||||
|
return []
|
||||||
|
return list(data.get("pages", []))
|
||||||
|
|
||||||
|
|
||||||
|
def _save_checkpoint(path: Path, key: dict, done: list[dict]) -> None:
|
||||||
|
companion.write_atomic(path, json.dumps({"key": key, "pages": done}))
|
||||||
|
|
||||||
|
|
||||||
|
def to_markdown(results: list[PageResult], title: str) -> str:
|
||||||
|
import datetime as _dt
|
||||||
|
lines = [
|
||||||
|
"---",
|
||||||
|
f'title: "{title}"',
|
||||||
|
f"extracted: {_dt.date.today().isoformat()}",
|
||||||
|
f"pages: {len(results)}",
|
||||||
|
"---",
|
||||||
|
"",
|
||||||
|
f"# {title}",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
for r in results:
|
||||||
|
lines.append(f"## {r.source}")
|
||||||
|
lines.append("")
|
||||||
|
lines.append(r.markdown if r.markdown else "_(empty)_")
|
||||||
|
lines.append("")
|
||||||
|
return "\n".join(lines) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
def to_plain_markdown(pages: list[str], title: str, transcribed_by: str) -> str:
|
||||||
|
"""Output for ollama runs outside companion mode: <!-- page N --> markers."""
|
||||||
|
import datetime as _dt
|
||||||
|
return "\n".join([
|
||||||
|
"---",
|
||||||
|
f'title: "{title}"',
|
||||||
|
f"extracted: {_dt.date.today().isoformat()}",
|
||||||
|
f"pages: {len(pages)}",
|
||||||
|
f"transcribed_by: {transcribed_by}",
|
||||||
|
"---",
|
||||||
|
"",
|
||||||
|
f"# {title}",
|
||||||
|
"",
|
||||||
|
companion.render_transcript(pages),
|
||||||
|
"",
|
||||||
|
])
|
||||||
|
|
||||||
|
|
||||||
|
def _fail(msg: str, code: int = 2) -> None:
|
||||||
|
print(f"error: {msg}", file=sys.stderr)
|
||||||
|
sys.exit(code)
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.argument(
|
||||||
|
"inputs", nargs=-1, required=True,
|
||||||
|
type=click.Path(exists=True, dir_okay=False, path_type=Path),
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"-o", "--output",
|
||||||
|
type=click.Path(dir_okay=False, path_type=Path),
|
||||||
|
help="Output .md path. Default: <first input stem>.md",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--backend", type=click.Choice(["claude", "ollama"]), default=None,
|
||||||
|
help="Transcription backend. Default: ollama for scans in confidential "
|
||||||
|
"folders, else the "
|
||||||
|
"config's backend (ollama unless set).",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--host", default=None,
|
||||||
|
help="Ollama only: use just this host from the config (e.g. remote, local) "
|
||||||
|
"and fail if it is unavailable.",
|
||||||
|
)
|
||||||
|
@click.option("--model", default=None,
|
||||||
|
help=f"Model name. Default: {DEFAULT_MODEL} for claude, the "
|
||||||
|
"config's per-host model for ollama.")
|
||||||
|
@click.option("--companion/--no-companion", "companion_flag", default=None,
|
||||||
|
help="Write the companion <stem>.md next to the scan. Default: on "
|
||||||
|
"for scans in confidential folders.")
|
||||||
|
@click.option("--allow-cloud", is_flag=True,
|
||||||
|
help="Permit --backend claude for scans in confidential folders.")
|
||||||
|
@click.option("--config", "config_path",
|
||||||
|
type=click.Path(dir_okay=False, path_type=Path), default=None,
|
||||||
|
help="Config file. Default: ~/.config/pdf-tools/scan-notes.toml")
|
||||||
|
@click.option("--dpi", type=int, default=None, help="PDF render resolution (config default 200).")
|
||||||
|
@click.option("--timeout", type=float, default=None,
|
||||||
|
help="Ollama per-page timeout in seconds (config default 300).")
|
||||||
|
@click.option(
|
||||||
|
"--title", default=None,
|
||||||
|
help="Title for the markdown document. Default: first input stem.",
|
||||||
|
)
|
||||||
|
@click.option("--stdout", is_flag=True, help="Write markdown to stdout.")
|
||||||
|
def main(inputs: tuple[Path, ...], output: Path | None, backend: str | None,
|
||||||
|
host: str | None, model: str | None, companion_flag: bool | None,
|
||||||
|
allow_cloud: bool, config_path: Path | None, dpi: int | None,
|
||||||
|
timeout: float | None, title: str | None, stdout: bool) -> None:
|
||||||
|
"""Transcribe scanned notebook pages to Markdown."""
|
||||||
|
cfg = load_config(config_path)
|
||||||
|
paths = list(inputs)
|
||||||
|
|
||||||
|
folders = [cfg.confidential_folder(p) for p in paths]
|
||||||
|
confidential = any(folders)
|
||||||
|
if confidential and not all(folders):
|
||||||
|
_fail("do not mix scans from confidential folders with other files in one run")
|
||||||
|
folder = folders[0]
|
||||||
|
companion_mode = confidential if companion_flag is None else companion_flag
|
||||||
|
if companion_mode and not confidential:
|
||||||
|
roots = ", ".join(str(c.root) for c in cfg.confidential) or "none configured"
|
||||||
|
_fail(f"--companion needs a scan in a confidential folder ({roots})")
|
||||||
|
if companion_mode:
|
||||||
|
if len(paths) != 1 or paths[0].suffix.lower() != ".pdf":
|
||||||
|
_fail("companion mode takes exactly one PDF")
|
||||||
|
if output or stdout:
|
||||||
|
_fail("companion mode writes <stem>.md next to the scan; "
|
||||||
|
"-o and --stdout need --no-companion")
|
||||||
|
|
||||||
|
backend = backend or ("ollama" if confidential else cfg.backend)
|
||||||
|
if backend not in ("claude", "ollama"):
|
||||||
|
_fail(f"unknown backend '{backend}' in config")
|
||||||
|
if backend == "claude" and confidential and not allow_cloud:
|
||||||
|
_fail(f"scans under the confidential folder {folder.root} are not "
|
||||||
|
"sent to Claude unless "
|
||||||
|
"--allow-cloud is given; use --backend ollama")
|
||||||
|
if host and backend != "ollama":
|
||||||
|
_fail("--host applies only to --backend ollama")
|
||||||
|
|
||||||
|
if backend == "claude":
|
||||||
|
if not os.environ.get("ANTHROPIC_API_KEY"):
|
||||||
|
_fail("ANTHROPIC_API_KEY not set")
|
||||||
|
claude_model = model or DEFAULT_MODEL
|
||||||
|
transcribed_by = f"claude/{claude_model}"
|
||||||
|
else:
|
||||||
|
hosts = cfg.hosts
|
||||||
|
if model:
|
||||||
|
hosts = [dataclasses.replace(h, model=model) for h in hosts]
|
||||||
|
try:
|
||||||
|
chosen = select_host(hosts, cfg.probe_timeout_s, forced=host)
|
||||||
|
except NoHostAvailable as e:
|
||||||
|
_fail(f"{e}. Not falling back to a cloud service.", EXIT_NO_HOST)
|
||||||
|
print(f"using Ollama host {chosen.name} "
|
||||||
|
f"({chosen.url}, {chosen.model})", file=sys.stderr)
|
||||||
|
transcribed_by = provenance(chosen)
|
||||||
|
# Global names and prompt_extra text, plus those of every folder
|
||||||
|
# the inputs come from.
|
||||||
|
names = dict.fromkeys(n for f in folders for n in cfg.names_for(f))
|
||||||
|
extras = dict.fromkeys(e for f in folders for e in cfg.extras_for(f))
|
||||||
|
prompt = build_prompt(list(names), list(extras))
|
||||||
|
page_timeout = timeout or cfg.timeout_s
|
||||||
|
|
||||||
|
dpi = dpi or cfg.dpi
|
||||||
|
pages = _iter_pages(paths, dpi=dpi)
|
||||||
|
if not pages:
|
||||||
|
_fail("no usable pages found", 1)
|
||||||
|
|
||||||
|
# Progress is saved after every page, so an interrupted run resumes
|
||||||
|
# where it stopped. The file sits next to the scan and is removed once
|
||||||
|
# the output is written.
|
||||||
|
ckpt = _checkpoint_path(paths[0])
|
||||||
|
key = _checkpoint_key(paths, transcribed_by,
|
||||||
|
TRANSCRIBE_PROMPT if backend == "claude" else prompt, dpi)
|
||||||
|
done = _load_checkpoint(ckpt, key)[:len(pages)]
|
||||||
|
if done:
|
||||||
|
print(f"resuming after page {len(done)} of {len(pages)}", file=sys.stderr)
|
||||||
|
|
||||||
|
stats_log = StatsLog(cfg.stats_log if backend == "ollama" else None)
|
||||||
|
run_id = f"{time.strftime('%Y%m%dT%H%M%S')}-{os.getpid()}"
|
||||||
|
loaded: dict | None = None # /api/ps snapshot, taken once the model is loaded
|
||||||
|
|
||||||
|
for i, (label, load) in enumerate(pages, start=1):
|
||||||
|
if i <= len(done):
|
||||||
|
continue
|
||||||
|
if backend == "claude":
|
||||||
|
print(f"[{i}/{len(pages)}] {label} ...", file=sys.stderr)
|
||||||
|
jpeg = _image_to_jpeg_bytes(load())
|
||||||
|
md = _transcribe_image(jpeg, model=claude_model)
|
||||||
|
else:
|
||||||
|
print(f"[{i}/{len(pages)}] {label} on {chosen.name} ...",
|
||||||
|
end="", file=sys.stderr, flush=True)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
img = load()
|
||||||
|
try:
|
||||||
|
page = transcribe_png(_image_to_png_bytes(img), chosen, prompt,
|
||||||
|
page_timeout, cfg.max_output_tokens)
|
||||||
|
except RepetitionError:
|
||||||
|
print("", file=sys.stderr)
|
||||||
|
print(f"warning: page {i} kept looping after {len(REPEAT_RETRIES)} "
|
||||||
|
"retries; writing a placeholder", file=sys.stderr)
|
||||||
|
page = PageOutput(REPEAT_PLACEHOLDER, {
|
||||||
|
"prompt_tokens": 0, "output_tokens": 0, "context_tokens": 0,
|
||||||
|
"done_reason": "repeat_abort", "retries": len(REPEAT_RETRIES),
|
||||||
|
"total_s": round(time.monotonic() - t0, 2)})
|
||||||
|
except OllamaError as e:
|
||||||
|
print("", file=sys.stderr)
|
||||||
|
saved = (f"pages 1-{len(done)} are saved; rerun to resume"
|
||||||
|
if done else "nothing was saved")
|
||||||
|
_fail(f"page {i}: {e}. {saved}.", EXIT_TRANSCRIBE_FAILED)
|
||||||
|
md, st = page.text, page.stats
|
||||||
|
if loaded is None:
|
||||||
|
loaded = loaded_model(chosen, cfg.probe_timeout_s)
|
||||||
|
retried = f", {st['retries']} retries" if st.get("retries") else ""
|
||||||
|
print(f" {time.monotonic() - t0:.0f} s, {st['context_tokens']} tokens{retried}",
|
||||||
|
file=sys.stderr)
|
||||||
|
nctx = loaded.get("loaded_num_ctx") or chosen.num_ctx
|
||||||
|
if st["done_reason"] == "length":
|
||||||
|
print(f"warning: page {i} output stopped at {st['output_tokens']} "
|
||||||
|
"tokens after retries; marked as cut off", file=sys.stderr)
|
||||||
|
elif nctx and st["context_tokens"] >= 0.9 * nctx:
|
||||||
|
print(f"warning: page {i} used {st['context_tokens']} of "
|
||||||
|
f"{nctx} context tokens", file=sys.stderr)
|
||||||
|
stats_log.write({
|
||||||
|
"time": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||||
|
"run": run_id,
|
||||||
|
"host": urlparse(chosen.url).hostname,
|
||||||
|
"model": chosen.model,
|
||||||
|
"num_ctx_setting": chosen.num_ctx,
|
||||||
|
"dpi": dpi,
|
||||||
|
"label": label,
|
||||||
|
"page": i,
|
||||||
|
"image_px": list(img.size),
|
||||||
|
**st,
|
||||||
|
**loaded,
|
||||||
|
})
|
||||||
|
done.append({"label": label, "text": md})
|
||||||
|
_save_checkpoint(ckpt, key, done)
|
||||||
|
|
||||||
|
texts = [d["text"] for d in done]
|
||||||
|
results = [PageResult(index=i, source=d["label"], markdown=d["text"])
|
||||||
|
for i, d in enumerate(done, start=1)]
|
||||||
|
|
||||||
|
if companion_mode:
|
||||||
|
path, created, warnings = companion.write_companion(
|
||||||
|
paths[0], folder.root, texts, transcribed_by, folder.companion,
|
||||||
|
folder.titles)
|
||||||
|
for w in warnings:
|
||||||
|
print(f"warning: {w}", file=sys.stderr)
|
||||||
|
verb = "created" if created else "updated draft transcript in"
|
||||||
|
print(f"{verb} {path}", file=sys.stderr)
|
||||||
|
ckpt.unlink(missing_ok=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
doc_title = title or paths[0].stem
|
||||||
|
if backend == "claude":
|
||||||
|
md_out = to_markdown(results, doc_title)
|
||||||
|
else:
|
||||||
|
md_out = to_plain_markdown(texts, doc_title, transcribed_by)
|
||||||
|
|
||||||
|
if stdout:
|
||||||
|
sys.stdout.write(md_out)
|
||||||
|
ckpt.unlink(missing_ok=True)
|
||||||
|
return
|
||||||
|
|
||||||
|
out_path = output or paths[0].with_suffix(".md")
|
||||||
|
companion.write_atomic(out_path, md_out)
|
||||||
|
ckpt.unlink(missing_ok=True)
|
||||||
|
print(f"wrote {len(results)} pages -> {out_path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
319
src/pdf_tools/ollama.py
Normal file
319
src/pdf_tools/ollama.py
Normal file
|
|
@ -0,0 +1,319 @@
|
||||||
|
"""Local transcription through the Ollama HTTP API.
|
||||||
|
|
||||||
|
Nothing in this module talks to a cloud service, and it never imports the
|
||||||
|
Anthropic SDK. If no configured Ollama host is usable, select_host() raises
|
||||||
|
NoHostAvailable and the caller exits; there is deliberately no fallback to
|
||||||
|
Claude, because this backend exists for confidential material.
|
||||||
|
|
||||||
|
HTTP goes through _http_json(), which tests replace with a fake.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import socket
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from .scan_config import OllamaHost
|
||||||
|
|
||||||
|
# The instruction sent with every page. Edit here to change it.
|
||||||
|
HANDWRITING_PROMPT = """\
|
||||||
|
Transcribe the handwriting in this image exactly as written. Preserve line
|
||||||
|
breaks, indentation, and bullets. Do not correct spelling, expand
|
||||||
|
abbreviations, or complete partial words. If you cannot read a word with
|
||||||
|
confidence, write [illegible] or [?word] instead of guessing; this applies
|
||||||
|
especially to names. Output only the transcript, in plain text, with no
|
||||||
|
commentary and no emojis.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Added only when the config lists names.
|
||||||
|
NAMES_PROMPT = """\
|
||||||
|
The notes may mention the following people and organizations. If a word
|
||||||
|
plausibly matches one of these names, use that spelling; otherwise write
|
||||||
|
[illegible].
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class PageOutput:
|
||||||
|
text: str
|
||||||
|
# Counts and timings Ollama reports with each response (see page_stats).
|
||||||
|
stats: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
class OllamaError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class NoHostAvailable(OllamaError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class RepetitionError(OllamaError):
|
||||||
|
"""Ollama stopped the model because it kept repeating the same tokens."""
|
||||||
|
|
||||||
|
|
||||||
|
# At temperature 0 a page that sends the model into a loop fails the same
|
||||||
|
# way every time. These retries add a little randomness and a repeat
|
||||||
|
# penalty; they are used only after that specific failure.
|
||||||
|
REPEAT_RETRIES = (
|
||||||
|
{"temperature": 0.3, "repeat_penalty": 1.15},
|
||||||
|
{"temperature": 0.6, "repeat_penalty": 1.3},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Written in place of a page that still loops after the retries.
|
||||||
|
REPEAT_PLACEHOLDER = "[page not transcribed: the model kept repeating itself; see scan]"
|
||||||
|
# Appended to a page whose output still hits max_output_tokens after the retries.
|
||||||
|
CUT_OFF_MARKER = "[transcript cut off at the output limit; see scan]"
|
||||||
|
|
||||||
|
|
||||||
|
def build_prompt(names: list[str], extras: list[str] | None = None) -> str:
|
||||||
|
"""HANDWRITING_PROMPT, then any configured prompt_extra text, then names."""
|
||||||
|
parts = [HANDWRITING_PROMPT]
|
||||||
|
parts += [e.strip("\n") + "\n" for e in (extras or []) if e.strip()]
|
||||||
|
if names:
|
||||||
|
listed = "\n".join(f"- {n}" for n in names)
|
||||||
|
parts.append(f"{NAMES_PROMPT}{listed}\n")
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def provenance(host: OllamaHost) -> str:
|
||||||
|
"""Value for the transcribed_by: field, e.g. ollama/localhost/gemma4:e4b."""
|
||||||
|
return f"ollama/{urlparse(host.url).hostname}/{host.model}"
|
||||||
|
|
||||||
|
|
||||||
|
def _http_json(method: str, url: str, payload: dict | None, timeout: float) -> dict:
|
||||||
|
data = json.dumps(payload).encode() if payload is not None else None
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url, data=data, method=method,
|
||||||
|
headers={"Content-Type": "application/json"} if data else {},
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||||
|
return json.loads(resp.read().decode("utf-8"))
|
||||||
|
|
||||||
|
|
||||||
|
def _model_present(wanted: str, available: list[str]) -> bool:
|
||||||
|
# Ollama reports untagged models as "name:latest".
|
||||||
|
if ":" not in wanted:
|
||||||
|
wanted = f"{wanted}:latest"
|
||||||
|
return wanted in available
|
||||||
|
|
||||||
|
|
||||||
|
def check_host(host: OllamaHost, timeout: float) -> str | None:
|
||||||
|
"""Return None if the host is up and has its model, else a short reason."""
|
||||||
|
try:
|
||||||
|
tags = _http_json("GET", f"{host.url}/api/tags", None, timeout)
|
||||||
|
except (urllib.error.URLError, socket.timeout, TimeoutError, OSError, ValueError):
|
||||||
|
return "unreachable"
|
||||||
|
available = [m.get("name", "") for m in tags.get("models", [])]
|
||||||
|
if not _model_present(host.model, available):
|
||||||
|
return f"does not have {host.model}"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _describe(host: OllamaHost) -> str:
|
||||||
|
return f"{host.name} ({urlparse(host.url).hostname}, {host.model})"
|
||||||
|
|
||||||
|
|
||||||
|
def select_host(hosts: list[OllamaHost], timeout: float,
|
||||||
|
forced: str | None = None) -> OllamaHost:
|
||||||
|
"""Return the first usable host, printing one line per host skipped.
|
||||||
|
|
||||||
|
With forced set, only that host is considered and failure is an error.
|
||||||
|
"""
|
||||||
|
if forced is not None:
|
||||||
|
match = [h for h in hosts if h.name == forced]
|
||||||
|
if not match:
|
||||||
|
names = ", ".join(h.name for h in hosts) or "none"
|
||||||
|
raise NoHostAvailable(f"no Ollama host named '{forced}' in config (have: {names})")
|
||||||
|
reason = check_host(match[0], timeout)
|
||||||
|
if reason:
|
||||||
|
raise NoHostAvailable(f"Ollama host {_describe(match[0])} {reason}")
|
||||||
|
return match[0]
|
||||||
|
|
||||||
|
for i, host in enumerate(hosts):
|
||||||
|
reason = check_host(host, timeout)
|
||||||
|
if reason is None:
|
||||||
|
return host
|
||||||
|
nxt = hosts[i + 1] if i + 1 < len(hosts) else None
|
||||||
|
tail = f"; falling back to {_describe(nxt)}" if nxt else ""
|
||||||
|
print(f"Ollama host {_describe(host)} {reason}{tail}", file=sys.stderr)
|
||||||
|
raise NoHostAvailable("no Ollama host is available")
|
||||||
|
|
||||||
|
|
||||||
|
_THINK_BLOCK = re.compile(r"<(think|thinking)>.*?</\1>", re.DOTALL | re.IGNORECASE)
|
||||||
|
_THINK_CLOSE = re.compile(r"^.*?</(think|thinking)>", re.DOTALL | re.IGNORECASE)
|
||||||
|
|
||||||
|
|
||||||
|
def strip_thinking(text: str) -> str:
|
||||||
|
"""Remove reasoning text that some models emit despite think: false."""
|
||||||
|
text = _THINK_BLOCK.sub("", text)
|
||||||
|
# An unmatched closing tag means the opening tag was never sent.
|
||||||
|
text = _THINK_CLOSE.sub("", text, count=1)
|
||||||
|
return text.strip("\n").rstrip()
|
||||||
|
|
||||||
|
|
||||||
|
def page_stats(resp: dict) -> dict:
|
||||||
|
"""Token counts, stop reason, and timings from an /api/chat response.
|
||||||
|
|
||||||
|
prompt_tokens includes the image; context_tokens is what the request
|
||||||
|
occupied in the model's context window (prompt plus output).
|
||||||
|
"""
|
||||||
|
ns = 1e-9
|
||||||
|
prompt = int(resp.get("prompt_eval_count") or 0)
|
||||||
|
output = int(resp.get("eval_count") or 0)
|
||||||
|
return {
|
||||||
|
"prompt_tokens": prompt,
|
||||||
|
"output_tokens": output,
|
||||||
|
"context_tokens": prompt + output,
|
||||||
|
"done_reason": resp.get("done_reason"),
|
||||||
|
"total_s": round((resp.get("total_duration") or 0) * ns, 2),
|
||||||
|
"load_s": round((resp.get("load_duration") or 0) * ns, 2),
|
||||||
|
"prompt_s": round((resp.get("prompt_eval_duration") or 0) * ns, 2),
|
||||||
|
"output_s": round((resp.get("eval_duration") or 0) * ns, 2),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def loaded_model(host: OllamaHost, timeout: float) -> dict:
|
||||||
|
"""How the host has the model loaded, from /api/ps: total size, the part
|
||||||
|
in GPU memory, and the context length it was loaded with. Empty if the
|
||||||
|
host does not report it."""
|
||||||
|
try:
|
||||||
|
ps = _http_json("GET", f"{host.url}/api/ps", None, timeout)
|
||||||
|
except (urllib.error.URLError, socket.timeout, TimeoutError, OSError, ValueError):
|
||||||
|
return {}
|
||||||
|
wanted = host.model if ":" in host.model else f"{host.model}:latest"
|
||||||
|
for m in ps.get("models", []):
|
||||||
|
if m.get("name") == wanted or m.get("model") == wanted:
|
||||||
|
size, vram = int(m.get("size") or 0), int(m.get("size_vram") or 0)
|
||||||
|
return {
|
||||||
|
"loaded_bytes": size,
|
||||||
|
"vram_bytes": vram,
|
||||||
|
"gpu_fraction": round(vram / size, 3) if size else None,
|
||||||
|
"loaded_num_ctx": m.get("context_length"),
|
||||||
|
}
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _chat(url: str, payload: dict, timeout: float) -> dict:
|
||||||
|
try:
|
||||||
|
return _http_json("POST", url, payload, timeout)
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
body = e.read().decode("utf-8", "replace")
|
||||||
|
if e.code == 400 and "think" in body.lower() and "think" in payload:
|
||||||
|
# Some models reject the think field outright; retry without it.
|
||||||
|
payload.pop("think")
|
||||||
|
return _chat(url, payload, timeout)
|
||||||
|
if "repeat limit" in body.lower():
|
||||||
|
raise RepetitionError(f"{url}: {body[:200]}") from None
|
||||||
|
raise OllamaError(f"{url} returned HTTP {e.code}: {body[:200]}") from None
|
||||||
|
except (socket.timeout, TimeoutError):
|
||||||
|
raise OllamaError(f"no response from {url} within {timeout:.0f} s") from None
|
||||||
|
except (urllib.error.URLError, OSError) as e:
|
||||||
|
raise OllamaError(f"cannot reach {url}: {e}") from None
|
||||||
|
|
||||||
|
|
||||||
|
def _page_request(png_bytes: bytes, host: OllamaHost, prompt: str,
|
||||||
|
max_output: int | None) -> tuple[dict, dict]:
|
||||||
|
"""First-attempt options and the /api/chat payload for one page image."""
|
||||||
|
options: dict = {"temperature": 0}
|
||||||
|
if host.num_ctx:
|
||||||
|
options["num_ctx"] = host.num_ctx
|
||||||
|
if max_output:
|
||||||
|
options["num_predict"] = max_output
|
||||||
|
payload = {
|
||||||
|
"model": host.model,
|
||||||
|
"messages": [{
|
||||||
|
"role": "user",
|
||||||
|
"content": prompt,
|
||||||
|
"images": [base64.b64encode(png_bytes).decode("ascii")],
|
||||||
|
}],
|
||||||
|
"stream": False,
|
||||||
|
"think": False,
|
||||||
|
"options": options,
|
||||||
|
}
|
||||||
|
return options, payload
|
||||||
|
|
||||||
|
|
||||||
|
def transcribe_png(png_bytes: bytes, host: OllamaHost, prompt: str,
|
||||||
|
timeout: float, max_output: int | None = None) -> PageOutput:
|
||||||
|
"""One page, one request. Returns the transcript text and its stats.
|
||||||
|
|
||||||
|
A model that loops either trips Ollama's repeat check (RepetitionError)
|
||||||
|
or runs on until max_output tokens (done_reason "length"). Either way the
|
||||||
|
page is retried with the REPEAT_RETRIES options; stats["retries"] records
|
||||||
|
how many were needed. If every retry loops, RepetitionError is raised
|
||||||
|
again, or the last output is returned with CUT_OFF_MARKER appended.
|
||||||
|
"""
|
||||||
|
options, payload = _page_request(png_bytes, host, prompt, max_output)
|
||||||
|
url = f"{host.url}/api/chat"
|
||||||
|
last = len(REPEAT_RETRIES)
|
||||||
|
reasons: list[str] = [] # why each earlier attempt was retried
|
||||||
|
for attempt, extra in enumerate(((), *REPEAT_RETRIES)):
|
||||||
|
payload["options"] = {**options, **dict(extra)}
|
||||||
|
try:
|
||||||
|
resp = _chat(url, payload, timeout)
|
||||||
|
except RepetitionError:
|
||||||
|
if attempt == last:
|
||||||
|
raise
|
||||||
|
reasons.append("repeat_limit")
|
||||||
|
continue
|
||||||
|
if resp.get("done_reason") == "length" and attempt < last:
|
||||||
|
reasons.append("output_cap")
|
||||||
|
continue
|
||||||
|
stats = page_stats(resp)
|
||||||
|
stats["retries"] = attempt
|
||||||
|
if attempt:
|
||||||
|
stats["retry_options"] = dict(extra)
|
||||||
|
stats["retry_reasons"] = reasons
|
||||||
|
text = strip_thinking((resp.get("message") or {}).get("content", ""))
|
||||||
|
if stats["done_reason"] == "length":
|
||||||
|
text = f"{text}\n{CUT_OFF_MARKER}"
|
||||||
|
return PageOutput(text, stats)
|
||||||
|
raise AssertionError("unreachable")
|
||||||
|
|
||||||
|
|
||||||
|
def chat_json(png_bytes: bytes, host: OllamaHost, prompt: str, schema: dict,
|
||||||
|
timeout: float, max_output: int | None = None) -> tuple[dict, dict]:
|
||||||
|
"""One page, one request, with the answer constrained to a JSON schema.
|
||||||
|
|
||||||
|
Returns (answer, stats). Retried with REPEAT_RETRIES when the model loops,
|
||||||
|
stops at max_output (the JSON would be incomplete), or returns something
|
||||||
|
that is not a JSON object with the schema's required keys. Raises
|
||||||
|
OllamaError if no attempt gives a usable answer.
|
||||||
|
"""
|
||||||
|
options, payload = _page_request(png_bytes, host, prompt, max_output)
|
||||||
|
payload["format"] = schema
|
||||||
|
url = f"{host.url}/api/chat"
|
||||||
|
required = schema.get("required", [])
|
||||||
|
reasons: list[str] = []
|
||||||
|
for attempt, extra in enumerate(((), *REPEAT_RETRIES)):
|
||||||
|
payload["options"] = {**options, **dict(extra)}
|
||||||
|
try:
|
||||||
|
resp = _chat(url, payload, timeout)
|
||||||
|
except RepetitionError:
|
||||||
|
reasons.append("repeat_limit")
|
||||||
|
continue
|
||||||
|
if resp.get("done_reason") == "length":
|
||||||
|
reasons.append("output_cap")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
answer = json.loads(strip_thinking((resp.get("message") or {}).get("content", "")))
|
||||||
|
except ValueError:
|
||||||
|
answer = None
|
||||||
|
if not isinstance(answer, dict) or any(k not in answer for k in required):
|
||||||
|
reasons.append("bad_json")
|
||||||
|
continue
|
||||||
|
stats = page_stats(resp)
|
||||||
|
stats["retries"] = attempt
|
||||||
|
if attempt:
|
||||||
|
stats["retry_options"] = dict(extra)
|
||||||
|
stats["retry_reasons"] = reasons
|
||||||
|
return answer, stats
|
||||||
|
raise OllamaError(f"no usable answer after {len(REPEAT_RETRIES)} retries "
|
||||||
|
f"({', '.join(reasons)})")
|
||||||
254
src/pdf_tools/scan_config.py
Normal file
254
src/pdf_tools/scan_config.py
Normal file
|
|
@ -0,0 +1,254 @@
|
||||||
|
"""Configuration for scan-notes-md.
|
||||||
|
|
||||||
|
Settings come from three layers, later ones winning:
|
||||||
|
1. DEFAULTS below
|
||||||
|
2. ~/.config/pdf-tools/scan-notes.toml (or $PDF_TOOLS_SCAN_CONFIG, or --config)
|
||||||
|
3. command-line flags (applied in extract_scan.main)
|
||||||
|
|
||||||
|
The file is optional. Only the keys you set are overridden; a [[ollama.hosts]]
|
||||||
|
list in the file replaces the default host list as a whole.
|
||||||
|
|
||||||
|
Confidential folders exist only in the config file; none are built in. Scans
|
||||||
|
under any listed root stay on Ollama hosts (Claude is refused without
|
||||||
|
--allow-cloud) and go to a companion <stem>.md shaped by the folder's
|
||||||
|
[confidential.companion] table (see companion.CompanionSpec). Each folder can
|
||||||
|
also carry its own `names` list and `prompt_extra` text for the Ollama
|
||||||
|
prompt, added to the top-level ones. The [index] table holds the settings
|
||||||
|
for scan-notes-index (see scan_index.py). See
|
||||||
|
scan-notes.example.toml for a worked example.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import copy
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tomllib
|
||||||
|
from dataclasses import dataclass, field, fields
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .companion import BASIC, CompanionSpec, PathRule
|
||||||
|
from .scan_stats import DEFAULT_STATS_LOG
|
||||||
|
|
||||||
|
CONFIG_ENV = "PDF_TOOLS_SCAN_CONFIG"
|
||||||
|
DEFAULT_CONFIG_PATH = Path.home() / ".config" / "pdf-tools" / "scan-notes.toml"
|
||||||
|
|
||||||
|
DEFAULTS: dict = {
|
||||||
|
"backend": "ollama",
|
||||||
|
"dpi": 200,
|
||||||
|
"names": [],
|
||||||
|
"prompt_extra": "",
|
||||||
|
# Per-page context and GPU statistics (no transcript text); "" disables.
|
||||||
|
"stats_log": DEFAULT_STATS_LOG,
|
||||||
|
"ollama": {
|
||||||
|
"timeout_s": 300,
|
||||||
|
"probe_timeout_s": 2.5,
|
||||||
|
# Output cap per page; a model that loops stops here instead of
|
||||||
|
# running until the timeout. 0 means no cap.
|
||||||
|
"max_output_tokens": 2048,
|
||||||
|
"hosts": [
|
||||||
|
{"name": "local", "url": "http://localhost:11434", "model": "gemma4:e4b"},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
"confidential": [],
|
||||||
|
# scan-notes-index: per-page search metadata from transcripts.
|
||||||
|
"index": {
|
||||||
|
# Added to the index prompt, on top of the transcription names.
|
||||||
|
"names": [],
|
||||||
|
"prompt_extra": "",
|
||||||
|
"kinds": ["calculation", "idea", "sketch", "talk notes", "meeting notes",
|
||||||
|
"reading notes", "to-do", "other", "blank"],
|
||||||
|
"max_output_tokens": 1536,
|
||||||
|
# Characters of the previous and next page transcripts sent as context.
|
||||||
|
"context_chars": 2000,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
_FOLDER_KEYS = {"root", "names", "prompt_extra", "titles", "companion"}
|
||||||
|
_SPEC_KEYS = {f.name for f in fields(CompanionSpec)}
|
||||||
|
_RULE_KEYS = {"match", "set", "title"}
|
||||||
|
_HOST_KEYS = {"name", "url", "model", "num_ctx"}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class OllamaHost:
|
||||||
|
name: str
|
||||||
|
url: str
|
||||||
|
model: str
|
||||||
|
num_ctx: int | None = None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ConfidentialFolder:
|
||||||
|
root: Path
|
||||||
|
companion: CompanionSpec = field(default_factory=lambda: BASIC)
|
||||||
|
# Display names for the "# Title" line of a new companion, keyed by
|
||||||
|
# folder name (or "<name>.short"). Others are turned into words.
|
||||||
|
titles: dict[str, str] = field(default_factory=dict)
|
||||||
|
# Added to the Ollama prompt only for scans in this folder.
|
||||||
|
names: list[str] = field(default_factory=list)
|
||||||
|
prompt_extra: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IndexSettings:
|
||||||
|
names: list[str] = field(default_factory=list)
|
||||||
|
prompt_extra: str = ""
|
||||||
|
kinds: list[str] = field(default_factory=list)
|
||||||
|
max_output_tokens: int = 1536
|
||||||
|
context_chars: int = 2000
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScanConfig:
|
||||||
|
backend: str
|
||||||
|
dpi: int
|
||||||
|
names: list[str]
|
||||||
|
prompt_extra: str
|
||||||
|
timeout_s: float
|
||||||
|
probe_timeout_s: float
|
||||||
|
max_output_tokens: int
|
||||||
|
hosts: list[OllamaHost]
|
||||||
|
confidential: list[ConfidentialFolder]
|
||||||
|
stats_log: Path | None = None
|
||||||
|
index: IndexSettings = field(default_factory=IndexSettings)
|
||||||
|
source: Path | None = None # file the settings were read from, if any
|
||||||
|
|
||||||
|
def confidential_folder(self, path: Path) -> ConfidentialFolder | None:
|
||||||
|
"""The deepest confidential folder containing path, if any."""
|
||||||
|
p = path.expanduser().resolve()
|
||||||
|
best = None
|
||||||
|
for c in self.confidential:
|
||||||
|
root = c.root.resolve()
|
||||||
|
if p == root or root in p.parents:
|
||||||
|
if best is None or len(root.parts) > len(best.root.resolve().parts):
|
||||||
|
best = c
|
||||||
|
return best
|
||||||
|
|
||||||
|
def names_for(self, folder: ConfidentialFolder | None) -> list[str]:
|
||||||
|
"""Global names plus the folder's own, without duplicates."""
|
||||||
|
extra = folder.names if folder else []
|
||||||
|
return list(dict.fromkeys([*self.names, *extra]))
|
||||||
|
|
||||||
|
def extras_for(self, folder: ConfidentialFolder | None) -> list[str]:
|
||||||
|
"""Top-level prompt_extra plus the folder's own, skipping empty ones."""
|
||||||
|
texts = [self.prompt_extra, folder.prompt_extra if folder else ""]
|
||||||
|
return [t for t in dict.fromkeys(texts) if t.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def _merge(base: dict, over: dict) -> dict:
|
||||||
|
out = copy.deepcopy(base)
|
||||||
|
for k, v in over.items():
|
||||||
|
if isinstance(v, dict) and isinstance(out.get(k), dict):
|
||||||
|
out[k] = _merge(out[k], v)
|
||||||
|
else:
|
||||||
|
out[k] = v
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _die(path: Path, msg: str) -> None:
|
||||||
|
print(f"error: {path}: {msg}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
|
||||||
|
def _unknown(path: Path, where: str, keys, allowed: set[str]) -> None:
|
||||||
|
extra = sorted(set(keys) - allowed)
|
||||||
|
if extra:
|
||||||
|
_die(path, f"unknown key(s) in {where}: {', '.join(extra)}")
|
||||||
|
|
||||||
|
|
||||||
|
def _spec(path: Path, raw) -> CompanionSpec:
|
||||||
|
if raw is None:
|
||||||
|
return BASIC
|
||||||
|
if not isinstance(raw, dict):
|
||||||
|
_die(path, "companion must be a table ([confidential.companion]); "
|
||||||
|
"leave it out for the basic layout")
|
||||||
|
_unknown(path, "[confidential.companion]", raw, _SPEC_KEYS)
|
||||||
|
rules = []
|
||||||
|
for r in raw.get("paths", []):
|
||||||
|
_unknown(path, "[[confidential.companion.paths]]", r, _RULE_KEYS)
|
||||||
|
if "match" not in r:
|
||||||
|
_die(path, "each [[confidential.companion.paths]] entry needs match")
|
||||||
|
rules.append(PathRule(match=str(r["match"]),
|
||||||
|
set={k: str(v) for k, v in r.get("set", {}).items()},
|
||||||
|
title=r.get("title")))
|
||||||
|
kw = {k: v for k, v in raw.items() if k != "paths"}
|
||||||
|
if "acronyms" in kw:
|
||||||
|
kw["acronyms"] = set(kw["acronyms"])
|
||||||
|
spec = CompanionSpec(**{**BASIC.__dict__, **kw, "paths": rules})
|
||||||
|
if "transcript_section" not in raw and spec.transcript_section not in spec.sections:
|
||||||
|
spec.transcript_section = spec.sections[0] if spec.sections else "Notes"
|
||||||
|
return spec
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path: Path | None = None) -> ScanConfig:
|
||||||
|
"""Read the config file (if present) on top of DEFAULTS."""
|
||||||
|
explicit = path is not None or bool(os.environ.get(CONFIG_ENV))
|
||||||
|
path = path or Path(os.environ.get(CONFIG_ENV) or DEFAULT_CONFIG_PATH)
|
||||||
|
raw = DEFAULTS
|
||||||
|
source = None
|
||||||
|
if path.exists():
|
||||||
|
try:
|
||||||
|
with path.open("rb") as f:
|
||||||
|
raw = _merge(DEFAULTS, tomllib.load(f))
|
||||||
|
except tomllib.TOMLDecodeError as e:
|
||||||
|
_die(path, f"cannot parse: {e}")
|
||||||
|
source = path
|
||||||
|
elif explicit:
|
||||||
|
print(f"error: config file not found: {path}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
_unknown(path, "the top level", raw, set(DEFAULTS))
|
||||||
|
o = raw["ollama"]
|
||||||
|
_unknown(path, "[ollama] (top-level keys must come before the first "
|
||||||
|
"[table])", o, set(DEFAULTS["ollama"]))
|
||||||
|
hosts = []
|
||||||
|
for h in o["hosts"]:
|
||||||
|
_unknown(path, "[[ollama.hosts]] (top-level keys must come before "
|
||||||
|
"the first [table])", h, _HOST_KEYS)
|
||||||
|
try:
|
||||||
|
hosts.append(OllamaHost(
|
||||||
|
name=str(h["name"]), url=str(h["url"]).rstrip("/"),
|
||||||
|
model=str(h["model"]), num_ctx=h.get("num_ctx"),
|
||||||
|
))
|
||||||
|
except KeyError as e:
|
||||||
|
_die(path, f"each [[ollama.hosts]] entry needs {e}")
|
||||||
|
|
||||||
|
confidential = []
|
||||||
|
for c in raw["confidential"]:
|
||||||
|
_unknown(path, "[[confidential]]", c, _FOLDER_KEYS)
|
||||||
|
if "root" not in c:
|
||||||
|
_die(path, "each [[confidential]] entry needs a root")
|
||||||
|
confidential.append(ConfidentialFolder(
|
||||||
|
root=Path(c["root"]).expanduser(),
|
||||||
|
companion=_spec(path, c.get("companion")),
|
||||||
|
titles={str(k): str(v) for k, v in c.get("titles", {}).items()},
|
||||||
|
names=[str(n) for n in c.get("names", [])],
|
||||||
|
prompt_extra=str(c.get("prompt_extra", "")),
|
||||||
|
))
|
||||||
|
|
||||||
|
ix = raw["index"]
|
||||||
|
_unknown(path, "[index]", ix, set(DEFAULTS["index"]))
|
||||||
|
kinds = [str(k) for k in ix["kinds"]]
|
||||||
|
if not kinds:
|
||||||
|
_die(path, "[index] kinds must not be empty")
|
||||||
|
index = IndexSettings(
|
||||||
|
names=[str(n) for n in ix["names"]],
|
||||||
|
prompt_extra=str(ix["prompt_extra"]),
|
||||||
|
kinds=kinds,
|
||||||
|
max_output_tokens=int(ix["max_output_tokens"]),
|
||||||
|
context_chars=int(ix["context_chars"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
return ScanConfig(
|
||||||
|
backend=raw["backend"],
|
||||||
|
dpi=int(raw["dpi"]),
|
||||||
|
names=[str(n) for n in raw.get("names", [])],
|
||||||
|
prompt_extra=str(raw.get("prompt_extra", "")),
|
||||||
|
timeout_s=float(o["timeout_s"]),
|
||||||
|
probe_timeout_s=float(o["probe_timeout_s"]),
|
||||||
|
max_output_tokens=int(o["max_output_tokens"]),
|
||||||
|
hosts=hosts,
|
||||||
|
confidential=confidential,
|
||||||
|
stats_log=Path(raw["stats_log"]).expanduser() if raw.get("stats_log") else None,
|
||||||
|
index=index,
|
||||||
|
source=source,
|
||||||
|
)
|
||||||
316
src/pdf_tools/scan_index.py
Normal file
316
src/pdf_tools/scan_index.py
Normal file
|
|
@ -0,0 +1,316 @@
|
||||||
|
"""scan-notes-index: per-page search metadata for a transcribed scan.
|
||||||
|
|
||||||
|
Reads a scan (PDF) and its transcript (the <stem>.md written by
|
||||||
|
scan-notes-md, plain or companion; any file with <!-- page N --> markers and
|
||||||
|
fenced page text). For each page it sends the page image, the page's
|
||||||
|
transcript, and the neighboring pages' transcripts to an Ollama host and asks
|
||||||
|
for a summary, key points, a kind of page, topics, references, the event the
|
||||||
|
page records, and the people named. Each page becomes one Markdown file,
|
||||||
|
"<stem> pNNN.md", in "<stem> pages/" next to the transcript, so Spotlight,
|
||||||
|
DEVONthink, or a later vector index finds the page itself rather than the
|
||||||
|
whole notebook.
|
||||||
|
|
||||||
|
Like the Ollama backend of scan-notes-md, this never calls a cloud service.
|
||||||
|
Existing page files are kept, so an interrupted run resumes where it stopped;
|
||||||
|
--force rewrites them. Per-page stats go to the stats log with task "index".
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
|
import datetime as dt
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
import click
|
||||||
|
|
||||||
|
from .companion import write_atomic
|
||||||
|
from .extract_scan import EXIT_NO_HOST, EXIT_TRANSCRIBE_FAILED, _image_to_png_bytes, _iter_pages
|
||||||
|
from .ollama import (NAMES_PROMPT, NoHostAvailable, OllamaError, chat_json,
|
||||||
|
loaded_model, provenance, select_host)
|
||||||
|
from .scan_config import load_config
|
||||||
|
from .scan_stats import StatsLog
|
||||||
|
|
||||||
|
INDEX_PROMPT = """\
|
||||||
|
You are indexing one page of a handwritten notebook so that it can be found
|
||||||
|
later by text search. A notebook may mix calculations, research ideas, notes
|
||||||
|
taken during talks and seminars, reading notes, to-do lists, and notes from
|
||||||
|
meetings.
|
||||||
|
|
||||||
|
You are given the page image and a draft transcript of it. The transcripts of
|
||||||
|
the previous and next pages are included only as context: describe THIS page
|
||||||
|
only.
|
||||||
|
|
||||||
|
Fill in the fields as follows.
|
||||||
|
- summary: a short paragraph (three to six sentences) on what this page is
|
||||||
|
about, in plain words the writer would use when searching. Say what the
|
||||||
|
page is (a calculation, notes from a meeting or talk, an idea), what it
|
||||||
|
covers, and what it concludes, decides, or leaves open. For meeting notes,
|
||||||
|
name the discussion items; for a talk, the speaker's main points; for a
|
||||||
|
calculation, what is computed, from what, and the result if written. If
|
||||||
|
the page names a law, model, method, or quantity, use that name. If an
|
||||||
|
equation is unmistakably a standard one, you may name it; otherwise
|
||||||
|
describe what it relates. Do not guess at a purpose that is not on the
|
||||||
|
page.
|
||||||
|
- key_points: up to eight short items, each one specific thing on the page:
|
||||||
|
a discussion item, decision, action, result, number with units, idea, or
|
||||||
|
question. Use the page's own terms. Empty for a blank page.
|
||||||
|
- kind: the one word or phrase from the list that fits best.{kind_note}
|
||||||
|
- topics: three to eight short search phrases (subjects, methods,
|
||||||
|
materials, projects, organizations). Empty for a blank page.
|
||||||
|
- references: every citation, paper, book, DOI, arXiv number, or URL written
|
||||||
|
on the page, copied as written and tidied into one line (authors, journal or
|
||||||
|
book, volume, pages, year, as far as written). Never add a title, year, or
|
||||||
|
author that is not on the page. Empty if there are none.
|
||||||
|
- event: the talk, seminar, meeting, or call the page records, with speaker,
|
||||||
|
organization, board, or committee if written or shown in a page heading.
|
||||||
|
Empty if the page does not record an event.
|
||||||
|
- event_date: a date written on the page, as YYYY-MM-DD if complete,
|
||||||
|
otherwise as written. Empty if none.
|
||||||
|
- people: names of people written on the page, spelled as written.
|
||||||
|
- continues_from_previous / continues_to_next: true if this page plainly
|
||||||
|
continues a calculation, list, or set of notes from the previous page, or
|
||||||
|
onto the next.
|
||||||
|
"""
|
||||||
|
|
||||||
|
SUMMARY_HEADING = "## Summary (machine-generated, unverified)"
|
||||||
|
_PAGE = re.compile(r"^<!-- page (\d+) -->\s*\n\s*(`{3,})text\n(.*?)\n?\2[ \t]*$",
|
||||||
|
re.MULTILINE | re.DOTALL)
|
||||||
|
_STR_FIELDS = ("summary", "event", "event_date")
|
||||||
|
_LIST_FIELDS = ("key_points", "topics", "references", "people")
|
||||||
|
|
||||||
|
|
||||||
|
def schema(kinds: list[str]) -> dict:
|
||||||
|
"""JSON schema the model's answer must follow."""
|
||||||
|
strings = {"type": "array", "items": {"type": "string"}}
|
||||||
|
props = {
|
||||||
|
"summary": {"type": "string"},
|
||||||
|
"key_points": strings,
|
||||||
|
"kind": {"type": "string", "enum": kinds},
|
||||||
|
"topics": strings,
|
||||||
|
"references": strings,
|
||||||
|
"event": {"type": "string"},
|
||||||
|
"event_date": {"type": "string"},
|
||||||
|
"people": strings,
|
||||||
|
"continues_from_previous": {"type": "boolean"},
|
||||||
|
"continues_to_next": {"type": "boolean"},
|
||||||
|
}
|
||||||
|
return {"type": "object", "properties": props, "required": list(props)}
|
||||||
|
|
||||||
|
|
||||||
|
def build_index_prompt(kinds: list[str], names: list[str], extras: list[str]) -> str:
|
||||||
|
"""INDEX_PROMPT, then any [index] prompt_extra text, then names."""
|
||||||
|
note = ('\n Use "calculation" for derivations and estimates as well.'
|
||||||
|
if "calculation" in kinds else "")
|
||||||
|
parts = [INDEX_PROMPT.format(kind_note=note)]
|
||||||
|
parts += [e.strip("\n") + "\n" for e in extras if e.strip()]
|
||||||
|
if names:
|
||||||
|
parts.append(NAMES_PROMPT + "\n".join(f"- {n}" for n in names) + "\n")
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def read_transcript(path: Path) -> tuple[dict[int, str], str | None]:
|
||||||
|
"""Page texts by page number, and the file's transcribed_by value."""
|
||||||
|
text = path.read_text(encoding="utf-8")
|
||||||
|
pages = {int(m.group(1)): m.group(3).strip("\n") for m in _PAGE.finditer(text)}
|
||||||
|
by = re.search(r"^transcribed_by:\s*(.+?)\s*$", text, re.MULTILINE)
|
||||||
|
return pages, by.group(1) if by else None
|
||||||
|
|
||||||
|
|
||||||
|
def page_prompt(base: str, pages: dict[int, str], n: int, chars: int) -> str:
|
||||||
|
def block(label: str, text: str | None, limit: int) -> str:
|
||||||
|
if text is None:
|
||||||
|
return f"{label}: (none)\n"
|
||||||
|
return f"{label}:\n<<<\n{text[:limit]}\n>>>\n"
|
||||||
|
return "\n".join([base,
|
||||||
|
block("Previous page transcript", pages.get(n - 1), chars),
|
||||||
|
block("THIS page's draft transcript", pages.get(n, ""), 4 * chars),
|
||||||
|
block("Next page transcript", pages.get(n + 1), chars)])
|
||||||
|
|
||||||
|
|
||||||
|
def clean(answer: dict, kinds: list[str]) -> dict:
|
||||||
|
"""Coerce the model's answer to the expected types."""
|
||||||
|
out = {k: str(answer.get(k) or "").strip() for k in _STR_FIELDS}
|
||||||
|
for k in _LIST_FIELDS:
|
||||||
|
v = answer.get(k) or []
|
||||||
|
out[k] = [str(x).strip() for x in (v if isinstance(v, list) else [v]) if str(x).strip()]
|
||||||
|
kind = str(answer.get("kind") or "")
|
||||||
|
out["kind"] = kind if kind in kinds else ("other" if "other" in kinds else kinds[-1])
|
||||||
|
for k in ("continues_from_previous", "continues_to_next"):
|
||||||
|
out[k] = answer.get(k) is True
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _q(s: str) -> str:
|
||||||
|
return json.dumps(s, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _qlist(items: list[str]) -> str:
|
||||||
|
return "[" + ", ".join(_q(x) for x in items) + "]"
|
||||||
|
|
||||||
|
|
||||||
|
def render_page(stem: str, scan: str, n: int, total: int, meta: dict, transcript: str,
|
||||||
|
indexed_by: str, transcribed_by: str | None, today: str) -> str:
|
||||||
|
fence = "```"
|
||||||
|
while fence in transcript:
|
||||||
|
fence += "`"
|
||||||
|
prev = n - 1 if meta["continues_from_previous"] and n > 1 else None
|
||||||
|
nxt = n + 1 if meta["continues_to_next"] and n < total else None
|
||||||
|
lines = [
|
||||||
|
"---",
|
||||||
|
f"notebook: {_q(stem)}",
|
||||||
|
f"scan: {_q(scan)}",
|
||||||
|
f"page: {n}",
|
||||||
|
f"kind: {_q(meta['kind'])}",
|
||||||
|
f"topics: {_qlist(meta['topics'])}",
|
||||||
|
f"references: {_qlist(meta['references'])}",
|
||||||
|
f"event: {_q(meta['event'])}",
|
||||||
|
f"event_date: {_q(meta['event_date'])}",
|
||||||
|
f"people: {_qlist(meta['people'])}",
|
||||||
|
f"continues_from: {prev if prev else 'null'}",
|
||||||
|
f"continues_to: {nxt if nxt else 'null'}",
|
||||||
|
f"indexed_by: {indexed_by}",
|
||||||
|
f"indexed: {today}",
|
||||||
|
]
|
||||||
|
if transcribed_by:
|
||||||
|
lines.append(f"transcribed_by: {transcribed_by}")
|
||||||
|
lines += ["---", "", f"# {stem}, page {n}", "", SUMMARY_HEADING, "",
|
||||||
|
meta["summary"] or "(no summary)", ""]
|
||||||
|
if meta["key_points"]:
|
||||||
|
lines += ["## Key points", "", *[f"- {k}" for k in meta["key_points"]], ""]
|
||||||
|
if meta["references"]:
|
||||||
|
lines += ["## References", "", *[f"- {r}" for r in meta["references"]], ""]
|
||||||
|
lines += ["## Transcript (unverified)", "", f"{fence}text", transcript, fence, ""]
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def page_path(out_dir: Path, stem: str, n: int) -> Path:
|
||||||
|
return out_dir / f"{stem} p{n:03d}.md"
|
||||||
|
|
||||||
|
|
||||||
|
def _fail(msg: str, code: int = 2) -> None:
|
||||||
|
print(f"error: {msg}", file=sys.stderr)
|
||||||
|
sys.exit(code)
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||||
|
@click.option("--transcript", "transcript_path",
|
||||||
|
type=click.Path(exists=True, dir_okay=False, path_type=Path),
|
||||||
|
help="Transcript with <!-- page N --> markers. Default: <stem>.md next to the scan.")
|
||||||
|
@click.option("-o", "--output-dir", type=click.Path(file_okay=False, path_type=Path),
|
||||||
|
help='Folder for the page files. Default: "<stem> pages" next to the transcript.')
|
||||||
|
@click.option("--host", default=None,
|
||||||
|
help="Use just this host from the config and fail if it is unavailable.")
|
||||||
|
@click.option("--model", default=None, help="Model name. Default: the config's per-host model.")
|
||||||
|
@click.option("--config", "config_path", type=click.Path(dir_okay=False, path_type=Path),
|
||||||
|
default=None, help="Config file. Default: ~/.config/pdf-tools/scan-notes.toml")
|
||||||
|
@click.option("--dpi", type=int, default=None, help="PDF render resolution (config default 200).")
|
||||||
|
@click.option("--timeout", type=float, default=None,
|
||||||
|
help="Per-page timeout in seconds (config default 300).")
|
||||||
|
@click.option("--force", is_flag=True, help="Rewrite page files that already exist.")
|
||||||
|
def main(pdf: Path, transcript_path: Path | None, output_dir: Path | None,
|
||||||
|
host: str | None, model: str | None, config_path: Path | None,
|
||||||
|
dpi: int | None, timeout: float | None, force: bool) -> None:
|
||||||
|
"""Write one searchable Markdown file of metadata per page of a transcribed scan."""
|
||||||
|
cfg = load_config(config_path)
|
||||||
|
ix = cfg.index
|
||||||
|
transcript_path = transcript_path or pdf.with_suffix(".md")
|
||||||
|
if not transcript_path.exists():
|
||||||
|
_fail(f"no transcript at {transcript_path}; run scan-notes-md first")
|
||||||
|
texts, transcribed_by = read_transcript(transcript_path)
|
||||||
|
if not texts:
|
||||||
|
_fail(f"{transcript_path} has no <!-- page N --> transcript blocks")
|
||||||
|
|
||||||
|
pages = _iter_pages([pdf], dpi=dpi or cfg.dpi)
|
||||||
|
total = len(pages)
|
||||||
|
if set(texts) != set(range(1, total + 1)):
|
||||||
|
_fail(f"{transcript_path} has {len(texts)} transcribed pages; {pdf.name} has {total}")
|
||||||
|
|
||||||
|
out_dir = output_dir or transcript_path.parent / f"{pdf.stem} pages"
|
||||||
|
todo = [n for n in range(1, total + 1)
|
||||||
|
if force or not page_path(out_dir, pdf.stem, n).exists()]
|
||||||
|
if not todo:
|
||||||
|
print(f"all {total} pages already indexed in {out_dir}", file=sys.stderr)
|
||||||
|
return
|
||||||
|
|
||||||
|
hosts = cfg.hosts
|
||||||
|
if model:
|
||||||
|
hosts = [dataclasses.replace(h, model=model) for h in hosts]
|
||||||
|
try:
|
||||||
|
chosen = select_host(hosts, cfg.probe_timeout_s, forced=host)
|
||||||
|
except NoHostAvailable as e:
|
||||||
|
_fail(f"{e}. Not falling back to a cloud service.", EXIT_NO_HOST)
|
||||||
|
print(f"using Ollama host {chosen.name} ({chosen.url}, {chosen.model})", file=sys.stderr)
|
||||||
|
if len(todo) < total:
|
||||||
|
print(f"{total - len(todo)} of {total} pages already indexed; skipping them",
|
||||||
|
file=sys.stderr)
|
||||||
|
|
||||||
|
folder = cfg.confidential_folder(pdf)
|
||||||
|
names = list(dict.fromkeys([*cfg.names_for(folder), *ix.names]))
|
||||||
|
base = build_index_prompt(ix.kinds, names, [ix.prompt_extra])
|
||||||
|
fmt = schema(ix.kinds)
|
||||||
|
indexed_by = provenance(chosen)
|
||||||
|
today = dt.date.today().isoformat()
|
||||||
|
page_timeout = timeout or cfg.timeout_s
|
||||||
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
stats_log = StatsLog(cfg.stats_log)
|
||||||
|
run_id = f"{time.strftime('%Y%m%dT%H%M%S')}-{os.getpid()}"
|
||||||
|
loaded: dict | None = None
|
||||||
|
failed: list[int] = []
|
||||||
|
t_start = time.monotonic()
|
||||||
|
for n in todo:
|
||||||
|
label, load = pages[n - 1]
|
||||||
|
print(f"[{n}/{total}] {label} on {chosen.name} ...", end="", file=sys.stderr, flush=True)
|
||||||
|
t0 = time.monotonic()
|
||||||
|
try:
|
||||||
|
answer, st = chat_json(_image_to_png_bytes(load()), chosen,
|
||||||
|
page_prompt(base, texts, n, ix.context_chars),
|
||||||
|
fmt, page_timeout, ix.max_output_tokens)
|
||||||
|
except OllamaError as e:
|
||||||
|
print(f" failed: {e}", file=sys.stderr)
|
||||||
|
failed.append(n)
|
||||||
|
continue
|
||||||
|
meta = clean(answer, ix.kinds)
|
||||||
|
write_atomic(page_path(out_dir, pdf.stem, n),
|
||||||
|
render_page(pdf.stem, pdf.name, n, total, meta, texts[n],
|
||||||
|
indexed_by, transcribed_by, today))
|
||||||
|
if loaded is None:
|
||||||
|
loaded = loaded_model(chosen, cfg.probe_timeout_s)
|
||||||
|
retried = f", {st['retries']} retries" if st.get("retries") else ""
|
||||||
|
print(f" {time.monotonic() - t0:.0f} s, {st['context_tokens']} tokens, "
|
||||||
|
f"{meta['kind']}{retried}", file=sys.stderr)
|
||||||
|
nctx = loaded.get("loaded_num_ctx") or chosen.num_ctx
|
||||||
|
if nctx and st["context_tokens"] >= 0.9 * nctx:
|
||||||
|
print(f"warning: page {n} used {st['context_tokens']} of {nctx} context tokens",
|
||||||
|
file=sys.stderr)
|
||||||
|
stats_log.write({
|
||||||
|
"time": time.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||||
|
"task": "index",
|
||||||
|
"run": run_id,
|
||||||
|
"host": urlparse(chosen.url).hostname,
|
||||||
|
"model": chosen.model,
|
||||||
|
"num_ctx_setting": chosen.num_ctx,
|
||||||
|
"label": label,
|
||||||
|
"page": n,
|
||||||
|
"kind": meta["kind"],
|
||||||
|
"n_references": len(meta["references"]),
|
||||||
|
**st,
|
||||||
|
**loaded,
|
||||||
|
})
|
||||||
|
|
||||||
|
mins = (time.monotonic() - t_start) / 60
|
||||||
|
done = len(todo) - len(failed)
|
||||||
|
print(f"indexed {done} page(s) in {mins:.1f} min -> {out_dir}", file=sys.stderr)
|
||||||
|
if failed:
|
||||||
|
_fail(f"{len(failed)} page(s) failed ({', '.join(map(str, failed))}); "
|
||||||
|
"rerun to retry them", EXIT_TRANSCRIBE_FAILED)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
146
src/pdf_tools/scan_stats.py
Normal file
146
src/pdf_tools/scan_stats.py
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
"""Per-page context and GPU statistics for scan-notes-md Ollama runs.
|
||||||
|
|
||||||
|
Each transcribed page appends one JSON line to the stats log (config key
|
||||||
|
`stats_log`; default ~/.local/state/pdf-tools/scan-stats.jsonl). A line
|
||||||
|
holds the page label, image size, token counts, stop reason, timings, and
|
||||||
|
how the model was loaded (total size, share in GPU memory, context length).
|
||||||
|
It never holds transcript text or images.
|
||||||
|
|
||||||
|
scan-notes-index writes the same kind of line with "task": "index".
|
||||||
|
`scan-notes-stats` summarizes the log, one block per task: token percentiles, how many pages
|
||||||
|
would exceed a given context size, truncated pages, GPU fit, and speed.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import sys
|
||||||
|
from collections import defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import click
|
||||||
|
|
||||||
|
DEFAULT_STATS_LOG = "~/.local/state/pdf-tools/scan-stats.jsonl"
|
||||||
|
THRESHOLDS = (4096, 8192, 16384, 32768)
|
||||||
|
|
||||||
|
|
||||||
|
class StatsLog:
|
||||||
|
"""Append-only JSON-lines log; a no-op when path is None."""
|
||||||
|
|
||||||
|
def __init__(self, path: Path | None):
|
||||||
|
self.path = path
|
||||||
|
if path is not None:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
def write(self, record: dict) -> None:
|
||||||
|
if self.path is None:
|
||||||
|
return
|
||||||
|
with self.path.open("a", encoding="utf-8") as f:
|
||||||
|
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def read_log(path: Path) -> list[dict]:
|
||||||
|
out = []
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
try:
|
||||||
|
out.append(json.loads(line))
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _pct(values: list[float], q: float) -> float:
|
||||||
|
s = sorted(values)
|
||||||
|
if not s:
|
||||||
|
return 0
|
||||||
|
k = max(0, min(len(s) - 1, math.ceil(q * len(s)) - 1))
|
||||||
|
return s[k]
|
||||||
|
|
||||||
|
|
||||||
|
def summarize(records: list[dict], thresholds=THRESHOLDS, top: int = 5) -> str:
|
||||||
|
"""Plain-text report, one block per (task, host, model, loaded context).
|
||||||
|
|
||||||
|
The task is "transcribe" (scan-notes-md) or "index" (scan-notes-index);
|
||||||
|
records written before the field existed are transcriptions.
|
||||||
|
"""
|
||||||
|
groups: dict[tuple, list[dict]] = defaultdict(list)
|
||||||
|
for r in records:
|
||||||
|
groups[(r.get("task", "transcribe"), r.get("host"), r.get("model"),
|
||||||
|
r.get("loaded_num_ctx"))].append(r)
|
||||||
|
if not groups:
|
||||||
|
return "no pages logged yet"
|
||||||
|
|
||||||
|
out = []
|
||||||
|
for (task, host, model, nctx), rs in sorted(groups.items(), key=lambda kv: str(kv[0])):
|
||||||
|
n = len(rs)
|
||||||
|
out.append(f"{task}: {model} on {host}, loaded context {nctx or 'unknown'}: {n} pages, "
|
||||||
|
f"{len({r.get('run') for r in rs})} run(s)")
|
||||||
|
for key, name in [("prompt_tokens", "prompt (incl. image)"),
|
||||||
|
("output_tokens", "output"),
|
||||||
|
("context_tokens", "context used")]:
|
||||||
|
v = [r.get(key, 0) for r in rs]
|
||||||
|
out.append(f" {name:21s} mean {sum(v) / n:7.0f} p50 {_pct(v, .5):6.0f}"
|
||||||
|
f" p90 {_pct(v, .9):6.0f} p99 {_pct(v, .99):6.0f} max {max(v):6.0f}")
|
||||||
|
ctx = [r.get("context_tokens", 0) for r in rs]
|
||||||
|
over = ", ".join(f"> {t}: {sum(c > t for c in ctx)} ({100 * sum(c > t for c in ctx) / n:.1f}%)"
|
||||||
|
for t in thresholds)
|
||||||
|
out.append(f" pages needing more context than {over}")
|
||||||
|
cut = [r for r in rs if r.get("done_reason") == "length"]
|
||||||
|
out.append(f" output stopped at a length limit: {len(cut)} page(s)")
|
||||||
|
retried = [r for r in rs if r.get("retries")]
|
||||||
|
aborted = [r for r in rs if r.get("done_reason") == "repeat_abort"]
|
||||||
|
why = defaultdict(int)
|
||||||
|
for r in retried:
|
||||||
|
for reason in r.get("retry_reasons", ["unrecorded"]):
|
||||||
|
why[reason] += 1
|
||||||
|
detail = ", ".join(f"{k} {v}" for k, v in sorted(why.items()))
|
||||||
|
out.append(f" model looped: {len(retried)} page(s) retried"
|
||||||
|
f"{f' ({detail})' if detail else ''}, "
|
||||||
|
f"{len(aborted)} left as placeholders")
|
||||||
|
gpu = [r["gpu_fraction"] for r in rs if r.get("gpu_fraction") is not None]
|
||||||
|
vram = [r["vram_bytes"] for r in rs if r.get("vram_bytes")]
|
||||||
|
size = [r["loaded_bytes"] for r in rs if r.get("loaded_bytes")]
|
||||||
|
if gpu:
|
||||||
|
out.append(f" model in GPU memory: {100 * min(gpu):.0f}-{100 * max(gpu):.0f}% "
|
||||||
|
f"(loaded {max(size) / 1e9:.1f} GB, in VRAM {max(vram) / 1e9:.1f} GB)")
|
||||||
|
secs = [r.get("total_s", 0) for r in rs]
|
||||||
|
out.append(f" seconds per page: mean {sum(secs) / n:.1f}, p90 {_pct(secs, .9):.1f}, "
|
||||||
|
f"max {max(secs):.1f}; total {sum(secs) / 3600:.1f} h")
|
||||||
|
out.append(f" largest pages by context used:")
|
||||||
|
for r in sorted(rs, key=lambda r: -r.get("context_tokens", 0))[:top]:
|
||||||
|
px = "x".join(str(x) for x in r.get("image_px", [])) or "?"
|
||||||
|
out.append(f" {r.get('context_tokens', 0):6d} ({r.get('prompt_tokens', 0)} in, "
|
||||||
|
f"{r.get('output_tokens', 0)} out, {px} px) {r.get('label')}")
|
||||||
|
out.append("")
|
||||||
|
return "\n".join(out).rstrip() + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.argument("log", required=False, type=click.Path(dir_okay=False, path_type=Path))
|
||||||
|
@click.option("--run", "run_id", default=None,
|
||||||
|
help="Only this run id; 'last' for the most recent run.")
|
||||||
|
@click.option("--threshold", "thresholds", type=int, multiple=True,
|
||||||
|
help="Context sizes to test against (repeatable). "
|
||||||
|
f"Default: {', '.join(map(str, THRESHOLDS))}.")
|
||||||
|
@click.option("--top", default=5, show_default=True, help="Largest pages to list.")
|
||||||
|
def main(log: Path | None, run_id: str | None, thresholds: tuple[int, ...],
|
||||||
|
top: int) -> None:
|
||||||
|
"""Summarize context use and GPU fit from the scan-notes-md stats log."""
|
||||||
|
if log is None:
|
||||||
|
from .scan_config import load_config
|
||||||
|
log = load_config().stats_log
|
||||||
|
if log is None or not log.exists():
|
||||||
|
print(f"error: no stats log at {log}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
records = read_log(log)
|
||||||
|
if run_id == "last" and records:
|
||||||
|
run_id = records[-1].get("run")
|
||||||
|
if run_id:
|
||||||
|
records = [r for r in records if r.get("run") == run_id]
|
||||||
|
sys.stdout.write(summarize(records, thresholds or THRESHOLDS, top))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
120
src/pdf_tools/scrub.py
Normal file
120
src/pdf_tools/scrub.py
Normal file
|
|
@ -0,0 +1,120 @@
|
||||||
|
"""Tool 1: strip annotations from a PDF.
|
||||||
|
|
||||||
|
Removes the /Annots array from every page. This catches Preview/Skim/
|
||||||
|
PDF Expert highlights, sticky notes, freehand drawings, text boxes,
|
||||||
|
strikethroughs, underlines — anything stored in the annotation layer.
|
||||||
|
|
||||||
|
What it does NOT remove:
|
||||||
|
- Markings that were "flattened" into the page content stream
|
||||||
|
(some apps offer this as an export option). Those become part of
|
||||||
|
the page itself and can only be removed by going back to an
|
||||||
|
unmarked source.
|
||||||
|
- Form field values. We deliberately leave AcroForm /Widget annots
|
||||||
|
alone unless --include-forms is given.
|
||||||
|
- Embedded files / attachments. Use a dedicated tool if you need
|
||||||
|
those gone too.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import click
|
||||||
|
import pikepdf
|
||||||
|
|
||||||
|
from .common import err, must_exist, with_suffix_before_ext
|
||||||
|
|
||||||
|
# Subtypes that represent user markup. Widget = form field, which we
|
||||||
|
# usually want to KEEP.
|
||||||
|
MARKUP_SUBTYPES = {
|
||||||
|
"/Text", "/FreeText", "/Line", "/Square", "/Circle", "/Polygon",
|
||||||
|
"/PolyLine", "/Highlight", "/Underline", "/Squiggly", "/StrikeOut",
|
||||||
|
"/Stamp", "/Caret", "/Ink", "/Popup", "/FileAttachment", "/Sound",
|
||||||
|
"/Movie", "/Redact", "/Watermark",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def scrub(in_path: Path, out_path: Path, include_forms: bool = False) -> dict:
|
||||||
|
"""Scrub annotations. Returns a count dict for reporting."""
|
||||||
|
counts = {"pages": 0, "removed": 0, "kept": 0}
|
||||||
|
|
||||||
|
with pikepdf.open(in_path) as pdf:
|
||||||
|
for page in pdf.pages:
|
||||||
|
counts["pages"] += 1
|
||||||
|
if "/Annots" not in page:
|
||||||
|
continue
|
||||||
|
|
||||||
|
annots = page["/Annots"]
|
||||||
|
kept = []
|
||||||
|
for a in annots:
|
||||||
|
try:
|
||||||
|
subtype = str(a.get("/Subtype", ""))
|
||||||
|
except Exception:
|
||||||
|
subtype = ""
|
||||||
|
is_widget = subtype == "/Widget"
|
||||||
|
is_markup = subtype in MARKUP_SUBTYPES
|
||||||
|
|
||||||
|
if is_widget and not include_forms:
|
||||||
|
kept.append(a)
|
||||||
|
counts["kept"] += 1
|
||||||
|
elif is_markup or (include_forms and is_widget):
|
||||||
|
counts["removed"] += 1
|
||||||
|
else:
|
||||||
|
# Unknown subtype — keep it to be safe.
|
||||||
|
kept.append(a)
|
||||||
|
counts["kept"] += 1
|
||||||
|
|
||||||
|
if kept:
|
||||||
|
page["/Annots"] = pikepdf.Array(kept)
|
||||||
|
else:
|
||||||
|
del page["/Annots"]
|
||||||
|
|
||||||
|
# Linearize + strip metadata that often leaks reviewer identity.
|
||||||
|
# We keep /Title and /Subject; drop /Author and /Producer hints
|
||||||
|
# from the editing app.
|
||||||
|
try:
|
||||||
|
with pdf.open_metadata() as meta:
|
||||||
|
for k in ("dc:creator", "xmp:CreatorTool", "pdf:Producer"):
|
||||||
|
if k in meta:
|
||||||
|
del meta[k]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
pdf.save(out_path, linearize=True)
|
||||||
|
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
@click.command()
|
||||||
|
@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||||
|
@click.option(
|
||||||
|
"-o", "--output",
|
||||||
|
type=click.Path(dir_okay=False, path_type=Path),
|
||||||
|
help="Output path. Default: <input>_clean.pdf",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--include-forms", is_flag=True,
|
||||||
|
help="Also strip form field widgets (default: keep them).",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--print-path", is_flag=True,
|
||||||
|
help="Print only the output path on stdout (for shell pipelines).",
|
||||||
|
)
|
||||||
|
def main(pdf: Path, output: Path | None, include_forms: bool, print_path: bool) -> None:
|
||||||
|
"""Strip annotations from a PDF for sending."""
|
||||||
|
in_path = must_exist(pdf)
|
||||||
|
out_path = output or with_suffix_before_ext(in_path, "_clean")
|
||||||
|
|
||||||
|
counts = scrub(in_path, out_path, include_forms=include_forms)
|
||||||
|
|
||||||
|
if print_path:
|
||||||
|
print(str(out_path))
|
||||||
|
else:
|
||||||
|
err(
|
||||||
|
f"scrubbed {counts['removed']} annotations across "
|
||||||
|
f"{counts['pages']} pages (kept {counts['kept']}) -> {out_path}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
308
tests/test_cli.py
Normal file
308
tests/test_cli.py
Normal file
|
|
@ -0,0 +1,308 @@
|
||||||
|
"""End-to-end CLI behavior with Ollama faked and Claude forbidden."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pypdfium2 as pdfium
|
||||||
|
import pytest
|
||||||
|
from click.testing import CliRunner
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from pdf_tools import extract_scan, ollama
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_path, monkeypatch):
|
||||||
|
root = tmp_path / "ACME"
|
||||||
|
cfg = tmp_path / "scan-notes.toml"
|
||||||
|
cfg.write_text(f"""
|
||||||
|
prompt_extra = "Mark drawings."
|
||||||
|
stats_log = "{tmp_path / "stats.jsonl"}"
|
||||||
|
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "remote"
|
||||||
|
url = "http://gpu-box:11434"
|
||||||
|
model = "gemma4:31b"
|
||||||
|
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "local"
|
||||||
|
url = "http://localhost:11434"
|
||||||
|
model = "gemma4:e4b"
|
||||||
|
|
||||||
|
[[confidential]]
|
||||||
|
root = "{root}"
|
||||||
|
names = ["Board Person"]
|
||||||
|
prompt_extra = "Board-only instruction."
|
||||||
|
|
||||||
|
[confidential.companion]
|
||||||
|
type = "meeting-notes"
|
||||||
|
fields = ["body", "committee"]
|
||||||
|
sections = ["Attendees", "Discussion", "Decisions", "Action items"]
|
||||||
|
transcript_section = "Discussion"
|
||||||
|
|
||||||
|
[[confidential.companion.paths]]
|
||||||
|
match = "{{body}}/committees/{{committee}}/**"
|
||||||
|
|
||||||
|
[[confidential]]
|
||||||
|
root = "{tmp_path / "lab"}"
|
||||||
|
names = ["Lab Person"]
|
||||||
|
""")
|
||||||
|
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(cfg))
|
||||||
|
monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
|
||||||
|
|
||||||
|
def no_claude(*a, **k):
|
||||||
|
raise AssertionError("Claude must not be called")
|
||||||
|
monkeypatch.setattr(extract_scan, "_transcribe_image", no_claude)
|
||||||
|
|
||||||
|
folder = root / "ACMEP-BOM" / "committees" / "finance-audit" / "2026-10-29-fa"
|
||||||
|
folder.mkdir(parents=True)
|
||||||
|
pdf = folder / "2026-10-29-acmep-fa-notes.pdf"
|
||||||
|
doc = pdfium.PdfDocument.new()
|
||||||
|
doc.new_page(200, 260)
|
||||||
|
doc.save(pdf)
|
||||||
|
return {"root": root, "pdf": pdf, "tmp": tmp_path}
|
||||||
|
|
||||||
|
|
||||||
|
def fake_ollama(monkeypatch, up: dict[str, list[str]]):
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
base, _, path = url.partition("/api/")
|
||||||
|
if path == "tags":
|
||||||
|
if base not in up:
|
||||||
|
raise OSError("unreachable")
|
||||||
|
return {"models": [{"name": m} for m in up[base]]}
|
||||||
|
if path == "ps":
|
||||||
|
return {"models": [{"name": m, "size": 100, "size_vram": 100,
|
||||||
|
"context_length": 4096} for m in up.get(base, [])]}
|
||||||
|
_fake.prompts.append(payload["messages"][0]["content"])
|
||||||
|
return {"message": {"content": f"transcript via {payload['model']}"},
|
||||||
|
"prompt_eval_count": 1000, "eval_count": 200, "done_reason": "stop",
|
||||||
|
"total_duration": 2_000_000_000}
|
||||||
|
_fake.prompts = []
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
return _fake
|
||||||
|
|
||||||
|
|
||||||
|
def run(*args):
|
||||||
|
return CliRunner().invoke(extract_scan.main, [str(a) for a in args])
|
||||||
|
|
||||||
|
|
||||||
|
def test_confidential_scan_defaults_to_ollama_and_writes_companion(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
r = run(env["pdf"])
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
md = env["pdf"].with_suffix(".md").read_text()
|
||||||
|
assert "body: ACMEP-BOM\ncommittee: finance-audit\ntype: meeting-notes" in md
|
||||||
|
assert "## Discussion\n\n### Draft transcript (unverified)" in md
|
||||||
|
assert "transcribed_by: ollama/localhost/gemma4:e4b" in md
|
||||||
|
assert "transcript via gemma4:e4b" in md
|
||||||
|
assert "falling back to local" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_host_exits_without_cloud(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {})
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "set-but-must-not-be-used")
|
||||||
|
r = run(env["pdf"])
|
||||||
|
assert r.exit_code == extract_scan.EXIT_NO_HOST
|
||||||
|
assert "Not falling back to a cloud service" in r.stderr
|
||||||
|
assert not env["pdf"].with_suffix(".md").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_claude_refused_for_confidential_scan(env, monkeypatch):
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
|
||||||
|
r = run("--backend", "claude", env["pdf"])
|
||||||
|
assert r.exit_code == 2 and "--allow-cloud" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_claude_refused_even_with_no_companion(env, monkeypatch):
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
|
||||||
|
r = run("--backend", "claude", "--no-companion", "--stdout", env["pdf"])
|
||||||
|
assert r.exit_code == 2 and "--allow-cloud" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_forced_remote_fails_clearly(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
r = run("--host", "remote", env["pdf"])
|
||||||
|
assert r.exit_code == extract_scan.EXIT_NO_HOST
|
||||||
|
assert "remote" in r.stderr and "falling back to local" not in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_flag_overrides_config(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["qwen3-vl:8b"]})
|
||||||
|
r = run("--model", "qwen3-vl:8b", "--host", "remote", env["pdf"])
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "ollama/gpu-box/qwen3-vl:8b" in env["pdf"].with_suffix(".md").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_companion_ollama_output_uses_page_markers(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
|
||||||
|
img = env["tmp"] / "page.png"
|
||||||
|
Image.new("RGB", (50, 50), "white").save(img)
|
||||||
|
r = run("--backend", "ollama", "--stdout", img)
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "<!-- page 1 -->" in r.stdout
|
||||||
|
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def test_companion_mode_rejects_stdout(env):
|
||||||
|
r = run("--stdout", env["pdf"])
|
||||||
|
assert r.exit_code == 2 and "--no-companion" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_confidential_folder_uses_its_layout(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
|
||||||
|
lab = env["tmp"] / "lab" / "reviews"
|
||||||
|
lab.mkdir(parents=True)
|
||||||
|
pdf = lab / "2026-11-02-review-notes.pdf"
|
||||||
|
doc = pdfium.PdfDocument.new()
|
||||||
|
doc.new_page(200, 260)
|
||||||
|
doc.save(pdf)
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "x")
|
||||||
|
assert run("--backend", "claude", pdf).exit_code == 2
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
md = pdf.with_suffix(".md").read_text()
|
||||||
|
assert "type: notes" in md and "## Notes" in md and "body:" not in md
|
||||||
|
|
||||||
|
|
||||||
|
def test_mixing_confidential_and_other_files_refused(env, monkeypatch):
|
||||||
|
img = env["tmp"] / "page.png"
|
||||||
|
Image.new("RGB", (50, 50), "white").save(img)
|
||||||
|
r = run("--backend", "ollama", "--no-companion", "--stdout", env["pdf"], img)
|
||||||
|
assert r.exit_code == 2 and "do not mix" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_names_come_only_from_the_scans_folder(env, monkeypatch):
|
||||||
|
fake = fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
assert run(env["pdf"]).exit_code == 0
|
||||||
|
assert "- Board Person" in fake.prompts[0]
|
||||||
|
assert "Lab Person" not in fake.prompts[0]
|
||||||
|
assert "Mark drawings." in fake.prompts[0]
|
||||||
|
assert "Board-only instruction." in fake.prompts[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_only_top_level_settings_outside_confidential_folders(env, monkeypatch):
|
||||||
|
fake = fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
img = env["tmp"] / "page.png"
|
||||||
|
Image.new("RGB", (50, 50), "white").save(img)
|
||||||
|
assert run("--backend", "ollama", "--stdout", img).exit_code == 0
|
||||||
|
assert fake.prompts[0] == ollama.build_prompt([], ["Mark drawings."])
|
||||||
|
assert "Board" not in fake.prompts[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_confidential_folders_without_config(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(tmp_path / "empty.toml"))
|
||||||
|
(tmp_path / "empty.toml").write_text("")
|
||||||
|
from pdf_tools.scan_config import load_config
|
||||||
|
assert load_config().confidential == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_interrupted_run_resumes_and_cleans_up(env, monkeypatch):
|
||||||
|
nb = env["tmp"] / "notebooks"
|
||||||
|
nb.mkdir()
|
||||||
|
pdf = nb / "2024-04 notebook.pdf"
|
||||||
|
doc = pdfium.PdfDocument.new()
|
||||||
|
for _ in range(3):
|
||||||
|
doc.new_page(200, 260)
|
||||||
|
doc.save(pdf)
|
||||||
|
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
if url.endswith("/api/tags"):
|
||||||
|
return {"models": [{"name": "gemma4:e4b"}]}
|
||||||
|
if url.endswith("/api/ps"):
|
||||||
|
return {"models": []}
|
||||||
|
calls.append(1)
|
||||||
|
if len(calls) == 2 and not getattr(_fake, "healthy", False):
|
||||||
|
raise OSError("connection reset")
|
||||||
|
return {"message": {"content": f"page text {len(calls)}"}}
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
|
||||||
|
r = run("--backend", "ollama", "--host", "local", pdf)
|
||||||
|
assert r.exit_code == extract_scan.EXIT_TRANSCRIBE_FAILED
|
||||||
|
assert "pages 1-1 are saved; rerun to resume" in r.stderr
|
||||||
|
ckpt = nb / ".2024-04 notebook.scan-partial.json"
|
||||||
|
assert ckpt.exists() and not pdf.with_suffix(".md").exists()
|
||||||
|
|
||||||
|
_fake.healthy = True
|
||||||
|
r = run("--backend", "ollama", "--host", "local", pdf)
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "resuming after page 1 of 3" in r.stderr
|
||||||
|
md = pdf.with_suffix(".md").read_text()
|
||||||
|
assert "page text 1" in md and "<!-- page 3 -->" in md
|
||||||
|
assert len(calls) == 4 # page 1 was not sent again
|
||||||
|
assert not ckpt.exists()
|
||||||
|
assert {p.name for p in nb.iterdir()} == {pdf.name, pdf.with_suffix(".md").name}
|
||||||
|
|
||||||
|
|
||||||
|
def test_checkpoint_from_other_model_is_ignored(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
img = env["tmp"] / "page.png"
|
||||||
|
Image.new("RGB", (50, 50), "white").save(img)
|
||||||
|
ckpt = env["tmp"] / ".page.scan-partial.json"
|
||||||
|
ckpt.write_text('{"key": {"transcribed_by": "other"}, "pages": [{"label": "x", "text": "stale"}]}')
|
||||||
|
r = run("--backend", "ollama", "--stdout", img)
|
||||||
|
assert r.exit_code == 0 and "stale" not in r.stdout
|
||||||
|
assert "ignoring saved progress" in r.stderr and not ckpt.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_stats_logged_per_page_without_text(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://localhost:11434": ["gemma4:e4b"]})
|
||||||
|
r = run(env["pdf"])
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "1200 tokens" in r.stderr
|
||||||
|
lines = (env["tmp"] / "stats.jsonl").read_text().splitlines()
|
||||||
|
assert len(lines) == 1
|
||||||
|
rec = json.loads(lines[0])
|
||||||
|
assert rec["model"] == "gemma4:e4b" and rec["host"] == "localhost"
|
||||||
|
assert (rec["prompt_tokens"], rec["output_tokens"], rec["context_tokens"]) == (1000, 200, 1200)
|
||||||
|
assert rec["gpu_fraction"] == 1.0 and rec["loaded_num_ctx"] == 4096
|
||||||
|
assert rec["image_px"][0] > 0 and rec["page"] == 1
|
||||||
|
assert "transcript via" not in lines[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_warns_when_output_hits_length_limit(env, monkeypatch):
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
if url.endswith("/api/tags"):
|
||||||
|
return {"models": [{"name": "gemma4:e4b"}]}
|
||||||
|
if url.endswith("/api/ps"):
|
||||||
|
return {"models": [{"name": "gemma4:e4b", "size": 1, "size_vram": 1,
|
||||||
|
"context_length": 4096}]}
|
||||||
|
return {"message": {"content": "partial"}, "prompt_eval_count": 3900,
|
||||||
|
"eval_count": 196, "done_reason": "length"}
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
r = run(env["pdf"])
|
||||||
|
assert r.exit_code == 0
|
||||||
|
assert "marked as cut off" in r.stderr
|
||||||
|
assert "cut off at the output limit" in env["pdf"].with_suffix(".md").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_looping_page_becomes_placeholder_and_run_finishes(env, monkeypatch):
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
if url.endswith("/api/tags"):
|
||||||
|
return {"models": [{"name": "gemma4:e4b"}]}
|
||||||
|
if url.endswith("/api/ps"):
|
||||||
|
return {"models": []}
|
||||||
|
raise urllib.error.HTTPError(
|
||||||
|
url, 500, "err", {},
|
||||||
|
io.BytesIO(b'{"error":"prediction aborted, token repeat limit reached"}'))
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
r = run(env["pdf"])
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "writing a placeholder" in r.stderr
|
||||||
|
assert ollama.REPEAT_PLACEHOLDER in env["pdf"].with_suffix(".md").read_text()
|
||||||
|
rec = json.loads((env["tmp"] / "stats.jsonl").read_text().splitlines()[0])
|
||||||
|
assert rec["done_reason"] == "repeat_abort" and rec["retries"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_ollama_is_the_default_backend_outside_confidential_folders(env, monkeypatch):
|
||||||
|
fake_ollama(monkeypatch, {"http://gpu-box:11434": ["gemma4:31b"]})
|
||||||
|
monkeypatch.setenv("ANTHROPIC_API_KEY", "set-but-must-not-be-used")
|
||||||
|
img = env["tmp"] / "page.png"
|
||||||
|
Image.new("RGB", (50, 50), "white").save(img)
|
||||||
|
r = run("--stdout", img)
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in r.stdout
|
||||||
291
tests/test_companion.py
Normal file
291
tests/test_companion.py
Normal file
|
|
@ -0,0 +1,291 @@
|
||||||
|
"""Companion files: front matter from paths, and preserving existing text."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pdf_tools import companion as comp
|
||||||
|
from pdf_tools.companion import CompanionSpec, PathRule
|
||||||
|
|
||||||
|
ROOT = Path("/tmp/acme-test-root")
|
||||||
|
|
||||||
|
# A meetings layout like the one in the user's config (test data only; the
|
||||||
|
# code knows nothing about any organization).
|
||||||
|
SPEC = CompanionSpec(
|
||||||
|
type="meeting-notes",
|
||||||
|
fields=["body", "committee"],
|
||||||
|
sections=["Attendees", "Discussion", "Decisions", "Action items"],
|
||||||
|
transcript_section="Discussion",
|
||||||
|
section_text={"Action items": "- [ ] Owner: task (due date)"},
|
||||||
|
title="{body}, {date}",
|
||||||
|
allowed={"body": ["ACME-BOD", "ACMEP-BOM", "ACMEF-BOT"]},
|
||||||
|
acronyms={"acmep", "ceo"},
|
||||||
|
paths=[
|
||||||
|
PathRule("{body}/committees/{committee}/**", title="{body.short} {committee}, {date}"),
|
||||||
|
PathRule("{body}/meetings/**", set={"committee": "board"}),
|
||||||
|
PathRule("chair/{committee}/**", set={"body": "chair"}, title="{committee}, {date}"),
|
||||||
|
PathRule("{body}/{committee}/**", title="{body.short} {committee}, {date}"),
|
||||||
|
PathRule("{body}/**"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
TITLES = {
|
||||||
|
"ACME-BOD": "ACME Board of Directors", "ACME-BOD.short": "ACME",
|
||||||
|
"ACMEP-BOM": "ACMEP Board of Managers", "ACMEP-BOM.short": "ACMEP",
|
||||||
|
"finance-audit": "Finance and Audit Committee",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def meta(rel: str) -> comp.NoteMeta:
|
||||||
|
return comp.meta_from_path(ROOT / rel, ROOT, SPEC)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("rel, date, body, committee", [
|
||||||
|
("ACMEP-BOM/committees/finance-audit/2026-10-29-fa/2026-10-29-acmep-fa-notes.pdf",
|
||||||
|
"2026-10-29", "ACMEP-BOM", "finance-audit"),
|
||||||
|
("ACME-BOD/meetings/2026-11-12/2026-11-12-acme-bod-notes.pdf",
|
||||||
|
"2026-11-12", "ACME-BOD", "board"),
|
||||||
|
("chair/acmep-ceo-search/2026-10-30-acmep-ceo-search-notes.pdf",
|
||||||
|
"2026-10-30", "chair", "acmep-ceo-search"),
|
||||||
|
("ACME-BOD/committees/iac/2026-06-xx/notes.pdf",
|
||||||
|
"2026-06-xx", "ACME-BOD", "iac"),
|
||||||
|
("ACMEP-BOM/meetings/2026-06-08/scan.pdf",
|
||||||
|
"2026-06-08", "ACMEP-BOM", "board"),
|
||||||
|
])
|
||||||
|
def test_meta_from_path(rel, date, body, committee):
|
||||||
|
m = meta(rel)
|
||||||
|
assert (m.date, m.fields["body"], m.fields["committee"]) == (date, body, committee)
|
||||||
|
assert m.scan == Path(rel).name
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_date_wins_over_folder():
|
||||||
|
m = meta("ACMEP-BOM/meetings/2026-06-08/2026-06-09-acmep-notes.pdf")
|
||||||
|
assert m.date == "2026-06-09"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_location_warns():
|
||||||
|
m = meta("inbox/scan.pdf")
|
||||||
|
assert m.fields["body"] == "" and m.date == ""
|
||||||
|
assert any("body" in w for w in m.warnings)
|
||||||
|
assert any("date" in w for w in m.warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def test_partial_date_warns():
|
||||||
|
assert any("no day" in w for w in meta("ACME-BOD/meetings/2026-06-xx/n.pdf").warnings)
|
||||||
|
|
||||||
|
|
||||||
|
def test_titles():
|
||||||
|
def t(rel, titles=TITLES):
|
||||||
|
return comp.title_for(meta(rel), SPEC, titles)
|
||||||
|
assert t("ACMEP-BOM/committees/finance-audit/x/2026-10-29-n.pdf") == \
|
||||||
|
"ACMEP Finance and Audit Committee, October 29, 2026"
|
||||||
|
assert t("ACME-BOD/meetings/2026-11-12/n.pdf") == \
|
||||||
|
"ACME Board of Directors, November 12, 2026"
|
||||||
|
assert t("chair/acmep-ceo-search/2026-10-30-n.pdf") == \
|
||||||
|
"ACMEP CEO Search, October 30, 2026"
|
||||||
|
assert t("chair/acmep-ceo-search/2026-10-30-n.pdf",
|
||||||
|
{"acmep-ceo-search": "ACMEP CEO search"}).startswith("ACMEP CEO search,")
|
||||||
|
assert t("ACME-BOD/meetings/x/n.pdf") == "ACME Board of Directors"
|
||||||
|
|
||||||
|
|
||||||
|
def test_allowed_values_and_rule_order():
|
||||||
|
m = meta("inbox/2026-01-02-n.pdf") # "inbox" is not an allowed body
|
||||||
|
assert m.fields == {"body": "", "committee": ""}
|
||||||
|
m = meta("ACME-BOD/2026-01-02-n.pdf") # matched by the last rule
|
||||||
|
assert m.fields == {"body": "ACME-BOD", "committee": ""}
|
||||||
|
assert any("committee" in w for w in m.warnings)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("pattern, folders, expected", [
|
||||||
|
("a/{x}", ("a", "b"), {"x": "b"}),
|
||||||
|
("a/{x}", ("a", "b", "c"), None),
|
||||||
|
("a/{x}/**", ("a", "b", "c"), {"x": "b"}),
|
||||||
|
("a/**", ("a",), {}),
|
||||||
|
("{x}/b", ("a", "c"), None),
|
||||||
|
])
|
||||||
|
def test_match(pattern, folders, expected):
|
||||||
|
assert comp._match(pattern, folders, {}) == expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_companion_matches_template():
|
||||||
|
m = meta("ACMEP-BOM/committees/finance-audit/2026-10-29-fa/2026-10-29-acmep-fa-notes.pdf")
|
||||||
|
text = comp.new_companion(m, ["page one", "page two"], "ollama/gpu-box/gemma4:31b",
|
||||||
|
SPEC, TITLES)
|
||||||
|
assert text.startswith(
|
||||||
|
"---\ndate: 2026-10-29\nbody: ACMEP-BOM\ncommittee: finance-audit\n"
|
||||||
|
"type: meeting-notes\nscan: 2026-10-29-acmep-fa-notes.pdf\nsubjects: []\n"
|
||||||
|
"transcribed_by: ollama/gpu-box/gemma4:31b\n---\n")
|
||||||
|
for h in ("## Attendees", "## Discussion", comp.DRAFT_HEADING,
|
||||||
|
"## Decisions", "## Action items"):
|
||||||
|
assert h in text
|
||||||
|
assert text.index("## Discussion") < text.index(comp.DRAFT_HEADING) < text.index("## Decisions")
|
||||||
|
assert "<!-- page 1 -->" in text and "<!-- page 2 -->" in text
|
||||||
|
assert comp.PENDING_LINE not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_transcript_keeps_layout_and_fences_headings():
|
||||||
|
page = "# 3 items\n - indented\n\nafter blank"
|
||||||
|
r = comp.render_transcript([page])
|
||||||
|
assert "```text\n# 3 items\n - indented\n\nafter blank\n```" in r
|
||||||
|
r2 = comp.render_transcript(["has ``` inside"])
|
||||||
|
assert "````text\nhas ``` inside\n````" in r2
|
||||||
|
|
||||||
|
|
||||||
|
# ---- preserving existing companion text -----------------------------------
|
||||||
|
|
||||||
|
EXISTING = """\
|
||||||
|
---
|
||||||
|
date: 2026-10-29
|
||||||
|
body: ACMEP-BOM
|
||||||
|
committee: finance-audit
|
||||||
|
type: meeting-notes
|
||||||
|
scan: 2026-10-29-acmep-fa-notes.pdf
|
||||||
|
subjects: [ACMEP performance, audit timeline]
|
||||||
|
---
|
||||||
|
|
||||||
|
# ACMEP Finance and Audit Committee, October 29, 2026
|
||||||
|
|
||||||
|
## Attendees
|
||||||
|
|
||||||
|
Pat, the committee, staff.
|
||||||
|
|
||||||
|
## Discussion
|
||||||
|
|
||||||
|
My own summary paragraph.
|
||||||
|
With an indented line.
|
||||||
|
|
||||||
|
{middle}
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
- Decision I wrote.
|
||||||
|
|
||||||
|
## Action items
|
||||||
|
|
||||||
|
- [ ] Owner: task (due date)
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _mine(text: str) -> list[str]:
|
||||||
|
"""Every line Pat wrote, which must survive any merge."""
|
||||||
|
return [l for l in EXISTING.format(middle="").splitlines() if l.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def assert_preserved(before: str, after: str):
|
||||||
|
after_lines = after.splitlines()
|
||||||
|
pos = 0
|
||||||
|
for line in _mine(before):
|
||||||
|
assert line in after_lines[pos:], f"lost or reordered: {line!r}"
|
||||||
|
pos = after_lines.index(line, pos) + 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_pending_line_replaced():
|
||||||
|
before = EXISTING.format(middle=comp.PENDING_LINE + "\n")
|
||||||
|
after = comp.merge_companion(before, ["draft"], "ollama/localhost/gemma4:e4b", "Discussion")
|
||||||
|
assert comp.PENDING_LINE not in after
|
||||||
|
assert after.count(comp.DRAFT_HEADING) == 1
|
||||||
|
assert_preserved(before, after)
|
||||||
|
assert after.index("My own summary") < after.index(comp.DRAFT_HEADING) \
|
||||||
|
< after.index("## Decisions")
|
||||||
|
|
||||||
|
|
||||||
|
def test_block_added_when_missing():
|
||||||
|
before = EXISTING.format(middle="")
|
||||||
|
after = comp.merge_companion(before, ["draft"], "ollama/localhost/gemma4:e4b", "Discussion")
|
||||||
|
assert after.count(comp.DRAFT_HEADING) == 1
|
||||||
|
assert_preserved(before, after)
|
||||||
|
assert after.index("With an indented line.") < after.index(comp.DRAFT_HEADING) \
|
||||||
|
< after.index("## Decisions")
|
||||||
|
assert "\n\n## Decisions" in after
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_block_replaced_not_duplicated():
|
||||||
|
before = EXISTING.format(middle="")
|
||||||
|
once = comp.merge_companion(before, ["first draft"], "ollama/localhost/gemma4:e4b", "Discussion")
|
||||||
|
twice = comp.merge_companion(once, ["second draft"], "ollama/gpu-box/gemma4:31b", "Discussion")
|
||||||
|
assert twice.count(comp.DRAFT_HEADING) == 1
|
||||||
|
assert "second draft" in twice and "first draft" not in twice
|
||||||
|
assert_preserved(before, twice)
|
||||||
|
assert "transcribed_by: ollama/gpu-box/gemma4:31b" in twice
|
||||||
|
assert twice.count("transcribed_by:") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_rerun_is_stable():
|
||||||
|
before = EXISTING.format(middle="")
|
||||||
|
once = comp.merge_companion(before, ["same"], "x", "Discussion")
|
||||||
|
assert comp.merge_companion(once, ["same"], "x", "Discussion") == once
|
||||||
|
|
||||||
|
|
||||||
|
def test_old_block_without_end_marker_stops_at_next_heading():
|
||||||
|
before = EXISTING.format(middle=(
|
||||||
|
f"{comp.DRAFT_HEADING}\n\nold draft text\n\n"
|
||||||
|
"### My corrections\n\nKeep this.\n"))
|
||||||
|
after = comp.merge_companion(before, ["new"], "x", "Discussion")
|
||||||
|
assert "old draft text" not in after
|
||||||
|
assert "### My corrections\n\nKeep this." in after
|
||||||
|
assert_preserved(before, after)
|
||||||
|
|
||||||
|
|
||||||
|
def test_hash_line_inside_transcript_does_not_end_block():
|
||||||
|
before = EXISTING.format(middle="")
|
||||||
|
once = comp.merge_companion(before, ["# not a heading\nmore"], "x", "Discussion")
|
||||||
|
twice = comp.merge_companion(once, ["replacement"], "x", "Discussion")
|
||||||
|
assert "not a heading" not in twice and "more\n" not in twice.split("## Decisions")[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_front_matter_only_gains_transcribed_by():
|
||||||
|
before = EXISTING.format(middle="")
|
||||||
|
after = comp.merge_companion(before, ["d"], "ollama/localhost/gemma4:e4b", "Discussion")
|
||||||
|
fm_before = before.split("\n---\n", 1)[0]
|
||||||
|
fm_after = after.split("\n---\n", 1)[0]
|
||||||
|
assert fm_after == fm_before + "\ntranscribed_by: ollama/localhost/gemma4:e4b"
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_discussion_section_appends_at_end():
|
||||||
|
before = "---\ndate: 2026-10-29\n---\n\n# Notes\n\nFree text.\n"
|
||||||
|
after = comp.merge_companion(before, ["d"], "x")
|
||||||
|
assert after.startswith("---\ndate: 2026-10-29\ntranscribed_by: x\n---\n\n# Notes\n\nFree text.\n\n")
|
||||||
|
assert after.rstrip().endswith(comp.END_MARKER)
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_companion_creates_then_preserves(tmp_path):
|
||||||
|
folder = tmp_path / "ACMEP-BOM" / "committees" / "finance-audit" / "2026-10-29-fa"
|
||||||
|
folder.mkdir(parents=True)
|
||||||
|
pdf = folder / "2026-10-29-acmep-fa-notes.pdf"
|
||||||
|
pdf.write_bytes(b"%PDF-1.4")
|
||||||
|
path, created, _ = comp.write_companion(pdf, tmp_path, ["one"], "x", SPEC)
|
||||||
|
assert created and path == pdf.with_suffix(".md")
|
||||||
|
|
||||||
|
edited = path.read_text().replace("subjects: []", "subjects: [audit]") \
|
||||||
|
.replace("## Attendees\n", "## Attendees\n\nSomeone.\n")
|
||||||
|
path.write_text(edited)
|
||||||
|
_, created, _ = comp.write_companion(pdf, tmp_path, ["two"], "y", SPEC)
|
||||||
|
final = path.read_text()
|
||||||
|
assert not created
|
||||||
|
assert "subjects: [audit]" in final and "Someone." in final
|
||||||
|
assert "two" in final and "\none\n" not in final
|
||||||
|
assert {p.name for p in folder.iterdir()} == {pdf.name, path.name}
|
||||||
|
|
||||||
|
|
||||||
|
# ---- basic layout (no companion table) -----------------------------------
|
||||||
|
|
||||||
|
def test_basic_layout_meta_has_no_body_or_warnings():
|
||||||
|
m = comp.meta_from_path(ROOT / "projects" / "2026-11-02-lab-review-notes.pdf", ROOT)
|
||||||
|
assert (m.date, m.fields, m.warnings) == ("2026-11-02", {}, [])
|
||||||
|
|
||||||
|
|
||||||
|
def test_basic_layout_template():
|
||||||
|
m = comp.meta_from_path(ROOT / "x" / "2026-11-02-lab-review-notes.pdf", ROOT)
|
||||||
|
text = comp.new_companion(m, ["p"], "ollama/localhost/gemma4:e4b")
|
||||||
|
assert text.startswith(
|
||||||
|
"---\ndate: 2026-11-02\ntype: notes\nscan: 2026-11-02-lab-review-notes.pdf\n"
|
||||||
|
"subjects: []\ntranscribed_by: ollama/localhost/gemma4:e4b\n---\n\n"
|
||||||
|
"# Lab Review Notes, November 2, 2026\n\n## Notes\n\n" + comp.DRAFT_HEADING)
|
||||||
|
assert "body:" not in text and "## Attendees" not in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_basic_layout_merge_keeps_text():
|
||||||
|
before = ("---\ndate: 2026-11-02\n---\n\n# Notes\n\n## Notes\n\nMine.\n\n"
|
||||||
|
"## Later\n\nAlso mine.\n")
|
||||||
|
after = comp.merge_companion(before, ["d"], "x")
|
||||||
|
assert after.index("Mine.") < after.index(comp.DRAFT_HEADING) < after.index("## Later")
|
||||||
|
assert "Also mine." in after
|
||||||
103
tests/test_config.py
Normal file
103
tests/test_config.py
Normal file
|
|
@ -0,0 +1,103 @@
|
||||||
|
"""Confidential-folder declarations in the config file."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pdf_tools.companion import BASIC
|
||||||
|
from pdf_tools.scan_config import load_config
|
||||||
|
|
||||||
|
EXAMPLE = Path(__file__).resolve().parent.parent / "scan-notes.example.toml"
|
||||||
|
|
||||||
|
|
||||||
|
def cfg_from(tmp_path, text: str):
|
||||||
|
p = tmp_path / "scan-notes.toml"
|
||||||
|
p.write_text(text)
|
||||||
|
return load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_nothing_built_in(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.delenv("PDF_TOOLS_SCAN_CONFIG", raising=False)
|
||||||
|
monkeypatch.setattr("pdf_tools.scan_config.DEFAULT_CONFIG_PATH", tmp_path / "none.toml")
|
||||||
|
cfg = load_config()
|
||||||
|
assert cfg.confidential == []
|
||||||
|
assert [h.url for h in cfg.hosts] == ["http://localhost:11434"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_example_file_loads():
|
||||||
|
cfg = load_config(EXAMPLE)
|
||||||
|
assert len(cfg.confidential) == 2
|
||||||
|
basic, meetings = cfg.confidential
|
||||||
|
assert basic.companion is BASIC
|
||||||
|
assert meetings.companion.transcript_section == "Discussion"
|
||||||
|
assert meetings.companion.fields == ["body", "committee"]
|
||||||
|
assert [r.match for r in meetings.companion.paths][0] == "{body}/committees/{committee}/**"
|
||||||
|
assert meetings.titles["BOD.short"] == "Board"
|
||||||
|
|
||||||
|
|
||||||
|
def test_folder_without_companion_is_basic(tmp_path):
|
||||||
|
cfg = cfg_from(tmp_path, f'[[confidential]]\nroot = "{tmp_path / "x"}"\n')
|
||||||
|
assert cfg.confidential[0].companion is BASIC
|
||||||
|
|
||||||
|
|
||||||
|
def test_transcript_section_defaults_to_first_section(tmp_path):
|
||||||
|
cfg = cfg_from(tmp_path, f'[[confidential]]\nroot = "{tmp_path}"\n'
|
||||||
|
'[confidential.companion]\nsections = ["Summary", "Log"]\n')
|
||||||
|
assert cfg.confidential[0].companion.transcript_section == "Summary"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tilde_expanded_and_deepest_root_wins(tmp_path):
|
||||||
|
outer, inner = tmp_path / "a", tmp_path / "a" / "b"
|
||||||
|
cfg = cfg_from(tmp_path,
|
||||||
|
f'[[confidential]]\nroot = "{outer}"\n\n'
|
||||||
|
f'[[confidential]]\nroot = "{inner}"\n')
|
||||||
|
assert cfg.confidential_folder(inner / "c" / "s.pdf").root == inner
|
||||||
|
assert cfg.confidential_folder(outer / "s.pdf").root == outer
|
||||||
|
assert cfg.confidential_folder(tmp_path / "other.pdf") is None
|
||||||
|
# A sibling whose name merely starts with the root's name is not inside it.
|
||||||
|
assert cfg.confidential_folder(tmp_path / "ab" / "s.pdf") is None
|
||||||
|
cfg = cfg_from(tmp_path, '[[confidential]]\nroot = "~/somewhere"\n')
|
||||||
|
assert cfg.confidential[0].root == Path.home() / "somewhere"
|
||||||
|
|
||||||
|
|
||||||
|
def test_names_are_per_folder(tmp_path):
|
||||||
|
a, b = tmp_path / "a", tmp_path / "b"
|
||||||
|
cfg = cfg_from(tmp_path, f'names = ["Everyone"]\n'
|
||||||
|
f'[[confidential]]\nroot = "{a}"\nnames = ["Alpha", "Everyone"]\n'
|
||||||
|
f'[[confidential]]\nroot = "{b}"\nnames = ["Beta"]\n')
|
||||||
|
fa, fb = cfg.confidential
|
||||||
|
assert cfg.names_for(fa) == ["Everyone", "Alpha"]
|
||||||
|
assert cfg.names_for(fb) == ["Everyone", "Beta"]
|
||||||
|
assert cfg.names_for(None) == ["Everyone"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("text", [
|
||||||
|
'[[confidential]]\nnames = []\n', # no root
|
||||||
|
'[[confidential]]\nroot = "/x"\ncompanion = "acme"\n', # old string form
|
||||||
|
'[[confidential]]\nroot = "/x"\nnmaes = []\n', # typo
|
||||||
|
'[[confidential]]\nroot = "/x"\n[confidential.companion]\nsectoins = []\n',
|
||||||
|
'[[confidential]]\nroot = "/x"\n[[confidential.companion.paths]]\ntitle = "t"\n',
|
||||||
|
# A top-level key written after a host table lands inside that table.
|
||||||
|
'[[ollama.hosts]]\nname = "l"\nurl = "http://x"\nmodel = "m"\nprompt_extra = "x"\n',
|
||||||
|
'[ollama]\ntimeout_s = 5\nnames = []\n',
|
||||||
|
'prompt_extar = "x"\n',
|
||||||
|
])
|
||||||
|
def test_bad_entries_rejected(tmp_path, text):
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
cfg_from(tmp_path, text)
|
||||||
|
|
||||||
|
|
||||||
|
def test_prompt_extra_top_level_and_per_folder(tmp_path):
|
||||||
|
a, b = tmp_path / "a", tmp_path / "b"
|
||||||
|
cfg = cfg_from(tmp_path, 'prompt_extra = "Figures."\n'
|
||||||
|
f'[[confidential]]\nroot = "{a}"\nprompt_extra = "Board terms."\n'
|
||||||
|
f'[[confidential]]\nroot = "{b}"\n')
|
||||||
|
fa, fb = cfg.confidential
|
||||||
|
assert cfg.extras_for(fa) == ["Figures.", "Board terms."]
|
||||||
|
assert cfg.extras_for(fb) == ["Figures."]
|
||||||
|
assert cfg.extras_for(None) == ["Figures."]
|
||||||
|
|
||||||
|
|
||||||
|
def test_example_prompt_extra_mentions_figures():
|
||||||
|
assert "[FIGURE:" in load_config(EXAMPLE).prompt_extra
|
||||||
240
tests/test_index.py
Normal file
240
tests/test_index.py
Normal file
|
|
@ -0,0 +1,240 @@
|
||||||
|
"""scan-notes-index: transcript parsing, page files, and the CLI with Ollama faked."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
import pypdfium2 as pdfium
|
||||||
|
import pytest
|
||||||
|
from click.testing import CliRunner
|
||||||
|
|
||||||
|
from pdf_tools import companion, ollama, scan_index
|
||||||
|
from pdf_tools.extract_scan import to_plain_markdown
|
||||||
|
from pdf_tools.scan_config import OllamaHost, load_config
|
||||||
|
from pdf_tools.scan_stats import read_log, summarize
|
||||||
|
|
||||||
|
HOST = OllamaHost("local", "http://localhost:11434", "gemma4:31b", num_ctx=8192)
|
||||||
|
KINDS = ["calculation", "idea", "meeting notes", "other", "blank"]
|
||||||
|
|
||||||
|
|
||||||
|
def answer(**over):
|
||||||
|
a = {"summary": "Estimate of a drag force.", "key_points": ["F = 6 pi eta a U"],
|
||||||
|
"kind": "calculation", "topics": ["Stokes drag"], "references": [],
|
||||||
|
"event": "", "event_date": "", "people": [],
|
||||||
|
"continues_from_previous": False, "continues_to_next": True}
|
||||||
|
return {**a, **over}
|
||||||
|
|
||||||
|
|
||||||
|
def make_scan(folder, pages=("first page", "second page", "third page")):
|
||||||
|
folder.mkdir(parents=True, exist_ok=True)
|
||||||
|
pdf = folder / "2009 notebook.pdf"
|
||||||
|
doc = pdfium.PdfDocument.new()
|
||||||
|
for _ in pages:
|
||||||
|
doc.new_page(200, 260)
|
||||||
|
doc.save(pdf)
|
||||||
|
md = pdf.with_suffix(".md")
|
||||||
|
md.write_text(to_plain_markdown(list(pages), pdf.stem, "ollama/localhost/gemma4:31b"))
|
||||||
|
return pdf
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def env(tmp_path, monkeypatch):
|
||||||
|
cfg = tmp_path / "scan-notes.toml"
|
||||||
|
cfg.write_text(f"""
|
||||||
|
names = ["Global Person"]
|
||||||
|
stats_log = "{tmp_path / "stats.jsonl"}"
|
||||||
|
|
||||||
|
[ollama]
|
||||||
|
max_output_tokens = 2048
|
||||||
|
|
||||||
|
[[ollama.hosts]]
|
||||||
|
name = "local"
|
||||||
|
url = "http://localhost:11434"
|
||||||
|
model = "gemma4:31b"
|
||||||
|
num_ctx = 8192
|
||||||
|
|
||||||
|
[index]
|
||||||
|
names = ["Board Member"]
|
||||||
|
prompt_extra = "Index-only instruction."
|
||||||
|
""")
|
||||||
|
monkeypatch.setenv("PDF_TOOLS_SCAN_CONFIG", str(cfg))
|
||||||
|
return tmp_path
|
||||||
|
|
||||||
|
|
||||||
|
def fake(monkeypatch, replies=None):
|
||||||
|
"""Ollama with gemma4:31b; chat replies come from the list, then answer()."""
|
||||||
|
replies = list(replies or [])
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
base, _, path = url.partition("/api/")
|
||||||
|
if path == "tags":
|
||||||
|
return {"models": [{"name": "gemma4:31b"}]}
|
||||||
|
if path == "ps":
|
||||||
|
return {"models": [{"name": "gemma4:31b", "size": 20, "size_vram": 20,
|
||||||
|
"context_length": 8192}]}
|
||||||
|
_fake.payloads.append(payload)
|
||||||
|
r = replies.pop(0) if replies else answer()
|
||||||
|
if isinstance(r, Exception):
|
||||||
|
raise r
|
||||||
|
if isinstance(r, dict) and "message" in r:
|
||||||
|
return r
|
||||||
|
return {"message": {"content": r if isinstance(r, str) else json.dumps(r)},
|
||||||
|
"prompt_eval_count": 2000, "eval_count": 300, "done_reason": "stop",
|
||||||
|
"total_duration": 9_000_000_000}
|
||||||
|
_fake.payloads = []
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
return _fake
|
||||||
|
|
||||||
|
|
||||||
|
def run(*args):
|
||||||
|
return CliRunner().invoke(scan_index.main, [str(a) for a in args])
|
||||||
|
|
||||||
|
|
||||||
|
def test_reads_plain_and_companion_transcripts(tmp_path):
|
||||||
|
plain = tmp_path / "a.md"
|
||||||
|
plain.write_text(to_plain_markdown(["one", "has ``` inside", ""], "a", "ollama/x/m"))
|
||||||
|
pages, by = scan_index.read_transcript(plain)
|
||||||
|
assert pages == {1: "one", 2: "has ``` inside", 3: ""} and by == "ollama/x/m"
|
||||||
|
|
||||||
|
comp = tmp_path / "b.md"
|
||||||
|
comp.write_text("---\ntranscribed_by: ollama/y/m\n---\n\n## Notes\n\n"
|
||||||
|
+ companion.draft_block(["alpha", "beta"]) + "\n")
|
||||||
|
pages, by = scan_index.read_transcript(comp)
|
||||||
|
assert pages == {1: "alpha", 2: "beta"} and by == "ollama/y/m"
|
||||||
|
|
||||||
|
|
||||||
|
def test_prompt_has_kinds_note_extras_and_names():
|
||||||
|
p = scan_index.build_index_prompt(KINDS, ["Pat Doe"], ["Extra rule."])
|
||||||
|
assert '"calculation" for derivations' in p
|
||||||
|
assert p.index("Extra rule.") < p.index(ollama.NAMES_PROMPT) < p.index("- Pat Doe")
|
||||||
|
assert '"calculation"' not in scan_index.build_index_prompt(["idea", "other"], [], [])
|
||||||
|
|
||||||
|
|
||||||
|
def test_page_prompt_marks_this_page_and_neighbors():
|
||||||
|
pages = {1: "p1", 2: "p2" * 50, 3: "p3"}
|
||||||
|
p = scan_index.page_prompt("BASE", pages, 1, chars=10)
|
||||||
|
assert "Previous page transcript: (none)" in p
|
||||||
|
assert "THIS page's draft transcript:\n<<<\np1\n>>>" in p
|
||||||
|
p = scan_index.page_prompt("BASE", pages, 3, chars=10)
|
||||||
|
assert "<<<\n" + "p2" * 5 + "\n>>>" in p and "Next page transcript: (none)" in p
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_coerces_types_and_unknown_kind():
|
||||||
|
m = scan_index.clean({"summary": None, "kind": "derivation", "topics": "one",
|
||||||
|
"people": ["", " A "], "continues_to_next": "yes"}, KINDS)
|
||||||
|
assert m["summary"] == "" and m["kind"] == "other"
|
||||||
|
assert m["topics"] == ["one"] and m["people"] == ["A"]
|
||||||
|
assert m["continues_to_next"] is False and m["references"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_page_front_matter_and_sections():
|
||||||
|
meta = scan_index.clean(answer(references=['Smith, "x", J. Fluid Mech. 1 (2001)'],
|
||||||
|
continues_from_previous=True), KINDS)
|
||||||
|
text = scan_index.render_page("nb", "nb.pdf", 1, 3, meta, "a ``` b", "ollama/h/m",
|
||||||
|
"ollama/h/t", "2026-11-02")
|
||||||
|
assert text.startswith('---\nnotebook: "nb"\nscan: "nb.pdf"\npage: 1\nkind: "calculation"')
|
||||||
|
assert 'references: ["Smith, \\"x\\", J. Fluid Mech. 1 (2001)"]' in text
|
||||||
|
assert "continues_from: null\ncontinues_to: 2\n" in text # page 1 has no previous
|
||||||
|
assert "transcribed_by: ollama/h/t" in text
|
||||||
|
assert scan_index.SUMMARY_HEADING + "\n\nEstimate of a drag force." in text
|
||||||
|
assert "## Key points\n\n- F = 6 pi eta a U" in text
|
||||||
|
assert "````text\na ``` b\n````" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_writes_one_file_per_page(env, monkeypatch):
|
||||||
|
f = fake(monkeypatch)
|
||||||
|
pdf = make_scan(env / "nb")
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == 0, r.output
|
||||||
|
out = env / "nb" / "2009 notebook pages"
|
||||||
|
files = sorted(p.name for p in out.iterdir())
|
||||||
|
assert files == [f"2009 notebook p00{n}.md" for n in (1, 2, 3)]
|
||||||
|
p2 = (out / "2009 notebook p002.md").read_text()
|
||||||
|
assert "page: 2\n" in p2 and "continues_to: 3" in p2 and "second page" in p2
|
||||||
|
assert "indexed_by: ollama/localhost/gemma4:31b" in p2
|
||||||
|
assert "continues_to: null" in (out / "2009 notebook p003.md").read_text()
|
||||||
|
|
||||||
|
req = f.payloads[1]
|
||||||
|
assert req["format"]["properties"]["kind"]["enum"][0] == "calculation"
|
||||||
|
assert req["options"] == {"temperature": 0, "num_ctx": 8192, "num_predict": 1536}
|
||||||
|
prompt = req["messages"][0]["content"]
|
||||||
|
assert "- Global Person" in prompt and "- Board Member" in prompt
|
||||||
|
assert "Index-only instruction." in prompt
|
||||||
|
assert "<<<\nsecond page\n>>>" in prompt and "<<<\nfirst page\n>>>" in prompt
|
||||||
|
|
||||||
|
log = read_log(env / "stats.jsonl")
|
||||||
|
assert [x["task"] for x in log] == ["index"] * 3 and log[0]["kind"] == "calculation"
|
||||||
|
assert "summary" not in json.dumps(log) # no generated text in the log
|
||||||
|
assert "index: gemma4:31b on localhost" in summarize(log)
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_pages_are_skipped_unless_forced(env, monkeypatch):
|
||||||
|
f = fake(monkeypatch)
|
||||||
|
pdf = make_scan(env / "nb")
|
||||||
|
out = env / "nb" / "2009 notebook pages"
|
||||||
|
out.mkdir()
|
||||||
|
(out / "2009 notebook p001.md").write_text("kept")
|
||||||
|
assert run(pdf).exit_code == 0
|
||||||
|
assert len(f.payloads) == 2 and (out / "2009 notebook p001.md").read_text() == "kept"
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == 0 and "already indexed" in r.stderr and len(f.payloads) == 2
|
||||||
|
assert run("--force", pdf).exit_code == 0 and len(f.payloads) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_bad_json_is_retried(env, monkeypatch):
|
||||||
|
f = fake(monkeypatch, ["not json", {"summary": "missing the other keys"}])
|
||||||
|
pdf = make_scan(env / "nb", pages=("only",))
|
||||||
|
assert run(pdf).exit_code == 0
|
||||||
|
assert len(f.payloads) == 3
|
||||||
|
log = read_log(env / "stats.jsonl")
|
||||||
|
assert log[0]["retries"] == 2 and log[0]["retry_reasons"] == ["bad_json", "bad_json"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_page_reported_and_others_written(env, monkeypatch):
|
||||||
|
def err():
|
||||||
|
return urllib.error.HTTPError("u", 500, "x", {}, io.BytesIO(b'{"error":"token repeat limit"}'))
|
||||||
|
fake(monkeypatch, [err(), err(), err()])
|
||||||
|
pdf = make_scan(env / "nb", pages=("a", "b"))
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == scan_index.EXIT_TRANSCRIBE_FAILED
|
||||||
|
assert "1 page(s) failed (1)" in r.stderr
|
||||||
|
out = env / "nb" / "2009 notebook pages"
|
||||||
|
assert [p.name for p in out.iterdir()] == ["2009 notebook p002.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_page_count_mismatch_refused(env, monkeypatch):
|
||||||
|
fake(monkeypatch)
|
||||||
|
pdf = make_scan(env / "nb")
|
||||||
|
pdf.with_suffix(".md").write_text(to_plain_markdown(["a"], "x", "y"))
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == 2 and "1 transcribed pages" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_transcript_refused(env, monkeypatch):
|
||||||
|
fake(monkeypatch)
|
||||||
|
pdf = make_scan(env / "nb")
|
||||||
|
pdf.with_suffix(".md").unlink()
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == 2 and "run scan-notes-md first" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_host_exits_without_cloud(env, monkeypatch):
|
||||||
|
def down(*a):
|
||||||
|
raise urllib.error.URLError("refused")
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", down)
|
||||||
|
pdf = make_scan(env / "nb")
|
||||||
|
r = run(pdf)
|
||||||
|
assert r.exit_code == scan_index.EXIT_NO_HOST
|
||||||
|
assert "Not falling back to a cloud service" in r.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_index_config_rejects_unknown_keys(tmp_path, capsys):
|
||||||
|
cfg = tmp_path / "c.toml"
|
||||||
|
cfg.write_text("[index]\nkind = ['x']\n")
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
load_config(cfg)
|
||||||
|
assert "unknown key(s) in [index]: kind" in capsys.readouterr().err
|
||||||
|
cfg.write_text("[index]\nkinds = []\n")
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
load_config(cfg)
|
||||||
247
tests/test_ollama_hosts.py
Normal file
247
tests/test_ollama_hosts.py
Normal file
|
|
@ -0,0 +1,247 @@
|
||||||
|
"""Host selection, fallback, and response cleanup. All HTTP is faked."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from pdf_tools import ollama
|
||||||
|
from pdf_tools.scan_config import OllamaHost
|
||||||
|
|
||||||
|
REMOTE = OllamaHost("remote", "http://gpu-box:11434", "gemma4:31b")
|
||||||
|
LOCAL = OllamaHost("local", "http://localhost:11434", "gemma4:e4b")
|
||||||
|
|
||||||
|
|
||||||
|
def fake_http(tags: dict[str, list[str] | None], chat=None):
|
||||||
|
"""tags maps base URL -> model names, or None for an unreachable host."""
|
||||||
|
calls = []
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
calls.append((method, url, payload, timeout))
|
||||||
|
base, _, path = url.partition("/api/")
|
||||||
|
if path == "tags":
|
||||||
|
models = tags.get(base)
|
||||||
|
if models is None:
|
||||||
|
raise urllib.error.URLError("connection refused")
|
||||||
|
return {"models": [{"name": m} for m in models]}
|
||||||
|
if path == "chat":
|
||||||
|
return chat(payload) if chat else {"message": {"content": "ok"}}
|
||||||
|
raise AssertionError(url)
|
||||||
|
|
||||||
|
_fake.calls = calls
|
||||||
|
return _fake
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_used_when_available(monkeypatch, capsys):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({
|
||||||
|
REMOTE.url: ["gemma4:31b"], LOCAL.url: ["gemma4:e4b"]}))
|
||||||
|
assert ollama.select_host([REMOTE, LOCAL], 2.5) is REMOTE
|
||||||
|
assert capsys.readouterr().err == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_falls_back_when_remote_unreachable(monkeypatch, capsys):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({
|
||||||
|
REMOTE.url: None, LOCAL.url: ["gemma4:e4b"]}))
|
||||||
|
assert ollama.select_host([REMOTE, LOCAL], 2.5) is LOCAL
|
||||||
|
err = capsys.readouterr().err.strip().splitlines()
|
||||||
|
assert len(err) == 1
|
||||||
|
assert "unreachable" in err[0] and "falling back to local" in err[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_falls_back_when_remote_lacks_model(monkeypatch, capsys):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({
|
||||||
|
REMOTE.url: ["qwen3-vl:8b"], LOCAL.url: ["gemma4:e4b"]}))
|
||||||
|
assert ollama.select_host([REMOTE, LOCAL], 2.5) is LOCAL
|
||||||
|
assert "does not have gemma4:31b" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_probe_uses_short_timeout(monkeypatch):
|
||||||
|
fake = fake_http({REMOTE.url: ["gemma4:31b"]})
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake)
|
||||||
|
ollama.select_host([REMOTE, LOCAL], 2.5)
|
||||||
|
assert fake.calls == [("GET", f"{REMOTE.url}/api/tags", None, 2.5)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_host_raises(monkeypatch):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({}))
|
||||||
|
with pytest.raises(ollama.NoHostAvailable):
|
||||||
|
ollama.select_host([REMOTE, LOCAL], 2.5)
|
||||||
|
|
||||||
|
|
||||||
|
def test_forced_host_skips_others(monkeypatch):
|
||||||
|
fake = fake_http({REMOTE.url: ["gemma4:31b"], LOCAL.url: ["gemma4:e4b"]})
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake)
|
||||||
|
assert ollama.select_host([REMOTE, LOCAL], 2.5, forced="local") is LOCAL
|
||||||
|
assert [c[1] for c in fake.calls] == [f"{LOCAL.url}/api/tags"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_forced_host_unavailable_is_error_not_fallback(monkeypatch):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({
|
||||||
|
REMOTE.url: None, LOCAL.url: ["gemma4:e4b"]}))
|
||||||
|
with pytest.raises(ollama.NoHostAvailable, match="remote"):
|
||||||
|
ollama.select_host([REMOTE, LOCAL], 2.5, forced="remote")
|
||||||
|
|
||||||
|
|
||||||
|
def test_forced_host_unknown_name(monkeypatch):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({}))
|
||||||
|
with pytest.raises(ollama.NoHostAvailable, match="no Ollama host named"):
|
||||||
|
ollama.select_host([REMOTE, LOCAL], 2.5, forced="laptop")
|
||||||
|
|
||||||
|
|
||||||
|
def test_untagged_model_matches_latest(monkeypatch):
|
||||||
|
host = OllamaHost("x", "http://x:11434", "llama3.2")
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({host.url: ["llama3.2:latest"]}))
|
||||||
|
assert ollama.select_host([host], 1) is host
|
||||||
|
|
||||||
|
|
||||||
|
def test_chat_request_shape(monkeypatch):
|
||||||
|
seen = {}
|
||||||
|
|
||||||
|
def chat(payload):
|
||||||
|
seen.update(payload)
|
||||||
|
return {"message": {"content": "line one\n line two"}}
|
||||||
|
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", fake_http({}, chat))
|
||||||
|
out = ollama.transcribe_png(b"\x89PNG", REMOTE, "PROMPT", 300)
|
||||||
|
assert out.text == "line one\n line two"
|
||||||
|
assert seen["stream"] is False and seen["think"] is False
|
||||||
|
assert seen["options"]["temperature"] == 0
|
||||||
|
assert seen["model"] == "gemma4:31b"
|
||||||
|
assert len(seen["messages"]) == 1 and seen["messages"][0]["images"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_retries_without_think_when_rejected(monkeypatch):
|
||||||
|
payloads = []
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
payloads.append(dict(payload))
|
||||||
|
if "think" in payload:
|
||||||
|
raise urllib.error.HTTPError(
|
||||||
|
url, 400, "Bad Request", {},
|
||||||
|
io.BytesIO(json.dumps({"error": "model does not support thinking"}).encode()))
|
||||||
|
return {"message": {"content": "text"}}
|
||||||
|
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
assert ollama.transcribe_png(b"x", LOCAL, "P", 10).text == "text"
|
||||||
|
assert "think" in payloads[0] and "think" not in payloads[1]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("raw, clean", [
|
||||||
|
("<think>hmm, looks like a list</think>\nA\nB", "A\nB"),
|
||||||
|
("<thinking>x</thinking>A", "A"),
|
||||||
|
("reasoning without an opening tag</think>\nA", "A"),
|
||||||
|
("A\n - indented\nB\n", "A\n - indented\nB"),
|
||||||
|
])
|
||||||
|
def test_strip_thinking(raw, clean):
|
||||||
|
assert ollama.strip_thinking(raw) == clean
|
||||||
|
|
||||||
|
|
||||||
|
def test_names_added_only_when_configured():
|
||||||
|
assert ollama.build_prompt([]) == ollama.HANDWRITING_PROMPT
|
||||||
|
p = ollama.build_prompt(["Pat Example", "ACME Foundation"])
|
||||||
|
assert p.startswith(ollama.HANDWRITING_PROMPT)
|
||||||
|
assert "- Pat Example" in p and "otherwise write\n[illegible]" in p
|
||||||
|
|
||||||
|
|
||||||
|
def test_provenance():
|
||||||
|
assert ollama.provenance(REMOTE) == "ollama/gpu-box/gemma4:31b"
|
||||||
|
assert ollama.provenance(LOCAL) == "ollama/localhost/gemma4:e4b"
|
||||||
|
|
||||||
|
|
||||||
|
def test_extras_go_between_base_prompt_and_names():
|
||||||
|
p = ollama.build_prompt(["Pat"], ["Draw figures as [FIGURE: x].", " "])
|
||||||
|
assert p.startswith(ollama.HANDWRITING_PROMPT + "\nDraw figures as [FIGURE: x].\n\n")
|
||||||
|
assert p.index("[FIGURE") < p.index(ollama.NAMES_PROMPT) < p.index("- Pat")
|
||||||
|
assert ollama.build_prompt([], ["", "\n"]) == ollama.HANDWRITING_PROMPT
|
||||||
|
|
||||||
|
|
||||||
|
def test_page_stats_from_response():
|
||||||
|
st = ollama.page_stats({"prompt_eval_count": 1200, "eval_count": 300,
|
||||||
|
"done_reason": "stop", "total_duration": 9_500_000_000,
|
||||||
|
"load_duration": 0, "prompt_eval_duration": 1_000_000_000,
|
||||||
|
"eval_duration": 8_000_000_000})
|
||||||
|
assert st["context_tokens"] == 1500 and st["done_reason"] == "stop"
|
||||||
|
assert (st["total_s"], st["prompt_s"], st["output_s"]) == (9.5, 1.0, 8.0)
|
||||||
|
assert ollama.page_stats({})["context_tokens"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_loaded_model_reports_gpu_fit(monkeypatch):
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
assert url.endswith("/api/ps")
|
||||||
|
return {"models": [{"name": "gemma4:31b", "size": 21_000_000_000,
|
||||||
|
"size_vram": 19_740_000_000, "context_length": 32768}]}
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
got = ollama.loaded_model(REMOTE, 2)
|
||||||
|
assert got == {"loaded_bytes": 21_000_000_000, "vram_bytes": 19_740_000_000,
|
||||||
|
"gpu_fraction": 0.94, "loaded_num_ctx": 32768}
|
||||||
|
|
||||||
|
|
||||||
|
def test_loaded_model_unreachable_is_empty(monkeypatch):
|
||||||
|
def _down(*a):
|
||||||
|
raise urllib.error.URLError("connection refused")
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _down)
|
||||||
|
assert ollama.loaded_model(REMOTE, 2) == {}
|
||||||
|
|
||||||
|
|
||||||
|
def _repeat_error(url):
|
||||||
|
return urllib.error.HTTPError(
|
||||||
|
url, 500, "Internal Server Error", {},
|
||||||
|
io.BytesIO(b'{"error":"prediction aborted, token repeat limit reached"}'))
|
||||||
|
|
||||||
|
|
||||||
|
def test_looping_page_retried_with_sampling(monkeypatch):
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
seen.append(dict(payload["options"]))
|
||||||
|
if len(seen) == 1:
|
||||||
|
raise _repeat_error(url)
|
||||||
|
return {"message": {"content": "ok"}, "eval_count": 5}
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
out = ollama.transcribe_png(b"x", REMOTE, "P", 10)
|
||||||
|
assert out.text == "ok" and out.stats["retries"] == 1
|
||||||
|
assert out.stats["retry_reasons"] == ["repeat_limit"]
|
||||||
|
assert seen[0] == {"temperature": 0}
|
||||||
|
assert seen[1] == {"temperature": 0.3, "repeat_penalty": 1.15}
|
||||||
|
assert out.stats["retry_options"] == ollama.REPEAT_RETRIES[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_try_keeps_temperature_zero(monkeypatch):
|
||||||
|
seen = []
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", lambda m, u, p, t: seen.append(p["options"])
|
||||||
|
or {"message": {"content": "ok"}})
|
||||||
|
out = ollama.transcribe_png(b"x", REMOTE, "P", 10)
|
||||||
|
assert seen == [{"temperature": 0}] and out.stats["retries"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_repetition_error_after_all_retries(monkeypatch):
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
raise _repeat_error(url)
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
with pytest.raises(ollama.RepetitionError):
|
||||||
|
ollama.transcribe_png(b"x", REMOTE, "P", 10)
|
||||||
|
|
||||||
|
|
||||||
|
def test_runaway_output_capped_and_retried(monkeypatch):
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def _fake(method, url, payload, timeout):
|
||||||
|
seen.append(dict(payload["options"]))
|
||||||
|
if len(seen) == 1:
|
||||||
|
return {"message": {"content": "x x x"}, "done_reason": "length", "eval_count": 2048}
|
||||||
|
return {"message": {"content": "fine"}, "done_reason": "stop", "eval_count": 300}
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", _fake)
|
||||||
|
out = ollama.transcribe_png(b"x", REMOTE, "P", 10, max_output=2048)
|
||||||
|
assert seen[0] == {"temperature": 0, "num_predict": 2048}
|
||||||
|
assert out.text == "fine" and out.stats["retries"] == 1
|
||||||
|
assert out.stats["retry_reasons"] == ["output_cap"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_runaway_on_every_attempt_is_marked_cut_off(monkeypatch):
|
||||||
|
monkeypatch.setattr(ollama, "_http_json", lambda m, u, p, t: {
|
||||||
|
"message": {"content": "long"}, "done_reason": "length", "eval_count": 2048})
|
||||||
|
out = ollama.transcribe_png(b"x", REMOTE, "P", 10, max_output=2048)
|
||||||
|
assert out.text == "long\n" + ollama.CUT_OFF_MARKER
|
||||||
|
assert out.stats["retries"] == 2 and out.stats["done_reason"] == "length"
|
||||||
36
tests/test_stats.py
Normal file
36
tests/test_stats.py
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
"""The scan-notes-stats summary."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pdf_tools.scan_stats import StatsLog, read_log, summarize
|
||||||
|
|
||||||
|
|
||||||
|
def rec(ctx, run="r1", reason="stop", label="nb.pdf p.1"):
|
||||||
|
return {"run": run, "host": "gpu-box", "model": "gemma4:31b",
|
||||||
|
"loaded_num_ctx": 32768, "label": label, "image_px": [2000, 1500],
|
||||||
|
"prompt_tokens": ctx - 300, "output_tokens": 300, "context_tokens": ctx,
|
||||||
|
"done_reason": reason, "total_s": 10.0, "gpu_fraction": 0.94,
|
||||||
|
"vram_bytes": 19.7e9, "loaded_bytes": 21e9}
|
||||||
|
|
||||||
|
|
||||||
|
def test_threshold_counts_and_extremes():
|
||||||
|
rs = [rec(1500), rec(3000), rec(5000), rec(9000, reason="length", label="big p.9")]
|
||||||
|
out = summarize(rs, thresholds=(4096, 8192))
|
||||||
|
assert "4 pages, 1 run(s)" in out
|
||||||
|
assert "> 4096: 2 (50.0%)" in out and "> 8192: 1 (25.0%)" in out
|
||||||
|
assert "length limit: 1 page(s)" in out
|
||||||
|
assert "model looped: 0 page(s) retried, 0 left as placeholders" in out
|
||||||
|
assert "model in GPU memory: 94-94%" in out
|
||||||
|
assert out.index("big p.9") < out.index("nb.pdf p.1")
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_roundtrip(tmp_path):
|
||||||
|
log = StatsLog(tmp_path / "sub" / "s.jsonl")
|
||||||
|
log.write(rec(100))
|
||||||
|
log.write(rec(200))
|
||||||
|
(tmp_path / "sub" / "s.jsonl").open("a").write("not json\n")
|
||||||
|
assert [r["context_tokens"] for r in read_log(tmp_path / "sub" / "s.jsonl")] == [100, 200]
|
||||||
|
StatsLog(None).write(rec(1)) # disabled log is a no-op
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty():
|
||||||
|
assert summarize([]) == "no pages logged yet"
|
||||||
Loading…
Add table
Add a link
Reference in a new issue