Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
This commit is contained in:
commit
e464871b69
29 changed files with 5296 additions and 0 deletions
142
docs/ollama-context-and-gpu.md
Normal file
142
docs/ollama-context-and-gpu.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# Ollama context size and GPU memory
|
||||
|
||||
What we have learned about how much context a handwritten page needs, how
|
||||
the context size decides whether a model fits in GPU memory, and how to
|
||||
measure both. Update the findings below as new runs add data.
|
||||
|
||||
## Why it matters
|
||||
|
||||
Ollama reserves memory for the model's context window (the KV cache) when it
|
||||
loads the model. A larger `num_ctx` reserves more memory. If weights plus
|
||||
cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every
|
||||
token gets slower. Too small a context has the opposite risk: the prompt,
|
||||
which includes the page image, is truncated, or the output stops early.
|
||||
|
||||
One page uses: prompt text (about 300 tokens with `prompt_extra`) + image
|
||||
tokens + output tokens. The context only needs to hold one page at a time,
|
||||
because each page is a separate request.
|
||||
|
||||
## How to measure
|
||||
|
||||
Every page transcribed with the Ollama backend appends one line to the stats
|
||||
log (`stats_log` in the config; default
|
||||
`~/.local/state/pdf-tools/scan-stats.jsonl`). A line records the page label,
|
||||
image size in pixels, prompt and output tokens, the stop reason, timings,
|
||||
and how the model was loaded: total size, bytes in VRAM, and the context
|
||||
length (from Ollama's `/api/ps`). It never records transcript text.
|
||||
|
||||
```bash
|
||||
scan-notes-stats # summary of everything logged
|
||||
scan-notes-stats --run last # most recent run only
|
||||
scan-notes-stats --threshold 6144 --threshold 8192 --top 10
|
||||
```
|
||||
|
||||
The summary reports, per model and loaded context size, token percentiles,
|
||||
how many pages would need more than each threshold, pages whose output hit a
|
||||
length limit, pages where the model looped, the share of the model in GPU
|
||||
memory, and seconds per page.
|
||||
|
||||
During a run, `scan-notes-md` warns when a page's output stops at a length
|
||||
limit or uses more than 90% of the loaded context.
|
||||
|
||||
To try a smaller context, set `num_ctx` on the host in the config:
|
||||
|
||||
```toml
|
||||
[[ollama.hosts]]
|
||||
name = "local"
|
||||
url = "http://localhost:11434"
|
||||
model = "gemma4:31b"
|
||||
num_ctx = 8192
|
||||
```
|
||||
|
||||
Changing `num_ctx` does not invalidate saved progress, so an interrupted
|
||||
notebook resumes. Check `ollama ps` afterwards: the PROCESSOR column should
|
||||
read `100% GPU`.
|
||||
|
||||
## Findings
|
||||
|
||||
### 2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13
|
||||
|
||||
- **Default context.** With no `num_ctx` set, Ollama loaded the model with a
|
||||
32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, so `ollama ps`
|
||||
showed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB).
|
||||
- **First pages.** On the first 10 logged pages (2200 x 1700 to 2339 x 1653
|
||||
px at 200 dpi), prompts were 768 to 787 tokens (base prompt,
|
||||
`prompt_extra`, and image) and outputs 351 to 462 tokens; the largest page
|
||||
used 1,230 tokens, under 4% of the loaded context. The image accounts for
|
||||
most of the prompt and varies little with page size. Pages averaged 15 s,
|
||||
so a 2,562-page batch takes about 11 hours at this setting.
|
||||
- **Looping.** Some pages make the model repeat itself until Ollama aborts
|
||||
the request with `prediction aborted, token repeat limit reached`. At
|
||||
temperature 0 the same page usually fails the same way: one page failed
|
||||
in two separate runs, though it went through on a third, because GPU
|
||||
inference at temperature 0 is not perfectly deterministic. It happened on
|
||||
2 of the first 70 pages. `scan-notes-md` now retries
|
||||
such a page with a little sampling (temperature 0.3, then 0.6, with a
|
||||
repeat penalty) and, if it still loops, writes a visible placeholder:
|
||||
`[page not transcribed: the model kept repeating itself; see scan]`.
|
||||
`scan-notes-stats` counts retried pages and placeholders.
|
||||
- **Runaway output.** A model can also loop without tripping the repeat
|
||||
check. Ollama's default output limit is unlimited, so such a page ran
|
||||
past six minutes and would have failed at the 600 s timeout on every
|
||||
rerun. `max_output_tokens` (default 2048, passed as `num_predict`) now
|
||||
stops it early. A page that hits the cap is retried like a looping page;
|
||||
if it hits the cap every time, its text is kept and marked
|
||||
`[transcript cut off at the output limit; see scan]`. The output cap also
|
||||
bounds the context a page can use: prompt plus at most 2,048 output
|
||||
tokens.
|
||||
|
||||
### 2026-11-02: full run, 40 notebooks, same setup
|
||||
|
||||
The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages)
|
||||
was transcribed before logging was added.
|
||||
|
||||
| Measure | mean | p50 | p90 | p99 | max |
|
||||
|---|---|---|---|---|---|
|
||||
| Prompt tokens (incl. image) | 781 | 787 | 787 | 787 | 797 |
|
||||
| Output tokens | 278 | 278 | 453 | 674 | 1,125 |
|
||||
| Context used | 1,059 | 1,058 | 1,232 | 1,459 | 1,912 |
|
||||
|
||||
- **Context.** No page needed more than 1,912 tokens, under 6% of the
|
||||
loaded 32,768 and under half of 4,096. No page reached the 2,048-token
|
||||
output cap on its final attempt. The prompt is nearly constant (768 to 797
|
||||
tokens), which suggests the model encodes each image into a fixed number of tokens; page size
|
||||
and resolution barely matter; context use is set by how much is written
|
||||
on the page. Even a page that hits the output cap needs only about 2,850
|
||||
tokens, so 4,096 would hold every page with this prompt.
|
||||
- **GPU fit.** The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM)
|
||||
for the whole run; the default context is what keeps it off the GPU
|
||||
entirely.
|
||||
- **Speed.** 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of
|
||||
model time for the logged pages. Wall-clock time for the batch was about
|
||||
8 h. Blank and nearly blank pages take 2 to 4 s.
|
||||
- **Retries.** 65 pages (2.6%) were retried: 53 after Ollama's repeat
|
||||
limit, 7 after reaching the output cap, and 5 before reasons were logged.
|
||||
Every one finished normally on its first retry (temperature 0.3, repeat
|
||||
penalty 1.15), with 67 to 888 output tokens, so these were loops rather
|
||||
than long pages. No page was left as a placeholder or marked cut off. The
|
||||
repeat penalty can also discourage repetition that is really on the page
|
||||
(the same value down a column), so retried pages are the ones most worth
|
||||
checking against the scan; they are the log lines with `"retries": 1`.
|
||||
|
||||
### 2026-11-02: 8192 context compared with 32768, one notebook
|
||||
|
||||
The same 85-page notebook, same prompt and settings except
|
||||
`num_ctx = 8192`:
|
||||
|
||||
| | 32,768 (default) | 8,192 |
|
||||
|---|---|---|
|
||||
| Loaded size, GPU share | 21.2 GB, 94% GPU | 20 GB, 100% GPU |
|
||||
| Seconds per page (mean) | 11.6 | 8.3 |
|
||||
| Total for the notebook | 16.4 min | 11.8 min |
|
||||
| Prompt time per page | 0.84 s | 0.69 s |
|
||||
| Output speed | 28.9 tokens/s | 42.7 tokens/s |
|
||||
| Output tokens (all pages) | 24,777 | 25,092 |
|
||||
| Pages retried | 2 | 3 |
|
||||
|
||||
At 8192 the whole model fits on the GPU, and output runs about 48% faster,
|
||||
so pages take about 28% less time. The two runs wrote nearly the same amount
|
||||
of text, so the gain is not from shorter output. The largest page in the full
|
||||
run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set
|
||||
`num_ctx = 8192` for gemma4:31b on a 24 GB card. At this speed the 40-notebook
|
||||
batch would take about 6 hours instead of 8.
|
||||
Loading…
Add table
Add a link
Reference in a new issue