Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
6.9 KiB
Ollama context size and GPU memory
What we have learned about how much context a handwritten page needs, how the context size decides whether a model fits in GPU memory, and how to measure both. Update the findings below as new runs add data.
Why it matters
Ollama reserves memory for the model's context window (the KV cache) when it
loads the model. A larger num_ctx reserves more memory. If weights plus
cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every
token gets slower. Too small a context has the opposite risk: the prompt,
which includes the page image, is truncated, or the output stops early.
One page uses: prompt text (about 300 tokens with prompt_extra) + image
tokens + output tokens. The context only needs to hold one page at a time,
because each page is a separate request.
How to measure
Every page transcribed with the Ollama backend appends one line to the stats
log (stats_log in the config; default
~/.local/state/pdf-tools/scan-stats.jsonl). A line records the page label,
image size in pixels, prompt and output tokens, the stop reason, timings,
and how the model was loaded: total size, bytes in VRAM, and the context
length (from Ollama's /api/ps). It never records transcript text.
scan-notes-stats # summary of everything logged
scan-notes-stats --run last # most recent run only
scan-notes-stats --threshold 6144 --threshold 8192 --top 10
The summary reports, per model and loaded context size, token percentiles, how many pages would need more than each threshold, pages whose output hit a length limit, pages where the model looped, the share of the model in GPU memory, and seconds per page.
During a run, scan-notes-md warns when a page's output stops at a length
limit or uses more than 90% of the loaded context.
To try a smaller context, set num_ctx on the host in the config:
[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:31b"
num_ctx = 8192
Changing num_ctx does not invalidate saved progress, so an interrupted
notebook resumes. Check ollama ps afterwards: the PROCESSOR column should
read 100% GPU.
Findings
2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13
- Default context. With no
num_ctxset, Ollama loaded the model with a 32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, soollama psshowed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB). - First pages. On the first 10 logged pages (2200 x 1700 to 2339 x 1653
px at 200 dpi), prompts were 768 to 787 tokens (base prompt,
prompt_extra, and image) and outputs 351 to 462 tokens; the largest page used 1,230 tokens, under 4% of the loaded context. The image accounts for most of the prompt and varies little with page size. Pages averaged 15 s, so a 2,562-page batch takes about 11 hours at this setting. - Looping. Some pages make the model repeat itself until Ollama aborts
the request with
prediction aborted, token repeat limit reached. At temperature 0 the same page usually fails the same way: one page failed in two separate runs, though it went through on a third, because GPU inference at temperature 0 is not perfectly deterministic. It happened on 2 of the first 70 pages.scan-notes-mdnow retries such a page with a little sampling (temperature 0.3, then 0.6, with a repeat penalty) and, if it still loops, writes a visible placeholder:[page not transcribed: the model kept repeating itself; see scan].scan-notes-statscounts retried pages and placeholders. - Runaway output. A model can also loop without tripping the repeat
check. Ollama's default output limit is unlimited, so such a page ran
past six minutes and would have failed at the 600 s timeout on every
rerun.
max_output_tokens(default 2048, passed asnum_predict) now stops it early. A page that hits the cap is retried like a looping page; if it hits the cap every time, its text is kept and marked[transcript cut off at the output limit; see scan]. The output cap also bounds the context a page can use: prompt plus at most 2,048 output tokens.
2026-11-02: full run, 40 notebooks, same setup
The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages) was transcribed before logging was added.
| Measure | mean | p50 | p90 | p99 | max |
|---|---|---|---|---|---|
| Prompt tokens (incl. image) | 781 | 787 | 787 | 787 | 797 |
| Output tokens | 278 | 278 | 453 | 674 | 1,125 |
| Context used | 1,059 | 1,058 | 1,232 | 1,459 | 1,912 |
- Context. No page needed more than 1,912 tokens, under 6% of the loaded 32,768 and under half of 4,096. No page reached the 2,048-token output cap on its final attempt. The prompt is nearly constant (768 to 797 tokens), which suggests the model encodes each image into a fixed number of tokens; page size and resolution barely matter; context use is set by how much is written on the page. Even a page that hits the output cap needs only about 2,850 tokens, so 4,096 would hold every page with this prompt.
- GPU fit. The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM) for the whole run; the default context is what keeps it off the GPU entirely.
- Speed. 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of model time for the logged pages. Wall-clock time for the batch was about 8 h. Blank and nearly blank pages take 2 to 4 s.
- Retries. 65 pages (2.6%) were retried: 53 after Ollama's repeat
limit, 7 after reaching the output cap, and 5 before reasons were logged.
Every one finished normally on its first retry (temperature 0.3, repeat
penalty 1.15), with 67 to 888 output tokens, so these were loops rather
than long pages. No page was left as a placeholder or marked cut off. The
repeat penalty can also discourage repetition that is really on the page
(the same value down a column), so retried pages are the ones most worth
checking against the scan; they are the log lines with
"retries": 1.
2026-11-02: 8192 context compared with 32768, one notebook
The same 85-page notebook, same prompt and settings except
num_ctx = 8192:
| 32,768 (default) | 8,192 | |
|---|---|---|
| Loaded size, GPU share | 21.2 GB, 94% GPU | 20 GB, 100% GPU |
| Seconds per page (mean) | 11.6 | 8.3 |
| Total for the notebook | 16.4 min | 11.8 min |
| Prompt time per page | 0.84 s | 0.69 s |
| Output speed | 28.9 tokens/s | 42.7 tokens/s |
| Output tokens (all pages) | 24,777 | 25,092 |
| Pages retried | 2 | 3 |
At 8192 the whole model fits on the GPU, and output runs about 48% faster,
so pages take about 28% less time. The two runs wrote nearly the same amount
of text, so the gain is not from shorter output. The largest page in the full
run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set
num_ctx = 8192 for gemma4:31b on a 24 GB card. At this speed the 40-notebook
batch would take about 6 hours instead of 8.