pdf-tools/docs/ollama-context-and-gpu.md
Eric Furst e464871b69 Initial public release of pdf-tools
Command-line tools for marked-up PDFs and scanned handwritten notes:
pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on
request), scan-notes-index (per-page search metadata), and
scan-notes-stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
2026-11-03 07:25:51 -05:00

6.9 KiB

Ollama context size and GPU memory

What we have learned about how much context a handwritten page needs, how the context size decides whether a model fits in GPU memory, and how to measure both. Update the findings below as new runs add data.

Why it matters

Ollama reserves memory for the model's context window (the KV cache) when it loads the model. A larger num_ctx reserves more memory. If weights plus cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every token gets slower. Too small a context has the opposite risk: the prompt, which includes the page image, is truncated, or the output stops early.

One page uses: prompt text (about 300 tokens with prompt_extra) + image tokens + output tokens. The context only needs to hold one page at a time, because each page is a separate request.

How to measure

Every page transcribed with the Ollama backend appends one line to the stats log (stats_log in the config; default ~/.local/state/pdf-tools/scan-stats.jsonl). A line records the page label, image size in pixels, prompt and output tokens, the stop reason, timings, and how the model was loaded: total size, bytes in VRAM, and the context length (from Ollama's /api/ps). It never records transcript text.

scan-notes-stats                       # summary of everything logged
scan-notes-stats --run last            # most recent run only
scan-notes-stats --threshold 6144 --threshold 8192 --top 10

The summary reports, per model and loaded context size, token percentiles, how many pages would need more than each threshold, pages whose output hit a length limit, pages where the model looped, the share of the model in GPU memory, and seconds per page.

During a run, scan-notes-md warns when a page's output stops at a length limit or uses more than 90% of the loaded context.

To try a smaller context, set num_ctx on the host in the config:

[[ollama.hosts]]
name = "local"
url = "http://localhost:11434"
model = "gemma4:31b"
num_ctx = 8192

Changing num_ctx does not invalidate saved progress, so an interrupted notebook resumes. Check ollama ps afterwards: the PROCESSOR column should read 100% GPU.

Findings

2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13

  • Default context. With no num_ctx set, Ollama loaded the model with a 32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, so ollama ps showed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB).
  • First pages. On the first 10 logged pages (2200 x 1700 to 2339 x 1653 px at 200 dpi), prompts were 768 to 787 tokens (base prompt, prompt_extra, and image) and outputs 351 to 462 tokens; the largest page used 1,230 tokens, under 4% of the loaded context. The image accounts for most of the prompt and varies little with page size. Pages averaged 15 s, so a 2,562-page batch takes about 11 hours at this setting.
  • Looping. Some pages make the model repeat itself until Ollama aborts the request with prediction aborted, token repeat limit reached. At temperature 0 the same page usually fails the same way: one page failed in two separate runs, though it went through on a third, because GPU inference at temperature 0 is not perfectly deterministic. It happened on 2 of the first 70 pages. scan-notes-md now retries such a page with a little sampling (temperature 0.3, then 0.6, with a repeat penalty) and, if it still loops, writes a visible placeholder: [page not transcribed: the model kept repeating itself; see scan]. scan-notes-stats counts retried pages and placeholders.
  • Runaway output. A model can also loop without tripping the repeat check. Ollama's default output limit is unlimited, so such a page ran past six minutes and would have failed at the 600 s timeout on every rerun. max_output_tokens (default 2048, passed as num_predict) now stops it early. A page that hits the cap is retried like a looping page; if it hits the cap every time, its text is kept and marked [transcript cut off at the output limit; see scan]. The output cap also bounds the context a page can use: prompt plus at most 2,048 output tokens.

2026-11-02: full run, 40 notebooks, same setup

The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages) was transcribed before logging was added.

Measure mean p50 p90 p99 max
Prompt tokens (incl. image) 781 787 787 787 797
Output tokens 278 278 453 674 1,125
Context used 1,059 1,058 1,232 1,459 1,912
  • Context. No page needed more than 1,912 tokens, under 6% of the loaded 32,768 and under half of 4,096. No page reached the 2,048-token output cap on its final attempt. The prompt is nearly constant (768 to 797 tokens), which suggests the model encodes each image into a fixed number of tokens; page size and resolution barely matter; context use is set by how much is written on the page. Even a page that hits the output cap needs only about 2,850 tokens, so 4,096 would hold every page with this prompt.
  • GPU fit. The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM) for the whole run; the default context is what keeps it off the GPU entirely.
  • Speed. 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of model time for the logged pages. Wall-clock time for the batch was about 8 h. Blank and nearly blank pages take 2 to 4 s.
  • Retries. 65 pages (2.6%) were retried: 53 after Ollama's repeat limit, 7 after reaching the output cap, and 5 before reasons were logged. Every one finished normally on its first retry (temperature 0.3, repeat penalty 1.15), with 67 to 888 output tokens, so these were loops rather than long pages. No page was left as a placeholder or marked cut off. The repeat penalty can also discourage repetition that is really on the page (the same value down a column), so retried pages are the ones most worth checking against the scan; they are the log lines with "retries": 1.

2026-11-02: 8192 context compared with 32768, one notebook

The same 85-page notebook, same prompt and settings except num_ctx = 8192:

32,768 (default) 8,192
Loaded size, GPU share 21.2 GB, 94% GPU 20 GB, 100% GPU
Seconds per page (mean) 11.6 8.3
Total for the notebook 16.4 min 11.8 min
Prompt time per page 0.84 s 0.69 s
Output speed 28.9 tokens/s 42.7 tokens/s
Output tokens (all pages) 24,777 25,092
Pages retried 2 3

At 8192 the whole model fits on the GPU, and output runs about 48% faster, so pages take about 28% less time. The two runs wrote nearly the same amount of text, so the gain is not from shorter output. The largest page in the full run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set num_ctx = 8192 for gemma4:31b on a 24 GB card. At this speed the 40-notebook batch would take about 6 hours instead of 8.