Command-line tools for marked-up PDFs and scanned handwritten notes: pdf-scrub, pdf-notes-md, scan-notes-md (Ollama by default, Claude on request), scan-notes-index (per-page search metadata), and scan-notes-stats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011ZSvbNaLhgFuTm7wmkgbWs
142 lines
6.9 KiB
Markdown
142 lines
6.9 KiB
Markdown
# Ollama context size and GPU memory
|
|
|
|
What we have learned about how much context a handwritten page needs, how
|
|
the context size decides whether a model fits in GPU memory, and how to
|
|
measure both. Update the findings below as new runs add data.
|
|
|
|
## Why it matters
|
|
|
|
Ollama reserves memory for the model's context window (the KV cache) when it
|
|
loads the model. A larger `num_ctx` reserves more memory. If weights plus
|
|
cache do not fit in VRAM, Ollama puts part of the model on the CPU, and every
|
|
token gets slower. Too small a context has the opposite risk: the prompt,
|
|
which includes the page image, is truncated, or the output stops early.
|
|
|
|
One page uses: prompt text (about 300 tokens with `prompt_extra`) + image
|
|
tokens + output tokens. The context only needs to hold one page at a time,
|
|
because each page is a separate request.
|
|
|
|
## How to measure
|
|
|
|
Every page transcribed with the Ollama backend appends one line to the stats
|
|
log (`stats_log` in the config; default
|
|
`~/.local/state/pdf-tools/scan-stats.jsonl`). A line records the page label,
|
|
image size in pixels, prompt and output tokens, the stop reason, timings,
|
|
and how the model was loaded: total size, bytes in VRAM, and the context
|
|
length (from Ollama's `/api/ps`). It never records transcript text.
|
|
|
|
```bash
|
|
scan-notes-stats # summary of everything logged
|
|
scan-notes-stats --run last # most recent run only
|
|
scan-notes-stats --threshold 6144 --threshold 8192 --top 10
|
|
```
|
|
|
|
The summary reports, per model and loaded context size, token percentiles,
|
|
how many pages would need more than each threshold, pages whose output hit a
|
|
length limit, pages where the model looped, the share of the model in GPU
|
|
memory, and seconds per page.
|
|
|
|
During a run, `scan-notes-md` warns when a page's output stops at a length
|
|
limit or uses more than 90% of the loaded context.
|
|
|
|
To try a smaller context, set `num_ctx` on the host in the config:
|
|
|
|
```toml
|
|
[[ollama.hosts]]
|
|
name = "local"
|
|
url = "http://localhost:11434"
|
|
model = "gemma4:31b"
|
|
num_ctx = 8192
|
|
```
|
|
|
|
Changing `num_ctx` does not invalidate saved progress, so an interrupted
|
|
notebook resumes. Check `ollama ps` afterwards: the PROCESSOR column should
|
|
read `100% GPU`.
|
|
|
|
## Findings
|
|
|
|
### 2026-11-01: gemma4:31b on an RTX 4090 (24 GB), Ollama 0.40.13
|
|
|
|
- **Default context.** With no `num_ctx` set, Ollama loaded the model with a
|
|
32,768-token context: 21.2 GB in all, 20.0 GB in VRAM, so `ollama ps`
|
|
showed 6%/94% CPU/GPU. The card was nearly full (23.4 of 24.6 GB).
|
|
- **First pages.** On the first 10 logged pages (2200 x 1700 to 2339 x 1653
|
|
px at 200 dpi), prompts were 768 to 787 tokens (base prompt,
|
|
`prompt_extra`, and image) and outputs 351 to 462 tokens; the largest page
|
|
used 1,230 tokens, under 4% of the loaded context. The image accounts for
|
|
most of the prompt and varies little with page size. Pages averaged 15 s,
|
|
so a 2,562-page batch takes about 11 hours at this setting.
|
|
- **Looping.** Some pages make the model repeat itself until Ollama aborts
|
|
the request with `prediction aborted, token repeat limit reached`. At
|
|
temperature 0 the same page usually fails the same way: one page failed
|
|
in two separate runs, though it went through on a third, because GPU
|
|
inference at temperature 0 is not perfectly deterministic. It happened on
|
|
2 of the first 70 pages. `scan-notes-md` now retries
|
|
such a page with a little sampling (temperature 0.3, then 0.6, with a
|
|
repeat penalty) and, if it still loops, writes a visible placeholder:
|
|
`[page not transcribed: the model kept repeating itself; see scan]`.
|
|
`scan-notes-stats` counts retried pages and placeholders.
|
|
- **Runaway output.** A model can also loop without tripping the repeat
|
|
check. Ollama's default output limit is unlimited, so such a page ran
|
|
past six minutes and would have failed at the 600 s timeout on every
|
|
rerun. `max_output_tokens` (default 2048, passed as `num_predict`) now
|
|
stops it early. A page that hits the cap is retried like a looping page;
|
|
if it hits the cap every time, its text is kept and marked
|
|
`[transcript cut off at the output limit; see scan]`. The output cap also
|
|
bounds the context a page can use: prompt plus at most 2,048 output
|
|
tokens.
|
|
|
|
### 2026-11-02: full run, 40 notebooks, same setup
|
|
|
|
The stats log covers 2,494 of the 2,562 pages; the first notebook (68 pages)
|
|
was transcribed before logging was added.
|
|
|
|
| Measure | mean | p50 | p90 | p99 | max |
|
|
|---|---|---|---|---|---|
|
|
| Prompt tokens (incl. image) | 781 | 787 | 787 | 787 | 797 |
|
|
| Output tokens | 278 | 278 | 453 | 674 | 1,125 |
|
|
| Context used | 1,059 | 1,058 | 1,232 | 1,459 | 1,912 |
|
|
|
|
- **Context.** No page needed more than 1,912 tokens, under 6% of the
|
|
loaded 32,768 and under half of 4,096. No page reached the 2,048-token
|
|
output cap on its final attempt. The prompt is nearly constant (768 to 797
|
|
tokens), which suggests the model encodes each image into a fixed number of tokens; page size
|
|
and resolution barely matter; context use is set by how much is written
|
|
on the page. Even a page that hits the output cap needs only about 2,850
|
|
tokens, so 4,096 would hold every page with this prompt.
|
|
- **GPU fit.** The model stayed at 94% GPU (21.2 GB loaded, 20.0 GB in VRAM)
|
|
for the whole run; the default context is what keeps it off the GPU
|
|
entirely.
|
|
- **Speed.** 11.1 s per page on average (p90 17.2 s, max 40.9 s), 7.7 h of
|
|
model time for the logged pages. Wall-clock time for the batch was about
|
|
8 h. Blank and nearly blank pages take 2 to 4 s.
|
|
- **Retries.** 65 pages (2.6%) were retried: 53 after Ollama's repeat
|
|
limit, 7 after reaching the output cap, and 5 before reasons were logged.
|
|
Every one finished normally on its first retry (temperature 0.3, repeat
|
|
penalty 1.15), with 67 to 888 output tokens, so these were loops rather
|
|
than long pages. No page was left as a placeholder or marked cut off. The
|
|
repeat penalty can also discourage repetition that is really on the page
|
|
(the same value down a column), so retried pages are the ones most worth
|
|
checking against the scan; they are the log lines with `"retries": 1`.
|
|
|
|
### 2026-11-02: 8192 context compared with 32768, one notebook
|
|
|
|
The same 85-page notebook, same prompt and settings except
|
|
`num_ctx = 8192`:
|
|
|
|
| | 32,768 (default) | 8,192 |
|
|
|---|---|---|
|
|
| Loaded size, GPU share | 21.2 GB, 94% GPU | 20 GB, 100% GPU |
|
|
| Seconds per page (mean) | 11.6 | 8.3 |
|
|
| Total for the notebook | 16.4 min | 11.8 min |
|
|
| Prompt time per page | 0.84 s | 0.69 s |
|
|
| Output speed | 28.9 tokens/s | 42.7 tokens/s |
|
|
| Output tokens (all pages) | 24,777 | 25,092 |
|
|
| Pages retried | 2 | 3 |
|
|
|
|
At 8192 the whole model fits on the GPU, and output runs about 48% faster,
|
|
so pages take about 28% less time. The two runs wrote nearly the same amount
|
|
of text, so the gain is not from shorter output. The largest page in the full
|
|
run used 1,912 tokens, so 8192 leaves a wide margin. Recommendation: set
|
|
`num_ctx = 8192` for gemma4:31b on a 24 GB card. At this speed the 40-notebook
|
|
batch would take about 6 hours instead of 8.
|