furst/art
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Art 0.2.0

Browse source 
Built from art-dev 2da005a.
This commit is contained in:
Eric Furst 2026-10-25 07:54:09 -04:00
commit d7e264bcf2
15 changed files with 1344 additions and 53 deletions
Download patch file Download diff file Expand all files Collapse all files

71
README.md
View file

@ -2,32 +2,33 @@
Art is a small, locally run LLM toolchain for rheology research. It accompanies the talk "Art: An LLM toolchain for rheology research" (Society of Rheology Annual Meeting, Boston, October 2026, session "AI and ML in Rheology"), and is meant as a starting point that other rheologists can read in one sitting, adopt, and adapt.
Everything runs on your own machine: embedding and re-ranking models are cached locally from HuggingFace, and text generation (coming in the next stage) goes to [Ollama](https://ollama.com) on `localhost`. No cloud APIs.
Everything runs on your own machine: embedding and re-ranking models are cached locally from HuggingFace, and text generation goes to [Ollama](https://ollama.com) on `localhost`. No cloud APIs are used.
Art has two parts:
1. **bingham**, a literature screener. It pulls new abstracts from the *Journal of Rheology* and arXiv, screens them against your interest profiles, and writes a daily Markdown digest of the papers worth reading. A Bingham plastic does not flow until the stress exceeds a yield threshold; most abstracts stay put, and only those that clear a calibrated threshold come through.
2. **Corpus search** (planned), hybrid retrieval over a locally indexed collection of rheology papers, with raw-evidence inspection and grounded synthesis with citations.
1. **bingham** is a literature screener. It pulls new abstracts from arXiv, the *Journal of Rheology*, and any other journals you switch on, screens them against your interest profiles, and writes a daily Markdown digest of the papers worth reading, each with a short summary written by a local LLM. A Bingham plastic does not flow until the stress exceeds a yield threshold; most abstracts stay put, and only those that clear a calibrated threshold come through.
2. **Corpus search** (planned) is a hybrid retrieval over a locally indexed collection of rheology papers, with raw-evidence inspection and grounded synthesis with citations.
## Status
bingham fetches, screens, and writes digests. The LLM passes (a short summary, a topic label, and a one-sentence match reason for each highlighted paper) are next; until then, each digest entry shows the opening sentences of the abstract and the screening scores. Thresholds are provisional until calibrated on a hand-labeled set.
bingham runs daily: it fetches, screens, summarizes the papers that pass, and writes the digest. Thresholds are provisional until calibrated on a hand-labeled set (`bingham label` and `bingham eval`, below). Still to come: a failure-mode report over the logged LLM calls, and corpus search.
## How bingham works
```
fetch -> store -> embed -> screen -> re-rank -> digest
(code) (SQLite) (bge) (cosine (cross- (code)
fetch -> store -> embed -> screen -> re-rank -> summarize -> label -> explain -> digest
(code) (SQLite) (bge) (cosine (cross- (LLM) (LLM) (LLM) (code)
+ BM25) encoder)
```
- **Fetch.** arXiv announcements from the per-category RSS feeds at `rss.arxiv.org`; *J. Rheol.* from Crossref (by ISSN) merged by DOI with the AIP current-issue feeds, which carry abstracts but no authors. Backfills and skipped days come from arXiv's OAI-PMH interface.
- **Fetch.** arXiv announcements from the per-category RSS feeds at `rss.arxiv.org`; *J. Rheol.* from Crossref (by ISSN) merged by DOI with the AIP current-issue feeds, which carry abstracts but no authors. Backfills and skipped days come from arXiv's OAI-PMH interface. Other journals come from their RSS feeds or from Crossref by ISSN, whichever carries abstracts (see "Journals" below).
- **Store.** One SQLite file holds papers, embeddings, scores, and a record of every run, so no stage is recomputed unless its model or profile has changed.
- **Screen.** Each paper (title and abstract) is embedded once with `BAAI/bge-large-en-v1.5`. For each profile it gets a cosine similarity (the best match over the profile description and its exemplar papers) and a BM25 score against the profile's keywords, which catches abbreviations such as SAOS, LAOS, and DST and names that embeddings blur. The top candidates by either score, plus any paper by a tracked author, go on.
- **Re-rank.** `BAAI/bge-reranker-v2-m3` reads each (profile description, paper) pair together and scores it from 0 to 1. A paper is highlighted when its score clears the profile's threshold.
- **Digest.** One Markdown file per day in `digests/`, grouped by profile and ordered by score, with a footer recording counts, models, host, device, and time per stage.
- **Summarize, label, explain.** For each highlighted paper only, a local model (`gemma4:e4b` through Ollama, with the researcher-tone system prompt in `modelfiles/researcher.Modelfile`) makes three separate calls: a two-to-four-sentence summary of the abstract, a short topic label, and one sentence on why the paper matches the profile. Each call does one task; prompts are versioned text files in `prompts/`. Every call sets `num_ctx` explicitly and is logged in the store with the model tag, Modelfile hash, Ollama version, prompt version, token counts, latency, and the raw response.
- **Digest.** One Markdown file per day in `digests/`, grouped by profile and ordered by score. Each entry gives the title, authors, source, summary, label, match reason, and link. A "Just below threshold" section lists near misses, and a field-pulse section groups papers by their topic labels. The footer records counts, models, prompt versions, host, device, and time per stage. If Ollama is not running, the digest is still written, with the opening sentences of each abstract in place of summaries.
Retrieval models decide relevance; the LLM, when added, only writes about papers that have already passed the screen, and never produces identifiers (DOIs, arXiv IDs, titles, authors, and dates come straight from the feed metadata).
Retrieval models decide relevance; the LLM only writes about papers that have already passed the screen, and never produces identifiers (DOIs, arXiv IDs, titles, authors, and dates come straight from the feed metadata).
## Getting started
@ -36,18 +37,19 @@ Retrieval models decide relevance; the LLM, when added, only writes about papers
- Python 3.12 and [uv](https://docs.astral.sh/uv/)
- About 3.5 GB of disk for the two HuggingFace models
- A GPU is optional. Tested on an NVIDIA RTX 4090 (CUDA) under Linux and an Apple M3 Pro (MPS) under macOS; the device is chosen automatically (`cuda`, then `mps`, then `cpu`).
- [Ollama](https://ollama.com), for the LLM passes once they are added (not needed yet)
- [Ollama](https://ollama.com) and the `gemma4:e4b` model (about 10 GB) for the summaries. Without Ollama, or with `--no-llm`, digests are written from the screen alone.
### Install
```
git clone https://lem.che.udel.edu/git/furst/art.git
git clone https://github.com/emfurst/art.git # or https://lem.che.udel.edu/git/furst/art.git
cd art
uv sync
uv run bingham download-models # once, with network access
ollama pull gemma4:e4b
```
`download-models` fetches the embedder and re-ranker into `./models/`. Every other command runs with HuggingFace in offline mode.
`download-models` fetches the embedder and re-ranker into `./models/`. Every other command runs with HuggingFace in offline mode. The first LLM call builds the researcher model from `modelfiles/researcher.Modelfile` and tags it with the file's hash (`art-researcher:<hash>`), so editing the Modelfile creates a new tag and every logged call records which version wrote it.
On Linux with an NVIDIA GPU, use a driver from the 580 series or later (CUDA 13). With older drivers, the standard PyTorch wheels cannot use the GPU, and Ollama 0.34 falls back to the CPU without an error (it still reports "100% GPU" in `ollama ps`, while generation slows to minutes per call). If you cannot update the driver, route `torch` to the CUDA 12.6 index in `pyproject.toml` (`[tool.uv.sources]`, index `https://download.pytorch.org/whl/cu126`) and use an Ollama release that supports your driver.
@ -77,6 +79,20 @@ uv run bingham topic "shear thickening and frictional contacts" [-n 15]
- **`topic`** ranks every stored paper against a free-text query, with no profile and no LLM.
- Raw feed responses are cached for `feed_cache_ttl_hours` (6); `--refresh` ignores the cache.
### How well is the screen doing?
Each digest ends with **Just below threshold**: per profile, up to three papers that scored between half the threshold and the threshold. To measure what the screen misses, build a labeled set a few papers at a time:
```
uv run bingham label # about 10 papers, sampled across the score range
uv run bingham label --near-misses
uv run bingham eval
```
`label` shows each paper's title, authors, and abstract with your profile names, but no scores, and asks which profiles it is relevant to. Papers are drawn at random within bands of their best re-ranker score (never re-ranked, below 0.1, 0.1 to 0.2, 0.2 to 0.4, 0.4 and up), so rare high-scoring papers and the large mass of papers the re-ranker never saw are both sampled. Labels go to `user/eval/labeled.jsonl` and sync with the rest of `user/`.
`eval` weights each band by its size in the store and reports, per profile: recall at the current threshold (with a bootstrap interval once there are a few relevant labels), precision, the share of relevant papers that reached the re-ranker, each missed paper and where it was lost, MRR, and recall and precision across a grid of thresholds with the highest one that keeps recall at `eval.target_recall` (0.9). Near-miss labels are reported separately, since they are not a random sample.
### Measured run times
| Machine | Screening a week (about 850 papers, 8 profiles) | A typical day |
@ -96,10 +112,24 @@ cp data/examples/profiles.yaml data/examples/config.yaml data/examples/interest_
- **`user/profiles.yaml`**: one entry per topic, each with a `description` (two or three sentences in the field's vocabulary, used for embedding and re-ranking), `keywords` for BM25, and optional `exemplars` (DOIs or arXiv IDs of papers that typify the topic; abstracts are fetched once, or can be pasted in). `tracked_authors` lists people whose papers are always shown. Editing a profile re-screens against that profile only.
- **`user/config.yaml`**: your contact email (sent to arXiv and Crossref), arXiv categories, and per-profile thresholds. Anything in `config.yaml` can be overridden here; nested keys override and lists replace.
- **`user/interest_profile.md`**: a prose version of your interests, used only by the LLM-as-classifier baseline.
- **`data/vocabulary.md`** and **`data/feeds.yaml`**: the rheology term list that keywords are drawn from, and journal RSS feeds (*Rheologica Acta*, *Soft Matter*, *JNNFM*, and *Macromolecules* are listed but disabled). Copy either into `user/` to customize it.
- **`data/vocabulary.md`** and **`data/feeds.yaml`**: the rheology term list that keywords are drawn from, and the journal feeds (see below). Copy either into `user/` to customize it.
Re-ranker scores depend on how broad a profile's description is, so thresholds are set per profile. The digest applies the current thresholds, so changing one needs no re-screen.
**Journals.** `data/feeds.yaml` lists the journals below. Only *J. Rheol.* is switched on; to add others, copy the file to `user/feeds.yaml` and set `enabled: true`. Each journal is read from whichever channel carries abstracts, and the coverage differs (checked October 2026):
| Journal | Channel | Abstracts |
|---|---|---|
| *J. Rheol.* | AIP RSS and Crossref, merged by DOI | full |
| *Phys. Rev. E*, *Lett.*, *Materials*, *Fluids* | APS RSS | first ~400 characters |
| *J. Chem. Phys.*, *Phys. Fluids* | Crossref | full for ~99% |
| *Macromolecules*, *Langmuir* | Crossref (ACS RSS is blocked) | full for ~95% |
| *Soft Matter* | Crossref | full for ~75% |
| *Rheologica Acta* | Springer RSS | full |
| *JNNFM*, *J. Colloid Interface Sci.* | Elsevier RSS or Crossref | none; screened on title only |
Papers screened on a truncated abstract or a title alone score lower against the same profile, so expect the screen to miss more of them. Any other journal with an RSS feed (`url:`) or Crossref deposits (`crossref_issn:`) can be added the same way.
**Drafting a profile from your own papers.** Instead of writing `user/profiles.yaml` from scratch, let `bingham init` draft it from papers you already have:
```
@ -127,13 +157,16 @@ When you are satisfied, save the draft as `user/profiles.yaml`.
config.yaml base settings (paths, arXiv, J. Rheol., screening, LLM)
data/examples/ example profiles, user config, and interest profile
data/vocabulary.md rheology vocabulary
data/feeds.yaml journal RSS feeds
data/feeds.yaml journal feeds (RSS or Crossref by ISSN)
user/ your own files (not tracked; see above)
modelfiles/ Ollama Modelfile for the researcher-tone model
prompts/ versioned prompt text files
grounding/ notation primers to paste into AI models (SoR nomenclature)
docs/failure-modes.md curated log of small-model failure modes
art/
cli.py the bingham command
env.py offline HuggingFace settings, config, device, host
llm.py the single Ollama wrapper; logs every call
store.py SQLite schema and access
feeds/fetch.py arXiv RSS, OAI-PMH, and API; journal RSS; Crossref
feeds/collect.py daily and backfill fetch logic, arXiv catch-up
@ -142,8 +175,10 @@ art/
screen/ embed.py, bm25.py, rerank.py, screen.py
digest.py Markdown digest
cluster.py average-linkage clustering (field pulse, profile drafts)
label.py evaluate.py the labeled set and the screen's recall, precision, and MRR
toolbox/ seeds.py (ORCID, ID lists, BibTeX, CSL JSON), draft.py (bingham init)
summarize/summarize.py LLM summary and topic label (being wired in)
summarize/summarize.py LLM summary, topic label, and match reason
baselines/llm_score.py LLM-as-classifier baseline, for comparison with the screen
tests/ uv run pytest
```
@ -155,7 +190,11 @@ Generated files (`data/art.sqlite`, `data/cache/`, `digests/`, `models/`) are no
uv run pytest
```
The tests cover the deterministic parts: feed parsing (against a saved arXiv RSS snapshot), merging papers from several sources, author matching, and the arXiv catch-up logic. They need no network and no models.
The tests cover the deterministic parts: feed parsing (against a saved arXiv RSS snapshot and APS-format RSS), merging papers from several sources, author matching, and the arXiv catch-up logic. They need no network and no models.
## Notation primer
`grounding/sor-nomenclature.md` is a separate, standalone file: the official SoR/ESR nomenclature (2018-10 edition) transcribed as a primer for AI models, with symbols in LaTeX, SI units, the conventions most often broken (for example, stress is $\sigma$, not $\tau$), and notes on errors in the source. Paste it as the first message of a chat with a cloud model, or keep it in a project so a coding assistant reads it. At about 12,500 tokens it is too long for Art's local model; the header and conventions section alone are about 2,300.
## Related work