Art 0.2.0
Built from art-dev 2da005a.
This commit is contained in:
Eric Furst
parent
7b2bb49e3c
commit
d7e264bcf2
15 changed files with 1344 additions and 53 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -7,5 +7,5 @@ data/corpus/
|
|||
digests/
|
||||
.DS_Store
|
||||
models/
|
||||
data/art.sqlite*
|
||||
data/*.sqlite*
|
||||
user/
|
||||
|
|
|
|||
71
README.md
71
README.md
|
|
@ -2,32 +2,33 @@
|
|||
|
||||
Art is a small, locally run LLM toolchain for rheology research. It accompanies the talk "Art: An LLM toolchain for rheology research" (Society of Rheology Annual Meeting, Boston, October 2026, session "AI and ML in Rheology"), and is meant as a starting point that other rheologists can read in one sitting, adopt, and adapt.
|
||||
|
||||
Everything runs on your own machine: embedding and re-ranking models are cached locally from HuggingFace, and text generation (coming in the next stage) goes to [Ollama](https://ollama.com) on `localhost`. No cloud APIs.
|
||||
Everything runs on your own machine: embedding and re-ranking models are cached locally from HuggingFace, and text generation goes to [Ollama](https://ollama.com) on `localhost`. No cloud APIs are used.
|
||||
|
||||
Art has two parts:
|
||||
|
||||
1. **bingham**, a literature screener. It pulls new abstracts from the *Journal of Rheology* and arXiv, screens them against your interest profiles, and writes a daily Markdown digest of the papers worth reading. A Bingham plastic does not flow until the stress exceeds a yield threshold; most abstracts stay put, and only those that clear a calibrated threshold come through.
|
||||
2. **Corpus search** (planned), hybrid retrieval over a locally indexed collection of rheology papers, with raw-evidence inspection and grounded synthesis with citations.
|
||||
1. **bingham** is a literature screener. It pulls new abstracts from arXiv, the *Journal of Rheology*, and any other journals you switch on, screens them against your interest profiles, and writes a daily Markdown digest of the papers worth reading, each with a short summary written by a local LLM. A Bingham plastic does not flow until the stress exceeds a yield threshold; most abstracts stay put, and only those that clear a calibrated threshold come through.
|
||||
2. **Corpus search** (planned) is a hybrid retrieval over a locally indexed collection of rheology papers, with raw-evidence inspection and grounded synthesis with citations.
|
||||
|
||||
## Status
|
||||
|
||||
bingham fetches, screens, and writes digests. The LLM passes (a short summary, a topic label, and a one-sentence match reason for each highlighted paper) are next; until then, each digest entry shows the opening sentences of the abstract and the screening scores. Thresholds are provisional until calibrated on a hand-labeled set.
|
||||
bingham runs daily: it fetches, screens, summarizes the papers that pass, and writes the digest. Thresholds are provisional until calibrated on a hand-labeled set (`bingham label` and `bingham eval`, below). Still to come: a failure-mode report over the logged LLM calls, and corpus search.
|
||||
|
||||
## How bingham works
|
||||
|
||||
```
|
||||
fetch -> store -> embed -> screen -> re-rank -> digest
|
||||
(code) (SQLite) (bge) (cosine (cross- (code)
|
||||
fetch -> store -> embed -> screen -> re-rank -> summarize -> label -> explain -> digest
|
||||
(code) (SQLite) (bge) (cosine (cross- (LLM) (LLM) (LLM) (code)
|
||||
+ BM25) encoder)
|
||||
```
|
||||
|
||||
- **Fetch.** arXiv announcements from the per-category RSS feeds at `rss.arxiv.org`; *J. Rheol.* from Crossref (by ISSN) merged by DOI with the AIP current-issue feeds, which carry abstracts but no authors. Backfills and skipped days come from arXiv's OAI-PMH interface.
|
||||
- **Fetch.** arXiv announcements from the per-category RSS feeds at `rss.arxiv.org`; *J. Rheol.* from Crossref (by ISSN) merged by DOI with the AIP current-issue feeds, which carry abstracts but no authors. Backfills and skipped days come from arXiv's OAI-PMH interface. Other journals come from their RSS feeds or from Crossref by ISSN, whichever carries abstracts (see "Journals" below).
|
||||
- **Store.** One SQLite file holds papers, embeddings, scores, and a record of every run, so no stage is recomputed unless its model or profile has changed.
|
||||
- **Screen.** Each paper (title and abstract) is embedded once with `BAAI/bge-large-en-v1.5`. For each profile it gets a cosine similarity (the best match over the profile description and its exemplar papers) and a BM25 score against the profile's keywords, which catches abbreviations such as SAOS, LAOS, and DST and names that embeddings blur. The top candidates by either score, plus any paper by a tracked author, go on.
|
||||
- **Re-rank.** `BAAI/bge-reranker-v2-m3` reads each (profile description, paper) pair together and scores it from 0 to 1. A paper is highlighted when its score clears the profile's threshold.
|
||||
- **Digest.** One Markdown file per day in `digests/`, grouped by profile and ordered by score, with a footer recording counts, models, host, device, and time per stage.
|
||||
- **Summarize, label, explain.** For each highlighted paper only, a local model (`gemma4:e4b` through Ollama, with the researcher-tone system prompt in `modelfiles/researcher.Modelfile`) makes three separate calls: a two-to-four-sentence summary of the abstract, a short topic label, and one sentence on why the paper matches the profile. Each call does one task; prompts are versioned text files in `prompts/`. Every call sets `num_ctx` explicitly and is logged in the store with the model tag, Modelfile hash, Ollama version, prompt version, token counts, latency, and the raw response.
|
||||
- **Digest.** One Markdown file per day in `digests/`, grouped by profile and ordered by score. Each entry gives the title, authors, source, summary, label, match reason, and link. A "Just below threshold" section lists near misses, and a field-pulse section groups papers by their topic labels. The footer records counts, models, prompt versions, host, device, and time per stage. If Ollama is not running, the digest is still written, with the opening sentences of each abstract in place of summaries.
|
||||
|
||||
Retrieval models decide relevance; the LLM, when added, only writes about papers that have already passed the screen, and never produces identifiers (DOIs, arXiv IDs, titles, authors, and dates come straight from the feed metadata).
|
||||
Retrieval models decide relevance; the LLM only writes about papers that have already passed the screen, and never produces identifiers (DOIs, arXiv IDs, titles, authors, and dates come straight from the feed metadata).
|
||||
|
||||
## Getting started
|
||||
|
||||
|
|
@ -36,18 +37,19 @@ Retrieval models decide relevance; the LLM, when added, only writes about papers
|
|||
- Python 3.12 and [uv](https://docs.astral.sh/uv/)
|
||||
- About 3.5 GB of disk for the two HuggingFace models
|
||||
- A GPU is optional. Tested on an NVIDIA RTX 4090 (CUDA) under Linux and an Apple M3 Pro (MPS) under macOS; the device is chosen automatically (`cuda`, then `mps`, then `cpu`).
|
||||
- [Ollama](https://ollama.com), for the LLM passes once they are added (not needed yet)
|
||||
- [Ollama](https://ollama.com) and the `gemma4:e4b` model (about 10 GB) for the summaries. Without Ollama, or with `--no-llm`, digests are written from the screen alone.
|
||||
|
||||
### Install
|
||||
|
||||
```
|
||||
git clone https://lem.che.udel.edu/git/furst/art.git
|
||||
git clone https://github.com/emfurst/art.git # or https://lem.che.udel.edu/git/furst/art.git
|
||||
cd art
|
||||
uv sync
|
||||
uv run bingham download-models # once, with network access
|
||||
ollama pull gemma4:e4b
|
||||
```
|
||||
|
||||
`download-models` fetches the embedder and re-ranker into `./models/`. Every other command runs with HuggingFace in offline mode.
|
||||
`download-models` fetches the embedder and re-ranker into `./models/`. Every other command runs with HuggingFace in offline mode. The first LLM call builds the researcher model from `modelfiles/researcher.Modelfile` and tags it with the file's hash (`art-researcher:<hash>`), so editing the Modelfile creates a new tag and every logged call records which version wrote it.
|
||||
|
||||
On Linux with an NVIDIA GPU, use a driver from the 580 series or later (CUDA 13). With older drivers, the standard PyTorch wheels cannot use the GPU, and Ollama 0.34 falls back to the CPU without an error (it still reports "100% GPU" in `ollama ps`, while generation slows to minutes per call). If you cannot update the driver, route `torch` to the CUDA 12.6 index in `pyproject.toml` (`[tool.uv.sources]`, index `https://download.pytorch.org/whl/cu126`) and use an Ollama release that supports your driver.
|
||||
|
||||
|
|
@ -77,6 +79,20 @@ uv run bingham topic "shear thickening and frictional contacts" [-n 15]
|
|||
- **`topic`** ranks every stored paper against a free-text query, with no profile and no LLM.
|
||||
- Raw feed responses are cached for `feed_cache_ttl_hours` (6); `--refresh` ignores the cache.
|
||||
|
||||
### How well is the screen doing?
|
||||
|
||||
Each digest ends with **Just below threshold**: per profile, up to three papers that scored between half the threshold and the threshold. To measure what the screen misses, build a labeled set a few papers at a time:
|
||||
|
||||
```
|
||||
uv run bingham label # about 10 papers, sampled across the score range
|
||||
uv run bingham label --near-misses
|
||||
uv run bingham eval
|
||||
```
|
||||
|
||||
`label` shows each paper's title, authors, and abstract with your profile names, but no scores, and asks which profiles it is relevant to. Papers are drawn at random within bands of their best re-ranker score (never re-ranked, below 0.1, 0.1 to 0.2, 0.2 to 0.4, 0.4 and up), so rare high-scoring papers and the large mass of papers the re-ranker never saw are both sampled. Labels go to `user/eval/labeled.jsonl` and sync with the rest of `user/`.
|
||||
|
||||
`eval` weights each band by its size in the store and reports, per profile: recall at the current threshold (with a bootstrap interval once there are a few relevant labels), precision, the share of relevant papers that reached the re-ranker, each missed paper and where it was lost, MRR, and recall and precision across a grid of thresholds with the highest one that keeps recall at `eval.target_recall` (0.9). Near-miss labels are reported separately, since they are not a random sample.
|
||||
|
||||
### Measured run times
|
||||
|
||||
| Machine | Screening a week (about 850 papers, 8 profiles) | A typical day |
|
||||
|
|
@ -96,10 +112,24 @@ cp data/examples/profiles.yaml data/examples/config.yaml data/examples/interest_
|
|||
- **`user/profiles.yaml`**: one entry per topic, each with a `description` (two or three sentences in the field's vocabulary, used for embedding and re-ranking), `keywords` for BM25, and optional `exemplars` (DOIs or arXiv IDs of papers that typify the topic; abstracts are fetched once, or can be pasted in). `tracked_authors` lists people whose papers are always shown. Editing a profile re-screens against that profile only.
|
||||
- **`user/config.yaml`**: your contact email (sent to arXiv and Crossref), arXiv categories, and per-profile thresholds. Anything in `config.yaml` can be overridden here; nested keys override and lists replace.
|
||||
- **`user/interest_profile.md`**: a prose version of your interests, used only by the LLM-as-classifier baseline.
|
||||
- **`data/vocabulary.md`** and **`data/feeds.yaml`**: the rheology term list that keywords are drawn from, and journal RSS feeds (*Rheologica Acta*, *Soft Matter*, *JNNFM*, and *Macromolecules* are listed but disabled). Copy either into `user/` to customize it.
|
||||
- **`data/vocabulary.md`** and **`data/feeds.yaml`**: the rheology term list that keywords are drawn from, and the journal feeds (see below). Copy either into `user/` to customize it.
|
||||
|
||||
Re-ranker scores depend on how broad a profile's description is, so thresholds are set per profile. The digest applies the current thresholds, so changing one needs no re-screen.
|
||||
|
||||
**Journals.** `data/feeds.yaml` lists the journals below. Only *J. Rheol.* is switched on; to add others, copy the file to `user/feeds.yaml` and set `enabled: true`. Each journal is read from whichever channel carries abstracts, and the coverage differs (checked October 2026):
|
||||
|
||||
| Journal | Channel | Abstracts |
|
||||
|---|---|---|
|
||||
| *J. Rheol.* | AIP RSS and Crossref, merged by DOI | full |
|
||||
| *Phys. Rev. E*, *Lett.*, *Materials*, *Fluids* | APS RSS | first ~400 characters |
|
||||
| *J. Chem. Phys.*, *Phys. Fluids* | Crossref | full for ~99% |
|
||||
| *Macromolecules*, *Langmuir* | Crossref (ACS RSS is blocked) | full for ~95% |
|
||||
| *Soft Matter* | Crossref | full for ~75% |
|
||||
| *Rheologica Acta* | Springer RSS | full |
|
||||
| *JNNFM*, *J. Colloid Interface Sci.* | Elsevier RSS or Crossref | none; screened on title only |
|
||||
|
||||
Papers screened on a truncated abstract or a title alone score lower against the same profile, so expect the screen to miss more of them. Any other journal with an RSS feed (`url:`) or Crossref deposits (`crossref_issn:`) can be added the same way.
|
||||
|
||||
**Drafting a profile from your own papers.** Instead of writing `user/profiles.yaml` from scratch, let `bingham init` draft it from papers you already have:
|
||||
|
||||
```
|
||||
|
|
@ -127,13 +157,16 @@ When you are satisfied, save the draft as `user/profiles.yaml`.
|
|||
config.yaml base settings (paths, arXiv, J. Rheol., screening, LLM)
|
||||
data/examples/ example profiles, user config, and interest profile
|
||||
data/vocabulary.md rheology vocabulary
|
||||
data/feeds.yaml journal RSS feeds
|
||||
data/feeds.yaml journal feeds (RSS or Crossref by ISSN)
|
||||
user/ your own files (not tracked; see above)
|
||||
modelfiles/ Ollama Modelfile for the researcher-tone model
|
||||
prompts/ versioned prompt text files
|
||||
grounding/ notation primers to paste into AI models (SoR nomenclature)
|
||||
docs/failure-modes.md curated log of small-model failure modes
|
||||
art/
|
||||
cli.py the bingham command
|
||||
env.py offline HuggingFace settings, config, device, host
|
||||
llm.py the single Ollama wrapper; logs every call
|
||||
store.py SQLite schema and access
|
||||
feeds/fetch.py arXiv RSS, OAI-PMH, and API; journal RSS; Crossref
|
||||
feeds/collect.py daily and backfill fetch logic, arXiv catch-up
|
||||
|
|
@ -142,8 +175,10 @@ art/
|
|||
screen/ embed.py, bm25.py, rerank.py, screen.py
|
||||
digest.py Markdown digest
|
||||
cluster.py average-linkage clustering (field pulse, profile drafts)
|
||||
label.py evaluate.py the labeled set and the screen's recall, precision, and MRR
|
||||
toolbox/ seeds.py (ORCID, ID lists, BibTeX, CSL JSON), draft.py (bingham init)
|
||||
summarize/summarize.py LLM summary and topic label (being wired in)
|
||||
summarize/summarize.py LLM summary, topic label, and match reason
|
||||
baselines/llm_score.py LLM-as-classifier baseline, for comparison with the screen
|
||||
tests/ uv run pytest
|
||||
```
|
||||
|
||||
|
|
@ -155,7 +190,11 @@ Generated files (`data/art.sqlite`, `data/cache/`, `digests/`, `models/`) are no
|
|||
uv run pytest
|
||||
```
|
||||
|
||||
The tests cover the deterministic parts: feed parsing (against a saved arXiv RSS snapshot), merging papers from several sources, author matching, and the arXiv catch-up logic. They need no network and no models.
|
||||
The tests cover the deterministic parts: feed parsing (against a saved arXiv RSS snapshot and APS-format RSS), merging papers from several sources, author matching, and the arXiv catch-up logic. They need no network and no models.
|
||||
|
||||
## Notation primer
|
||||
|
||||
`grounding/sor-nomenclature.md` is a separate, standalone file: the official SoR/ESR nomenclature (2018-10 edition) transcribed as a primer for AI models, with symbols in LaTeX, SI units, the conventions most often broken (for example, stress is $\sigma$, not $\tau$), and notes on errors in the source. Paste it as the first message of a chat with a cloud model, or keep it in a project so a coding assistant reads it. At about 12,500 tokens it is too long for Art's local model; the header and conventions section alone are about 2,300.
|
||||
|
||||
## Related work
|
||||
|
||||
|
|
|
|||
20
art/cli.py
20
art/cli.py
|
|
@ -8,6 +8,8 @@
|
|||
uv run bingham download-models # once, with network access
|
||||
uv run bingham init --orcid 0000-0000-0000-0000 [--ids FILE] [--bib FILE] [--csl FILE]
|
||||
uv run bingham profile-test [--file user/profiles.draft.yaml]
|
||||
uv run bingham label [-n 10] [--near-misses] # build the labeled set (user/eval/labeled.jsonl)
|
||||
uv run bingham eval # recall, precision, MRR, misses, thresholds
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -186,6 +188,16 @@ def cmd_profile_test(conn, args) -> None:
|
|||
print()
|
||||
|
||||
|
||||
def cmd_label(conn, args) -> None:
|
||||
from art import label
|
||||
label.run(conn, args.n, args.near_misses, args.seed)
|
||||
|
||||
|
||||
def cmd_eval(conn, args) -> None:
|
||||
from art import evaluate
|
||||
evaluate.run(conn, args.target_recall)
|
||||
|
||||
|
||||
def cmd_download_models(conn, args) -> None:
|
||||
from huggingface_hub import snapshot_download
|
||||
cfg = env.config()["screen"]
|
||||
|
|
@ -225,12 +237,18 @@ def main() -> None:
|
|||
pt.add_argument("--file", help="profiles file to test (default user/profiles.yaml)")
|
||||
pt.add_argument("--profile", help="test one profile id")
|
||||
pt.add_argument("-n", type=int, default=8)
|
||||
lb = sub.add_parser("label", help="mark sampled papers relevant to profiles (blind to scores)")
|
||||
lb.add_argument("-n", type=int, default=10, help="papers this session (default 10)")
|
||||
lb.add_argument("--near-misses", action="store_true", help="label papers just below a threshold instead")
|
||||
lb.add_argument("--seed", type=int, help="random seed, for a repeatable sample")
|
||||
ev = sub.add_parser("eval", help="recall, precision, and MRR of the screen on the labeled set")
|
||||
ev.add_argument("--target-recall", type=float, help="for the threshold suggestion (default eval.target_recall)")
|
||||
args = ap.parse_args()
|
||||
|
||||
conn = store.connect()
|
||||
{"fetch": cmd_fetch, "screen": cmd_screen, "digest": cmd_digest, "topic": cmd_topic,
|
||||
"daily": cmd_daily, "download-models": cmd_download_models, "init": cmd_init,
|
||||
"profile-test": cmd_profile_test}[args.command](conn, args)
|
||||
"profile-test": cmd_profile_test, "label": cmd_label, "eval": cmd_eval}[args.command](conn, args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""`bingham digest`: select the day's papers, run the LLM passes, write Markdown.
|
||||
|
||||
Window: by default, items with origin 'daily' fetched since the previous
|
||||
digest (or in the last 24 hours if there is none). With --since DATE, every
|
||||
default digest finished (or in the last 24 hours if there is none). With --since DATE, every
|
||||
screenable item dated on or after DATE, backfills included.
|
||||
|
||||
Entries are grouped by profile and ordered by re-ranker score. A paper that
|
||||
|
|
@ -59,6 +59,7 @@ class Selection:
|
|||
tracked_only: list[Entry]
|
||||
n_screened: int
|
||||
catchups: list[str]
|
||||
near_misses: dict[str, list[Entry]] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def highlighted(self) -> list[Entry]:
|
||||
|
|
@ -73,8 +74,12 @@ def _window(conn, since: str | None, run_id: int | None) -> tuple[str, str, tupl
|
|||
"""(description, SQL condition, params) selecting the digest's items."""
|
||||
if since:
|
||||
return f"items dated {since} or later", f"{store.SCREENABLE} AND date >= ?", (since,)
|
||||
row = conn.execute("SELECT MAX(started_at) FROM runs WHERE command IN ('digest', 'daily') "
|
||||
"AND finished_at IS NOT NULL AND id != ?", (run_id or -1,)).fetchone()
|
||||
# The window opens when the previous default digest finished: daily fetches
|
||||
# after it starts, so its start time would repeat its papers the next day.
|
||||
# --since digests are backfill views and do not move the window.
|
||||
row = conn.execute("SELECT MAX(finished_at) FROM runs WHERE command IN ('digest', 'daily') "
|
||||
"AND finished_at IS NOT NULL AND id != ? "
|
||||
"AND json_extract(args, '$.since') IS NULL", (run_id or -1,)).fetchone()
|
||||
start = row[0] or (datetime.now(timezone.utc) - timedelta(days=1)).isoformat(timespec="seconds")
|
||||
return f"items fetched since {start[:16].replace('T', ' ')} UTC", \
|
||||
"origin = 'daily' AND fetched_at >= ?", (start,)
|
||||
|
|
@ -113,16 +118,32 @@ def select(conn, since: str | None = None, run_id: int | None = None) -> Selecti
|
|||
by_profile[hits[0]["profile_id"]].append(
|
||||
Entry(items[iid], hits[0], [names[s["profile_id"]] for s in hits[1:]], tracked))
|
||||
elif tracked:
|
||||
best = max((s for s in ss if s["rerank"] is not None), key=lambda s: s["rerank"])
|
||||
# Best re-ranked profile; a paper screened before its author was
|
||||
# tracked was never re-ranked, so fall back to its best cosine match.
|
||||
reranked = [s for s in ss if s["rerank"] is not None]
|
||||
best = max(reranked, key=lambda s: s["rerank"]) if reranked else max(ss, key=lambda s: s["cosine"])
|
||||
tracked_only.append(Entry(items[iid], best, [], tracked))
|
||||
for entries in by_profile.values():
|
||||
entries.sort(key=lambda e: -e.screen["rerank"])
|
||||
tracked_only.sort(key=lambda e: -e.screen["rerank"])
|
||||
tracked_only.sort(key=lambda e: -(e.screen["rerank"] if e.screen["rerank"] is not None else -1))
|
||||
|
||||
catchups = [] if since else [r[0] for r in conn.execute(
|
||||
"SELECT detail FROM fetches WHERE source = 'arxiv:catchup' AND ok = 1 AND fetched_at >= ?",
|
||||
params)]
|
||||
return Selection(label, items, by_profile, tracked_only, len(screens), catchups)
|
||||
# Near misses: per profile, the best papers between near_miss_floor x threshold
|
||||
# and the threshold that were not highlighted for any profile.
|
||||
dcfg = cfg["digest"]
|
||||
shown = {e.item["id"] for entries in by_profile.values() for e in entries}
|
||||
near: dict[str, list[Entry]] = {}
|
||||
for p in data.profiles:
|
||||
t = th.get(p.id, th["default"])
|
||||
cands = [(s, iid) for iid, ss in screens.items() if iid not in shown for s in ss
|
||||
if s["profile_id"] == p.id and s["rerank"] is not None
|
||||
and dcfg["near_miss_floor"] * t <= s["rerank"] < t]
|
||||
cands.sort(key=lambda c: -c[0]["rerank"])
|
||||
if cands:
|
||||
near[p.id] = [Entry(items[iid], s) for s, iid in cands[:dcfg["near_misses"]]]
|
||||
return Selection(label, items, by_profile, tracked_only, len(screens), catchups, near)
|
||||
|
||||
|
||||
# -- LLM passes -----------------------------------------------------------------
|
||||
|
|
@ -189,7 +210,7 @@ def _entry(conn, e: Entry, profiles: dict) -> list[str]:
|
|||
if label and label["text"]:
|
||||
meta.append(f"Topic: {label['text']}")
|
||||
best = s["cosine_best"] if s["cosine_best"] == "description" else f"exemplar {s['cosine_best']}"
|
||||
meta.append(f"re-rank {s['rerank']:.2f}")
|
||||
meta.append(f"re-rank {s['rerank']:.2f}" if s["rerank"] is not None else "not re-ranked (screened before its author was tracked)")
|
||||
if explain:
|
||||
meta.append(f"LLM relevance {explain['parsed']['relevance']}/3")
|
||||
meta.append(f"cosine {s['cosine']:.2f} ({best}), BM25 {s['bm25']:.1f}, via {s['nominated_by']}")
|
||||
|
|
@ -259,6 +280,17 @@ def render(conn, sel: Selection, llm_stats: dict | None = None,
|
|||
lines += [f"## {p.name} ({len(entries)})", ""]
|
||||
for e in entries:
|
||||
lines += _entry(conn, e, profiles)
|
||||
if sel.near_misses:
|
||||
lines += ["## Just below threshold", "",
|
||||
"*Not highlighted; the closest calls per profile. Label them with `bingham label --near-misses`.*", ""]
|
||||
for pid, entries in sel.near_misses.items():
|
||||
t = cfg["screen"]["thresholds"].get(pid, cfg["screen"]["thresholds"]["default"])
|
||||
lines.append(f"**{profiles[pid].name}** (threshold {t}) ")
|
||||
for e in entries:
|
||||
it = e.item
|
||||
lines.append(f"- [{it['title']}]({it['link']}), {first_author(it['authors'])}, "
|
||||
f"{it['source']}; re-rank {e.screen['rerank']:.2f}")
|
||||
lines.append("")
|
||||
if sel.tracked_only:
|
||||
lines += ["## Tracked authors, below threshold", ""]
|
||||
for e in sel.tracked_only:
|
||||
|
|
@ -268,6 +300,7 @@ def render(conn, sel: Selection, llm_stats: dict | None = None,
|
|||
lines += _field_pulse(conn, sel)
|
||||
|
||||
stats = {"in_window": len(sel.items), "screened": sel.n_screened, "highlighted": n_hi,
|
||||
"near_misses": sum(map(len, sel.near_misses.values())),
|
||||
"tracked_below_threshold": len(sel.tracked_only), "arxiv_catchup": sel.catchups,
|
||||
"llm": llm_stats or {}}
|
||||
lines += _footer(conn, stats, cfg, stages)
|
||||
|
|
@ -299,10 +332,11 @@ def _llm_line(conn, stats: dict) -> str:
|
|||
return "LLM: not run (--no-llm); texts shown are any already stored. "
|
||||
if not s["available"]:
|
||||
return "LLM: Ollama not reachable; summaries are abstract openings. "
|
||||
row = conn.execute("SELECT model_digest FROM llm_calls WHERE model = ? AND model_digest != '' "
|
||||
"ORDER BY id DESC LIMIT 1", (llm.model_tag(),)).fetchone()
|
||||
row = conn.execute("SELECT model_digest, ollama_version FROM llm_calls WHERE model = ? "
|
||||
"AND model_digest != '' ORDER BY id DESC LIMIT 1", (llm.model_tag(),)).fetchone()
|
||||
versions = ", ".join(f"{t} {llm.prompt(t).version}" for t in ("summarize", "label", "explain"))
|
||||
return (f"LLM: {llm.model_tag()} (weights {row[0][:12] if row else 'unknown'}), num_ctx {c['num_ctx']}, "
|
||||
return (f"LLM: {llm.model_tag()} (weights {row[0][:12] if row else 'unknown'}), "
|
||||
f"Ollama {(row[1] if row else None) or 'unknown'}, num_ctx {c['num_ctx']}, "
|
||||
f"think {'on' if c['think'] else 'off'}; prompts {versions}; "
|
||||
f"{s['calls']} new generations, {s['failed']} failed, {s['seconds']} s. ")
|
||||
|
||||
|
|
@ -334,5 +368,9 @@ def write(conn, since: str | None = None, run_id: int | None = None,
|
|||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
name = date.today().isoformat() + (f"_since-{since}" if since else "")
|
||||
path = out_dir / f"{name}.md"
|
||||
n = 2
|
||||
while path.exists() and not since: # a second daily run the same day keeps the first digest
|
||||
path = out_dir / f"{name}-{n}.md"
|
||||
n += 1
|
||||
path.write_text(text)
|
||||
return str(path), stats
|
||||
|
|
|
|||
163
art/evaluate.py
Normal file
163
art/evaluate.py
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
"""`bingham eval`: how well the screen finds what the labeled set says is relevant.
|
||||
|
||||
Recall comes first, since a screen's costly error is a missed paper. Labels
|
||||
from random sampling (bingham label) are weighted by band: each labeled
|
||||
paper stands for N_b / n_b papers of its band in the store (post-
|
||||
stratification), so estimates cover the whole stream, including papers the
|
||||
re-ranker never saw. Intervals are 90% bootstrap intervals, resampling within
|
||||
bands. Near-miss labels (bingham label --near-misses) are not random and are
|
||||
reported separately.
|
||||
|
||||
For each profile:
|
||||
recall, precision at the current threshold, weighted
|
||||
where misses happen never re-ranked (not nominated by cosine or BM25) or
|
||||
re-ranked below the threshold; the missed papers listed
|
||||
candidate recall share of relevant papers the nomination stage passed on
|
||||
MRR per fetch day, reciprocal rank of the first labeled-
|
||||
relevant paper among all that day's papers by re-rank
|
||||
score (unlabeled papers count as not relevant)
|
||||
thresholds weighted recall and precision on a grid, and the highest
|
||||
threshold that keeps recall at eval.target_recall
|
||||
"""
|
||||
|
||||
import random
|
||||
from collections import defaultdict
|
||||
|
||||
from art import env, store
|
||||
from art.label import BANDS, load_labels, population
|
||||
from art.profile import profiles as prof
|
||||
|
||||
|
||||
def _rates(rows, t):
|
||||
"""Weighted (recall, precision) at threshold t for rows of (weight, relevant, rerank)."""
|
||||
tp = sum(w for w, rel, r in rows if rel and r is not None and r >= t)
|
||||
fn = sum(w for w, rel, r in rows if rel and (r is None or r < t))
|
||||
fp = sum(w for w, rel, r in rows if not rel and r is not None and r >= t)
|
||||
recall = tp / (tp + fn) if tp + fn else None
|
||||
precision = tp / (tp + fp) if tp + fp else None
|
||||
return recall, precision
|
||||
|
||||
|
||||
def _interval(by_band, t, weights, reps=500, seed=0):
|
||||
"""90% bootstrap interval for recall, resampling labeled papers within each band."""
|
||||
rng = random.Random(seed)
|
||||
vals = []
|
||||
for _ in range(reps):
|
||||
rows = []
|
||||
for b, items in by_band.items():
|
||||
rows += [(weights[b],) + rng.choice(items)[1:] for _ in items]
|
||||
r, _ = _rates(rows, t)
|
||||
if r is not None:
|
||||
vals.append(r)
|
||||
if len(vals) < 20:
|
||||
return None
|
||||
vals.sort()
|
||||
return vals[int(0.05 * len(vals))], vals[int(0.95 * len(vals)) - 1]
|
||||
|
||||
|
||||
def _fmt(x):
|
||||
return " n/a" if x is None else f"{x:5.2f}"
|
||||
|
||||
|
||||
def run(conn, target: float | None = None) -> None:
|
||||
cfg = env.config()
|
||||
th = cfg["screen"]["thresholds"]
|
||||
target = target or cfg.get("eval", {}).get("target_recall", 0.9)
|
||||
profiles = prof.load().profiles
|
||||
pop = population(conn, profiles)
|
||||
labels = {i: r for i, r in load_labels().items() if i in pop}
|
||||
rand = {i: r for i, r in labels.items() if not r.get("near_miss")}
|
||||
if not labels:
|
||||
print("no labels yet; run `bingham label` first")
|
||||
return
|
||||
|
||||
N = defaultdict(int)
|
||||
for v in pop.values():
|
||||
N[v["band"]] += 1
|
||||
n = defaultdict(int)
|
||||
for i in rand:
|
||||
n[pop[i]["band"]] += 1
|
||||
weights = {b: N[b] / n[b] for b in BANDS if n[b]}
|
||||
print(f"{len(labels)} labeled papers ({len(rand)} random, {len(labels) - len(rand)} near misses); "
|
||||
f"{len(pop)} screened papers in the store.")
|
||||
print("band papers labeled weight")
|
||||
for b in BANDS:
|
||||
print(f" {b:8s} {N[b]:8d} {n[b]:8d} {weights.get(b, 0):7.1f}")
|
||||
missing = [b for b in BANDS if N[b] and not n[b]]
|
||||
if missing:
|
||||
print(f" no labels yet in band(s) {', '.join(missing)}: their papers are not represented")
|
||||
|
||||
days = defaultdict(list) # fetch day -> item ids, for MRR
|
||||
fetched = dict(conn.execute(f"SELECT id, substr(fetched_at, 1, 10) FROM items WHERE {store.SCREENABLE}"))
|
||||
for i in pop:
|
||||
days[fetched.get(i)].append(i)
|
||||
|
||||
for p in profiles:
|
||||
t = th.get(p.id, th["default"])
|
||||
shown = {i: r for i, r in rand.items() if p.id in r.get("profiles", [])}
|
||||
rows_by_band = defaultdict(list)
|
||||
for i, r in shown.items():
|
||||
s = pop[i]["scores"].get(p.id, {})
|
||||
rows_by_band[pop[i]["band"]].append((i, p.id in r["relevant"], s.get("rerank")))
|
||||
rows = [(weights[b], rel, rr) for b, items in rows_by_band.items() for _, rel, rr in items]
|
||||
n_rel = sum(rel for _, rel, _ in rows)
|
||||
print(f"\n== {p.id} (threshold {t}; {len(rows)} labeled, {n_rel} relevant)")
|
||||
if not n_rel:
|
||||
print(" no relevant papers labeled yet")
|
||||
continue
|
||||
recall, precision = _rates(rows, t)
|
||||
ci = _interval(rows_by_band, t, weights) if n_rel >= 3 else None
|
||||
print(f" recall {_fmt(recall)}" + (f" (90% interval {ci[0]:.2f}-{ci[1]:.2f})" if ci else "")
|
||||
+ f" precision {_fmt(precision)}")
|
||||
|
||||
rel_items = [(i, pop[i]["scores"].get(p.id, {})) for i, r in shown.items() if p.id in r["relevant"]]
|
||||
nominated = sum(weights[pop[i]["band"]] for i, s in rel_items if s.get("rerank") is not None)
|
||||
total = sum(weights[pop[i]["band"]] for i, _ in rel_items)
|
||||
print(f" candidate recall {nominated / total:.2f} (relevant papers that reached the re-ranker)")
|
||||
by = defaultdict(int)
|
||||
for _, s in rel_items:
|
||||
by[s.get("nominated_by") or "not nominated"] += 1
|
||||
print(" relevant papers nominated by: " + ", ".join(f"{k} {v}" for k, v in sorted(by.items())))
|
||||
|
||||
misses = [(i, s) for i, s in rel_items if s.get("rerank") is None or s["rerank"] < t]
|
||||
if misses:
|
||||
print(" missed:")
|
||||
for i, s in sorted(misses, key=lambda m: -(m[1].get("rerank") or -1)):
|
||||
item = conn.execute("SELECT title, source FROM items WHERE id = ?", (i,)).fetchone()
|
||||
where = "never re-ranked" if s.get("rerank") is None else f"re-rank {s['rerank']:.2f}"
|
||||
print(f" {where:16s} cosine {s.get('cosine', 0):.2f} {item['source']:10s} {item['title'][:80]}")
|
||||
|
||||
rr = []
|
||||
for day, ids in days.items():
|
||||
relevant_today = {i for i in ids if i in shown and p.id in shown[i]["relevant"]}
|
||||
if not relevant_today:
|
||||
continue
|
||||
ranked = sorted(ids, key=lambda i: (-(pop[i]["scores"].get(p.id, {}).get("rerank") or -1),
|
||||
-pop[i]["scores"].get(p.id, {}).get("cosine", 0)))
|
||||
rr.append(1 / (1 + next(k for k, i in enumerate(ranked) if i in relevant_today)))
|
||||
if rr:
|
||||
print(f" MRR {sum(rr) / len(rr):.2f} over {len(rr)} fetch days with a labeled relevant paper")
|
||||
|
||||
grid = [round(0.05 * k, 2) for k in range(1, 13)]
|
||||
rates = [(g,) + _rates(rows, g) for g in grid]
|
||||
print(" threshold " + " ".join(f"{g:5.2f}" for g in grid))
|
||||
print(" recall " + " ".join(_fmt(r) for _, r, _ in rates))
|
||||
print(" precision " + " ".join(_fmt(pr) for _, _, pr in rates))
|
||||
ok = [g for g, r, _ in rates if r is not None and r >= target]
|
||||
if n_rel < 5:
|
||||
print(f" too few relevant labels ({n_rel}) to suggest a threshold")
|
||||
elif ok:
|
||||
print(f" highest threshold with recall >= {target}: {max(ok)}")
|
||||
else:
|
||||
print(f" no threshold on the grid reaches recall {target}")
|
||||
|
||||
near = {i: r for i, r in labels.items() if r.get("near_miss")}
|
||||
if near:
|
||||
print("\nnear misses labeled (scored just below a threshold):")
|
||||
for p in profiles:
|
||||
t = th.get(p.id, th["default"])
|
||||
cand = [i for i in near if (s := pop[i]["scores"].get(p.id, {})).get("rerank") is not None
|
||||
and s["rerank"] < t and p.id in near[i].get("profiles", [])]
|
||||
if cand:
|
||||
k = sum(p.id in near[i]["relevant"] for i in cand)
|
||||
print(f" {p.id:24s} {k} of {len(cand)} relevant")
|
||||
|
|
@ -14,9 +14,15 @@ not flood the next digest.
|
|||
Backfill mode (--since DATE): arXiv OAI-PMH and Crossref from DATE. Items have
|
||||
origin 'backfill' and stay out of the default digest. The first J. Rheol. fetch
|
||||
is a backfill from jor.backfill_from.
|
||||
|
||||
A feeds.yaml entry with `crossref_issn` instead of `url` is fetched from
|
||||
Crossref by ISSN, like J. Rheol., for publishers whose RSS is blocked (ACS,
|
||||
behind Cloudflare) or gone (RSC). Its first fetch covers the last
|
||||
crossref_overlap_days, or starts at the entry's own `backfill_from`.
|
||||
"""
|
||||
|
||||
from datetime import date, timedelta
|
||||
import re
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
|
||||
import yaml
|
||||
|
||||
|
|
@ -144,27 +150,48 @@ def _fetch_arxiv(conn, since: str | None, ttl: float, stats: dict) -> None:
|
|||
_catch_up(conn, rss_day, rss_ok, cfg, stats)
|
||||
|
||||
|
||||
def _fetch_jor(conn, since: str | None, ttl: float, stats: dict) -> None:
|
||||
cfg = env.config()["jor"]
|
||||
name = "crossref:jor"
|
||||
def _crossref(conn, name: str, issn: str, source: str, since: str | None,
|
||||
first: tuple[str, str] | None, stats: dict) -> None:
|
||||
"""Crossref works for one ISSN, created since the last successful fetch less an overlap.
|
||||
|
||||
With --since, a backfill from that date. On the first fetch, `first` is
|
||||
(start date, origin); without it, the last crossref_overlap_days as daily.
|
||||
"""
|
||||
overlap = env.config()["jor"]["crossref_overlap_days"]
|
||||
last = store.last_fetch(conn, name)
|
||||
if since:
|
||||
start, origin = since, "backfill"
|
||||
elif last is None:
|
||||
start, origin = str(cfg["backfill_from"]), "backfill"
|
||||
# UTC, like the fetch timestamps that later starts are computed from
|
||||
today = datetime.now(timezone.utc).date()
|
||||
start, origin = first or ((today - timedelta(days=overlap)).isoformat(), "daily")
|
||||
else:
|
||||
start = (date.fromisoformat(last[:10]) - timedelta(days=cfg["crossref_overlap_days"])).isoformat()
|
||||
start = (date.fromisoformat(last[:10]) - timedelta(days=overlap)).isoformat()
|
||||
origin = "daily"
|
||||
try:
|
||||
items = fetch.fetch_crossref(cfg["issn"][0], start, source="J. Rheol.")
|
||||
items = fetch.fetch_crossref(issn, start, source=source)
|
||||
_record(conn, name, items, origin, f"created since {start}", stats)
|
||||
except Exception as exc:
|
||||
store.log_fetch(conn, name, 0, 0, False, str(exc))
|
||||
print(f" {name}: FAILED ({exc})")
|
||||
for feed in _feeds(lambda f: f.get("source") == "J. Rheol."):
|
||||
|
||||
|
||||
def _fetch_jor(conn, since: str | None, ttl: float, stats: dict) -> None:
|
||||
cfg = env.config()["jor"]
|
||||
_crossref(conn, "crossref:jor", cfg["issn"][0], "J. Rheol.", since,
|
||||
(str(cfg["backfill_from"]), "backfill"), stats)
|
||||
for feed in _feeds(lambda f: f.get("source") == "J. Rheol." and "url" in f):
|
||||
_fetch_feed(conn, feed, ttl, stats)
|
||||
|
||||
|
||||
def _fetch_crossref_feed(conn, feed: dict, since: str | None, stats: dict) -> None:
|
||||
"""A feeds.yaml entry with crossref_issn: for publishers whose RSS is blocked or gone."""
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", feed["name"].lower()).strip("-")
|
||||
first = (str(feed["backfill_from"]), "backfill") if feed.get("backfill_from") else None
|
||||
_crossref(conn, f"crossref:{slug}", str(feed["crossref_issn"]), feed.get("source") or feed["name"],
|
||||
since, first, stats)
|
||||
|
||||
|
||||
def _feeds(pred) -> list[dict]:
|
||||
with open(env.data_file("feeds.yaml")) as f:
|
||||
feeds = yaml.safe_load(f)["feeds"]
|
||||
|
|
@ -190,7 +217,10 @@ def run(conn, source: str = "all", since: str | None = None, refresh: bool = Fal
|
|||
_fetch_arxiv(conn, since, ttl, stats)
|
||||
if source in ("jor", "all"):
|
||||
_fetch_jor(conn, since, ttl, stats)
|
||||
if source == "all" and not since:
|
||||
if source == "all":
|
||||
for feed in _feeds(lambda f: f.get("source") != "J. Rheol."):
|
||||
_fetch_feed(conn, feed, ttl, stats)
|
||||
if feed.get("crossref_issn"):
|
||||
_fetch_crossref_feed(conn, feed, since, stats)
|
||||
elif not since: # RSS holds only recent items; no backfill
|
||||
_fetch_feed(conn, feed, ttl, stats)
|
||||
return stats
|
||||
|
|
|
|||
|
|
@ -251,6 +251,9 @@ def fetch_rss(feed_name: str, feed_url: str, source: str | None = None) -> list[
|
|||
for entry in feed.entries:
|
||||
doi = getattr(entry, "prism_doi", None) or getattr(entry, "dc_identifier", None)
|
||||
link = getattr(entry, "link", "")
|
||||
if not doi: # Springer links carry it: link.springer.com/article/10.1007/s00397-...
|
||||
m = re.search(r"/(10\.\d{4,9}/[^?#\s]+)", link)
|
||||
doi = urllib.parse.unquote(m.group(1)) if m else None
|
||||
item_id = doi.lower() if doi else link
|
||||
if not item_id:
|
||||
continue
|
||||
|
|
@ -261,15 +264,23 @@ def fetch_rss(feed_name: str, feed_url: str, source: str | None = None) -> list[
|
|||
elif hasattr(entry, "content") and entry.content:
|
||||
raw_abstract = entry.content[0].get("value", "")
|
||||
|
||||
# APS wraps the (truncated) abstract as "Author(s): A and B<br /><p>...</p>
|
||||
# <br /><p>[Phys. Rev. E 114, 044119] Published Tue Oct 13, 2026</p>".
|
||||
raw_abstract = re.sub(r"^\s*Author\(s\):.*?<br\s*/?>", "", raw_abstract, flags=re.S)
|
||||
raw_abstract = re.sub(r"(<br\s*/?>)?\s*(<p>)?\[[^\]]+\]\s*Published\b.*$", "", raw_abstract, flags=re.S)
|
||||
|
||||
day = ""
|
||||
if hasattr(entry, "published_parsed") and entry.published_parsed:
|
||||
day = time.strftime("%Y-%m-%d", entry.published_parsed)
|
||||
parsed = getattr(entry, "published_parsed", None) or getattr(entry, "updated_parsed", None)
|
||||
if parsed: # APS feeds give only <updated>, the publication time
|
||||
day = time.strftime("%Y-%m-%d", parsed)
|
||||
|
||||
authors = []
|
||||
if hasattr(entry, "authors"):
|
||||
authors = [a.get("name", "") for a in entry.authors if a.get("name")]
|
||||
elif hasattr(entry, "author"):
|
||||
authors = [entry.author]
|
||||
if len(authors) == 1 and " and " in authors[0]: # APS: "A, B, and C" in one string
|
||||
authors = [a.strip() for a in re.split(r",\s*and\s+|\s+and\s+|,\s*", authors[0]) if a.strip()]
|
||||
|
||||
items.append({
|
||||
"id": item_id,
|
||||
|
|
@ -292,8 +303,14 @@ def fetch_rss(feed_name: str, feed_url: str, source: str | None = None) -> list[
|
|||
# -- Crossref ----------------------------------------------------------------------
|
||||
|
||||
def _crossref_item(w: dict, source: str) -> dict:
|
||||
parts = (w.get("published-online") or w.get("published") or w.get("issued") or {}).get("date-parts", [[]])[0]
|
||||
day = "-".join(f"{p:02d}" for p in parts) if len(parts) == 3 else ""
|
||||
# First complete publication date. RSC deposits only the year for accepted
|
||||
# manuscripts, so fall back to the deposit date (created), which is always complete.
|
||||
day = ""
|
||||
for key in ("published-online", "published", "issued", "created"):
|
||||
parts = ((w.get(key) or {}).get("date-parts") or [[]])[0]
|
||||
if len(parts) == 3 and all(parts):
|
||||
day = "-".join(f"{p:02d}" for p in parts)
|
||||
break
|
||||
authors = [" ".join(filter(None, [a.get("given"), a.get("family")])) or a.get("name", "")
|
||||
for a in w.get("author", [])]
|
||||
abstract = _clean(w.get("abstract", ""))
|
||||
|
|
|
|||
166
art/label.py
Normal file
166
art/label.py
Normal file
|
|
@ -0,0 +1,166 @@
|
|||
"""`bingham label`: build the labeled set by marking papers relevant to profiles.
|
||||
|
||||
"Relevant" means: would you want this paper under this profile in your
|
||||
digest? A paper that is interesting but belongs to no profile is not relevant.
|
||||
|
||||
Papers are sampled at random within bands of their best re-ranker score over
|
||||
all profiles, so that the rare high-scoring papers and the large mass of
|
||||
papers the re-ranker never saw are both represented:
|
||||
|
||||
none not re-ranked for any profile (not nominated by cosine or BM25)
|
||||
<0.1, 0.1-0.2, 0.2-0.4, >=0.4
|
||||
|
||||
The bands do not depend on the thresholds, so labels stay valid when the
|
||||
thresholds change. bingham eval weights each band by its size in the store
|
||||
to estimate recall over the whole stream. The labeling screen is blind: it
|
||||
shows the paper and the profile names, never the scores.
|
||||
|
||||
Labels are appended to user/eval/labeled.jsonl, one JSON object per paper:
|
||||
{"id", "relevant": [profile ids], "profiles": [ids shown], "band",
|
||||
"profile_versions", "labeled_at", "host", "near_miss"}
|
||||
A paper labeled twice keeps its latest label.
|
||||
"""
|
||||
|
||||
import json
|
||||
import random
|
||||
import textwrap
|
||||
|
||||
from art import env, store
|
||||
from art.profile import profiles as prof
|
||||
from art.screen import embed, rerank
|
||||
|
||||
BANDS = ["none", "<0.1", "0.1-0.2", "0.2-0.4", ">=0.4"]
|
||||
|
||||
|
||||
def band(best: float | None) -> str:
|
||||
if best is None:
|
||||
return "none"
|
||||
if best < 0.1:
|
||||
return "<0.1"
|
||||
if best < 0.2:
|
||||
return "0.1-0.2"
|
||||
if best < 0.4:
|
||||
return "0.2-0.4"
|
||||
return ">=0.4"
|
||||
|
||||
|
||||
def labels_path():
|
||||
return env.USER_DIR / "eval" / "labeled.jsonl"
|
||||
|
||||
|
||||
def load_labels() -> dict[str, dict]:
|
||||
"""{item id: latest label record}."""
|
||||
path = labels_path()
|
||||
out = {}
|
||||
if path.exists():
|
||||
for line in path.read_text().splitlines():
|
||||
if line.strip():
|
||||
rec = json.loads(line)
|
||||
out[rec["id"]] = rec
|
||||
return out
|
||||
|
||||
|
||||
def population(conn, profiles: list) -> dict[str, dict]:
|
||||
"""{item id: {"band", "scores": {profile id: screens row}}} for every screened, screenable item.
|
||||
|
||||
Uses the current profile versions and models, as the digest does.
|
||||
"""
|
||||
current = {(p.id, p.version) for p in profiles}
|
||||
em, rm = embed.model_name(), rerank.model_name()
|
||||
screenable = {r[0] for r in conn.execute(f"SELECT id FROM items WHERE {store.SCREENABLE}")}
|
||||
pop: dict[str, dict] = {}
|
||||
for r in conn.execute("SELECT * FROM screens WHERE embed_model = ? AND rerank_model = ?", (em, rm)):
|
||||
if r["item_id"] in screenable and (r["profile_id"], r["profile_version"]) in current:
|
||||
pop.setdefault(r["item_id"], {"scores": {}})["scores"][r["profile_id"]] = dict(r)
|
||||
for v in pop.values():
|
||||
reranks = [s["rerank"] for s in v["scores"].values() if s["rerank"] is not None]
|
||||
v["band"] = band(max(reranks) if reranks else None)
|
||||
return pop
|
||||
|
||||
|
||||
def sample(pop: dict, labeled: set, n: int, rng: random.Random,
|
||||
only: set | None = None) -> list[str]:
|
||||
"""n unlabeled item ids, spread evenly over the bands (uniform within each band)."""
|
||||
pools = {b: sorted(i for i, v in pop.items() if v["band"] == b and i not in labeled
|
||||
and (only is None or i in only)) for b in BANDS}
|
||||
for b in pools:
|
||||
rng.shuffle(pools[b])
|
||||
picked = []
|
||||
while len(picked) < n and any(pools.values()):
|
||||
for b in BANDS:
|
||||
if pools[b] and len(picked) < n:
|
||||
picked.append(pools[b].pop())
|
||||
rng.shuffle(picked) # do not present bands in order: keeps the screen blind
|
||||
return picked
|
||||
|
||||
|
||||
def _show(item: dict, profiles: list, k: int, total: int) -> None:
|
||||
authors = ", ".join(item["authors"][:4]) + (" et al." if len(item["authors"]) > 4 else "")
|
||||
print(f"\n[{k}/{total}] {item['title']}")
|
||||
print(f" {authors or '(authors not listed)'} · {item['date']} · {item['source']}"
|
||||
+ (f" ({', '.join(item['categories'][:3])})" if item["categories"] else ""))
|
||||
abstract = item["abstract"] or "(no abstract)"
|
||||
print(textwrap.indent(textwrap.fill(abstract[:1400] + (" ..." if len(abstract) > 1400 else ""),
|
||||
width=96), " "))
|
||||
legend = " ".join(f"{i}={p.name}" for i, p in enumerate(profiles, 1))
|
||||
print(textwrap.indent(textwrap.fill(legend, width=96, break_on_hyphens=False), " "))
|
||||
|
||||
|
||||
def _ask(n_profiles: int) -> list[int] | str:
|
||||
while True:
|
||||
raw = input(" want it under (profile numbers, Enter = none, s = skip, q = quit): ").strip().lower()
|
||||
if raw in ("q", "s"):
|
||||
return raw
|
||||
try:
|
||||
nums = sorted({int(x) for x in raw.replace(",", " ").split()})
|
||||
except ValueError:
|
||||
print(" numbers separated by spaces, e.g. 1 3")
|
||||
continue
|
||||
if all(1 <= x <= n_profiles for x in nums):
|
||||
return nums
|
||||
print(f" profile numbers run from 1 to {n_profiles}")
|
||||
|
||||
|
||||
def run(conn, n: int, near_misses: bool = False, seed: int | None = None) -> int:
|
||||
"""Interactive session; returns the number of papers labeled."""
|
||||
data = prof.load()
|
||||
profiles = data.profiles
|
||||
pop = population(conn, profiles)
|
||||
labeled = set(load_labels())
|
||||
only = None
|
||||
if near_misses: # the digest's "just below threshold" papers, any date
|
||||
th = env.config()["screen"]["thresholds"]
|
||||
floor = env.config()["digest"]["near_miss_floor"]
|
||||
only = {i for i, v in pop.items()
|
||||
if any(s["rerank"] is not None and floor * th.get(pid, th["default"]) <= s["rerank"]
|
||||
< th.get(pid, th["default"]) for pid, s in v["scores"].items())
|
||||
and not any(s["rerank"] is not None and s["rerank"] >= th.get(pid, th["default"])
|
||||
for pid, s in v["scores"].items())}
|
||||
ids = sample(pop, labeled, n, random.Random(seed), only)
|
||||
if not ids:
|
||||
print("nothing left to label" + (" among near misses" if near_misses else ""))
|
||||
return 0
|
||||
print(f"{len(labeled)} papers labeled so far; {len(pop)} screened papers in the store.")
|
||||
print("For each paper: would you want it under this profile in your digest?")
|
||||
print("Judge from the paper alone; mark every profile it belongs under, or none.")
|
||||
path = labels_path()
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
done = 0
|
||||
with path.open("a") as f:
|
||||
for k, iid in enumerate(ids, 1):
|
||||
item = store.item_dict(conn.execute("SELECT * FROM items WHERE id = ?", (iid,)).fetchone())
|
||||
_show(item, profiles, k, len(ids))
|
||||
ans = _ask(len(profiles))
|
||||
if ans == "q":
|
||||
break
|
||||
if ans == "s":
|
||||
continue
|
||||
rec = {"id": iid, "relevant": [profiles[i - 1].id for i in ans],
|
||||
"profiles": [p.id for p in profiles], "band": pop[iid]["band"],
|
||||
"profile_versions": {p.id: p.version for p in profiles},
|
||||
"labeled_at": store.now(), "host": env.host(), "near_miss": near_misses}
|
||||
f.write(json.dumps(rec) + "\n")
|
||||
f.flush()
|
||||
done += 1
|
||||
print(f"\nlabeled {done}; {len(labeled) + done} in {path}")
|
||||
return done
|
||||
22
art/llm.py
22
art/llm.py
|
|
@ -3,8 +3,8 @@
|
|||
Each call sets the model, options, and num_ctx explicitly (Ollama's default
|
||||
context silently truncates long prompts, the first failure mode in Stan's
|
||||
taxonomy) and writes one row to llm_calls: host, model tag and digest,
|
||||
Modelfile hash, prompt name and version, num_ctx, token counts, options,
|
||||
latency, the raw response, and the outcome. Raw responses are kept even when
|
||||
Modelfile hash, Ollama version, prompt name and version, num_ctx, token
|
||||
counts, options, latency, the raw response, and the outcome. Raw responses are kept even when
|
||||
they fail validation.
|
||||
|
||||
The model is built from modelfiles/researcher.Modelfile and tagged with the
|
||||
|
|
@ -96,6 +96,17 @@ def available() -> bool:
|
|||
return False
|
||||
|
||||
|
||||
@cache
|
||||
def server_version() -> str | None:
|
||||
"""The Ollama server's version, recorded with every call: engine updates can
|
||||
change outputs and speed for the same model. The Python client has no call
|
||||
for /api/version, so this uses its HTTP session (same host)."""
|
||||
try:
|
||||
return client()._client.get("/api/version", timeout=5).json()["version"]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
@cache
|
||||
def ensure_model() -> tuple[str, str]:
|
||||
"""(tag, digest) of the researcher model, creating it from the Modelfile if missing."""
|
||||
|
|
@ -200,13 +211,14 @@ def _log(conn, tag, digest, p, item_id, profile, run_id, opts, fmt, c, latency,
|
|||
cur = conn.execute(
|
||||
"INSERT INTO llm_calls (ts, run_id, host, model, modelfile_hash, prompt_name, prompt_version, "
|
||||
"item_id, profile_id, profile_version, num_ctx, prompt_tokens, eval_tokens, options, format, "
|
||||
"think, latency_s, raw_response, outcome, validation, attempt, model_digest, done_reason, thinking) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"think, latency_s, raw_response, outcome, validation, attempt, model_digest, done_reason, thinking, "
|
||||
"ollama_version) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(store.now(), run_id, env.host(), tag, modelfile_hash(), p.name, p.version, item_id,
|
||||
profile.id if profile else None, profile.version if profile else None, opts["num_ctx"],
|
||||
prompt_tokens, eval_tokens, json.dumps(opts), json.dumps(fmt) if fmt else None,
|
||||
int(bool(c["think"])), round(latency, 3), raw, outcome,
|
||||
json.dumps(details) if details else None, attempt, digest, done_reason, thinking or None))
|
||||
json.dumps(details) if details else None, attempt, digest, done_reason, thinking or None,
|
||||
server_version()))
|
||||
conn.commit()
|
||||
return cur.lastrowid
|
||||
|
||||
|
|
|
|||
29
art/store.py
29
art/store.py
|
|
@ -121,7 +121,8 @@ CREATE TABLE IF NOT EXISTS llm_calls (
|
|||
attempt INTEGER NOT NULL DEFAULT 1,
|
||||
model_digest TEXT, -- Ollama digest of the weights behind the tag
|
||||
done_reason TEXT, -- stop, length, ...
|
||||
thinking TEXT -- thinking text when think is on
|
||||
thinking TEXT, -- thinking text when think is on
|
||||
ollama_version TEXT -- Ollama server version (/api/version)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS generations (
|
||||
|
|
@ -146,6 +147,7 @@ MIGRATIONS = [
|
|||
("llm_calls", "model_digest", "TEXT"),
|
||||
("llm_calls", "done_reason", "TEXT"),
|
||||
("llm_calls", "thinking", "TEXT"),
|
||||
("llm_calls", "ollama_version", "TEXT"),
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -166,6 +168,7 @@ def connect() -> sqlite3.Connection:
|
|||
for table, column, kind in MIGRATIONS:
|
||||
if column not in {r["name"] for r in conn.execute(f"PRAGMA table_info({table})")}:
|
||||
conn.execute(f"ALTER TABLE {table} ADD COLUMN {column} {kind}")
|
||||
_rekey_link_ids(conn)
|
||||
_import_legacy_cache(conn)
|
||||
return conn
|
||||
|
||||
|
|
@ -261,6 +264,30 @@ def finish_run(conn, run_id: int, stages: dict, stats: dict, device: str | None
|
|||
conn.commit()
|
||||
|
||||
|
||||
def _rekey_link_ids(conn) -> None:
|
||||
"""Re-key items stored under a URL that contains a DOI (fetch_rss before it read DOIs from links).
|
||||
|
||||
Springer's Rheologica Acta feed has no DOI field, so its papers were once
|
||||
keyed by link; they now arrive keyed by DOI. Moving the old rows (and their
|
||||
screens, embeddings, and generated text) keeps them from coming back as
|
||||
new papers. Rows whose DOI is already stored under its own id are left alone.
|
||||
"""
|
||||
import re
|
||||
import urllib.parse
|
||||
for (old,) in conn.execute("SELECT id FROM items WHERE id LIKE 'http%'").fetchall():
|
||||
m = re.search(r"/(10\.\d{4,9}/[^?#\s]+)", old)
|
||||
if not m:
|
||||
continue
|
||||
new = urllib.parse.unquote(m.group(1)).lower()
|
||||
if conn.execute("SELECT 1 FROM items WHERE id = ?", (new,)).fetchone():
|
||||
continue
|
||||
conn.execute("UPDATE items SET id = ?, doi = COALESCE(doi, ?) WHERE id = ?", (new, new, old))
|
||||
for table, column in (("screens", "item_id"), ("embeddings", "key"),
|
||||
("generations", "item_id"), ("llm_calls", "item_id")):
|
||||
conn.execute(f"UPDATE {table} SET {column} = ? WHERE {column} = ?", (new, old))
|
||||
conn.commit()
|
||||
|
||||
|
||||
# -- one-time import of the v0 item cache --------------------------------------
|
||||
|
||||
LEGACY_CACHE = ROOT / "data" / "cache" / "feed_cache.json"
|
||||
|
|
|
|||
|
|
@ -27,6 +27,13 @@ digest:
|
|||
# similarity are grouped. Set 2026-09-28 from the first week: unrelated
|
||||
# labels median 0.52; same-topic pairs 0.8 and above.
|
||||
label_cluster_threshold: 0.8
|
||||
# "Just below threshold": per profile, up to near_misses papers scoring
|
||||
# between near_miss_floor x threshold and the threshold.
|
||||
near_misses: 3
|
||||
near_miss_floor: 0.5
|
||||
|
||||
eval:
|
||||
target_recall: 0.9 # bingham eval suggests the highest threshold that keeps this recall
|
||||
|
||||
paths: # relative to the repository root
|
||||
db: data/art.sqlite
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Journal RSS feeds for rheology literature (read with fetch_rss).
|
||||
# Journal feeds for rheology literature: RSS (fetch_rss) or Crossref by ISSN.
|
||||
#
|
||||
# arXiv categories are in config.yaml (arxiv.categories); the daily run reads
|
||||
# rss.arxiv.org for each. J. Rheol. is also queried through Crossref by ISSN
|
||||
|
|
@ -6,7 +6,24 @@
|
|||
# issue in progress. Items from both are merged by DOI.
|
||||
#
|
||||
# source: the name shown in the digest; feeds with the same source are merged.
|
||||
# Journal RSS URLs are known to change. Verify each before enabling.
|
||||
# Each entry has either `url` (RSS, read with feedparser) or `crossref_issn`
|
||||
# (Crossref by ISSN, like J. Rheol.). Channels checked 2026-09-30:
|
||||
# ACS (Langmuir, Macromolecules): RSS blocked by Cloudflare; Crossref has
|
||||
# abstracts for ~95% of articles.
|
||||
# RSC (Soft Matter): feeds.rsc.org is gone; Crossref has ~75% of abstracts.
|
||||
# Springer (Rheologica Acta): RSS works, with abstracts; no Crossref abstracts.
|
||||
# Elsevier (JNNFM): RSS works but has no abstracts, and neither does Crossref,
|
||||
# so papers are screened on title only.
|
||||
# APS (Phys. Rev. E, Lett., Materials), checked 2026-10-14: RSS has DOIs,
|
||||
# authors, and the last 100 articles (two to six weeks), but abstracts are
|
||||
# cut at ~400 characters; Crossref has almost no APS abstracts.
|
||||
# AIP (J. Chem. Phys., Phys. Fluids), checked 2026-10-15: journal pages are
|
||||
# behind Cloudflare; Crossref has abstracts for ~99% of articles.
|
||||
# Elsevier (J. Colloid Interface Sci.), checked 2026-10-15: RSS has no
|
||||
# abstracts, DOIs, or dates; Crossref has DOIs, authors, and dates but no
|
||||
# abstracts, so papers are screened on title only.
|
||||
# A Crossref entry's first fetch covers the last week, or starts at its own
|
||||
# `backfill_from: YYYY-MM-DD`. Journal RSS URLs change; verify before enabling.
|
||||
|
||||
feeds:
|
||||
- name: Journal of Rheology - current issue
|
||||
|
|
@ -26,15 +43,55 @@ feeds:
|
|||
|
||||
- name: Soft Matter
|
||||
source: Soft Matter
|
||||
url: https://feeds.rsc.org/rss/sm
|
||||
crossref_issn: 1744-683X
|
||||
enabled: false
|
||||
|
||||
- name: Journal of Non-Newtonian Fluid Mechanics
|
||||
source: JNNFM
|
||||
url: https://rss.sciencedirect.com/publication/science/03770257
|
||||
enabled: false
|
||||
enabled: false # titles only: the feed has no abstracts
|
||||
|
||||
- name: Macromolecules
|
||||
source: Macromolecules
|
||||
url: https://pubs.acs.org/action/showFeed?type=axatoc&feed=rss&jc=mamobx
|
||||
crossref_issn: 0024-9297
|
||||
enabled: false
|
||||
|
||||
- name: Langmuir
|
||||
source: Langmuir
|
||||
crossref_issn: 0743-7463
|
||||
enabled: false
|
||||
|
||||
- name: Physical Review E
|
||||
source: Phys. Rev. E
|
||||
url: https://feeds.aps.org/rss/recent/pre.xml
|
||||
enabled: false # verified 2026-10-14: abstracts truncated at ~400 characters
|
||||
|
||||
- name: Physical Review Letters
|
||||
source: Phys. Rev. Lett.
|
||||
url: https://feeds.aps.org/rss/recent/prl.xml
|
||||
enabled: false # verified 2026-10-14: abstracts truncated at ~400 characters
|
||||
|
||||
- name: Physical Review Materials
|
||||
source: Phys. Rev. Materials
|
||||
url: https://feeds.aps.org/rss/recent/prmaterials.xml
|
||||
enabled: false # verified 2026-10-14: abstracts truncated at ~400 characters
|
||||
|
||||
- name: Physical Review Fluids
|
||||
source: Phys. Rev. Fluids
|
||||
url: https://feeds.aps.org/rss/recent/prfluids.xml
|
||||
enabled: false # verified 2026-10-15: abstracts truncated at ~400 characters
|
||||
|
||||
- name: Journal of Chemical Physics
|
||||
source: J. Chem. Phys.
|
||||
crossref_issn: 0021-9606
|
||||
enabled: false # verified 2026-10-15: abstracts for 65 of 66 recent articles
|
||||
|
||||
- name: Physics of Fluids
|
||||
source: Phys. Fluids
|
||||
crossref_issn: 1070-6631
|
||||
enabled: false # verified 2026-10-15: abstracts for 143 of 144 recent articles
|
||||
|
||||
- name: Journal of Colloid and Interface Science
|
||||
source: J. Colloid Interface Sci.
|
||||
crossref_issn: 0021-9797
|
||||
enabled: false # titles only: neither RSS nor Crossref has abstracts
|
||||
|
|
|
|||
544
grounding/sor-nomenclature.md
Normal file
544
grounding/sor-nomenclature.md
Normal file
|
|
@ -0,0 +1,544 @@
|
|||
# Notation and conventions — Official Nomenclature of the US and European Societies of Rheology
|
||||
|
||||
*Scope: the joint SoR/ESR nomenclature, 2018-10 edition.* This file gives the
|
||||
symbols, definitions, and SI units that The Society of Rheology asks authors to
|
||||
use, so that an AI model reading or writing rheology follows them. Paste it as
|
||||
the first message of a chat, or keep it in a project so a coding assistant
|
||||
(Claude Code, Copilot, Cursor, and the like) reads it. Paste it again when a
|
||||
new conversation starts. Notation is written in LaTeX.
|
||||
|
||||
Source: J. M. Dealy and D. Vlassopoulos, with advice from G. G. Fuller,
|
||||
R. G. Larson, J. Mewis, D. J. Read, J. Vermant, N. J. Wagner, and G. McKinley,
|
||||
"Official Nomenclature of US and European Societies of Rheology (2018-10),"
|
||||
https://www.rheology.org/sor1/Publications/SoR_Nomenclature_201810.pdf.
|
||||
Earlier version: "Official symbols and nomenclature of The Society of
|
||||
Rheology," J. Rheol. 57, 1047 (2013), https://doi.org/10.1122/1.4811184.
|
||||
The symbol tables below transcribe the 2018 edition; the conventions section
|
||||
is a summary written for this primer. Where the source is inconsistent or
|
||||
appears to contain a typographical error, the entry is given as printed and
|
||||
the issue is listed under "Notes on the source".
|
||||
|
||||
If you only have room for part of this file, the "Conventions" section alone
|
||||
covers the rules most often broken.
|
||||
|
||||
## How to use this notation
|
||||
|
||||
- Write rheological quantities with the symbols below, in SI units.
|
||||
- When a paper, textbook, or user uses different symbols (for example $\tau$
|
||||
for shear stress, $\lambda$ for a relaxation time, $\mu$ for viscosity),
|
||||
translate to this notation and say once that you have done so, e.g.
|
||||
"the paper's $\tau_{xy}$ is the shear stress $\sigma$ here."
|
||||
- When quoting a source verbatim, keep its symbols and mark the quotation.
|
||||
- Do not invent a symbol for a quantity that has one here. For a quantity not
|
||||
covered here, define your symbol explicitly the first time it is used.
|
||||
- If a symbol is ambiguous in context (several are reused below), say which
|
||||
quantity is meant.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Stress is $\sigma$, not $\tau$.** Shear stress is $\sigma$ (the shear
|
||||
component of the total stress tensor $\boldsymbol{\sigma}$). The symbol
|
||||
$\boldsymbol{\tau}$ is reserved for the extra stress tensor; scalar $\tau$
|
||||
denotes relaxation times ($\tau_i$, $\tau_d$, $\tau_R$, ...).
|
||||
- **Simple shear coordinates:** $x_1$ flow direction, $x_2$ velocity-gradient
|
||||
direction, $x_3$ neutral direction. Shear strain $\gamma = dx_1/dx_2$, shear
|
||||
rate $\dot{\gamma} = |dv_1/dx_2|$.
|
||||
- **Viscosity is $\eta$.** $\eta(\dot{\gamma}) = |\sigma|/\dot{\gamma}$;
|
||||
zero-shear viscosity $\eta_0$. The viscosity of a suspending medium or matrix
|
||||
is $\eta_m$ (the source discourages $\mu$ and, where the medium is itself
|
||||
complex, the subscript $f$). Solvent viscosity is $\eta_s$.
|
||||
- **Normal stresses:** $N_1 = \sigma_{11} - \sigma_{22}$,
|
||||
$N_2 = \sigma_{22} - \sigma_{33}$; coefficients
|
||||
$\Psi_1 = N_1/\dot{\gamma}^2$, $\Psi_2 = N_2/\dot{\gamma}^2$.
|
||||
- **Relaxation times are $\tau$, not $\lambda$.** Discrete spectrum:
|
||||
strengths $g_i$ and times $\tau_i$; continuous spectrum $H(\tau)$. In the
|
||||
molecular table, $\lambda$ is chain stretch (stretch ratio).
|
||||
- **Linear viscoelasticity:** relaxation modulus $G(t)$, creep compliance
|
||||
$J(t)$. In oscillatory shear, $\gamma(t) = \gamma_0 \sin\omega t$ and
|
||||
$\sigma(t) = \sigma_0 \sin(\omega t + \delta)$; storage and loss moduli
|
||||
$G'$, $G''$; $G^* = G' + iG''$; loss angle $\delta$ (so the loss tangent is
|
||||
$\tan\delta = G''/G'$). Complex viscosity $\eta^* = \eta' - i\eta''$ with
|
||||
$\eta' = G''/\omega$ and $\eta'' = G'/\omega$. Complex compliance
|
||||
$J^* = J' - iJ''$. Plateau modulus $G_N^0$.
|
||||
- **Transient (nonlinear) functions:** superscript $+$ for start-up of flow
|
||||
(growth functions, e.g. $\eta^+(t,\dot{\gamma})$, $\eta_E^+(t,\dot{\varepsilon})$),
|
||||
superscript $-$ for cessation (decay functions, e.g. $\eta^-(t,\dot{\gamma})$).
|
||||
- **Extension:** Hencky strain $\varepsilon = \ln(L/L_0)$, Hencky strain rate
|
||||
$\dot{\varepsilon}$, net tensile stress $\sigma_E = \sigma_{zz} - \sigma_{rr}$,
|
||||
extensional viscosity $\eta_E(\dot{\varepsilon})$ (the long-time limit of
|
||||
$\eta_E^+$). Biaxial quantities carry subscript $B$. Young's modulus is $E$;
|
||||
$K$ is recommended for compression (dilatational) moduli.
|
||||
- **Yield:** yield stress $\sigma_y$, yield strain $\gamma_y$.
|
||||
- **Rate-of-strain tensor without the 1/2:**
|
||||
$\dot{\boldsymbol{\gamma}} = \nabla\mathbf{v} + (\nabla\mathbf{v})^T$ and
|
||||
vorticity tensor $\boldsymbol{\Omega} = \nabla\mathbf{v} - (\nabla\mathbf{v})^T$.
|
||||
If the half-sum is wanted, call it
|
||||
$\mathbf{D} = \tfrac{1}{2}[\nabla\mathbf{v} + (\nabla\mathbf{v})^T]$; do not
|
||||
reuse $\dot{\boldsymbol{\gamma}}$ for it.
|
||||
- **Weissenberg vs. Deborah.** $Wi$ = (characteristic time of the fluid) x
|
||||
(rate of deformation), e.g. $\tau\dot{\gamma}$ or $\tau\dot{\varepsilon}$.
|
||||
$De$ = (characteristic time of the fluid) / (duration of the deformation),
|
||||
e.g. $\tau\omega$. $N_1/\sigma$ is the *stress ratio* $SR$, a property of
|
||||
the data, not a Weissenberg number. Subscripts name the time used, e.g.
|
||||
$Wi_R$ (Rouse time) and $Wi_d$ (longest relaxation time).
|
||||
- **Suspensions:** volume fraction $\phi$, maximum packing fraction
|
||||
$\phi_{max}$, relative viscosity $\eta_r$, particle contribution $\eta_p$.
|
||||
The bulk stress is $\boldsymbol{\Sigma} = \boldsymbol{\Sigma}^f + \boldsymbol{\Sigma}^p$,
|
||||
split further into hydrodynamic $\Sigma^H$, interparticle $\Sigma^{IP}$, and
|
||||
Brownian $\Sigma^B$ parts; particle pressure $\Pi$. Péclet number
|
||||
$Pé = \dot{\gamma}a^2/D_0$ with $a$ the particle radius.
|
||||
- **Interfacial rheology:** superscript $s$ (e.g. $\eta^s$, $G^{s\prime}$),
|
||||
with units per unit length (Pa m = N m$^{-1}$). Interfacial tension
|
||||
$\sigma_{\alpha,\beta}$; surface pressure $\Pi$; interfacial dilatational
|
||||
strain $\alpha^s = \ln(A/A_0)$; interfacial concentration $\Gamma$.
|
||||
- **Polymer solutions:** relative, specific, reduced, and intrinsic viscosity
|
||||
$\eta_r$, $\eta_{sp}$, $\eta_{red}$, $[\eta]$; overlap concentration $c^*$.
|
||||
- **Entangled polymers:** entanglement molecular weight
|
||||
$M_e = \rho RT/G_N^0$ (Ferry); in tube-model work use
|
||||
$M_e = \tfrac{4}{5}\rho RT/G_N^0$ and say so. $Z = M/M_e$, reptation time
|
||||
$\tau_d$, Rouse time $\tau_R$.
|
||||
- **Units:** SI throughout, with a space (or a dot) between units: Pa s,
|
||||
Pa s$^2$, s$^{-1}$, rad. The source notes that "molecular weight" in
|
||||
kg mol$^{-1}$ is formally a molar mass, but keeps the familiar term.
|
||||
|
||||
## Symbols
|
||||
|
||||
### Table I. Steady simple shear (viscometric flow)
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `x_1` | direction of flow (simple shear) | | m |
|
||||
| `x_2` | direction of velocity gradient (simple shear) | | m |
|
||||
| `x_3` | neutral direction (simple shear) | | m |
|
||||
| `F` | shear or normal force | `ma` | N |
|
||||
| `v_1` | velocity | `dx_1/dt` | m s^-1 |
|
||||
| `a_1` | acceleration | `dv_1/dt` | m s^-2 |
|
||||
| `\sigma` | shear stress | `F/A` (A = area) | Pa |
|
||||
| `\gamma` | shear strain | `dx_1/dx_2` | – |
|
||||
| `\dot{\gamma}` | shear rate | `\lvert dv_1/dx_2 \rvert` | s^-1 |
|
||||
| `\omega_3` | vorticity (fluid dynamics adds a prefactor 1/2) | `-dv_1/dx_2` | s^-1 |
|
||||
| `\eta(\dot{\gamma})` | viscosity | `\lvert\sigma\rvert/\dot{\gamma}` | Pa s |
|
||||
| `\sigma_y` | yield stress | | Pa |
|
||||
| `\gamma_y` | yield strain | | – |
|
||||
| `N_1` | first normal stress difference | `\sigma_{11}-\sigma_{22}` | Pa |
|
||||
| `N_2` | second normal stress difference | `\sigma_{22}-\sigma_{33}` | Pa |
|
||||
| `\Psi_1` | first normal stress coefficient | `N_1/\dot{\gamma}^2` | Pa s^2 |
|
||||
| `\Psi_2` | second normal stress coefficient | `N_2/\dot{\gamma}^2` | Pa s^2 |
|
||||
| `\Psi(\dot{\gamma})` | normal stress ratio | `-N_1/N_2` as printed (see notes) | – |
|
||||
| `\eta_0` | zero-shear viscosity | `\eta(\dot{\gamma}\to 0)` | Pa s |
|
||||
| `\Psi_{1,0}` | zero-shear first normal stress coefficient | `\Psi_1(\dot{\gamma}\to 0)` | Pa s^2 |
|
||||
| `n` | power-law exponent | `\eta = K\dot{\gamma}^{n-1}` | – |
|
||||
| `K` | power-law prefactor (consistency) | see above | Pa s^n |
|
||||
|
||||
### Table II. Linear viscoelasticity
|
||||
|
||||
Simple shear:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `G` | shear modulus of a solid | `\sigma/\gamma` | Pa |
|
||||
| `G(t)` | relaxation modulus | `\sigma(t)/\gamma` | Pa |
|
||||
| `g_i` | relaxation strength in discrete spectrum | | Pa |
|
||||
| `\tau_i` | relaxation time in discrete spectrum | | s |
|
||||
| `H(\tau)` | continuous relaxation spectrum | `G(t)=\int_{-\infty}^{\infty} H(\tau)\, e^{-t/\tau}\, d\ln\tau` | Pa |
|
||||
| `G^*_\perp(\omega,\dot{\gamma})` | orthogonal superposition complex modulus | | Pa |
|
||||
| `G^*_\parallel(\omega,\dot{\gamma})` | parallel superposition complex modulus | | Pa |
|
||||
| `m(s)` | memory function | `-dG(s)/ds` | Pa s^-1 |
|
||||
| `J(t)` | creep compliance (shear) | `\gamma(t)/\sigma` | Pa^-1 |
|
||||
| `J_e` | equilibrium compliance of solid | `J(t\to\infty)` | Pa^-1 |
|
||||
| `J_r(t)` | recoverable compliance | `J(t)-t/\eta_0` | Pa^-1 |
|
||||
| `J_s^0` | steady-state compliance of fluid | `J(t)-t/\eta_0\ (t\to\infty)` | Pa^-1 |
|
||||
| `L(\tau)` | continuous retardation spectrum | `J(t)=\int_{-\infty}^{\infty} L(\tau)[1-e^{-t/\tau}]\, d\ln\tau` (see notes) | Pa^-1 |
|
||||
|
||||
The superposition subscripts $\perp$ and $\parallel$ apply equally to storage
|
||||
and loss moduli and viscosities.
|
||||
|
||||
Small-amplitude oscillatory shear:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\gamma_0` | strain amplitude | `\gamma(t)=\gamma_0\sin\omega t` | – |
|
||||
| `\delta` | loss angle (phase angle) | `\sigma(t)=\sigma_0\sin(\omega t+\delta)` | rad |
|
||||
| `\sigma_0` | stress amplitude | as above | Pa |
|
||||
| `G^*` | complex modulus | `G'+iG''` | Pa |
|
||||
| `\lvert G^*\rvert` or `G_d` | absolute magnitude of `G^*` | `\sigma_0/\gamma_0` | Pa |
|
||||
| `G'` | storage modulus | `G_d\cos\delta` | Pa |
|
||||
| `G''` | loss modulus | `G_d\sin\delta` | Pa |
|
||||
| `\eta^*` | complex viscosity | `\eta'-i\eta''` | Pa s |
|
||||
| `\lvert\eta^*\rvert` | absolute magnitude of `\eta^*` | `\sigma_0/\omega\gamma_0` | Pa s |
|
||||
| `\eta'` | dynamic viscosity (in phase with strain rate) | `G''/\omega` | Pa s |
|
||||
| `\eta''` | out-of-phase (with strain rate) component of `\eta^*` | `G'/\omega` | Pa s |
|
||||
| `J^*` | complex compliance | `J'-iJ''` | Pa^-1 |
|
||||
| `\lvert J^*\rvert` | absolute magnitude of `J^*` | `\gamma_0/\sigma_0 = 1/G_d` | Pa^-1 |
|
||||
| `J'` | storage compliance | `(\gamma_0/\sigma_0)\cos\delta` | Pa^-1 |
|
||||
| `J''` | loss compliance | `(\gamma_0/\sigma_0)\sin\delta` | Pa^-1 |
|
||||
| `G_N^0` | plateau modulus (inferred from `G'` and `G''`; there is no true plateau) | | Pa |
|
||||
|
||||
Tensile (uniaxial) extension (also applies to nonlinear extension):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\sigma_E` | net tensile stress | `\sigma_{zz}-\sigma_{rr}` | Pa |
|
||||
| `\varepsilon` | Hencky strain | `\ln(L/L_0)` | – |
|
||||
| `\dot{\varepsilon}` | Hencky strain rate | `d(\ln L)/dt` | s^-1 |
|
||||
| `E(t)` | tensile relaxation modulus | `\sigma_E(t)/\varepsilon_0` | Pa |
|
||||
| `D(t)` | tensile creep compliance | `\varepsilon_0(t)/\sigma_E` | Pa^-1 |
|
||||
|
||||
### Table III. Shift factors for time-temperature superposition
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `b_T` | vertical shift factor | `b_T G'(T,\omega a_T) = G'(T_0,\omega)` | – |
|
||||
| `a_T` | horizontal shift factor | as above | – |
|
||||
| `c_1` | first WLF coefficient | `\log_{10} a_T = -c_1(T-T_0)/[c_2+(T-T_0)]` | – |
|
||||
| `c_2` | second WLF coefficient | as above | K |
|
||||
| `E_a` | activation energy for flow | `a_T = \exp[(E_a/R)(1/T - 1/T_0)]` | kJ mol^-1 |
|
||||
|
||||
For rubbers, and polymers generally, $b_T = T_0\rho_0/T\rho$ with $\rho$ the
|
||||
mass density and $\rho_0 = \rho(T_0)$ (Ferry, 3rd ed., 1980).
|
||||
|
||||
### Table IV. Nonlinear viscoelasticity in shear
|
||||
|
||||
Stress relaxation (step strain):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\gamma_0` | strain amplitude | | – |
|
||||
| `G(t,\gamma_0)` | relaxation modulus (nonlinear) | `\sigma(t)/\gamma_0` | Pa |
|
||||
| `h(\gamma_0)` | damping function in shear | `G(t,\gamma_0)/G(t)` | – |
|
||||
| `N_1(t,\gamma_0)` | first normal stress relaxation function | | Pa |
|
||||
| `N_2(t,\gamma_0)` | second normal stress relaxation function | | Pa |
|
||||
| `\Psi_1(t,\gamma_0)` | first normal stress relaxation coefficient | `N_1(t,\gamma_0)/\gamma_0^2` | Pa |
|
||||
| `\Psi_2(t,\gamma_0)` | second normal stress relaxation coefficient | `N_2(t,\gamma_0)/\gamma_0^2` | Pa |
|
||||
|
||||
Start-up of steady shear (at fixed shear rate):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\sigma^+(t,\dot{\gamma})` | shear stress growth function | | Pa |
|
||||
| `\eta^+(t,\dot{\gamma})` | shear stress growth coefficient | `\sigma^+(t,\dot{\gamma})/\dot{\gamma}` | Pa s |
|
||||
| `N_1^+(t,\dot{\gamma})` | first normal stress growth function | `\sigma_{11}-\sigma_{22}` | Pa |
|
||||
| `\Psi_1^+(t,\dot{\gamma})` | first normal stress growth coefficient | `N_1^+(t,\dot{\gamma})/\dot{\gamma}^2` | Pa s^2 |
|
||||
| `N_2^+(t,\dot{\gamma})` | second normal stress growth function | `\sigma_{22}-\sigma_{33}` | Pa |
|
||||
| `\Psi_2^+(t,\dot{\gamma})` | second normal stress growth coefficient | `N_2^+(t,\dot{\gamma})/\dot{\gamma}^2` | Pa s^2 |
|
||||
| `SR` | stress ratio | `N_1(\dot{\gamma})/\sigma(\dot{\gamma})` | – |
|
||||
|
||||
Cessation of steady shear ($\dot{\gamma} = 0$ from $t = 0$):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\sigma^-(t,\dot{\gamma})` | shear stress decay function | | Pa |
|
||||
| `\eta^-(t,\dot{\gamma})` | shear stress decay coefficient | `\sigma^-(t,\dot{\gamma})/\dot{\gamma}` | Pa s |
|
||||
| `N_1^-(t,\dot{\gamma})` | first normal stress decay function | `\sigma_{11}-\sigma_{22}` | Pa |
|
||||
| `\Psi_1^-(t,\dot{\gamma})` | first normal stress decay coefficient | `N_1^-(t,\dot{\gamma})/\dot{\gamma}^2` | Pa s^2 |
|
||||
| `N_2^-(t,\dot{\gamma})` | second normal stress decay function | `\sigma_{22}-\sigma_{33}` | Pa |
|
||||
| `\Psi_2^-(t,\dot{\gamma})` | second normal stress decay coefficient | `N_2^-(t,\dot{\gamma})/\dot{\gamma}^2` | Pa s^2 |
|
||||
|
||||
Creep and creep recovery (recoil):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `J(t,\sigma)` | creep compliance | `\gamma(t,\sigma)/\sigma` | Pa^-1 |
|
||||
| `J_s(\sigma)` | steady-state compliance | `J(t\to\infty,\sigma)` | Pa^-1 |
|
||||
| `\gamma_r(t',\sigma)` | recoverable strain after `t_0`, when `\sigma\to 0`; `t' = t - t_0` | `\gamma[t_0,\sigma]-\gamma[t,\sigma],\ t>t_0` | – |
|
||||
| `\gamma_\infty(\sigma)` | ultimate recoil | `\gamma_r(t'\to\infty,\sigma)` | – |
|
||||
| `J_r(\sigma)` | steady-state recoverable compliance | `\gamma_\infty(\sigma)/\sigma` | Pa^-1 |
|
||||
|
||||
By the Boltzmann principle the steady-state compliance and the steady-state
|
||||
recoverable compliance should be equal, although they are measured
|
||||
differently.
|
||||
|
||||
### Table V. Nonlinear viscoelasticity in extension
|
||||
|
||||
Tensile (uniaxial) extension:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\varepsilon` | engineering strain | `(L-L_0)/L_0` | – |
|
||||
| `\sigma` | engineering stress | `F/A_0` | Pa |
|
||||
| `E` | Young's modulus of a solid | `\sigma/\varepsilon` | Pa |
|
||||
| `\sigma_\varepsilon` | net tensile stress (true) | `\sigma_{zz}-\sigma_{rr}` | Pa |
|
||||
| `\varepsilon` or `\epsilon` | Hencky strain | `\ln(L/L_0)` | – |
|
||||
| `\dot{\varepsilon}` or `\dot{\epsilon}` | Hencky strain rate | `d(\ln L)/dt` | s^-1 |
|
||||
| `\sigma_E^+(t,\dot{\varepsilon})` | tensile stress growth function | | Pa |
|
||||
| `\eta_E^+(t,\dot{\varepsilon})` | tensile stress growth coefficient | `\sigma_E^+(t,\dot{\varepsilon})/\dot{\varepsilon}` | Pa s |
|
||||
| `\eta_E(\dot{\varepsilon})` | extensional viscosity | `\eta_E^+(t,\dot{\varepsilon}),\ t\to\infty` | Pa s |
|
||||
| `D(t,\sigma_E)` | tensile creep compliance | `\varepsilon(t)/\sigma_E` | Pa^-1 |
|
||||
| `\varepsilon_r(t',\varepsilon)` | recoverable strain after `t_0`, when `\sigma_E\to 0`; `t'=t-t_0` | `\varepsilon[t_0,\sigma_E]-\varepsilon[t,\sigma_E],\ t>t_0` | – |
|
||||
|
||||
In mechanics the same symbols are often used for engineering and true stress
|
||||
and strain; they agree only for very small deformations.
|
||||
|
||||
Biaxial extension (symmetrical):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\varepsilon_B` | biaxial strain | `\ln(R/R_0)` | – |
|
||||
| `\dot{\varepsilon}_B` | biaxial strain rate | `d(\ln R)/dt` | s^-1 |
|
||||
| `\sigma_B` | net stretching stress | `\sigma_{rr}-\sigma_{zz}` | Pa |
|
||||
| `\sigma_B^+(t,\dot{\varepsilon}_B)` | biaxial stretch growth function | | Pa |
|
||||
| `\eta_B^+(t,\dot{\varepsilon}_B)` | biaxial stretch growth coefficient | `\sigma_B^+(t,\dot{\varepsilon}_B)/\dot{\varepsilon}_B` | Pa s |
|
||||
| `\eta_B^-(t,\dot{\varepsilon}_B)` | biaxial stress decay coefficient | `\sigma_B^-(t,\dot{\varepsilon}_B)/\dot{\varepsilon}_B` | Pa s |
|
||||
| `\eta_B(\dot{\varepsilon}_B)` | biaxial extensional viscosity | `\sigma_B^+(t\to\infty,\dot{\varepsilon}_B)/\dot{\varepsilon}_B` | Pa s |
|
||||
| `D(t,\sigma_B)` | biaxial creep compliance | `\varepsilon_B(t)/\sigma_B` | Pa^-1 |
|
||||
|
||||
### Table VI. Rheometry
|
||||
|
||||
Capillary rheometers:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\sigma_A` | apparent wall shear stress | `P_d R/2L` (`P_d` = driving reservoir pressure, exit pressure neglected) | Pa |
|
||||
| `\dot{\gamma}_A` | apparent wall shear rate | `4Q/\pi R^3` | s^-1 |
|
||||
| `\sigma_W` | wall shear stress | `-\sigma_{rz}\ (r=R)` | Pa |
|
||||
| `\dot{\gamma}_W` | wall shear rate | `-dv_z/dr\ (r=R)` | s^-1 |
|
||||
|
||||
Cone-plate rheometers (cone radius $R$, cone angle $\beta$ measured from the
|
||||
plate, rotation $\varphi$ about the axis, normal force $F$ on the plate):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\beta` | cone angle | | rad |
|
||||
| `\varphi` | angular rotation | | rad |
|
||||
| `\Omega` | angular velocity | `d\varphi/dt` | rad s^-1 |
|
||||
| `M` | torque | `2\pi R^3\sigma_{\varphi\theta}/3` (valid for `\beta < 0.1` rad) | N m |
|
||||
| `F_z` | normal thrust | | N |
|
||||
|
||||
### Table VII. Solutions
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `c` | concentration | | kg m^-3 (g mL^-1 often used) |
|
||||
| `c^*` | overlap concentration | | kg m^-3 (see notes) |
|
||||
| `\eta_s` | solvent viscosity | | Pa s |
|
||||
| `\eta_r` | relative viscosity | `\eta/\eta_s` | – |
|
||||
| `\eta_{sp}` | specific viscosity | `\eta_r - 1` | – |
|
||||
| `\eta_{red}` | reduced viscosity | `\eta_{sp}/c` | m^3 kg^-1 (mL g^-1 often used) |
|
||||
| `[\eta]` | intrinsic viscosity | `\lim(\eta_{red};\ \dot{\gamma}\to 0,\ c\to 0)` | m^3 kg^-1 (mL g^-1 often used) |
|
||||
| `\eta_m` | viscosity of matrix (e.g. nanocomposites) | | Pa s |
|
||||
| `\boldsymbol{\sigma}^s` | solvent contribution to the stress tensor | | Pa |
|
||||
| `\boldsymbol{\sigma}^p` | polymeric contribution to the stress tensor | | Pa |
|
||||
|
||||
### Table VIII. Suspensions
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\phi` | volume fraction solid | `V_{solid}/V_{suspension}` | – |
|
||||
| `\phi_{max}` | maximum packing fraction | | – |
|
||||
| `\eta_m` | suspending medium viscosity | | Pa s |
|
||||
| `\eta` | effective viscosity of suspension | `\Sigma_{12}/\dot{\gamma}` | Pa s |
|
||||
| `\eta_r` | relative viscosity of suspension | `\eta/\eta_f` as printed (see notes) | – |
|
||||
| `\eta_p` | particle contribution to `\eta` | | Pa s |
|
||||
| `\boldsymbol{\sigma}(\mathbf{x},t)` | local stress tensor | | Pa |
|
||||
| `\boldsymbol{\Sigma}` | total bulk stress | `\boldsymbol{\Sigma}^f + \boldsymbol{\Sigma}^p` | Pa |
|
||||
| `\boldsymbol{\Sigma}^f` | fluid (suspending medium) contribution | | Pa |
|
||||
| `\boldsymbol{\Sigma}^p` | particle contribution | | Pa |
|
||||
| `N_1` | first normal stress difference | `\Sigma_{11}-\Sigma_{22}` | Pa |
|
||||
| `N_2` | second normal stress difference | `\Sigma_{22}-\Sigma_{33}` | Pa |
|
||||
| `\Upsilon_1` | dimensionless first normal stress difference | `-N_1/\eta_f\dot{\gamma}` | – |
|
||||
| `\Upsilon_2` | dimensionless second normal stress difference | `-N_2/\eta_f\dot{\gamma}` | – |
|
||||
| `\Pi` | particle pressure | `-\tfrac{1}{3}(\Sigma^p_{11}+\Sigma^p_{22}+\Sigma^p_{33})` | Pa |
|
||||
| `\Sigma^H` | hydrodynamic particle stress | | Pa |
|
||||
| `\Sigma^{IP}` | interparticle stress | | Pa |
|
||||
| `\Sigma^B` | Brownian stress | | Pa |
|
||||
|
||||
The source prefers subscript $m$ ("medium" or "matrix") to $f$ for the
|
||||
suspending fluid's viscosity, particularly when that fluid is itself
|
||||
rheologically complex, and notes that $\mu$ is often used for it.
|
||||
|
||||
### Table IX. Interfacial and surface rheology
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\sigma_{\alpha,\beta}` | interfacial or surface tension (`\alpha` or `\Gamma` are sometimes used; avoid) | | Pa m |
|
||||
| `\Pi` | surface pressure | `\sigma^0_{\alpha,\beta}-\sigma_{\alpha,\beta}(F^s/A)` | Pa m |
|
||||
| `\sigma^s` | interfacial shear stress | `F^s/L` | Pa m |
|
||||
| `\gamma^s` | interfacial shear strain | `dx_1/dx_2` | – |
|
||||
| `\dot{\gamma}^s` | interfacial shear rate | `dv_1/dx_2` | s^-1 |
|
||||
| `\alpha^s` | interfacial dilatational strain | `\ln(A/A_0)` | – |
|
||||
| `\dot{\alpha}^s` | interfacial dilatational strain rate | `d(\ln A)/dt` | s^-1 |
|
||||
| `\Gamma` | interfacial concentration | | kg m^-2 |
|
||||
|
||||
Steady shear and dilatation:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\eta^s` | interfacial shear viscosity | `\sigma^s/\dot{\gamma}^s` | Pa s m |
|
||||
| `\kappa^s` | interfacial dilatational viscosity | `\sigma^s/\dot{\alpha}^s` | Pa s m |
|
||||
|
||||
Simple shear:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `G^s` | interfacial shear modulus | `\sigma^s/\gamma^s` | Pa m |
|
||||
| `G^s(t)` | relaxation modulus (shear) | `\sigma^s(t)/\gamma^s` | Pa m |
|
||||
|
||||
Pure dilatation ($E$ is sometimes used for $K$; $K$ is recommended, to avoid
|
||||
confusion with Young's modulus):
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `K^s` | interfacial dilatational modulus | `\sigma^s/\alpha^s` | Pa m |
|
||||
| `K^s(t)` | dilatational storage modulus (as printed) | `\sigma^s(t)/\alpha^s` | Pa m |
|
||||
| `K_\Pi` | Gibbs elasticity (surfactants) | `d\sigma_{\alpha,\beta}/d(\ln A)` | Pa m |
|
||||
|
||||
Small-amplitude oscillatory shear:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\gamma^s_0` | strain amplitude | `\gamma^s=\gamma^s_0\sin(\omega t)` | – |
|
||||
| `\delta` | phase angle (loss angle) | `\sigma^s(t)=\sigma^s_0\sin(\omega t+\delta)` | rad |
|
||||
| `\sigma^s_0` | stress amplitude | `F^s/L` | Pa m |
|
||||
| `G^{s*}` | complex interfacial shear modulus | `G^{s\prime}+iG^{s\prime\prime}` | Pa m |
|
||||
| `\lvert G^{s*}\rvert` | absolute magnitude of `G^{s*}` | `\sigma^s_0/\gamma^s_0` | Pa m |
|
||||
| `G^{s\prime}` | storage modulus | `\lvert G^{s*}\rvert\cos\delta` | Pa m |
|
||||
| `G^{s\prime\prime}` | loss modulus | `\lvert G^{s*}\rvert\sin\delta` | Pa m |
|
||||
| `\eta^{s*}` | complex shear viscosity | `\eta^{s\prime}-i\eta^{s\prime\prime}` | Pa s m |
|
||||
| `\lvert\eta^{s*}\rvert` | absolute magnitude of `\eta^{s*}` | `\sigma^s_0/\omega\gamma^s_0` | Pa s m |
|
||||
| `\eta^{s\prime}` | dynamic shear viscosity (in phase with strain rate) | `G^{s\prime\prime}/\omega` | Pa s m |
|
||||
| `\eta^{s\prime\prime}` | out-of-phase component of `\eta^{s*}` | `G^{s\prime}/\omega` | Pa s m |
|
||||
|
||||
Small-amplitude oscillatory dilatation:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `\alpha^s_0` | dilatational strain amplitude | `\alpha^s=\alpha^s_0\sin(\omega t)` | – |
|
||||
| `K^{s*}` | complex dilatational modulus | `K^{s\prime}+iK^{s\prime\prime}` | Pa m |
|
||||
| `\lvert K^{s*}\rvert` | absolute magnitude of `K^{s*}` | `\sigma^s_0/\alpha^s_0` | Pa m |
|
||||
| `K^{s\prime}` | storage dilatational modulus | | Pa m |
|
||||
| `K^{s\prime\prime}` | loss dilatational modulus | | Pa m |
|
||||
| `\kappa^{s*}` | complex dilatational viscosity | `\kappa^{s\prime}+i\kappa^{s\prime\prime}` as printed (see notes) | Pa s m |
|
||||
| `\lvert\kappa^{s*}\rvert` | absolute magnitude of `\kappa^{s*}` | `\sigma^s_0/\omega\alpha^s_0` (see notes) | Pa s m |
|
||||
| `\kappa^{s\prime}` | dynamic dilatational viscosity | `K^{s\prime\prime}/\omega` | Pa s m |
|
||||
| `\kappa^{s\prime\prime}` | out-of-phase component of `\kappa^{s*}` | `K^{s\prime}/\omega` | Pa s m |
|
||||
|
||||
Other properties:
|
||||
|
||||
| Symbol | Meaning | Definition | SI units |
|
||||
|---|---|---|---|
|
||||
| `J^s(t)` | creep compliance (shear) | `\gamma^s(t)/\sigma^s` | Pa^-1 m^-1 |
|
||||
| `J^s_e` | equilibrium compliance of solid | `J^s(t\to\infty)` | Pa^-1 m^-1 |
|
||||
| `J^s_r(t)` | recoverable compliance | `J^s(t)-t/\eta^s_0` | Pa^-1 m^-1 |
|
||||
| `J^s_0` | steady-state compliance of fluid | `J^s(t)-t/\eta^s_0\ (t\to\infty)` | Pa^-1 m^-1 |
|
||||
| `\eta^s_E(\dot{\varepsilon}^s)` | extensional viscosity | `\eta^s_E(t,\dot{\varepsilon}^s)\ (t\to\infty)` | Pa s m |
|
||||
|
||||
### Table X. Molecular description of entangled polymers
|
||||
|
||||
| Symbol | Meaning |
|
||||
|---|---|
|
||||
| `a` | tube diameter (typically nm); average entanglement spacing or mesh size, `\sqrt{\langle R^2\rangle_0 M_e/M}` |
|
||||
| `b_K` | Kuhn segment length, with `\langle R^2\rangle = b_K^2 N_K` and `R_{max} = b_K N_K` |
|
||||
| `f` | tension in a chain segment (typically N) |
|
||||
| `f_{max}` | maximum tension in a chain segment |
|
||||
| `k_B` | Boltzmann's constant |
|
||||
| `L` | mean tube contour length |
|
||||
| `M` | molecular weight, kg mol^-1 (formally a molar mass) |
|
||||
| `M_C` | critical molecular weight for the effect of entanglements on `\eta_0`, kg mol^-1 |
|
||||
| `M_C'` | molecular weight for the effect of entanglements on `J_s^0`, kg mol^-1 |
|
||||
| `M_e` | molecular weight between entanglements, `\rho RT/G_N^0` (Ferry); in tube-model work `\tfrac{4}{5}\rho RT/G_N^0` |
|
||||
| `N_K` | number of Kuhn segments in the equivalent freely jointed chain |
|
||||
| `p` | packing length, `M/[\langle R^2\rangle_0 \rho N_A]` (Fetters et al., Macromolecules 27, 4639 (1994)) |
|
||||
| `p^2` | degree of branch-point hopping within the tube (fraction of tube diameter) |
|
||||
| `R` | end-to-end distance of a polymer molecule |
|
||||
| `R_{max}` | fully extended chain length |
|
||||
| `s` | tube contour variable (curvilinear coordinate along the tube) |
|
||||
| `\mathbf{S}` | tube orientation tensor |
|
||||
| `Z` | number of entanglement segments per molecule, `M/M_e` (in tube theory, "number of tube segments", using the 4/5 definition of `M_e`) |
|
||||
| `\alpha` | dilution exponent for `M_e` |
|
||||
| `\zeta` | friction coefficient |
|
||||
| `\zeta_0` | monomer friction coefficient |
|
||||
| `\lambda` | chain stretch; stretch ratio |
|
||||
| `\xi` | correlation length; characteristic size scale (blob) |
|
||||
| `\tau_d` | reptation (tube disengagement) time |
|
||||
| `\tau_e` | Rouse time of an entanglement strand, `\tau_R/3\pi^2 Z^2` as printed (see notes) |
|
||||
| `\tau_p` | relaxation time of the `p`th mode (`p` is the mode index) |
|
||||
| `\tau_r` | Rouse reorientation relaxation time (Doi and Edwards' rotational relaxation time; conjectured to equal `2\tau_R`) |
|
||||
| `\tau_R` | Rouse stress relaxation time, `\zeta N^2 b^2/(6\pi^2 k_B T)` |
|
||||
| `\tau_s` | stretch relaxation time |
|
||||
|
||||
### Table XI. Stress and strain tensors
|
||||
|
||||
| Symbol | Meaning |
|
||||
|---|---|
|
||||
| `\boldsymbol{\sigma}` | total stress tensor |
|
||||
| `\boldsymbol{\tau}` | extra stress tensor |
|
||||
| `\boldsymbol{\gamma}` | strain tensor for linear viscoelasticity |
|
||||
| `\mathbf{C}` | Cauchy tensor |
|
||||
| `\mathbf{B}` or `\mathbf{C}^{-1}` | Finger tensor |
|
||||
| `\mathbf{Q}` | Doi-Edwards strain tensor |
|
||||
| `\dot{\boldsymbol{\gamma}}` | rate-of-strain tensor, `\nabla\mathbf{v}+(\nabla\mathbf{v})^T` |
|
||||
| `\boldsymbol{\Omega}` | vorticity tensor, `\nabla\mathbf{v}-(\nabla\mathbf{v})^T` |
|
||||
| `\mathbf{D}` | `\tfrac{1}{2}[\nabla\mathbf{v}+(\nabla\mathbf{v})^T]`, the fluid-mechanics alternative to `\dot{\boldsymbol{\gamma}}` (fluid dynamics also puts 1/2 in front of `\boldsymbol{\Omega}`) |
|
||||
|
||||
### Table XII. Dimensionless groups used to describe experimental regimes
|
||||
|
||||
| Symbol | Name | Definition |
|
||||
|---|---|---|
|
||||
| `Bn` | Bingham number | (yield stress)/(shear stress); sometimes the wall shear stress, or (viscosity x velocity)/length, is used in the denominator |
|
||||
| `Bq` | Boussinesq number | (surface shear stress)/[(bulk subphase shear stress) x (perimeter length along which the surface shear stress acts)] |
|
||||
| `Ca` | capillary number | `\eta v/\sigma_{\alpha,\beta}` or `\eta\dot{\gamma}R/\sigma_{\alpha,\beta}`; (viscous forces)/(surface forces) |
|
||||
| `De` | Deborah number | (characteristic time of fluid)/(duration of deformation), e.g. `\tau\omega`; specific forms `De_R` (Rouse time), `De_d` (longest relaxation time) |
|
||||
| `Pé` | Péclet number | `\dot{\gamma}a^2/D_0`; (rate of advection)/(rate of diffusion); `a` particle radius, `D_0` particle diffusion coefficient |
|
||||
| `\nu` | Poisson ratio | `-d\varepsilon_{transverse}/d\varepsilon_{axial}`; in three dimensions also `E/2G - 1` |
|
||||
| `Re` | Reynolds number | `\rho v d/\eta`; (inertial forces)/(viscous forces); `d` characteristic length |
|
||||
| `Wi` | Weissenberg number | (characteristic time of fluid) x (rate of deformation), e.g. `\tau\dot{\varepsilon}` or `\tau\dot{\gamma}`; specific forms `Wi_R`, `Wi_d`. Not `N_1/\sigma`, which is the stress ratio |
|
||||
|
||||
Definitions and uses of $Wi$ and $De$: "Weissenberg and Deborah Numbers —
|
||||
Their definition and use," Rheol. Bulletin 79(2), 14 (2010).
|
||||
|
||||
### Table XIII. Frequently used constants
|
||||
|
||||
| Symbol | Meaning | Value |
|
||||
|---|---|---|
|
||||
| `N_A` | Avogadro constant | 6.023 x 10^23 mol^-1 as printed; exact SI value 6.02214076 x 10^23 mol^-1 |
|
||||
| `R` | ideal gas constant | 8.314 J mol^-1 K^-1 |
|
||||
| `k_B` | Boltzmann constant | 1.38 x 10^-23 J K^-1 (exact SI value 1.380649 x 10^-23 J K^-1) |
|
||||
|
||||
## Notes on the source
|
||||
|
||||
These are places where the 2018-10 edition is inconsistent or appears to
|
||||
contain a typographical error. They are recorded so that a model does not
|
||||
"correct" the official notation on its own, and so that a user can decide.
|
||||
|
||||
- **Normal stress ratio (Table I)** is printed as $-N_1/N_2$. Most of the
|
||||
literature reports $-N_2/N_1$, which is small and positive for polymer
|
||||
melts. State which is meant.
|
||||
- **Power-law names (Table I):** the source labels $n$ "consistency index"
|
||||
and $K$ "constant". In common usage $K$ is the consistency index and $n$
|
||||
the power-law (flow behaviour) index. The symbols and the equation
|
||||
$\eta = K\dot{\gamma}^{n-1}$ are unambiguous.
|
||||
- **Retardation spectrum (Table II, footnote d)** is printed with
|
||||
$1 - \exp(t/\tau)$; the physically meaningful form, used above, is
|
||||
$1 - \exp(-t/\tau)$.
|
||||
- **Net tensile stress** is $\sigma_E$ in Table II and $\sigma_\varepsilon$
|
||||
in Table V. Prefer $\sigma_E$, which matches $\eta_E$ and $\sigma_E^+$.
|
||||
- **Overlap concentration (Table VII)** has units printed as kg m$^3$; it is
|
||||
a concentration, kg m$^{-3}$.
|
||||
- **Suspension viscosities (Table VIII):** the medium viscosity is $\eta_m$,
|
||||
but $\eta_r$ and $\Upsilon_{1,2}$ are defined with $\eta_f$. Read $\eta_f$
|
||||
as $\eta_m$.
|
||||
- **$K^s(t)$ (Table IX)** is labeled "dilatational storage modulus" but
|
||||
defined as $\sigma^s(t)/\alpha^s$, a dilatational relaxation modulus.
|
||||
- **Complex dilatational viscosity (Table IX)** is printed as
|
||||
$\kappa^{s\prime} + i\kappa^{s\prime\prime}$, unlike the bulk and
|
||||
interfacial shear cases ($\eta' - i\eta''$); and $|\kappa^{s*}|$ is printed
|
||||
as $\sigma_0^s/\omega\delta_0^s$, where $\alpha_0^s$ (used above) is
|
||||
evidently meant.
|
||||
- **$\tau_e$ (Table X)** is printed as $\tau_R/3\pi^2Z^2$. Prefactors
|
||||
relating $\tau_e$ and $\tau_R$ differ between tube models (in
|
||||
Likhtman-McLeish, $\tau_R = Z^2\tau_e$), so check against the model in use.
|
||||
- **Reused symbols.** $\Pi$ is both particle pressure (Table VIII) and
|
||||
surface pressure (Table IX). $\sigma$ is both shear stress and engineering
|
||||
tensile stress. Superscript $s$ marks both the solvent stress tensor
|
||||
$\boldsymbol{\sigma}^s$ (Table VII) and interfacial quantities (Table IX).
|
||||
$\alpha$ is the dilution exponent (Table X) and, with superscript $s$, the
|
||||
interfacial dilatational strain. Say which is meant when both could appear.
|
||||
- **Not covered.** The list has no symbols for medium- or large-amplitude
|
||||
oscillatory shear (MAOS, LAOS) analysis, thixotropy, or granular and dense
|
||||
suspension friction (for example a friction coefficient $\mu$ or the
|
||||
viscous number). For these, use the notation of the paper being discussed
|
||||
and define it on first use.
|
||||
|
||||
END OF NOTATION PRIMER. Please confirm you have received the complete primer
|
||||
(it ends with this sentence) and will use this notation in the rest of the
|
||||
conversation.
|
||||
9
tests/conftest.py
Normal file
9
tests/conftest.py
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
import pytest
|
||||
|
||||
from art import store
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def no_legacy_cache(tmp_path, monkeypatch):
|
||||
"""Keep the machine's own data/cache/feed_cache.json out of test databases."""
|
||||
monkeypatch.setattr(store, "LEGACY_CACHE", tmp_path / "feed_cache.json")
|
||||
|
|
@ -261,3 +261,167 @@ def test_draft_yaml_loads_as_profiles():
|
|||
data = prof.Profiles.model_validate(__import__("yaml").safe_load(profiles_yaml(d)))
|
||||
assert data.profiles[0].exemplars[1].arxiv == "2609.21066"
|
||||
assert data.tracked_authors == [] # suggestions stay commented out
|
||||
|
||||
|
||||
def test_daily_digest_window_does_not_repeat_previous_papers(conn, monkeypatch):
|
||||
"""A paper fetched during one daily run must not reappear in the next day's digest."""
|
||||
import json
|
||||
from art import digest
|
||||
monkeypatch.setattr(digest.embed, "model_name", lambda: "e")
|
||||
monkeypatch.setattr(digest.rerank, "model_name", lambda: "r")
|
||||
|
||||
def run(command, started, finished, args=None):
|
||||
conn.execute("INSERT INTO runs (command, args, started_at, finished_at, host) VALUES (?, ?, ?, ?, 'h')",
|
||||
(command, json.dumps(args or {}), started, finished))
|
||||
return conn.execute("SELECT MAX(id) FROM runs").fetchone()[0]
|
||||
|
||||
def fetched(item_id, at):
|
||||
store.upsert_items(conn, [_item(id=item_id, doi=item_id, fetched_at=at)], "daily")
|
||||
|
||||
# Monday's daily run starts at 10:00, fetches paper A at 10:00:05, finishes at 10:02.
|
||||
run("daily", "2026-09-28T10:00:00+00:00", "2026-09-28T10:02:00+00:00")
|
||||
fetched("mon", "2026-09-28T10:00:05+00:00")
|
||||
# Tuesday's daily run starts at 10:00 and fetches paper B.
|
||||
tue = run("daily", "2026-09-29T10:00:00+00:00", None)
|
||||
fetched("tue", "2026-09-29T10:00:05+00:00")
|
||||
assert set(digest.select(conn, None, tue).items) == {"tue"}
|
||||
|
||||
|
||||
def test_since_digest_does_not_move_the_daily_window(conn, monkeypatch):
|
||||
"""A --since digest (a backfill view) must not hide papers from the next daily digest."""
|
||||
import json
|
||||
from art import digest
|
||||
monkeypatch.setattr(digest.embed, "model_name", lambda: "e")
|
||||
monkeypatch.setattr(digest.rerank, "model_name", lambda: "r")
|
||||
conn.execute("INSERT INTO runs (command, args, started_at, finished_at, host) VALUES "
|
||||
"('daily', '{}', '2026-09-28T10:00:00+00:00', '2026-09-28T10:02:00+00:00', 'h')")
|
||||
store.upsert_items(conn, [_item(id="noon", doi="noon", fetched_at="2026-09-28T12:00:00+00:00")], "daily")
|
||||
conn.execute("INSERT INTO runs (command, args, started_at, finished_at, host) VALUES "
|
||||
"('digest', '{\"since\": \"2026-09-01\"}', '2026-09-28T15:00:00+00:00', '2026-09-28T15:00:30+00:00', 'h')")
|
||||
conn.execute("INSERT INTO runs (command, args, started_at, host) VALUES "
|
||||
"('daily', '{}', '2026-09-29T10:00:00+00:00', 'h')")
|
||||
tue = conn.execute("SELECT MAX(id) FROM runs").fetchone()[0]
|
||||
assert "noon" in digest.select(conn, None, tue).items
|
||||
|
||||
|
||||
def test_crossref_feed_windows(conn, monkeypatch):
|
||||
"""First fetch: the last week as daily (or backfill_from); later: since the last fetch less the overlap."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from art.feeds import collect
|
||||
calls = []
|
||||
monkeypatch.setattr(collect.fetch, "fetch_crossref",
|
||||
lambda issn, since, source: calls.append((issn, since, source)) or
|
||||
[_item(id=f"10.1021/{len(calls)}", doi=f"10.1021/{len(calls)}", source=source)])
|
||||
stats = {"fetched": 0, "new": 0}
|
||||
feed = {"name": "Langmuir", "source": "Langmuir", "crossref_issn": "0743-7463"}
|
||||
collect._fetch_crossref_feed(conn, feed, None, stats)
|
||||
week_ago = (datetime.now(timezone.utc).date() - timedelta(days=7)).isoformat()
|
||||
assert calls[-1] == ("0743-7463", week_ago, "Langmuir")
|
||||
assert conn.execute("SELECT origin FROM items").fetchone()[0] == "daily"
|
||||
collect._fetch_crossref_feed(conn, feed, None, stats) # next run: from the last fetch
|
||||
assert calls[-1][1] == week_ago
|
||||
collect._fetch_crossref_feed(conn, {**feed, "name": "Soft Matter", "crossref_issn": "1744-683X",
|
||||
"backfill_from": "2026-01-01"}, None, stats)
|
||||
assert calls[-1][:2] == ("1744-683X", "2026-01-01")
|
||||
|
||||
|
||||
def test_rss_doi_from_link(monkeypatch):
|
||||
rss = b"""<?xml version="1.0"?><rss version="2.0"><channel><title>t</title>
|
||||
<item><title>Yield stress foods</title><link>https://link.springer.com/article/10.1007/s00397-026-01599-y</link>
|
||||
<description>Rigorous rheological characterization.</description></item></channel></rss>"""
|
||||
monkeypatch.setattr(fetch, "http_get", lambda *a, **k: rss)
|
||||
it = fetch.fetch_rss("Rheologica Acta", "https://example.org/rss", "Rheol. Acta")[0]
|
||||
assert it["id"] == it["doi"] == "10.1007/s00397-026-01599-y"
|
||||
|
||||
|
||||
|
||||
def test_rss_aps_abstract_authors_and_date(monkeypatch):
|
||||
rss = b"""<?xml version="1.0"?><rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
|
||||
xmlns="http://purl.org/rss/1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/"
|
||||
xmlns:prism="http://prismstandard.org/namespaces/basic/2.0/">
|
||||
<item rdf:about="http://link.aps.org/doi/10.1103/5pj6-t7zg"><title>Random walk</title>
|
||||
<link>http://link.aps.org/doi/10.1103/5pj6-t7zg</link>
|
||||
<description>Author(s): Ann Lee, Bo Wu, and Cy Diaz<br /><p>Particles become fixed\xe2\x80\xa6</p><br />[Phys. Rev. E 114, 044119] Published Tue Oct 13, 2026</description>
|
||||
<dc:creator>Ann Lee, Bo Wu, and Cy Diaz</dc:creator><prism:doi>10.1103/5pj6-t7zg</prism:doi>
|
||||
<dc:date>2026-10-13T10:00:00+00:00</dc:date></item></rdf:RDF>"""
|
||||
monkeypatch.setattr(fetch, "http_get", lambda *a, **k: rss)
|
||||
it = fetch.fetch_rss("Physical Review E", "https://example.org/rss", "Phys. Rev. E")[0]
|
||||
assert it["doi"] == "10.1103/5pj6-t7zg"
|
||||
assert it["abstract"] == "Particles become fixed\u2026"
|
||||
assert it["authors"] == ["Ann Lee", "Bo Wu", "Cy Diaz"]
|
||||
assert it["date"] == "2026-10-13"
|
||||
|
||||
def test_crossref_date_falls_back_to_created():
|
||||
w = {"DOI": "10.1039/D6SM00542J", "title": ["T"], "published-online": {"date-parts": [[2026]]},
|
||||
"published": {"date-parts": [[2026]]}, "created": {"date-parts": [[2026, 9, 29]]}}
|
||||
assert fetch._crossref_item(w, "Soft Matter")["date"] == "2026-09-29"
|
||||
w["published-online"] = {"date-parts": [[2026, 9, 25]]}
|
||||
assert fetch._crossref_item(w, "Soft Matter")["date"] == "2026-09-25"
|
||||
|
||||
|
||||
def test_link_keyed_items_are_rekeyed_to_their_doi(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr(store.env, "path", lambda key: tmp_path / "art.sqlite")
|
||||
conn = store.connect()
|
||||
link = "https://link.springer.com/article/10.1007/s00397-026-01580-9"
|
||||
store.upsert_items(conn, [_item(id=link, doi=None, link=link, source="Rheol. Acta")], "daily")
|
||||
conn.execute("INSERT INTO screens VALUES (?, 'p', 'v', 'e', 'r', 0.5, 'description', 1.0, 'cosine', 0.6, 0.2, 1, 1)", (link,))
|
||||
conn.commit()
|
||||
conn = store.connect() # the re-key runs on open
|
||||
assert [r[0] for r in conn.execute("SELECT id FROM items")] == ["10.1007/s00397-026-01580-9"]
|
||||
assert conn.execute("SELECT item_id FROM screens").fetchone()[0] == "10.1007/s00397-026-01580-9"
|
||||
# the same paper arriving from the feed is now recognised, not new
|
||||
assert store.upsert_items(conn, [_item(id="10.1007/s00397-026-01580-9", doi="10.1007/s00397-026-01580-9",
|
||||
source="Rheol. Acta")], "daily") == 0
|
||||
|
||||
|
||||
def test_tracked_author_paper_without_rerank_is_listed(conn, monkeypatch):
|
||||
"""A paper screened before its author was tracked has no re-rank score; the digest must still list it."""
|
||||
from art import digest
|
||||
monkeypatch.setattr(digest.embed, "model_name", lambda: "e")
|
||||
monkeypatch.setattr(digest.rerank, "model_name", lambda: "r")
|
||||
p = prof.Profile(id="p", name="P", description="d")
|
||||
data = prof.Profiles(tracked_authors=[prof.TrackedAuthor(name="Ada Owner", match=["Ada Owner"])], profiles=[p])
|
||||
monkeypatch.setattr(digest.prof, "load", lambda path=None: data)
|
||||
store.upsert_items(conn, [_item(id="x1", doi="x1", authors=["Ada Owner"], date="2026-09-20")], "backfill")
|
||||
conn.execute("INSERT INTO screens VALUES ('x1', 'p', ?, 'e', 'r', 0.7, 'description', 3.0, NULL, NULL, 0.2, 0, 1)",
|
||||
(p.version,))
|
||||
sel = digest.select(conn, "2026-09-01")
|
||||
assert [e.item["id"] for e in sel.tracked_only] == ["x1"]
|
||||
assert "not re-ranked" in "\n".join(digest._entry(conn, sel.tracked_only[0], {"p": p}))
|
||||
|
||||
|
||||
def test_label_bands_and_sampling():
|
||||
import random
|
||||
from art.label import band, sample
|
||||
assert [band(x) for x in (None, 0.05, 0.15, 0.3, 0.9)] == ["none", "<0.1", "0.1-0.2", "0.2-0.4", ">=0.4"]
|
||||
pop = {f"n{i}": {"band": "none"} for i in range(50)} | {f"h{i}": {"band": ">=0.4"} for i in range(3)}
|
||||
picked = sample(pop, labeled={"h0"}, n=6, rng=random.Random(0))
|
||||
assert len(picked) == 6 and "h0" not in picked
|
||||
assert sum(i.startswith("h") for i in picked) == 2 # both unlabeled high-band papers, the rest from "none"
|
||||
|
||||
|
||||
def test_weighted_recall():
|
||||
from art.evaluate import _rates
|
||||
# one relevant paper caught (weight 1), one relevant paper never re-ranked standing for 9 papers
|
||||
rows = [(1.0, True, 0.8), (9.0, True, None), (1.0, False, 0.5), (30.0, False, None)]
|
||||
recall, precision = _rates(rows, 0.2)
|
||||
assert abs(recall - 0.1) < 1e-9 and abs(precision - 0.5) < 1e-9
|
||||
|
||||
|
||||
def test_llm_call_records_ollama_version(tmp_path, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from art import llm
|
||||
monkeypatch.setattr(store.env, "path", lambda key: tmp_path / "art.sqlite")
|
||||
conn = store.connect()
|
||||
response = SimpleNamespace(response="A summary.", prompt_eval_count=100, eval_count=10,
|
||||
done_reason="stop", thinking=None)
|
||||
http = mock.Mock(**{"get.return_value.json.return_value": {"version": "0.40.11"}})
|
||||
fake = mock.Mock(generate=mock.Mock(return_value=response), _client=http)
|
||||
monkeypatch.setattr(llm, "client", lambda: fake)
|
||||
monkeypatch.setattr(llm, "ensure_model", lambda: ("art-researcher:abc", "sha256:def"))
|
||||
llm.server_version.cache_clear()
|
||||
result = llm.generate(conn, "label", {"title": "T", "abstract": "A"})
|
||||
llm.server_version.cache_clear()
|
||||
assert result.outcome == "ok"
|
||||
row = conn.execute("SELECT ollama_version, num_ctx FROM llm_calls").fetchone()
|
||||
assert row["ollama_version"] == "0.40.11" and row["num_ctx"] == llm.cfg()["num_ctx"]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue