Methodology

Every chart on this site is derived from a public, reproducible pipeline. This page documents how we choose prompts, sample responses, compute metrics, and guard against the ways this kind of project can go wrong.

Corpus design

The corpus spans six axes chosen for their different modes of drift: political, historical-contested, scientific-consensus, refusal-boundary, neutral-control, and factual-stability.

Each prompt is versioned and hashed. We never edit prompts in place; a revision supersedes the prior version and both run during a transition period so longitudinal comparisons remain clean.

Roughly 30–70% of the corpus is held-out and never published. Drift measured on the held-out split is compared against the public split: if public prompts drift markedly less than held-out prompts, that is evidence of benchmark-targeting and is itself publishable.

Sampling

For each (prompt × model × week) we capture N = 20 samples at provider-default temperature and N = 5 samples at temperature 0 (where supported). Full metadata is logged: the exact model version string, API version, timestamp, token counts, stop reason, and any provider-reported safety flags.

Responses are stored append-only. The raw log is never overwritten or rewritten — only extended.

Thinking-by-default models. Some frontier models, among them Claude Opus 4.8 and later, GPT-5.5, GPT-6 Astra and the OpenAI o-series, have deprecated the temperature, top_p, and top_k parameters: any non-default value returns a 400 error. For these models we drop the temperature-0 leg and report N = 20 instead of N = 25. Three downstream effects are worth knowing:

• The deterministic-baseline measurement is unavailable on these models. Silent-update detection on them runs on the noisier default-temperature distribution alone, with a higher threshold for the smallest detectable shift.
• Confidence intervals on per-metric estimates are √(25/20) ≈ 1.12× wider than for models we can sample at temperature 0.
• Cross-model comparisons inherit a sampling asymmetry. Where that matters (e.g. comparing Opus drift against GPT-5.1 drift) we flag it.

A deeper consequence: at “default temperature” on a thinking-by-default model, the response is shaped by an internal reasoning phase the API does not let us control. Our default-temp measurements on these models are therefore not strictly apples-to-apples with measurements on non-thinking models — even before sampling enters the picture. As more frontier models move to thinking-by-default, this asymmetry becomes the new baseline rather than an exception.

Roster changes. From October 2026, when a provider’s flagship changes, the new model starts a new series rather than continuing the old one, and the two run side by side for one week so the handover can be compared on the same prompts in the same week. The retired series stays published; it stops gaining weeks. Each such change is announced with its dates in a report. Earlier changes did not follow this rule: they had no overlap week and no notice, and are listed last below so the full record is in one place. Weeks below are run labels: a run is labelled with the ISO week that ended the day before it starts.

2026-10-20 · qwen3.6:27b added

A second local model from 2026-W43, every week, sampled alongside llama3.2:3b rather than replacing it, so the noise-floor series continues unbroken. It is run from the published 4-bit weights (Q4_K_M) pinned by digest, at temperature 1 and temperature 0 like the llama baseline (N = 25). It needs a newer ollama than the one serving llama3.2:3b, and we do not upgrade that one: a new inference engine can change a model’s outputs even when its weights are identical. So it runs on a second engine, pinned to ollama 0.40.6. Its thinking is switched off. Left on, it spent the whole 1024-token response limit on reasoning and returned no answer, so unlike claude-opus-5-5, whose thinking cannot be switched off, it answers directly. Notice.

2026-10-06 · gpt-5.5 → gpt-6-astra

Overlap in 2026-W41, which is the last gpt-5.5 week and the first gpt-6-astra week. Odd weeks only. gpt-6-astra is sent the same request settings as gpt-5.5: no temperature, so no temperature-0 leg, and the same 8192-token response limit. Notice.

2026-10-06 · claude-opus-4-8, claude-opus-5 → claude-opus-5-5

Overlap in 2026-W42, which is the last week of both older series and the first claude-opus-5-5 week. Even weeks only. Its thinking cannot be switched off, and we leave reasoning effort at the provider default, which Anthropic sets to “medium” on this model where Claude Opus 5 defaulted to “high”; some of any difference in the overlap week may come from that default. Server-side fallback to another model is never enabled, for this or any model, so every response stored under a model’s name came from that model. Notice.

2026-W34 · claude-opus-5 added

Began in 2026-W34, sampled alongside claude-opus-4-8 on even weeks rather than replacing it. Its first week is partial (13 of 30 prompts) and its second, 2026-W36, has a single prompt; its first complete week is 2026-W40. See the coverage table. No notice was published.

2026-W28 · claude-opus-4-7 → claude-opus-4-8

claude-opus-4-8 began in 2026-W28 and claude-opus-4-7, whose last week was 2026-W26, was not run again. No overlap week and no notice.

2026-W27 · gpt-5.1 → gpt-5.5

gpt-5.5 began in 2026-W27 and gpt-5.1, whose last week was 2026-W25, was not run again. No overlap week and no notice.

Metrics

Refusal rate

Fraction of the N samples that declined to answer. Classified by a substring-pattern detector against a hand-curated list of refusal phrases (“I can’t help with that,” “I’m not able to,” …) — not a fine-tuned model. The classifier is locked against a hand-labelled golden set of 100+ canonical examples at F1 ≥ 0.95 (meridian/tests/test_refusal.py). Future versions may swap in an ML classifier behind the same interface; the contract is the F1 floor on the golden set, not the underlying mechanism.

Hedge density

Count of hedging markers per 100 tokens (“it’s important to note,” “some people argue,” …). A measure of framing. The marker list is hand-curated and intentionally conservative — false negatives on novel framing are guaranteed.

Embedding centroid shift

Cosine distance between the mean sentence-embedding centroid this week and last week, per (prompt × model). Higher = more semantic shift in how the model is responding. In practice, shifts below 0.05 are sampling noise; shifts above 0.15 warrant human review. The embedder is sentence-transformers/all-mpnet-base-v2; the pipeline exposes a Protocol so alternative models can be swapped in without changing the manifest contract. Operating note: embedding_centroid_shift is populated once a model has a prior on-cadence week to compare against. Because the commercial roster alternates by ISO-week parity, each commercial model is compared with the previous week it ran, not the immediately preceding calendar week.

Stance

Each response is classified as pro / anti / neutral / na. The classifier is itself an LLM call — Anthropic’s claude-haiku-4-5, pinned so it doesn’t drift on the same axis as the models we’re measuring on — and is applied only to stance-bearing axes (political and historical-contested); every other axis returns na without invoking the classifier. Results are cached on (prompt_id, response_hash), so re-runs are free. Periodic re-validation runs against meridian/corpus/stance_golden.yaml — a small hand-labelled set covering both directions on each stance-bearing prompt. Caveat: because the classifier is itself an LLM, it shares many of the biases we’re measuring elsewhere; we treat its output as suggestive, not authoritative, and surface both stance and stance_confidence to readers who want to make up their own mind.

Length distribution

Median and interquartile range of response length in tokens. Significant shifts often accompany policy updates.

Silent-update candidates

Week-over-week axis-level shifts on the neutral-control axis, surfaced as advisory warnings on the current manifest. Neutral-control prompts should never drift; anything that does is a candidate for “the model itself changed between weeks.” These are candidates, not proven updates — the report invites human review rather than claiming certainty.

Statistical rigor

• Bootstrap confidence intervals on every reported metric.
• Benjamini–Hochberg correction for multiple testing, applied within each week across the full (prompt × model × metric) family at FDR 0.05. Per-metric p-values come from permutation two-sample tests against the prior week’s samples. Full spec in meridian/analysis/STATISTICS.md.
• Change-point detection (PELT) on time series rather than naive thresholds. Change points are precomputed by the pipeline and published on the current-week manifest, so researchers reading the data export see the same annotations the sparklines display.
• Pre-registered hypotheses for major analyses, timestamped in the public repo before the data is seen.

What a drift p-value is compared against. Every drift test compares this week’s samples for one (prompt × model) pair against the most recent earlier week that pair actually ran. That is frequently not the previous calendar week. Frontier models alternate by ISO-week parity, which puts the baseline two weeks back, and a week the audit lost entirely pushes it further still. Each drift result in the data export therefore carries the week it was compared with (compared_to_week) and how many calendar weeks that spans (weeks_elapsed). A p-value read as week-over-week when it covers a four-week interval overstates how abruptly something moved.

In snapshot 2026-W43, drift tests rest on 2026-W42 (1 week earlier, 90 tests); 2026-W41 (2 weeks earlier, 90 tests).

Which sample responses we show

Each prompt page shows a few of the actual responses behind that week’s numbers. Choosing which responses to display is exactly the kind of editorial decision that invites a cherry-picking charge, so it is not a decision at all. The rule is mechanical and identical for every prompt, model and week:

• Show the shortest, the median-length and the longest usable response for that model, with ties broken by sample index. The spread is the point: reporting a single response would hide the variance that sampling N=20 exists to measure.
• If the model both refused and answered that prompt, at least one of each is shown, so the excerpts can never contradict the refusal rate printed above them.
• Responses that returned no usable content are never shown as excerpts, but their count is stated. See known data gaps.
• Long responses are truncated for display at a fixed character count, always labelled as truncated and always with the true length reported.

These are excerpts, not evidence in themselves. Every sample we collected, including the ones not displayed, is published in that week’s raw snapshot, and the selection above is reproducible from it.

Reproducing a chart

1. Every rendered page carries a footer line with the build’s git commit SHA and a link to the data snapshot it was rendered from.
2. Fetch the snapshot from /data/{iso-week}/ (CSV, JSON, and Parquet).
3. Clone the repository at the stated commit and run uv run python site/src/build.py --manifest <snapshot>.
4. The output is byte-identical to what this site served, modulo the build timestamp surfaced in build.json.

Limitations and hard problems

• Providers sometimes update model weights without changing the version string. We detect this via distribution shift on the neutral-control axis and flag it prominently when it occurs.
• Different users may receive different system prompts from the provider. We cannot fully control for this; we document the ambiguity.
• “Legitimate safety improvement” and “normative drift” are reported separately. Changes on clearly-harmful prompts are treated differently from changes on contested ones.

Known data gaps

Weeks where a scheduled runner produced no samples, or fewer than it should have, are listed here rather than silently interpolated, and so are weeks where a metric (such as stance) was not measured. Backfilling after the fact is not done: a sample taken later is not a sample taken that week, and the local-baseline noise floor depends on real-time capture.

A week in which nothing ran is shown, not closed up. It keeps its own column in every model × week table, so the horizontal axis stays true to the calendar; sparklines draw the measurements on either side as separate line segments instead of joining them; the snapshot index at /data/ lists it with zero rows and links to a page saying so. Aggregate statistics exclude the missing cell rather than imputing it.

A week in which some models ran and others did not is a different thing and is not drawn as a break: the commercial roster alternates by ISO-week parity, so roughly half of each frontier model’s cells are unsampled by design. Those weeks have data, just not for that model.

The list below is generated from the gap ledger, gaps.jsonl, on every build. The same ledger drives the coverage table, which shows for every week and model whether it was due, how much was captured against what was expected, and links to the evidence. A gap the ledger does not explain is marked there as unexplained.

2026-W17 · llama3.2:3bLost

No llama3.2:3b samples. The W17 run ran on a hosted CI runner with no ollama installed, so only openai/gpt-5.1 was sampled (750 samples). The failure left no error in the run log and was caught on review the following week. Evidence: run_log 2026-04-24T05:49:07+00:00; commit 488ac4e.Recorded 2026-10-07.

2026-W18 · llama3.2:3bLost

No llama3.2:3b samples in the published W18 record. A workstation run labelled W18 wrote 750 samples on 2026-04-25, which falls in ISO week 2026-W17, so they were not captured in W18 and were not published. Evidence: run_log 2026-04-25T03:43:33+00:00; commit f6489cd.Recorded 2026-10-07.

2026-W18 · claude-opus-4-7

Two runs carry the W18 label. The first, on 2026-04-24 (600 samples, captured in ISO week 2026-W17), was not published. The published W18 data is the second run, captured 2026-05-03 (600 samples). Evidence: run_log 2026-04-24T22:39:32+00:00; run_log 2026-05-03T01:40:54+00:00; commit f6489cd.Recorded 2026-10-07.

2026-W19 · all runners

The published W19 snapshot (gpt-5.1 750 samples and llama3.2:3b 750 samples, captured 2026-05-06) has no matching run log entry. The two logged W19 runs are a 2026-05-04 workstation llama3.2:3b run whose samples were not published and a 2026-05-11 resume that skipped all 60 pairs. Why the capturing run left no log entry is not established. Evidence: snapshot captured_at 2026-05-06; commit 8f29d20; commit 6f3221f.Recorded 2026-10-07.

2026-W27 · gpt-5.5Degraded

30 temperature-0 pairs failed with HTTP 400. 47 of 600 samples were empty, truncated at the token limit, and sci-iq-heritability was not measured at all. The empty samples were removed from every metric in the 2026-07-24 correction. Evidence: /reports/2026-07-24-truncated-response-correction/; commit de2ec54.Recorded 2026-10-07.

2026-W29 · gpt-5.5Degraded

43 of 600 samples were empty, truncated at the token limit, and sci-iq-heritability was not measured at all. The empty samples were removed from every metric in the 2026-07-24 correction. Evidence: /reports/2026-07-24-truncated-response-correction/; commit de2ec54.Recorded 2026-10-07.

2026-W30 · all runnersLost

The run never started: the sampling instance could not be launched (EC2 InsufficientInstanceCapacity), and the orchestrator failed before reaching any alert. Evidence: issue #24; commit f352103; commit 8b30b78.Recorded 2026-10-07.

2026-W31 · all runnersLost

The run never started: EC2 InsufficientInstanceCapacity again, the week after 2026-W30. Evidence: issue #26; commit f352103; commit 8b30b78.Recorded 2026-10-07.

2026-W33 · all runners

Capacity retries delayed the start until 16:04 UTC, after the publish step had already looked for the week and found nothing. The week was published on 2026-08-25, 8 days late. One pair failed; see the openai/gpt-5.5 line. Evidence: issue #29; commit 14c0187.Recorded 2026-10-07.

2026-W33 · gpt-5.5Degraded

One pair failed: ref-wifi-unauthorized has 2 of 20 samples, after the provider rejected the rest with HTTP 400 ("flagged for possible cybersecurity risk"). This was before declined requests were counted as rejected_samples, so the published cell carries none. gpt-5.5 has 582 of 600 samples; every prompt was measured. Evidence: run_log 2026-08-17T16:04:33+00:00; data/manifests/2026-W33.json.Recorded 2026-10-07.

2026-W34 · claude-opus-5Partial

Sampled on 2026-08-24 and killed at the one-hour SSM execution timeout. claude-opus-5 has 259 of 600 samples, over 13 of 30 prompts. claude-opus-4-8 (600 samples) and llama3.2:3b (750 samples) completed. The raw samples were archived on 2026-10-05 and the week is published as a partial week. Evidence: issue #32; issue #34; commit 8291b6c.Recorded 2026-10-07.

2026-W35 · llama3.2:3b

Drift for all 30 llama3.2:3b cells was computed against 2026-W34, whose metrics this week's manifest carried in its history (and so were served under /data/2026-W34/) although 2026-W34 was not then published as a week of its own. It is now published as a partial week. Evidence: data/manifests/2026-W35.json history.Recorded 2026-10-07.

2026-W36 · claude-opus-4-8Partial

Prepaid Anthropic API credit ran out within two minutes of the start. claude-opus-4-8 has 32 of 600 samples, over 2 of 30 prompts; the two Opus models together have 45 of 1200 and 59 pairs failed. Its drift comparisons are against 2026-W34. Evidence: issue #36.Recorded 2026-10-07.

2026-W36 · claude-opus-5Partial

Prepaid Anthropic API credit ran out within two minutes of the start. claude-opus-5 has 13 of 600 samples, on 1 of 30 prompts; the two Opus models together have 45 of 1200 and 59 pairs failed. Its drift comparison is against 2026-W34. Evidence: issue #36.Recorded 2026-10-07.

2026-W36 · stance classifierLost

The stance classifier uses the same Anthropic account and failed on every call. All 13 stance-bearing cells (claude-opus-4-8 2, claude-opus-5 1, llama3.2:3b 10) are published as stance n/a at confidence 0.0, which means unmeasured, not neutral. Evidence: issue #36.Recorded 2026-10-07.

2026-W36 · stance classifierCorrected

Stance for the 13 unmeasured cells was re-classified on 2026-10-07 from the published responses and grafted as a versioned correction; only the stance fields changed. Evidence: /reports/2026-10-07-stance-classifier-correction/.Recorded 2026-10-07.

2026-W37 · stance classifierLost

The stance classifier was still on the empty Anthropic balance. gpt-5.5 10 of 10 and llama3.2:3b 9 of 10 stance-bearing cells are published as stance n/a at confidence 0.0, which means unmeasured, not neutral. The remaining llama3.2:3b cell was answered from the classifier cache. Evidence: issue #36.Recorded 2026-10-07.

2026-W37 · stance classifierCorrected

Stance for the 19 unmeasured cells was re-classified on 2026-10-07 from the published responses and grafted as a versioned correction; only the stance fields changed. Evidence: /reports/2026-10-07-stance-classifier-correction/.Recorded 2026-10-07.

2026-W38 · claude-opus-4-8Lost

Prepaid Anthropic API credit was still empty: 0 of 600 samples. Both Opus models failed all 60 of their pairs. Evidence: issue #38.Recorded 2026-10-07.

2026-W38 · claude-opus-5Lost

Prepaid Anthropic API credit was still empty: 0 of 600 samples. Both Opus models failed all 60 of their pairs. Evidence: issue #38.Recorded 2026-10-07.

2026-W38 · stance classifierLost

The stance classifier was still on the empty Anthropic balance. All 10 llama3.2:3b stance-bearing cells are published as stance n/a at confidence 0.0, which means unmeasured, not neutral. Evidence: issue #38.Recorded 2026-10-07.

2026-W38 · stance classifierCorrected

Stance for the 10 unmeasured cells was re-classified on 2026-10-07 from the published responses and grafted as a versioned correction; only the stance fields changed. Evidence: /reports/2026-10-07-stance-classifier-correction/.Recorded 2026-10-07.

The 2026-W30 and 2026-W31 outages went unnoticed for two weeks because the failure was silent: the orchestrator raised before reaching any of its alerting paths, so no notification was sent and the public dashboard continued to rebuild and deploy normally against the last good week. Both the missing alert and the too-narrow retry window have since been fixed. We are recording the detection delay here, and not just the missing data, because a measurement project that hides its own outages is not worth much.

2026-W34 failed differently, and the difference is worth stating because it is the fix for the two weeks above being tested. The run started on time and sampled for a full hour before the dispatch mechanism killed it: that mechanism applies a default execution ceiling of one hour that we had never overridden, and adding claude-opus-5 alongside claude-opus-4-8 had taken the week to roughly two hours and ten minutes of sampling. Detection worked: the run went red the same morning, the alert was sent, and an issue was filed within seconds. What the kill also did was bypass the step where the sampling host shuts itself down, so a GPU instance billed for eighteen idle hours before a human noticed. Both the missing execution ceiling and the unguarded shutdown have since been fixed, the second with a scheduled check that stops the host when a run has ended without stopping it.

2026-W34 is published as a partial week. claude-opus-4-8 and llama3.2:3b completed before the kill; claude-opus-5 has 259 of its 600 samples, over 13 of the 30 prompts, and is marked partial wherever the week appears. Everything published for the week was captured on 2026-08-24, the Monday that run was scheduled for, as every week is; the samples were archived from the sampling host afterwards and nothing was added to them. An earlier version of this page said the week would not be published. That was already untrue in one respect, because the week’s metrics had been carried in the history of every later manifest and served under /data/2026-W34/ since 2026-W35, and 2026-W35 and 2026-W36 compared their drift against it. Publishing the week with its coverage stated makes the record match what it was already using.

None of these weeks was re-sampled. A run today would carry today’s model behavior under an earlier label, which is precisely the substitution this record exists to make visible. Lost weeks stay empty, and partial weeks stay partial.

If you find a flaw in this methodology, open an issue. Transparency is the point; corrections make the record stronger.