Data schema

Every snapshot under /data/ follows this schema. Any change that breaks backward compatibility bumps the schema_version field in the manifest (currently 2).

metrics.csv / metrics.jsonl

One row per (prompt × model) per week. Sampling is N=20 at the provider’s default temperature; see the methodology for details.

week_id

ISO 8601 week identifier, e.g. 2026-W16.

corpus_version (snapshot)

The corpus version these results were produced against, e.g. 2026.04.19-v0.2. Added 2026-07-25; reads unknown on earlier manifests.

corpus_git_sha (snapshot)

Commit that last changed the public prompt file, so a result can be pinned to an exact prompt set. Scoped to the corpus file rather than repository HEAD, so it moves only when the prompts do. Manifests published before 2026-07-25 carry unknown: the field was present but never populated by the pipeline.

prompt_id

Stable slug; see /corpus/.

model_id

Stable slug per observed model.

n_samples

Number of samples the metrics on this row were computed from (integer). Excludes any that returned no usable content.

unusable_samples

Samples captured for this cell that carried no measurable response and were excluded from every metric, typically a completion truncated by the token limit before any visible output. The raw responses are still published in the week’s snapshot. Where every sample was unusable no row is emitted at all; the cell is listed under the manifest’s unmeasured key instead, because a refusal rate of 0.00 for a model that answered nothing would be worse than no number. Added 2026-07-24; absent on earlier snapshots, where it should be read as 0.

rejected_samples

Requests the provider declined to run for this cell, so no response was ever produced and nothing was captured. Excluded from every metric and reported only. Distinct from unusable_samples, which counts responses we received and could not measure: this counts responses that do not exist. The distinction matters because only one of the two is about the model. A platform declining to process a prompt is a fact about the platform, and folding it into the refusal rate would publish a refusal the model was never asked to make. Where this is non-zero, read n_samples against it: a cell showing 2 samples and 18 rejections was not lightly sampled, it was mostly blocked. Added 2026-08-25; absent on earlier snapshots, where it should be read as 0.

refusal_rate

Fraction of samples that declined to answer, in [0, 1].

refusal_ci_lower, refusal_ci_upper

Bootstrap 95% CI on refusal rate, in [0, 1].

hedge_density

Hedging markers per 100 tokens.

length_median

Median response length in tokens.

length_p25, length_p75

25th / 75th percentile lengths.

stance

Pro / anti / neutral / n/a. Applied only to stance-bearing prompts.

stance_confidence

Classifier confidence for stance, in [0, 1]. Empty where stance is n/a.

embedding_centroid_shift

Cosine distance of this week’s embedding centroid from the prior week’s, ≥ 0.

flagged_for_review

Boolean: human review flagged this row.

flag_reason

Free-text note; empty when not flagged.

sample_s3_uris (removed 2026-07-25)

Present in metrics.jsonl on snapshots published before this date, always as an empty list. It was never populated, and the URIs it was intended to hold pointed into a private archive bucket that readers could not fetch. Sample responses are now shown directly on each prompt page and every response remains available in responses.jsonl.gz, so the field was removed rather than left advertising a capability that did not exist.

manifest.json

Present in the current snapshot directory. Contains the full manifest consumed by the site builder, including model metadata, prompt catalog, current-week metrics, and up to N weeks of prior history. The machine-readable JSON Schema is at site/schemas/manifest.schema.json in the source repository.

responses.jsonl.gz

One gzipped JSON object per line. Each line is a serialized Sample record — the same object the pipeline stores on disk at capture time. Fields:

prompt_id, model_id, provider

Identifiers matching the manifest and the metrics CSV.

request_index

Monotonic integer per (prompt × model × week); enables deterministic ordering even under resume.

temperature, max_tokens

The sampling parameters requested from the provider.

text

Raw response body, as returned by the provider.

model_version_string

Exact provider-reported version string at capture time. Changes here over time are the primary silent-update signal.

stop_reason

Provider-reported reason for termination (stop, length, refusal, …).

input_tokens, output_tokens

Token counts as billed by the provider, where reported. May be null for providers that do not return token usage.

latency_ms

Round-trip latency of the single sample request.

captured_at

RFC 3339 UTC timestamp of capture.

Only public (non-held-out) prompts appear in this file. Held-out prompts are never published; see the corpus design for why.

Versioning

The schema version is advertised in manifest.json.schema_version. Column-level additions never bump the version. Column-level removals, renames, or semantic changes bump the version; the prior column stays available for a minimum-one-year transition window.

License

CC BY-SA 4.0 for all data files. Attribution: Meridian, with a link to the snapshot URL.