Reproducible evidence

Source: core repository guide, snapshot e58778bef40b. Synced 2026-10-24. Package versions are independent of milestone labels.

ArtemisKit 0.5 introduces versioned workload identity so an assessment artifact can state which declared workload and evaluation criteria produced a result. This is the foundation for defensible provider comparisons, assessment reports, and scenario-specific leaderboard entries.

0.5.0 workload identity contract

Manifests generated by normal scenario runs use version 1.4 and include a workload identity:

{
  "workload_identity": {
    "schema_version": "1",
    "workload": {
      "schema_version": "1",
      "algorithm": "sha256",
      "digest": "..."
    },
    "rubric": {
      "schema_version": "1",
      "algorithm": "sha256",
      "digest": "..."
    }
  }
}

workload covers the declared scenario inputs: prompts, variables, case setup, tool declarations, fixtures, metadata, and retry/timeout configuration. rubric covers each case’s expected assertion, including strict LLM-grader settings. This separation means a changed prompt changes the workload identity, while a changed scoring criterion changes the rubric identity.

Provider/model configuration is intentionally not included in either identity. It is recorded separately as execution provenance. Profile and reviewed-pack identities are deferred until ArtemisKit has their explicit public contracts.

0.5.1 execution provenance contract

Normal scenario-run manifests also include execution_provenance with distinct target and evaluator identities. Target evidence records the requested model(s), observed model identifiers returned by the provider where available, and declared temperature, token limit, and seed. Each case also retains its own requested/observed target identity so an error can state that no observed model was available instead of inventing one.

Evaluator model identities are retained separately and never merged with target identities. This release does not claim evaluator usage or costs; those require a dedicated evaluator usage contract in a later 0.5.x increment.

0.5.2 attempt and cost evidence contract

Normal scenario-run manifests record a run-level attempt_evidence envelope and bounded per-case attempt_evidence entries. Every entry carries a durable attempt ID, retry-chain ID, one-based repetition and attempt coordinates, terminal status, inclusion decision, latency, and a sanitized error classification where applicable. Failed retry attempts are explicitly marked included_in_outcome: false; only the terminal case measurement can affect the outcome-rate denominator. A standard run declares repetition 1 of 1; a future comparative orchestrator can create distinct runs with explicit independent repetition coordinates without presenting retries as samples.

The run envelope declares default retry policy (including its exponential backoff) and timeout. Case overrides remain part of the workload identity and the per-case attempt trail records the effective execution rather than silently collapsing it into a counter.

metrics.cost_provenance replaces generic token-price estimates as the assurance cost contract:

State	Required evidence	Reporting behavior
known	Provider billing amount, ISO currency, and recorded timestamp	May be summed only with compatible currency evidence.
user_supplied	Operator-entered amount, ISO currency, and recorded timestamp	Visible as user supplied; never represented as provider-attested.
unavailable	Bounded reason code	No amount is invented from token counts or a generic price table.

Normal CLI runs currently produce unavailable because adapters expose usage tokens, not billing records. This deliberately includes OpenAI-compatible local endpoints such as Ollama: any generic estimate is not local spend evidence. A CLI --budget now fails closed when compatible, attested USD cost evidence is unavailable.

Canonicalization and redaction

Identity inputs are sorted recursively before serialization and SHA-256 hashing. Values under keys such as apiKey, token, secret, password, authorization, and credential are replaced before hashing. ArtemisKit’s built-in secret patterns and declared scenario redaction patterns are also applied to string values before hashing.

Do not interpolate credentials or customer secrets into prompts, fixtures, or variables. The identity layer reduces accidental retention and makes recognized secrets non-material to a digest; it cannot infer every sensitive literal in arbitrary text. Use the existing redaction controls and keep secret material outside declarative scenario files.

Interpretation and compatibility

Matching digests mean the declared, sanitized inputs match under this contract. They are not a digital signature or an attestation of target-provider behaviour.

0.5.3 comparison eligibility contract

Every standard-run comparison now emits a versioned eligibility decision before a metric or case-level delta is calculated:

Status	Meaning	Delta behavior
compatible	Declared scenario, workload, rubric, and execution configuration match.	Available.
qualified	A delta is available, but historical evidence is missing or declared target/execution configuration changed.	Available only with visible reason codes.
incomparable	Scenario, workload, or rubric differs.	Withheld. No regression decision is made.

Reason codes are bounded and machine-readable. Current codes cover missing workload/rubric or execution provenance, scenario/workload/rubric mismatch, and target provider/model or generation setting changes. CLI output, SDK comparison results, storage comparisons, JSON output, and HTML comparison reports carry the same eligibility decision.

The target model is intentionally treated as a qualification rather than a workload mismatch: a model-to-model assessment is useful, but it must not be confused with a same-configuration regression claim. Historical v1.0–v1.2 manifests remain readable and are qualified, never silently upgraded to compatible.

This contract only decides compatibility from currently declared evidence. Assessment profiles, organizational policies, reviewed packs, language/region applicability, and cost equivalence remain outside its decision scope. Attempt/repetition evidence is now recorded, but provider-neutral comparative orchestration comes later. Absence of a dedicated future field does not mean it was reviewed or equivalent.

Historical v1.0–v1.3 manifests remain readable. They do not gain inferred workload identity, execution provenance, retry-chain evidence, or cost provenance; a missing field means the historical record cannot make that particular reproducibility claim.