FLUXFLUX
  • Introduction
  • Quickstart
  • From Your Data to a Synthetic Copy
  • Two Specs, One Substrate
  • The Six Families
  • The FLUID Seam
  • Deterministic & Governed
  • Runtime & Evidence
  • Anatomy
  • The Nineteen Kinds
  • Versions
  • Changelog
  • JSON Schema 0.5.4
  • UI Hints 0.5.4
  • Enforcement Contract 0.5.0
  • Bundle Manifest 0.5.0
  • Overview
  • Tutorial
  • How-to guides
  • Reference
  • Explanation
  • Command line
  • Error codes
  • Accuracy program
  • The vNext Plan
  • The Nine RFCs
  • Playback Source (proposed)
  • Conformance Profiles
Examples
What's New
FLUID
GitHub
  • Introduction
  • Quickstart
  • From Your Data to a Synthetic Copy
  • Two Specs, One Substrate
  • The Six Families
  • The FLUID Seam
  • Deterministic & Governed
  • Runtime & Evidence
  • Anatomy
  • The Nineteen Kinds
  • Versions
  • Changelog
  • JSON Schema 0.5.4
  • UI Hints 0.5.4
  • Enforcement Contract 0.5.0
  • Bundle Manifest 0.5.0
  • Overview
  • Tutorial
  • How-to guides
  • Reference
  • Explanation
  • Command line
  • Error codes
  • Accuracy program
  • The vNext Plan
  • The Nine RFCs
  • Playback Source (proposed)
  • Conformance Profiles
Examples
What's New
FLUID
GitHub
  • FLUX Simulation Engine
  • Tutorial
    • First evidence pack in five minutes
  • How-to guides
    • Model your customers
    • Add a journey
    • Use the FLUID seam
    • Calibrate from aggregates: declare a scorecard
    • Draw a column given another
    • Ship with evidence
    • Write a bundle from a profile
    • Score a run against a profile, in full or sampled
    • Seal and score a prediction
    • Load a simulation into a database
  • Reference
    • Command line
      • fluxsim init
      • fluxsim validate
      • fluxsim plan
      • fluxsim apply
      • fluxsim run
      • fluxsim verify
      • fluxsim ship
      • fluxsim replay
      • fluxsim calibrate
      • fluxsim accuracy
      • fluxsim gate
      • fluxsim expr
      • fluxsim describe
      • fluxsim plugins
      • fluxsim explain
    • Error codes
    • Python API
    • Schemas
    • Specifications
      • FLUX Simulation Engine: kernel normative specification (v0.1)
      • FLUX Simulation Engine: kernel v2 normative specification (v0.2)
      • Seam generator and fidelity check: normative specification (v0.1)
      • Profile document: normative specification (format 1-1-0)
      • Scorecard: normative specification (format 1-1-0)
      • Sample-based verification: normative specification
      • Parameter register: normative specification (format 1-1-0)
      • flux-expr/1: normative specification (v0.1)
      • flux-expr/2: normative specification (v0.1)
      • flux-expr/3: normative specification (v0.1)
  • Explanation
    • Architecture and patterns
    • The six-step loop
    • Determinism and the seed tree
    • The FLUID seam
    • The evidence pack, and what it does not claim
  • Accuracy program

    • The accuracy program: design
    • Accuracy report

Seal and score a prediction

A run shows what a scenario predicts. It does not show whether the prediction comes true, which is why every report says validity: not asserted. The fluxsim accuracy commands keep that score, apart from the evidence pack:

  1. register seals a prediction before its outcome is known;
  2. score checks it against what was observed, once the outcome is known;
  3. record publishes the outcomes, misses included, for everything registered at one time.

1. Start from a shipped run

A prediction names the run it rests on by the run's digest. Ship a run and keep its id and digest in variables, as the tutorial does:

$ fluxsim init demo
created demo with a small FLUX bundle and its FLUID contract (10 files):
...
$ cd demo
$ fluxsim run . --verify --ship -o ../out
...
run ... complete; shipped to ../out
$ RUN=$(ls .flux/runs)
$ DIGEST=$(grep 'Run digest' .flux/runs/$RUN/EVIDENCE.md | grep -o 'sha256:[0-9a-f]*')

2. Seal the prediction

Write the prediction down before its horizon ends. predicted is the value you expect, and interval_low and interval_high are the range you claim it falls in; a prediction without an interval is never scored. Numbers are decimal strings, and times are RFC 3339. The <<EOF here, without quotes, lets the shell put the run's digest in:

$ cat > ../prediction.json <<EOF
{
  "prediction_id": "recovery-rate-q3",
  "run_digest": "$DIGEST",
  "metric_id": "payment_recovery_rate",
  "statement": "Of the subscribers the detector flags in the third quarter, 62 in 100 pay within seven days.",
  "predicted": "0.62",
  "interval_low": "0.55",
  "interval_high": "0.69",
  "horizon_end": "2026-09-30T23:59:59Z",
  "registered_at": "2026-06-30T09:00:00Z"
}
EOF
$ fluxsim accuracy register --file ../prediction.json
registered recovery-rate-q3 (sha256:...)

register prints the prediction's seal: a digest of the prediction, its registration time included. The record is written once, into the state directory, which is ./.flux unless --state-dir names another:

$ ls .flux/accuracy/predictions
recovery-rate-q3.json

Publish the seal where others can see it when you register, in a ticket or a commit for example. The seal is a digest, not a signature: anyone who can write the state directory can write a matching one, so it shows that a record is the one first registered only to a reader who has the digest from before the outcome was known.

Registering the same claim again writes nothing:

$ fluxsim accuracy register --file ../prediction.json
recovery-rate-q3 is already registered with this seal (sha256:...); nothing was written

A different claim under the same id is refused, so a prediction cannot be swapped once its outcome is known:

$ sed 's/"0.62"/"0.60"/' ../prediction.json > ../changed.json
$ fluxsim accuracy register --file ../changed.json   # exit 1
FLX-7007: prediction id is already registered with a different seal -- ...
more: fluxsim explain FLX-7007  ...

Register a changed prediction under a new prediction_id.

3. Score it once the outcome is known

After the horizon, write down what was observed. basis says how it was measured: live-ab, historical-holdout, or none when nothing was measured against the world.

$ cat > ../readout.json <<'EOF'
{
  "prediction_id": "recovery-rate-q3",
  "metric_id": "payment_recovery_rate",
  "observed": "0.58",
  "sample_size": 1840,
  "basis": "historical-holdout",
  "observed_at": "2026-10-07T12:00:00Z"
}
EOF
$ fluxsim accuracy score --prediction-id recovery-rate-q3 --readout ../readout.json
recovery-rate-q3: within-interval

0.58 lies between 0.55 and 0.69, so the prediction is within-interval. Outside them it would be outside-interval. A prediction with no interval, or a readout whose basis is none, is not-scored: nothing is passed or failed without a measurement. The verdict is written beside the prediction, under .flux/accuracy/verdicts/.

score refuses a readout observed before the prediction was registered (FLX-7001), because it could have shaped the prediction, and one whose prediction_id or metric_id is not the prediction's (FLX-7002).

4. Publish the record

record writes one accuracy record for every prediction registered at the time you give, exactly as the predictions spell it:

$ fluxsim accuracy record --registered-at 2026-06-30T09:00:00Z --out ../accuracy.json
wrote ../accuracy.json
$ grep -E '"(passed|predictions_registered|predictions_scored)"' ../accuracy.json
  "passed": 1,
  "predictions_registered": 1,
  "predictions_scored": 1,

The record holds outcomes only: how many predictions were registered, how many could be scored and how many were within their interval, the metrics, and the seal of each prediction, so a reader can check each record against the digest published at registration. It holds no predicted or observed value and no statement. A prediction whose horizon has not closed is counted as registered and not as scored.

When the store refuses

Every record is checked against its seal each time it is read. A record edited by hand, for example to move its interval after the outcome is known, is refused by score and by record, which names every record that fails:

$ sed -i.bak 's/"0.55"/"0.50"/' .flux/accuracy/predictions/recovery-rate-q3.json
$ fluxsim accuracy record --registered-at 2026-06-30T09:00:00Z --out ../edited.json   # exit 1
FLX-7008: sealed prediction record does not verify -- ...
more: fluxsim explain FLX-7008  ...
$ mv .flux/accuracy/predictions/recovery-rate-q3.json.bak .flux/accuracy/predictions/recovery-rate-q3.json

The other refusals:

  • FLX-7003: a prediction_id that is not one plain identifier, such as one that climbs out of the store with ../.
  • FLX-7004: record was given a registration time that no prediction carries.
  • FLX-7005: a prediction or readout that is not the right shape; the message lists every field at fault.
  • FLX-7006: no file at the path given.
  • FLX-7009: a symbolic link, or the wrong kind of file, where a record or one of the store's directories belongs. The store follows no link below <state-dir>/accuracy.

See the evidence pack for how this fits beside the pack, and the accuracy reference for every option.

Last Updated: 11/19/26, 8:45 AM
Prev
Score a run against a profile, in full or sampled
Next
Load a simulation into a database