Accuracy report
What each family measures, and the pass criterion every number below is held to, is in DESIGN.md. Results are measured at the stated sizes and seeds, not established in general.
Latest nightly run: 2026-11-11, commit bd36cc7dcad4, base seed 8201423603942286921, confirmation seed 8201423603942286922, nightly workflow.
Verdict: passed.
Summary
pytest: 486 tests, 0 failed, 0 errored, 14 skipped, 0 expected failures (strict xfails).
| Family | Cases | Passed |
|---|---|---|
| 1. Fidelity metrics on the profile S | 9 | 9 |
| 2. Goodness of fit: kernel samplers | 25 | 25 |
| 2. Goodness of fit: power against a 2% shift | 4 | 4 |
| 2. Goodness of fit: the decimal Φ and Φ⁻¹ | 3 | 3 |
| 2. Goodness of fit: declared distributions, by kind | 94 | 94 |
| 3. The fidelity gate's own accuracy | 9 | 9 |
| 4. Properties and metamorphic relations | 32 | 32 |
| 5. Seed-tree stream quality | 28 | 28 |
| 6. Journey conformance | 18 | 18 |
| 7. Determinism | 5 | 5 |
| Recovery tests: calibrate and accuracy | 76 | 76 |
1. Fidelity metrics on the profile S
9 of 9 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| ContingencySimilarity-definition | 50,000 | pass | aggregate=0.8493, records=0.8493 | aggregate ContingencySimilarity equals the record-level crosstab value exactly |
| CorrelationSimilarity-definition | 50,000 | pass | similarity=0.9339, worstEncodedError=3.33e-16, worstRawError=2.49e-09 | r from moments equals numpy corrcoef to 1e-12 (encoded) and 1e-6 (raw) |
| KSComplement-definition | 50,000 | pass | aggregateDistance=0.1507, edgeDistance=0.1507, recordDistance=0.162 | aggregate KS equals the record KS at the edges; never exceeds the full record KS |
| TVComplement-definition | 50,000 | pass | aggregate=0.8493, records=0.8493 | aggregate TVComplement equals the record-level value exactly |
| engine-contingency-dependent | 100,000 | pass | ContingencySimilarity=0.8372, expectedDistance=0.008074, independenceCeiling=0.8392 | ContingencySimilarity within 3 x sampling distance of the independence ceiling |
| engine-contingency-independent | 100,000 | pass | ContingencySimilarity=0.9894, expectedDistance=0.008327 | 1 - ContingencySimilarity <= 3 x expected two-sample distance |
| engine-gate-agreement | 100,000 | pass | histogramDistance=0.00143, meanRelativeDeviation=0.1089 | the fidelity check's verdicts equal deviation <= margin computed from S |
| engine-histogram-family-mean | 100,000 | pass | committedMean=300.5, predictedMean=332.7, relativeShiftFromCommitted=0.1089, standardError=0.5512 | synthetic mean within 4 standard errors of the within-bin uniform prediction |
| engine-histogram-scores | 100,000 | pass | KSComplement=0.9986, TVComplement=0.9986, bound=0.006584, committedRecords=50,000 | 1 - TVComplement and 1 - KSComplement <= 3 x expected sampling distance |
2. Goodness of fit: kernel samplers
25 of 25 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| bernoulli-0.001 | 250,000 | pass | binomial p=0.9244, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bernoulli-0.35 | 250,000 | pass | binomial p=0.9615, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bernoulli-0.5 | 250,000 | pass | binomial p=0.1984, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bounded_uint-1000 | 100,000 | pass | chi2 p=0.3493, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bounded_uint-2 | 100,000 | pass | chi2 p=0.4555, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bounded_uint-3x2^61-residue | 100,000 | pass | chi2 p=0.9436, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| bounded_uint-7 | 100,000 | pass | chi2 p=0.5717, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| categorical-ten-with-a-rare-level | 100,000 | pass | chi2 p=0.1419, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| categorical-three | 100,000 | pass | chi2 p=0.9997, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| exponential-gap-ms | 100,000 | pass | cvm p=0.09709, ks p=0.1544, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| exponential-unit | 250,000 | pass | ad p=0.1401, cvm p=0.5162, ks p=0.4346, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| permutation-4 | 100,000 | pass | chi2 p=0.1167, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| permutation-6 | 50,000 | pass | chi2 p=0.07896, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| poisson-0.5 | 100,000 | pass | chi2 p=0.5647, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| poisson-1.5 | 100,000 | pass | chi2 p=0.4525, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| poisson-20 | 100,000 | pass | chi2 p=0.9212, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| poisson-250 | 50,000 | pass | chi2 p=0.4652, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| truncated_normal-above-mean | 50,000 | pass | cvm p=0.4244, ks p=0.3838, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| truncated_normal-integer | 50,000 | pass | chi2 p=0.998, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| truncated_normal-lower-tail | 50,000 | pass | cvm p=0.8619, ks p=0.7585, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| truncated_normal-one-digit | 50,000 | pass | chi2 p=0.6868, method=bh | no statistic rejected by Benjamini-Hochberg at q=0.01 across the discrete battery |
| truncated_normal-two-sided | 50,000 | pass | cvm p=0.1296, ks p=0.2514, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| truncated_normal-upper-tail | 50,000 | pass | cvm p=0.3196, ks p=0.2673, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| truncated_normal-wide | 50,000 | pass | ad p=0.0655, cvm p=0.2072, ks p=0.1913, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
| uniform53 | 250,000 | pass | cvm p=0.6337, ks p=0.7047, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the continuous battery |
2. Goodness of fit: power against a 2% shift
4 of 4 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| bernoulli-0.35 | 250,000 | pass | detected=20, power=1, repetitions=20, threshold=0.000588 | a 2% mean shift is detected in at least 90% of repetitions at the battery's rank-one critical value |
| exponential-unit | 250,000 | pass | detected=20, power=1, repetitions=20, threshold=0.000159 | a 2% mean shift is detected in at least 90% of repetitions at the battery's rank-one critical value |
| poisson-20 | 100,000 | pass | detected=20, power=1, repetitions=20, threshold=0.000588 | a 2% mean shift is detected in at least 90% of repetitions at the battery's rank-one critical value |
| truncated_normal-two-sided | 50,000 | pass | detected=20, power=1, repetitions=20, threshold=0.000159 | a 2% mean shift is detected in at least 90% of repetitions at the battery's rank-one critical value |
2. Goodness of fit: the decimal Φ and Φ⁻¹
3 of 3 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| cdf-ppf-round-trip | - | pass | - | relative round-trip error within the bound measured for each range of p |
| normal_cdf | 380 | pass | worstRelativeError=1.14e-13 | relative error against scipy.special.ndtr <= 1e-12 |
| normal_ppf | 799 | pass | worstScaledError=4.46e-16 | error against scipy.special.ndtri <= 1e-12 x max(1, |x|) |
2. Goodness of fit: declared distributions, by kind
94 of 94 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| arrivals/burst-start | 34,959 | pass | cvm p=0.8535, ks p=0.9267, method=by, window_ms=7,948,799,000 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| arrivals/emitters/engagement-0 | 20,000 | pass | emitters=7,000, expected=7,000 | the largest-remainder share is exact and the structural invariants hold |
| arrivals/emitters/engagement-0.5-frozen | 20,000 | pass | emitters=7,000, expected=7,000 | the largest-remainder share is exact and the structural invariants hold |
| arrivals/emitters/engagement-0.5-stochastic | 20,000 | pass | binomial p=0.5287, method=by, p=0.35 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| arrivals/emitters/engagement-1 | 100,000 | pass | binomial p=0.7883, emitters=34,959, method=by, p=0.35 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| arrivals/gaps | 52,252 | pass | cvm p=0.113, ks p=0.2497, mean_ms=3e+05, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| arrivals/occurrences | 34,959 | pass | chi2 p=0.8484, cells=9, mean=1.5, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| arrivals/structure | 34,959 | pass | emitters_whose_first_event_is_not_occurrence_0=0, occurrences_over_the_cap=0 | the largest-remainder share is exact and the structural invariants hold |
| campaign-delivery/0.3 | 80,000 | pass | binomial p=0.6798, delivered=23,946, method=by, share=0.3 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| campaign-delivery/0.3/ledger | 80,000 | pass | delivered=23,946, suppressed=56,054 | the largest-remainder share is exact and the structural invariants hold |
| campaign-delivery/default | 80,000 | pass | binomial p=0.8632, delivered=64,020, method=by, share=0.8 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| campaign-delivery/default/ledger | 80,000 | pass | delivered=64,020, suppressed=15,980 | the largest-remainder share is exact and the structural invariants hold |
| dial/0.5/frozen-share | 60 | pass | frozen=1,500, frozen_positives=450 | the largest-remainder share is exact and the structural invariants hold |
| dial/0.5/stochastic-share | 90,000 | pass | binomial p=0.785, method=by, p=0.3 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| dial/0/frozen-share | 60 | pass | frozen=3,000, frozen_positives=900 | the largest-remainder share is exact and the structural invariants hold |
| dial/1/frozen-share | 60 | pass | frozen=0, frozen_positives=0 | the largest-remainder share is exact and the structural invariants hold |
| dial/1/stochastic-share | 180,000 | pass | binomial p=0.3599, method=by, p=0.3 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| experiment/ordinal-0 | 2,000 | pass | chi2 p=0.6983, cells=3, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the experiment-assignment battery |
| experiment/ordinal-49 | 2,000 | pass | chi2 p=0.831, cells=3, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the experiment-assignment battery |
| experiment/ordinal-99 | 2,000 | pass | chi2 p=0.6408, cells=3, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the experiment-assignment battery |
| experiment/sizes | 4 | pass | - | the largest-remainder share is exact and the structural invariants hold |
| experiment/sizes-every-assignment | 2,000 | pass | - | the largest-remainder share is exact and the structural invariants hold |
| experiment/uniform | 4,000 | pass | method=by, smallest_adjusted=1 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the experiment-assignment battery |
| generator/battery | 150,000 | pass | - | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/cat_levels | 150,000 | pass | chi2 p=0.1592, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/cat_support | 150,000 | pass | chi2 p=0.9465, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/hist_dec | 150,000 | pass | chi2-bins p=0.6445, chi2-within-bin0 p=0.3782, cvm-within-bin1 p=0.9599, cvm-within-bin2 p=0.3038 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/hist_int | 150,000 | pass | chi2-bins p=0.6364, chi2-within-bin0 p=0.4275, chi2-within-bin1 p=0.03643, chi2-within-bin2 p=0.05601 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/identifiers | 150,000 | pass | distinct=yes, repeatable=yes, seedFree=yes, seedMovesDraws=yes | identifiers are pairwise distinct, well formed, identical across runs and seeds, while a drawn column moves with the seed |
| generator/norm_dec_declared | 150,000 | pass | chi2 p=0.9587, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/norm_dec_default | 150,000 | pass | chi2 p=0.5483, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/norm_int | 150,000 | pass | chi2 p=0.01226, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/null_cat | 119,951 | pass | binomial-null p=0.7518, chi2-nonnull p=0.1121, declared=0.2, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/null_norm | 97,391 | pass | binomial-null p=0.5552, chi2 p=0.1394, declared=0.35, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/power/cat_levels-1pt-moved | 150,000 | pass | defect=0.01 of probability moved from alpha to beta, pvalue=2.02e-10, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/cat_levels-3pt-moved | 150,000 | pass | defect=0.03 of probability moved from alpha to beta, pvalue=1.22e-128, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/hist_dec-bin-weights-swapped | 150,000 | pass | defect=bin weights 0.39 and 0.30 swapped, pvalue=0, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/norm_dec_declared-floor-not-round | 150,000 | pass | defect=values floored to the grid instead of rounded (a half-step shift), pvalue=1.28e-23, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/norm_dec_declared-mean+2% | 150,000 | pass | defect=the declared mean 0.70 read 2% high, pvalue=0, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/norm_dec_default-mean+2% | 150,000 | pass | defect=the midpoint mean 1.2 read 2% high, pvalue=3.14e-55, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/norm_int-mean+2% | 150,000 | pass | defect=the declared INT mean 30 read 2% high, pvalue=9.92e-33, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/null_cat-fraction-0.21 | 150,000 | pass | defect=nullFraction 0.21 where 0.20 is declared, pvalue=2.33e-20, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/null_cat-fraction-0.23 | 150,000 | pass | defect=nullFraction 0.23 where 0.20 is declared, pvalue=1.88e-169, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/rate_dec_both-lower-clamp-mass-dropped | 150,000 | pass | defect=values below range.min not clamped onto it: the lower cell holds one cell's mass, pvalue=0, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/rate_dec_both-rate+10% | 150,000 | pass | defect=the exponential rate 0.8 read 10% high, pvalue=4.74e-195, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/rate_int_free-poisson-mean+2% | 150,000 | pass | defect=the Poisson mean 12 read 2% high, pvalue=1.73e-164, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/power/uni_dec-every-tenth-point | 150,000 | pass | defect=a uniform sampler snapped to every tenth grid point, pvalue=0, threshold=8.01e-05 | the module's own statistic rejects the defect at the battery's rank-one value |
| generator/ranges | 150,000 | pass | outside=0 | no draw outside its declared range or off its declared scale |
| generator/rate_dec_both | 150,000 | pass | chi2 p=0.3732, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/rate_dec_upper | 150,000 | pass | chi2 p=0.7925, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/rate_int_clamped | 150,000 | pass | chi2 p=0.4007, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/rate_int_free | 150,000 | pass | chi2 p=0.2585, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/uni_dec | 150,000 | pass | chi2 p=0.006871, chi2-mod10 p=0.2294, chi2-mod7 p=0.4649, method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/uni_int | 150,000 | pass | chi2 p=0.2109, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the generator battery |
| generator/uniform-ends | 150,000 | pass | max=100.000, min=0.000 | the integer range reaches both of its declared ends |
| journey-reaction/persona-high | 60,000 | pass | binomial p=0.1842, method=by, p=0.93, successes=55,717 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| journey-reaction/persona-low | 60,000 | pass | binomial p=0.8441, method=by, p=0.05, successes=2,989 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| journey-reaction/persona-pragmatic-family | 60,000 | pass | binomial p=0.7345, method=by, p=0.42, successes=25,241 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
| journey-reaction/structure | 180,000 | pass | transitions=180,000 | the largest-remainder share is exact and the structural invariants hold |
| population/battery | 150,001 | pass | violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/cohort-ties/declaration-order-is-not-key-order | 2 | pass | - | a tied remainder goes to the NFC-first key, whatever the seed |
| population/cohort-ties/four-way-ties | 2 | pass | - | a tied remainder goes to the NFC-first key, whatever the seed |
| population/cohort-ties/key-order-differs-after-nfc | 1 | pass | - | a tied remainder goes to the NFC-first key, whatever the seed |
| population/cohort-ties/two-halves-odd-size | 1,001 | pass | - | a tied remainder goes to the NFC-first key, whatever the seed |
| population/cohorts | 150,001 | pass | - | cohort sizes equal largest_remainder(size, lifecycleMix) exactly |
| population/market | 150,001 | pass | chi2 p=0.04885, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/personas | 150,001 | pass | - | persona split equals largest_remainder over equal weights exactly |
| population/power/above_mean-sd+1% | 150,001 | pass | defect=the declared stddev 0.1 read 1% high on the mirrored branch, pvalue=3.24e-10, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/above_mean-sd+10% | 150,001 | pass | defect=the declared stddev 0.1 read 10% high on the mirrored branch, pvalue=0, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/above_mean-sd+3% | 150,001 | pass | defect=the declared stddev 0.1 read 3% high on the mirrored branch, pvalue=1.7e-58, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/appetite-mean+0.02 | 150,001 | pass | defect=the declared mean 0.5 read 0.02 high (4%), pvalue=4.94e-216, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/appetite-mean+1% | 150,001 | pass | defect=the declared mean 0.5 read 1% high, pvalue=7.41e-17, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/market-weight-0.01-moved | 150,001 | pass | defect=0.01 of the market probability moved from de to pl, pvalue=8.05e-40, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/market-weight-0.02-moved | 150,001 | pass | defect=0.02 of the market probability moved from de to pl, pvalue=5.18e-145, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/segment-3pt-moved | 150,001 | pass | defect=0.03 of probability moved from core to edge, pvalue=7.04e-151, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/power/tight-upper-bound-0.9-read-0.85 | 150,001 | pass | defect=the declared max 0.9 read as 0.85, pvalue=6.69e-85, threshold=0.00046 | the module's own statistic rejects the defect at the battery's rank-one value |
| population/scalar-trait | 150,001 | pass | constant=yes, pathsForDrawnTrait=150,001, pathsForScalar=0 | a scalar trait is its declared constant and materialises no stream |
| population/trait-above_mean | 150,001 | pass | cvm p=0.7743, ks p=0.6937, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/trait-above_mean-bounds | 150,001 | pass | hi=0.9, lo=0.45, offScale=0, outside=0 | every draw inside its declared interval, at the trait scale |
| population/trait-appetite | 150,001 | pass | cvm p=0.5166, ks p=0.6717, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/trait-appetite-bounds | 150,001 | pass | hi=2.1, lo=-1.1, offScale=0, outside=0 | every draw inside its declared interval, at the trait scale |
| population/trait-segment | 150,001 | pass | chi2 p=0.8984, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/trait-tight | 150,001 | pass | cvm p=0.3756, ks p=0.3762, method=by, violations=0 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the population battery |
| population/trait-tight-bounds | 150,001 | pass | hi=0.9, lo=0.4, offScale=0, outside=0 | every draw inside its declared interval, at the trait scale |
| power/biased-ordinal | 63 | pass | defect=ordinal 0 arm probabilities (0.6, 0.25, 0.15), rank_one_threshold=3.36e-05, smallest_p=2.04e-23, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/burst-window | 14 | pass | defect=window cut by 0.03, rank_one_threshold=0.00022, smallest_p=2.04e-27, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/campaign | 14 | pass | defect=declared probability 0.8 moved by +0.01, rank_one_threshold=0.00022, smallest_p=2.85e-12, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/dial | 14 | pass | defect=declared probability 0.3 moved by +0.01, rank_one_threshold=0.00022, smallest_p=1.16e-16, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/emitter-share | 14 | pass | defect=declared probability 0.35 moved by +0.01, rank_one_threshold=0.00022, smallest_p=6.35e-12, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/gap-mean | 14 | pass | defect=gap mean x1.05, rank_one_threshold=0.00022, smallest_p=2.14e-11, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/poisson-mean | 14 | pass | defect=extra-occurrence mean x1.05, rank_one_threshold=0.00022, smallest_p=1.65e-27, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/reaction | 14 | pass | defect=declared probability 0.42 moved by +0.01, rank_one_threshold=0.00022, smallest_p=4.03e-06, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| power/skewed-shuffle | 63 | pass | defect=shuffle key skew 0.2, rank_one_threshold=3.36e-05, smallest_p=2.81e-10, tier=nightly | the injected defect is rejected: smallest p at or below the battery's rank-one critical value q/(m*c(m)) |
| runtime-kinds/battery | 14 | pass | method=by | no statistic rejected by Benjamini-Yekutieli at q=0.01 across the runtime-kind battery |
3. The fidelity gate's own accuracy
9 of 9 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| mean-exactness | 10,000 | pass | trials=500 | every verdict equals the declared rule on the sample deviation, in rationals |
| mean-roc | 10,000 | pass | auc=0.9999, committedRecords=10,000, falseAcceptRateAtTwice=0, falseAcceptUpper95=0.02429 | ROC area >= 0.99 between f <= 0.5 and f >= 1.5; nightly: FRR(f=0) and FAR(f=2) 95% upper bounds <= 5% |
| quantile-exactness | 10,000 | pass | trials=500 | every verdict equals the declared rule on the sample deviation, in rationals |
| quantile-roc | 10,000 | pass | auc=1, committedRecords=10,000, falseAcceptRateAtTwice=0, falseAcceptUpper95=0.02429 | ROC area >= 0.99 between f <= 0.5 and f >= 1.5; nightly: FRR(f=0) and FAR(f=2) 95% upper bounds <= 5% |
| ship-end-to-end-offset-0 | 64 | pass | bundleWritten=yes, exitCode=0, meanDeviation=0, tolerance=50 | ship exits 0 within tolerance and 1 with FLX-8001 outside it, as the rule decides |
| ship-end-to-end-offset-200 | 64 | pass | bundleWritten=no, exitCode=1, meanDeviation=200, tolerance=50 | ship exits 0 within tolerance and 1 with FLX-8001 outside it, as the rule decides |
| support-exactness | 10,000 | pass | max=0 | withheld exactly when the off-support record count exceeds the declared max |
| tv-exactness | 10,000 | pass | trials=500 | every verdict equals the declared rule on the sample deviation, in rationals |
| tv-roc | 10,000 | pass | auc=1, committedRecords=10,000, falseAcceptRateAtTwice=0, falseAcceptUpper95=0.02429 | ROC area >= 0.99 between f <= 0.5 and f >= 1.5; nightly: FRR(f=0) and FAR(f=2) 95% upper bounds <= 5% |
mean: tolerance 0.02, 10,000 committed and 10,000 synthetic records per trial; ROC area 0.9999.
| Injected deviation | Population deviation | Withheld | Withhold rate |
|---|---|---|---|
| 0 × tolerance | 0 | 0 of 150 | 0 |
| 0.5 × tolerance | 0.009995 | 1 of 50 | 0.02 |
| 0.9 × tolerance | 0.01799 | 11 of 50 | 0.22 |
| 1.1 × tolerance | 0.02199 | 32 of 50 | 0.64 |
| 1.5 × tolerance | 0.02999 | 50 of 50 | 1 |
| 2 × tolerance | 0.03998 | 150 of 150 | 1 |
False rejects at no deviation: 0 (95% upper bound 0.02429). False accepts at twice the tolerance: 0 (95% upper bound 0.02429).
quantile: tolerance 10, 10,000 committed and 10,000 synthetic records per trial; ROC area 1.
| Injected deviation | Population deviation | Withheld | Withhold rate |
|---|---|---|---|
| 0 × tolerance | 0 | 0 of 150 | 0 |
| 0.5 × tolerance | 5 | 0 of 50 | 0 |
| 0.9 × tolerance | 9 | 20 of 50 | 0.4 |
| 1.1 × tolerance | 11 | 40 of 50 | 0.8 |
| 1.5 × tolerance | 15 | 50 of 50 | 1 |
| 2 × tolerance | 20 | 150 of 150 | 1 |
False rejects at no deviation: 0 (95% upper bound 0.02429). False accepts at twice the tolerance: 0 (95% upper bound 0.02429).
tv: tolerance 0.05, 10,000 committed and 10,000 synthetic records per trial; ROC area 1.
| Injected deviation | Population deviation | Withheld | Withhold rate |
|---|---|---|---|
| 0 × tolerance | 0 | 0 of 150 | 0 |
| 0.5 × tolerance | 0.025 | 0 of 50 | 0 |
| 0.9 × tolerance | 0.045 | 10 of 50 | 0.2 |
| 1.1 × tolerance | 0.055 | 45 of 50 | 0.9 |
| 1.5 × tolerance | 0.075 | 50 of 50 | 1 |
| 2 × tolerance | 0.1 | 150 of 150 | 1 |
False rejects at no deviation: 0 (95% upper bound 0.02429). False accepts at twice the tolerance: 0 (95% upper bound 0.02429).
4. Properties and metamorphic relations
32 of 32 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| mr1-generator-same-bytes | 200 | pass | - | rows and generatorDigest identical, in a hostile decimal context too |
| mr1-run-child-hashseed | - | pass | pythonHashSeed=822991084, runDigest=sha256:2d1c6fafdb22d3ab785b1e418668a7a0bd4eb1f4447827bd01010087ad0249fb | byte-identical output under another PYTHONHASHSEED |
| mr1-run-hostile-decimal | - | pass | completed=yes | byte-identical output under prec=5 ROUND_UP |
| mr1-run-twice | - | pass | runDigest=sha256:2d1c6fafdb22d3ab785b1e418668a7a0bd4eb1f4447827bd01010087ad0249fb | byte-identical output |
| mr2-column-order | 24 | pass | columns=9, examples=400 | every column's values identical under any permutation of the column order |
| mr3-arrival-gaps-1800000 | 600 | pass | - | each gap within 1 ms of twice the original; emitters, starts, payloads equal |
| mr3-arrival-gaps-300000 | 600 | pass | - | each gap within 1 ms of twice the original; emitters, starts, payloads equal |
| mr3-arrival-gaps-60000 | 600 | pass | - | each gap within 1 ms of twice the original; emitters, starts, payloads equal |
| mr3-exponential-scaling | - | pass | examples=5,000 | |exp(rate/c) - c*exp(rate)| <= (1+c)0.510^-scale on one stream |
| mr3-truncated-normal-location-scale | - | pass | checked=5,000, vacuous=402 | |tn(meanb+a, sdb, lob+a, hib+a) - (b*tn(mean, sd, lo, hi) + a)| <= (1+b)*10^-scale |
| mr4-marginals-ks | 10,000 | pass | method=bh, negativeControlP=0 | no two-sample KS rejected by Benjamini-Hochberg at q=0.01 across the numeric columns |
| mr4-prefix-stable | - | pass | examples=200 | the first n rows of a 2n-row generation equal the n-row generation |
| mr5-end-to-end-added-trait | - | pass | addedTraitRecords=1,000, population=3,504 | emits, events, decisions and every other population record byte-identical |
| mr5-generator-added-column | - | pass | examples=400 | every other column's values identical after adding a column at any position |
| mr5-world-added-trait | - | pass | examples=300 | every other trait of every subject identical after adding a trait |
| regression-mr1_ambient_decimal_context--control-prec28 | - | pass | knownDefect=no, property=mr1_ambient_decimal_context | the committed counterexample for mr1_ambient_decimal_context passes |
| regression-mr1_ambient_decimal_context--persona-quantise-prec5 | - | pass | knownDefect=no, property=mr1_ambient_decimal_context | the committed counterexample for mr1_ambient_decimal_context passes |
| regression-mr1_ambient_decimal_context--signal-gap-prec6 | - | pass | knownDefect=no, property=mr1_ambient_decimal_context | the committed counterexample for mr1_ambient_decimal_context passes |
| regression-mr1_generator_same_bytes--seed42 | - | pass | knownDefect=no, property=mr1_generator_same_bytes | the committed counterexample for mr1_generator_same_bytes passes |
| regression-mr2_column_order--reversed | - | pass | knownDefect=no, property=mr2_column_order | the committed counterexample for mr2_column_order passes |
| regression-mr3_arrival_gap_scaling--default-gap | - | pass | knownDefect=no, property=mr3_arrival_gap_scaling | the committed counterexample for mr3_arrival_gap_scaling passes |
| regression-mr3_exponential_scaling--half-rate | - | pass | knownDefect=no, property=mr3_exponential_scaling | the committed counterexample for mr3_exponential_scaling passes |
| regression-mr3_truncated_normal_equivariance--zero-stddev | - | pass | knownDefect=no, property=mr3_truncated_normal_equivariance | the committed counterexample for mr3_truncated_normal_equivariance passes |
| regression-mr4_prefix_stable--n7 | - | pass | knownDefect=no, property=mr4_prefix_stable | the committed counterexample for mr4_prefix_stable passes |
| regression-mr5_generator_added_column--normal-decimal-front | - | pass | knownDefect=no, property=mr5_generator_added_column | the committed counterexample for mr5_generator_added_column passes |
| regression-mr5_world_added_trait--categorical | - | pass | knownDefect=no, property=mr5_world_added_trait | the committed counterexample for mr5_world_added_trait passes |
| regression-schema_fluid_columns--decimal-and-timestamp | - | pass | knownDefect=no, property=schema_fluid_columns | the committed counterexample for schema_fluid_columns passes |
| regression-schema_flux_persona--normal-stddev-overflow | - | pass | knownDefect=no, property=schema_flux_persona | the committed counterexample for schema_flux_persona passes |
| regression-schema_flux_persona--scalar-categorical-normal | - | pass | knownDefect=no, property=schema_flux_persona | the committed counterexample for schema_flux_persona passes |
| schema-fluid-columns-as-generated | - | pass | ok=25, refused=575 | FluxError or conforming rows, never another exception |
| schema-fluid-columns-supported-types | - | pass | ok=521, refused=279 | FluxError or conforming rows, never another exception |
| schema-flux-persona-specs | - | pass | ok=819, refused=681 | FluxError or a population conforming to its declared traits, never another exception |
5. Seed-tree stream quality
28 of 28 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| avalanche/name-bit/entropy | 20,000 | pass | bits=256, flipFraction=0.4996, intervalLevel=0.9988, maxBitFraction=0.5091 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/name-bit/first-word | 20,000 | pass | bits=64, flipFraction=0.5006, intervalLevel=0.9988, maxBitFraction=0.5084 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/occurrence+-1/entropy | 20,000 | pass | bits=256, flipFraction=0.4998, intervalLevel=0.9988, maxBitFraction=0.5111 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/occurrence+-1/first-word | 20,000 | pass | bits=64, flipFraction=0.4995, intervalLevel=0.9988, maxBitFraction=0.5088 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/ordinal+-1/entropy | 20,000 | pass | bits=256, flipFraction=0.5001, intervalLevel=0.9988, maxBitFraction=0.5111 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/ordinal+-1/first-word | 20,000 | pass | bits=64, flipFraction=0.5002, intervalLevel=0.9988, maxBitFraction=0.5061 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/root-seed-lsb/entropy | 20,000 | pass | bits=256, flipFraction=0.5001, intervalLevel=0.9988, maxBitFraction=0.5157 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| avalanche/root-seed-lsb/first-word | 20,000 | pass | bits=64, flipFraction=0.4993, intervalLevel=0.9988, maxBitFraction=0.5067 | every output bit flips with probability 1/2: no exact binomial test rejected by BY at q=0.01; overall flip fraction inside its Clopper-Pearson interval |
| collisions/generator-row | 4,194,304 | pass | boundEntropy128BitPool=5.17e-26, entropyCollisions=0, expectedFirstWordCollisions=4.77e-07, firstWordCollisions=0 | no collision among sibling 256-bit entropies or among first 64-bit words |
| collisions/nfc/'café' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'naïve-trait' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'xéyèz' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'Ångström' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'é' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'ṩ' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/nfc/'한글' | - | pass | differsInCodePoints=yes | NFC-equivalent spellings give the same label and entropy |
| collisions/one-character-names | 4,194,304 | pass | boundEntropy128BitPool=5.17e-26, entropyCollisions=0, expectedFirstWordCollisions=4.77e-07, firstWordCollisions=0 | no collision among sibling 256-bit entropies or among first 64-bit words |
| collisions/separator/U+001E/first | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/separator/U+001E/last | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/separator/U+001E/middle | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/separator/U+001F/first | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/separator/U+001F/last | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/separator/U+001F/middle | - | pass | - | a part holding U+001F or U+001E is refused with FLX-2202 |
| collisions/signal-occurrence | 4,194,304 | pass | boundEntropy128BitPool=5.17e-26, entropyCollisions=0, expectedFirstWordCollisions=4.77e-07, firstWordCollisions=0 | no collision among sibling 256-bit entropies or among first 64-bit words |
| collisions/subject-trait | 4,194,304 | pass | boundEntropy128BitPool=5.17e-26, entropyCollisions=0, expectedFirstWordCollisions=4.77e-07, firstWordCollisions=0 | no collision among sibling 256-bit entropies or among first 64-bit words |
| sibling-quality/bit-balance | 1,048,576 | pass | method=by, minAdjusted=1, minPvalue=0.03084 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across byte frequency, per-bit balance and lag-1 serial correlation of sibling first words |
| sibling-quality/byte-frequency | 1,048,576 | pass | method=by, minAdjusted=1, minPvalue=0.3947 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across byte frequency, per-bit balance and lag-1 serial correlation of sibling first words |
| sibling-quality/serial-correlation | 1,048,576 | pass | method=by, minAdjusted=1, minPvalue=0.5282 | no statistic rejected by Benjamini-Yekutieli at q=0.01 across byte frequency, per-bit balance and lag-1 serial correlation of sibling first words |
6. Journey conformance
18 of 18 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| dwell/power-c1.1 | 10,000 | pass | caught=20, largest_smallest_p=1.1e-14, rank_one_threshold=0.0012, rate=1 | the unscaled c=1.1 comparison has min(KS, AD) p at or below the battery's rank-one critical value in >= 90% of repetitions |
| dwell/sample-sanity | 10,000 | pass | - | each sample has at least one dwell per emitter, all >= 1 ms, mean within 5 SE |
| dwell/scaled-2 | 60,022 | pass | anderson_ksamp p=0.1939, ks_2samp p=0.2221, method=by | no two-sample KS or k-sample Anderson-Darling statistic rejected by Benjamini-Yekutieli at q=0.01 |
| dwell/seeds | 59,638 | pass | anderson_ksamp p=0.25, ks_2samp p=0.3771, method=by | no two-sample KS or k-sample Anderson-Darling statistic rejected by Benjamini-Yekutieli at q=0.01 |
| transitions/battery | 144,000 | pass | method=bh | no (Persona, from-state, step) cell rejected by the exact binomial test under Benjamini-Hochberg at q=0.01, both seeds in one battery |
| transitions/dial/0 | 60,000 | pass | frozen=60,000, frozen_successes=22,200, stochastic=0 | frozen successes equal LRM(frozen count, p) and are the first frozen ordinals, in every (step, seed); no stochastic-share binomial rejected under Benjamini-Hochberg at q=0.01 |
| transitions/dial/0.5 | 60,000 | pass | frozen=30,000, frozen_successes=11,100, stochastic=30,000 | frozen successes equal LRM(frozen count, p) and are the first frozen ordinals, in every (step, seed); no stochastic-share binomial rejected under Benjamini-Hochberg at q=0.01 |
| transitions/dial/1 | 60,000 | pass | frozen=0, frozen_successes=0, stochastic=60,000 | frozen successes equal LRM(frozen count, p) and are the first frozen ordinals, in every (step, seed); no stochastic-share binomial rejected under Benjamini-Hochberg at q=0.01 |
| transitions/persona-a/atrisk | 144,000 | pass | cells=6, declared=0.42, method=bh | no (Persona, from-state, step) cell rejected by the exact binomial test under Benjamini-Hochberg at q=0.01, both seeds in one battery |
| transitions/persona-a/new | 144,000 | pass | cells=6, declared=0.42, method=bh | no (Persona, from-state, step) cell rejected by the exact binomial test under Benjamini-Hochberg at q=0.01, both seeds in one battery |
| transitions/persona-b/atrisk | 144,000 | pass | cells=6, declared=0.65, method=bh | no (Persona, from-state, step) cell rejected by the exact binomial test under Benjamini-Hochberg at q=0.01, both seeds in one battery |
| transitions/persona-c/active | 144,000 | pass | cells=6, declared=0.08, method=bh | no (Persona, from-state, step) cell rejected by the exact binomial test under Benjamini-Hochberg at q=0.01, both seeds in one battery |
| transitions/power | 12,000 | pass | caught=8, of=8, rank_one_threshold=0.000417, worst_p=9.44e-101 | a 0.1 shift of the declared probability is caught in every row, both seeds |
| transitions/structure | 144,000 | pass | transitions=193,457 | no transition from a row the taxonomy forbids; every success/failure recorded as declared |
| transitions/two-seeds/persona-a/atrisk | 288,000 | pass | pvalue=0.6286 | no row's seed-vs-seed 2x2 chi-square rejected by Benjamini-Hochberg at q=0.01 |
| transitions/two-seeds/persona-a/new | 288,000 | pass | pvalue=0.6101 | no row's seed-vs-seed 2x2 chi-square rejected by Benjamini-Hochberg at q=0.01 |
| transitions/two-seeds/persona-b/atrisk | 288,000 | pass | pvalue=0.8385 | no row's seed-vs-seed 2x2 chi-square rejected by Benjamini-Hochberg at q=0.01 |
| transitions/two-seeds/persona-c/active | 288,000 | pass | pvalue=0.6511 | no row's seed-vs-seed 2x2 chi-square rejected by Benjamini-Hochberg at q=0.01 |
7. Determinism
5 of 5 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| ambient-decimal-5:ROUND_UP | - | pass | ambient=5:ROUND_UP, differing=0, exit=0 | the end-to-end run does not depend on the ambient decimal context |
| ambient-decimal-7:ROUND_UP | - | pass | ambient=7:ROUND_UP, differing=0, exit=0 | the end-to-end run does not depend on the ambient decimal context |
| hostile-subprocess | - | pass | ambient=8:ROUND_UP, digests=35 | PYTHONHASHSEED, LC_ALL=C and an ambient decimal context change no digest |
| in-process-twice | - | pass | differing=0, digests=35 | two builds in one process have identical digests |
| pinned-run | - | pass | runDigest=2d1c6fafdb22d3ab785b1e418668a7a0bd4eb1f4447827bd01010087ad0249fb | the small fixture's run digest and emit digest equal their pins |
Recovery tests: calibrate and accuracy
76 of 76 measured cases passed.
| Case | n | Result | Measured | Criterion |
|---|---|---|---|---|
| accuracy-boundary-at-high | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-at-low | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-deep-decimal-above | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-deep-decimal-at-high | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-deep-decimal-below | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-degenerate-miss | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-degenerate-point | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-inside | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-integers-as-numbers | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-just-above-high | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-just-below-low | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-negative-interval | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-negative-outside | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-number-observed-string-bounds | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-numbers-above | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-numbers-at-high | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-numbers-at-low | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-straddles-zero | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-string-observed-number-bounds | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-zero-string-high | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-boundary-zero-string-low | 1 | pass | - | verdict equals containment at the boundary |
| accuracy-containment-c0.8 | 1,000 | pass | mismatches=0 | each verdict equals low <= observed <= high on exact decimals |
| accuracy-containment-c0.95 | 1,000 | pass | mismatches=0 | each verdict equals low <= observed <= high on exact decimals |
| accuracy-coverage-c0.8 | 1,000 | pass | ciHigh=0.8398, ciLow=0.7687, level=0.995, nominal=0.8 | nominal coverage lies inside the 99.5% Clopper-Pearson interval of the measured rate |
| accuracy-coverage-c0.95 | 1,000 | pass | ciHigh=0.9649, ciLow=0.924, level=0.995, nominal=0.95 | nominal coverage lies inside the 99.5% Clopper-Pearson interval of the measured rate |
| accuracy-record-c0.8 | 1,000 | pass | passed=806, registered=1,000 | the record covers every registered prediction and counts the independent passes |
| accuracy-record-c0.95 | 1,000 | pass | passed=947, registered=1,000 | the record covers every registered prediction and counts the independent passes |
| accuracy-record-seal-digests-c0.8 | 1,000 | pass | matching=1,000 | the record's predictionDigests are the sealDigests register returned |
| accuracy-record-seal-digests-c0.95 | 1,000 | pass | matching=1,000 | the record's predictionDigests are the sealDigests register returned |
| accuracy-zero-bound--5-0--1-interval_high | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound--5-0-1-interval_high | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound--5-0-1-strings | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound--5-0.0--1-interval_high | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-0-0-interval_low-interval_high | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-0-0-strings | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-5-0-interval_low-observed | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-5-2-interval_low | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-5-2-strings | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0-5-7-interval_low | 1 | pass | - | a registered zero bound is scored, not dropped |
| accuracy-zero-bound-0.0-5-2-interval_low | 1 | pass | - | a registered zero bound is scored, not dropped |
| calibrate-cli-fixture | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-A | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-B | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-C | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-D | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-E | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-grade-F | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-integers | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-just-below-one | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-months | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-ones | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-required-only | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-tiny | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-weeks | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-years | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-cli-zeros | 1 | pass | - | the CLI reports the authored scorecard exactly and grades nothing |
| calibrate-fixed-fixture | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-A | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-B | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-C | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-D | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-E | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-grade-F | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-integers | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-just-below-one | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-months | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-ones | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-required-only | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-tiny | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-weeks | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-years | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-fixed-zeros | 1 | pass | - | authored scorecard fields, credibility and digest recovered exactly |
| calibrate-property-api | - | pass | - | every schema-valid scorecard is recovered exactly through the API |
| calibrate-property-cli | - | pass | - | the CLI reports every schema-valid scorecard exactly |
| calibrate-property-dials | - | pass | - | dials pass through the calibrator unchanged |
| calibrate-property-digest-moves | - | pass | - | the evidence digest moves if and only if the authored block moves |
5. PractRand
| Feed | Bytes tested | Verdict | Anomalies by evaluation |
|---|---|---|---|
| first-words | 1,073,741,824 | pass | unusual: 2 |
| interleave | 1,073,741,824 | pass | unusual: 1 |
7. Determinism matrix
6 cells, 35 digests each: every cell's manifest is identical.
| Cell | Platform | Python | NumPy |
|---|---|---|---|
| ubuntu-latest-py3.11-numpy1.26.0 | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.11.18 | 1.26.0 |
| ubuntu-latest-py3.11-numpylatest | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.11.18 | 2.4.6 |
| ubuntu-latest-py3.12-numpy1.26.0 | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.12.16 | 1.26.0 |
| ubuntu-latest-py3.12-numpylatest | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.12.16 | 2.5.4 |
| ubuntu-latest-py3.13-numpy2.1.0 | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.13.18 | 2.1.0 |
| ubuntu-latest-py3.13-numpylatest | Linux-7.0.0-1019-azure-x86_64-with-glibc2.43 | 3.13.18 | 2.5.4 |
8. Speed
Median wall time per call on the nightly runner. A benchmark more than 3 times slower than the committed snapshot fails the night; more than 1.5 times is flagged.
| Benchmark | Median | Rounds | Against snapshot |
|---|---|---|---|
| test_bench[fidelity-check-evaluate-x5000] | 115,699.4 µs | 9 | 0.98× |
| test_bench[fidelity-compute-statistic-x5000] | 114,403.0 µs | 9 | 0.98× |
| test_bench[generator-rows-x1000] | 354,828.8 µs | 5 | 0.99× |
| test_bench[sampler-bernoulli-x500] | 8,244.1 µs | 120 | 0.99× |
| test_bench[sampler-bounded_uint-x500] | 17,499.6 µs | 57 | 0.99× |
| test_bench[sampler-categorical-x500] | 13,872.3 µs | 71 | 0.98× |
| test_bench[sampler-exponential-x500] | 30,128.1 µs | 33 | 1.02× |
| test_bench[sampler-poisson-x500] | 22,561.3 µs | 44 | 1.00× |
| test_bench[sampler-truncated_normal-x500] | 135,834.5 µs | 8 | 0.98× |
| test_bench[sampler-uniform53-x500] | 6,340.5 µs | 153 | 0.99× |
| test_bench[stream-entropy-x1000] | 2,362.1 µs | 420 | 1.00× |
| test_bench[stream-from-entropy-x1000] | 12,258.4 µs | 80 | 1.00× |
Mutation testing
Mutation testing runs weekly; this artefact is from a night without it.
What was not tested
- Simulation-based calibration. It waits for the fitted calibrator. Today
calibratereads a declared scorecard and fits nothing, so there is no posterior to calibrate. The recovery tests above cover what the declared path does. - Drift-detection accuracy. It waits for drift detection on S, which does not exist yet.
- Joint structure in generated data. The generator draws each column independently. Section 1 measures the resulting shortfall in ContingencySimilarity and CorrelationSimilarity; nothing here claims joint fidelity.
- Privacy and re-identification resistance. S is aggregate by construction. Its privacy properties are outside this program.
- The agentic tier and the stochastic sweep executor. Neither is built; the engine refuses both, and the main suite tests those refusals.
- Real data. Every record set here is synthetic, drawn from stated parameters (ADR 0003). A fit to an owner's real committed state is measured only by that owner's own fidelity check.