FLUXFLUX
  • Introduction
  • Quickstart
  • From Your Data to a Synthetic Copy
  • Two Specs, One Substrate
  • The Six Families
  • The FLUID Seam
  • Deterministic & Governed
  • Runtime & Evidence
  • Anatomy
  • The Nineteen Kinds
  • Versions
  • Changelog
  • JSON Schema 0.5.4
  • UI Hints 0.5.4
  • Enforcement Contract 0.5.0
  • Bundle Manifest 0.5.0
  • Overview
  • Tutorial
  • How-to guides
  • Reference
  • Explanation
  • Command line
  • Error codes
  • Accuracy program
  • The vNext Plan
  • The Nine RFCs
  • Playback Source (proposed)
  • Conformance Profiles
Examples
What's New
FLUID
GitHub
  • Introduction
  • Quickstart
  • From Your Data to a Synthetic Copy
  • Two Specs, One Substrate
  • The Six Families
  • The FLUID Seam
  • Deterministic & Governed
  • Runtime & Evidence
  • Anatomy
  • The Nineteen Kinds
  • Versions
  • Changelog
  • JSON Schema 0.5.4
  • UI Hints 0.5.4
  • Enforcement Contract 0.5.0
  • Bundle Manifest 0.5.0
  • Overview
  • Tutorial
  • How-to guides
  • Reference
  • Explanation
  • Command line
  • Error codes
  • Accuracy program
  • The vNext Plan
  • The Nine RFCs
  • Playback Source (proposed)
  • Conformance Profiles
Examples
What's New
FLUID
GitHub
  • FLUX Simulation Engine
  • Tutorial
    • First evidence pack in five minutes
  • How-to guides
    • Model your customers
    • Add a journey
    • Use the FLUID seam
    • Calibrate from aggregates: declare a scorecard
    • Draw a column given another
    • Ship with evidence
    • Write a bundle from a profile
    • Score a run against a profile, in full or sampled
    • Seal and score a prediction
    • Load a simulation into a database
  • Reference
    • Command line
      • fluxsim init
      • fluxsim validate
      • fluxsim plan
      • fluxsim apply
      • fluxsim run
      • fluxsim verify
      • fluxsim ship
      • fluxsim replay
      • fluxsim calibrate
      • fluxsim accuracy
      • fluxsim gate
      • fluxsim expr
      • fluxsim describe
      • fluxsim plugins
      • fluxsim explain
    • Error codes
    • Python API
    • Schemas
    • Specifications
      • FLUX Simulation Engine: kernel normative specification (v0.1)
      • FLUX Simulation Engine: kernel v2 normative specification (v0.2)
      • Seam generator and fidelity check: normative specification (v0.1)
      • Profile document: normative specification (format 1-1-0)
      • Scorecard: normative specification (format 1-1-0)
      • Sample-based verification: normative specification
      • Parameter register: normative specification (format 1-1-0)
      • flux-expr/1: normative specification (v0.1)
      • flux-expr/2: normative specification (v0.1)
      • flux-expr/3: normative specification (v0.1)
  • Explanation
    • Architecture and patterns
    • The six-step loop
    • Determinism and the seed tree
    • The FLUID seam
    • The evidence pack, and what it does not claim
  • Accuracy program

    • The accuracy program: design
    • Accuracy report

Load a simulation into a database

Every run writes its rows to the run directory as NDJSON. Use this page to write the same rows, as typed columns, to Parquet or Arrow IPC files or into a database table, in the same fluxsim run.

The stream's outputDigest does not change when you add a sink. The engine computes it before a sink sees a row, so a sink decides where the rows land and nothing about what they are.

Install what the sink needs

The engine installs without any columnar or database library. Each sink lives behind an extra, and the database sink also needs the ADBC driver for your database.

pip install 'flux-sim-engine[arrow]'                        # Parquet and Arrow IPC files
pip install 'flux-sim-engine[adbc]' duckdb                  # DuckDB
pip install 'flux-sim-engine[adbc]' adbc-driver-postgresql  # PostgreSQL
DatabaseDriver package
SQLiteadbc-driver-sqlite
DuckDBduckdb, 1.5 or later (it ships its own driver)
PostgreSQLadbc-driver-postgresql
Snowflakeadbc-driver-snowflake
BigQueryadbc-driver-bigquery

fluxsim plugins lists the sinks and says whether what each one needs is installed. It imports none of them to find out.

$ fluxsim plugins
vertical packs:
  telco
  towers
sinks (fluxsim run --sink NAME:TARGET):
  adbc       built-in  needs flux-sim-engine[adbc] (installed)
  arrow-ipc  built-in  needs flux-sim-engine[arrow] (installed)
  parquet    built-in  needs flux-sim-engine[arrow] (installed)

Write Parquet files

Name the sink and its target with --sink NAME:TARGET. For a file sink the target is a directory, and there is no default: a sink never writes anywhere you did not name.

$ fluxsim init demo
created demo with a small FLUX bundle and its FLUID contract (10 files):
...
$ cd demo
$ fluxsim run . --sink parquet:../parquet
plan written to .flux/runs/8835218acf7b4f8e/plan.json (runId=8835218acf7b4f8e, planDigest=sha256:8835218acf7b4f8e28b738c7d7e9e8d6b5c5e8bca59e47533fdda09eb90cf501)
applied to .flux/runs/8835218acf7b4f8e (runDigest=sha256:eb54ef239a1004cb7f3b31a7df97560d766eeb562c2f86147308aa6b1739cf8e)
emitted payment_recovery_moment: 154 rows (scenario)
delivered payment_recovery_moment: 154 rows in 1 parquet file under ../parquet/payment_recovery_moment/
run 8835218acf7b4f8e complete
$ ls ../parquet/payment_recovery_moment
_manifest-d9e491cf29bf8c57-parquet.json
d9e491cf29bf8c57-00000.parquet

The file name starts with the first 16 hex characters of the stream's outputDigest. The same bundle and seed always write the same name, so running again replaces the file instead of adding a second copy, and a run with another seed lands beside it. The manifest lists every file of the stream with its SHA-256, size and row count, the stream header, and the writer settings, and its name says which sink wrote it. A stream is published whole or not at all: files are written under hidden names and renamed into place once every row is in, the manifest is written last, and a stream that fails part-way leaves no manifest and no files behind. Names that start with _ or . are skipped by Parquet dataset readers, so the manifest is never read as data.

The columns are typed from the FLUID contract, and no value goes through a float:

Logical typeParquet / Arrow
STRING, JSONstring
INTint64
DECIMAL(p,s)decimal, exactly p digits at scale s (up to 76 digits)
BOOLboolean
DATEdate
TIMESTAMPtimestamp in milliseconds, UTC
BYTESbinary

Files are compressed with zstd. Each file's metadata also carries the stream header, the run id, the outputDigest and flux.synthetic = true.

Check that the files hold the run

Reading the files back and rendering the rows again gives the stream's outputDigest. The run's receipt.json records the digest the engine computed.

import json
from pathlib import Path

from flux_engine.sinks.arrow import output_digest_of
from flux_engine.sinks.parquet import read_stream

work = Path(workdir)
receipt = json.loads((work / "demo/.flux/runs/8835218acf7b4f8e/receipt.json").read_text())
expected = receipt["emits"]["payment_recovery_moment"]["outputDigest"]

table = read_stream(work / "parquet/payment_recovery_moment/_manifest-d9e491cf29bf8c57-parquet.json")
assert table.num_rows == 154
assert output_digest_of(table).value == expected

Add --sink-opt verify=true to have the sink make the same check on its own files before it publishes them.

Query them

Any Parquet reader works. With DuckDB:

import duckdb

markets = duckdb.connect().execute(
    f"SELECT market, count(*) FROM '{workdir}/parquet/payment_recovery_moment/*.parquet' "
    "GROUP BY market ORDER BY market"
).fetchall()
assert sum(count for _, count in markets) == 154

Load into DuckDB or SQLite

For the database sink the target is the driver, and the connection is a sink option. SQLite and DuckDB take a database file.

$ fluxsim run . --sink adbc:duckdb --sink-opt uri=../sim.duckdb
...
delivered payment_recovery_moment: 154 rows into sim_payment_recovery_moment at duckdb:../sim.duckdb
run 8835218acf7b4f8e complete

The table is named after the expose with the prefix sim_, a namespace of its own so a simulated table cannot be mistaken for a real one. Change the prefix with --sink-opt prefix=..., or put the tables in a schema of their own with --sink-opt schema=sim, which is created when missing.

Run it again and the table still holds 154 rows. Each load first deletes the rows of its own outputDigest, then inserts the stream, in one transaction, so a replay never duplicates rows. A run with another seed has another digest and adds its rows beside the first.

$ fluxsim run . --sink adbc:duckdb --sink-opt uri=../sim.duckdb --sink-opt verify=true
...
delivered payment_recovery_moment: 154 rows into sim_payment_recovery_moment at duckdb:../sim.duckdb
run 8835218acf7b4f8e complete

With verify=true the sink reads its rows back inside the transaction and commits only when they digest to the stream's outputDigest.

con = duckdb.connect(f"{workdir}/sim.duckdb", read_only=True)
assert con.execute("SELECT count(*) FROM sim_payment_recovery_moment").fetchone()[0] == 154
columns = [row[0] for row in con.execute("DESCRIBE sim_payment_recovery_moment").fetchall()]
assert columns[-4:] == ["_flux_synthetic", "_flux_run_id", "_flux_output_digest", "_flux_row"]
label = con.execute(
    "SELECT comment FROM duckdb_tables() WHERE table_name = 'sim_payment_recovery_moment'"
).fetchone()[0]
assert json.loads(label)["synthetic"] is True
con.close()

After the stream's own columns, every table has four of the sink's:

  • _flux_synthetic: always true, on every row.
  • _flux_run_id: the run the row came from.
  • _flux_output_digest: the stream the row belongs to. A slice is the rows with one digest.
  • _flux_row: the row's place in the stream. ORDER BY _flux_row returns the stream in its own order.

Where the database has table comments (DuckDB, PostgreSQL, Snowflake) or descriptions (BigQuery), the table is also labelled with a JSON comment that says "synthetic": true and names the expose, the run and the latest load's digest. SQLite has neither, so there the four columns are the label.

SQLite has no exact decimal type, and its driver would write a timestamp without its Z. On SQLite the sink stores DECIMAL and TIMESTAMP values as the text the engine renders, such as 6998.9770 and 2026-07-01T12:00:00.125Z, which SQLite's date functions read. A DECIMAL wider than 38 digits goes to every database as text, since 38 digits is the widest decimal most of them store exactly.

$ fluxsim run . --sink adbc:sqlite --sink-opt uri=../sim.sqlite --sink-opt prefix=twin_
...
delivered payment_recovery_moment: 154 rows into twin_payment_recovery_moment at sqlite:../sim.sqlite
run 8835218acf7b4f8e complete

Modes

--sink-opt mode= picks what happens to a table that already exists:

ModeTable missingTable exists
append (default)createdthe stream's slice is replaced, other slices are kept
createcreatedrefused
replacecreateddropped and created again, with the stream alone

The sink only ever changes tables it created. A table without its four _flux_ columns is refused in every mode, so a real table that happens to share a name is never appended to, emptied or dropped (FLX-3506). In append mode the columns must also match the stream's; when the contract changed, use mode=replace.

PostgreSQL, Snowflake and BigQuery

A server database's URI usually holds a password, and a command line is visible to every user of the machine and kept in shell history. Put the URI in an environment variable and pass its name with uri-env. A uri= with a password or a secret query parameter is refused.

export SIM_PG_URI='postgresql://sim:secret@db.example.com:5432/analytics'
fluxsim run . --sink adbc:postgresql --sink-opt uri-env=SIM_PG_URI --sink-opt schema=sim

PostgreSQL loads through COPY in binary. Its driver creates a DECIMAL column as an unconstrained NUMERIC, which keeps the number exactly and drops trailing zeros.

export SIM_SNOWFLAKE_URI='sim_loader:secret@my-account/ANALYTICS/SIM?warehouse=LOAD_WH'
fluxsim run . --sink adbc:snowflake --sink-opt uri-env=SIM_SNOWFLAKE_URI

Snowflake loads through a stage and COPY INTO.

BigQuery takes driver options instead of a URI. Any --sink-opt db.KEY=VALUE goes to the driver as an ADBC database option, and by default the driver authenticates with your Google application default credentials. A driver option that names a secret, such as a token or a password, is refused inline; pass --sink-opt db.KEY-env=VAR to read it from the environment.

fluxsim run . --sink adbc:bigquery \
  --sink-opt db.adbc.bigquery.sql.project_id=my-project \
  --sink-opt db.adbc.bigquery.sql.dataset_id=sim

The test suite runs the database sink against SQLite and DuckDB on every change, and against PostgreSQL in a container when Docker is available. Snowflake and BigQuery are not exercised by the test suite: the sink drives them through their own ADBC drivers the same way, and a report from a run against either is welcome. Snowflake commits DDL on its own and BigQuery has no transaction here, so on those two a failed first load can leave an empty table, which the next run replaces.

Settings

Pass each setting as --sink-opt KEY=VALUE, as many times as you need. An unknown key, or a value of the wrong type or out of range, stops the command before the simulation runs, and the message lists the keys the sink takes. A run writes to one sink: to deliver it to a second place, run again with the other --sink. The run is the same, so nothing is simulated differently, and its receipt is added beside the first.

SinkSettingDefaultMeaning
parquetcompressionzstdzstd, snappy, gzip, lz4, brotli or none
parquetcompression-level3 for zstd, the codec's own otherwisethe codec's level, where it has one; checked against the codec's range
parquetrow-group-size1048576rows per row group
parquet, arrow-ipcrows-per-fileone filestart a new file after this many rows
arrow-ipccompressionzstdzstd, lz4 or none
every sinkbatch-size65536rows per Arrow batch (for arrow-ipc, per record batch in the file)
every sinkverifyfalseread what was written back and check its digest before publishing
adbcurinonethe database file, or a URI without a password
adbcuri-envnonethe name of an environment variable holding the URI
adbcschemathe defaultthe schema (BigQuery: dataset) to write into
adbcprefixsim_the table name prefix; may be empty
adbcmodeappendappend, create or replace
adbcdb.KEYnonean ADBC database option for the driver
adbcdb.KEY-envnonethe name of an environment variable holding that driver option; required for one that names a secret

Receipts and the evidence pack

Each sink returns a receipt: the rows, and either the files with their SHA-256 or the table. fluxsim run records each one in sink-receipts.json in the run directory as soon as its stream is published, so a delivery that fails on a later stream still leaves the earlier ones on record. A receipt is kept per sink, target, table and expose: delivering the run to another place adds one, and delivering it to the same place again replaces that one.

$ grep -E '"(sink|table|target)"' .flux/runs/8835218acf7b4f8e/sink-receipts.json
      "sink": "adbc",
      "table": "sim_payment_recovery_moment",
      "target": "duckdb:../sim.duckdb",
      "sink": "adbc",
      "table": "twin_payment_recovery_moment",
      "target": "sqlite:../sim.sqlite",
      "sink": "parquet",
      "target": "../parquet",

A receipt never holds a user, a password, a query string or a fragment: a target such as postgresql://sim:secret@db.example.com/analytics?sslmode=require is recorded as postgresql://db.example.com/analytics. With --ship, the sink runs after verify and before ship, so ship packs sink-receipts.json into the evidence archive with the rest of the run.

When something is wrong

A mistake in the command line stops it before the simulation runs and names what to change: an unknown sink, a missing extra or driver, a missing target, an unknown or out-of-range setting, or a password on the command line.

$ fluxsim run . --sink parquet  # exit 2
FLX-3504: a sink was given a target or an option it cannot use -- the parquet sink needs a directory: --sink parquet:DIRECTORY
...
$ fluxsim run . --sink adbc:postgresql --sink-opt uri=postgresql://sim:secret@db.example.com/analytics  # exit 2
FLX-3504: a sink was given a target or an option it cannot use -- the uri carries a password or a secret, and a command line is not the place for one: put the uri in an environment variable and pass --sink-opt uri-env=NAME
...

What only the run's own streams or the destination can tell stops the delivery after the run instead, with exit status 1: two exposes whose tables would be one (checked before any stream is written), a table the sink will not touch, or a write the destination refuses. The run directory is written, a stream that failed is left out whole, and --ship does not run.

CodeWhat happened
FLX-3501the sink's extra or driver is not installed; the message has the pip install line
FLX-3504a target or a setting is missing or wrong
FLX-3505writing failed part-way; nothing partial was left
FLX-3506the database sink will not touch a table it did not create
FLX-3507there is no sink by that name

Write your own sink

A package adds a sink without changing the engine, through the flux_engine.sinks entry-point group:

[project.entry-points."flux_engine.sinks"]
mysink = "my_package.sink:target"

my_package.sink.target(target, options) receives the text after mysink: and the --sink-opt settings as strings. It checks them and returns an object with a name and a sink_for(context) method. context is a flux_engine.sinks.receipts.StreamContext: the run id, the expose id, the row count and the outputDigest. Each sink it returns implements flux_engine.api.emit.EmitSink (open, write, close). When close has seen every row, it sets receipt to a flux_engine.sinks.receipts.SinkReceipt; otherwise it leaves receipt as None and discards what it wrote. The engine loads the entry point only when you name the sink. A built-in name always wins over a package's.

These sink types are new and are not yet part of the governed flux_engine.api surface, so they may still change before they join it.

Iceberg tables and Kafka topics are not sinks here. Write Parquet and load it with the tools you already use for those.

Last Updated: 11/19/26, 8:45 AM
Prev
Seal and score a prediction