AWS Provider

Deploy data products to Amazon Web Services (S3, Glue, Athena, Lake Formation) with the same contract and CLI commands you use for the other providers.

Docs Baseline: CLI 0.18.1
fluid apply creates: S3 buckets, Glue databases and tables, Lake Formation registrations and grants, KMS keys, S3 lifecycle rules
fluid verify reads: Glue, Athena, S3, KMS, Lake Formation

Why it matters Set binding.platform: aws with a format and a location, and fluid apply emits an OpenTofu module for S3, Glue and Lake Formation. Athena queries the Glue table it creates.

Overview

The AWS provider turns a FLUID contract into cloud resources:

• Plan and apply — S3 buckets, Glue databases and Glue tables. No Athena workgroup is created: Athena queries run in a workgroup you already have, and fluid verify uses primary unless you pass --athena-workgroup.
• Lake Formation governance — location registration, principal grants, column-limited grants, LF-tags (TBAC), row filters and the bucket policy that goes with the grants. See Lake Formation.
• Column restrictions — policy.authz.columnRestrictions becomes each Lake Formation grant's excluded columns. See Column restrictions.
• Encryption and retention (0.7.6) — a KMS key per bucket and an S3 lifecycle rule. See Encryption at rest and retention.
• Masking at landing — the DuckDB runner treats policy.privacy.masking columns before they reach S3. See Masking at landing.
• Verification — fluid verify reads Glue, runs an Athena COUNT(*) and checks the governance above on the live account. See Verify on AWS.
• Sovereignty validation — region allow and deny lists are enforced before deployment. See Data Sovereignty.
• IAM policy compilation — fluid policy-compile writes IAM bindings from accessPolicy, as advisory output only. See accessPolicy on AWS.

Working example: Bitcoin price tracker

This contract passes fluid validate --strict on CLI 0.18.1. Its read access is Lake Formation grants on the binding, and one of the two grantees has two columns restricted from it.

fluidVersion: "0.7.5"
kind: DataProduct
id: crypto.market_data.bitcoin_prices_aws_v1
name: Bitcoin Price Tracker (AWS Athena)
description: >
  Bitcoin price data on S3, cataloged in Glue and queried through Athena.
domain: Market Data

metadata:
  layer: Gold
  owner:
    team: data-engineering
    email: data-eng@company.example.com

# The binding's region must be one of these. With enforcementMode: strict,
# a region written as an {{ env.* }} placeholder fails `fluid validate`.
sovereignty:
  jurisdiction: "EU"
  dataResidency: true
  allowedRegions:
    - eu-central-1
    - eu-west-1
  enforcementMode: strict

exposes:
  - exposeId: bitcoin_prices_table
    title: Bitcoin Price Feed
    version: "1.0.0"
    kind: table

    binding:
      platform: aws
      format: parquet
      location:
        database: crypto_data
        table: bitcoin_prices
        bucket: "{{ env.S3_BUCKET }}"
        path: data/bitcoin/prices/
        region: eu-west-1

      # On AWS, who may read the table is declared here, as Lake Formation
      # grants. accessPolicy is not written to AWS.
      governance:
        lakeFormation:
          registerLocation: true
          grants:
            - principal: "arn:aws:iam::{{ env.AWS_ACCOUNT_ID }}:role/data-analyst"
              permissions: [SELECT]
            - principal: "arn:aws:iam::{{ env.AWS_ACCOUNT_ID }}:role/intern"
              permissions: [SELECT]

    policy:
      classification: Internal
      authn: iam
      authz:
        readers:
          - "arn:aws:iam::{{ env.AWS_ACCOUNT_ID }}:role/data-analyst"
          - "arn:aws:iam::{{ env.AWS_ACCOUNT_ID }}:role/intern"
        columnRestrictions:
          - principal: "arn:aws:iam::{{ env.AWS_ACCOUNT_ID }}:role/intern"
            columns: [market_cap_usd, volume_24h_usd]
            access: deny

    contract:
      schema:
        - name: price_timestamp
          type: timestamp
          required: true
          description: UTC timestamp when the price was recorded
        - name: price_usd
          type: decimal(18,2)
          required: true
          description: Bitcoin price in USD
        - name: market_cap_usd
          type: decimal(20,2)
          required: false
          description: Total market capitalization in USD
          sensitivity: internal
        - name: volume_24h_usd
          type: decimal(20,2)
          required: false
          description: 24-hour trading volume in USD
          sensitivity: internal

# A placeholder for your own ingest code in ./runtime. This page does not
# ship that script.
builds:
  - id: bitcoin_price_ingestion
    description: Fetch Bitcoin prices from the CoinGecko API
    pattern: hybrid-reference
    engine: python
    repository: ./runtime
    properties:
      model: ingest
    execution:
      trigger:
        type: manual
      runtime:
        image: python:3.11-slim
        dependencies: [boto3, requests, pyarrow]
        env: [AWS_REGION, S3_BUCKET]
    outputs:
      - bitcoin_prices_table

Set the two placeholders, validate, and review the module fluid apply would run:

export S3_BUCKET=my-fluid-data-bucket AWS_ACCOUNT_ID=123456789012

fluid validate contract.fluid.yaml --strict
fluid generate iac contract.fluid.yaml --provider aws --out ./review

✅ Valid FLUID contract (schema v0.7.5)
...
Wrote OpenTofu module: ./review/main.tf.json  (provider: aws, 7 resources)

fluid validate keeps {{ env.* }} placeholders literal. fluid generate iac and fluid apply resolve them before they write the module, A variable that is unset is not an error: the placeholder is written into the module as literal text, so set both before you generate or apply.

Infrastructure created

The seven resources the example above produces:

Declaring more fields adds resources: the Lake Formation fields add aws_lakeformation_data_lake_settings, aws_lakeformation_lf_tag, aws_lakeformation_resource_lf_tags and aws_lakeformation_data_cells_filter, and the 0.7.6 fields add aws_kms_key, aws_kms_alias, aws_s3_bucket_server_side_encryption_configuration and aws_s3_bucket_lifecycle_configuration.

The module also looks up the caller's account (data.aws_caller_identity) whenever a Lake Formation, KMS or {account}-fluid-data bucket reference needs it.

After the build lands data under the table's prefix, query it in Athena:

SELECT price_timestamp, price_usd
FROM crypto_data.bitcoin_prices
ORDER BY price_timestamp DESC
LIMIT 5;

The Glue table exists, empty, as soon as fluid apply finishes; it has rows once a build has written objects under data/bitcoin/prices/.

Shared vs. isolated containers

Key schema patterns

The binding schema uses three fields to identify platform resources:

GCP uses platform: gcp with format: bigquery_table, and Snowflake platform: snowflake with format: snowflake_table. Retargeting a contract changes platform, format and location together; see Switch clouds.

Region

How --env picks the overlay that carries the region is in Environments and overlays.

Name a real region code in the binding:

location:
  region: eu-west-1

When every AWS binding in the contract names the same region code, the emitted module pins it on the provider, and it wins over AWS_REGION in your shell:

"provider": { "aws": { "region": "eu-west-1" } }

That is also what keeps the sovereignty check and the placement of the resources in agreement. What the module does not pin:

fluid generate iac and fluid apply resolve a placeholder such as {{ env.AWS_REGION }} first, so it is pinned when it resolves to a region code. fluid validate and fluid plan keep it literal. Under enforcementMode: strict (the schema default) a literal placeholder fails fluid validate with two errors, a region outside allowedRegions and a region that does not match the jurisdiction; under advisory the same findings are warnings, and --strict turns warnings into exit 1. Put the real region in the binding, in an overlay per environment if the environments differ (per-environment overlays).

Moving a contract between regions

Before planning, fluid apply reads the regions its existing state records. If they differ from the pinned region, it stops. This is what a contract first applied from a shell in another region meets: without the guard, tofu would plan the resources as new in the pinned region, plan no destroy, and leave the originals unmanaged. fluid diff runs the same check. The guard is described in fluid apply.

CLI command error
❌ opentofu_region_moved  [ERR_OPENTOFU_REGION_MOVED]
  error: state holds this contract's resources in us-east-1, but its bindings
name eu-west-1. Applying would create them again in eu-west-1 and leave the
originals unmanaged
  state: .fluid/iac/aws/crypto_market_data_bitcoin_prices_aws_v1

💡 Suggestions:
  • If the resources should stay where they are, set the binding's
location.region to the region the error names
  • If they should move, empty and remove them there first (tofu destroy in the
state directory the error names, with AWS_REGION set to the old region), then
apply again

To keep the resources, set location.region to the region the error names (us-east-1 here). To move them, run tofu destroy in the state directory the error prints, with AWS_REGION set to the old region, then apply again. The bucket this product owns is created with force_destroy, so that destroy deletes its objects.

That output comes from the real apply path with the state read replaced by one that records us-east-1; it was not produced against an AWS account.

State

OpenTofu state covers the keys, backends and the commands that read state; One contract, two clouds deploys one contract to both clouds.

fluid apply keeps OpenTofu state per provider, so one contract deployed to AWS and to GCP through two --env overlays has two states and neither plan reads the other cloud's resources as orphans.

• Local state (the default) is in .fluid/iac/aws/<id>/ under the workspace directory (--workspace-dir, default the current directory), where <id> is the contract id with . and - written as _. The apply prints the module's path and a state: line.
• Remote state takes --state-backend s3://<bucket>/<key> or the FLUID_STATE_BACKEND variable. A key you write is used as written. A bucket with no key gets fluid/<contract id>/aws/terraform.tfstate when the spec comes from FLUID_STATE_BACKEND or the contract has a packaging block.
• A bucket-only --state-backend on a contract with no packaging block gets the shared key fluid/terraform.tfstate, which every such contract on that bucket uses. Name a key, or use FLUID_STATE_BACKEND, so two products do not share a state.

State that a release before the per-provider key wrote at fluid/<contract id>/terraform.tfstate is moved to the new key by the first apply, with OpenTofu's own init -migrate-state. When the old key holds another provider's state (a gcp apply that finds the aws state there), the apply leaves it and logs that the old key holds the other provider's state. A key that names no provider and holds another cloud's resources is refused with state_shared_with_another_provider.

CLI commands

The CLI reads the provider from the contract's binding.platform; no --provider flag is needed.

--provider disambiguates a contract that spans clouds or declares none; it is not a retargeting switch, and since 0.15.0 a --provider that contradicts every cloud the contract declares is rejected before anything is written, on both fluid generate iac and fluid apply. Retarget by editing binding — see the switch-clouds recipe.

The aliases the provider detector accepts — platform: s3, glue, athena, redshift as well as aws — are now also what the AWS emitter filters exposes[] with (since 0.15.0). They used to disagree: one of those aliases auto-detected as AWS and was then skipped by every AWS filter, so a correctly detected provider emitted an empty module.

# Validate against the bundled schema; --strict also fails on warnings
fluid validate contract.fluid.yaml --strict

# Review the module apply would run
fluid generate iac contract.fluid.yaml --provider aws --out ./review

# Generate execution plan
fluid plan contract.fluid.yaml --env dev --out plans/plan-dev.json

# Drift gate: compare the deployed Glue table with the contract
fluid diff contract.fluid.yaml --env dev --exit-on-drift

# Deploy S3 bucket, Glue DB and table, Lake Formation resources
fluid apply contract.fluid.yaml --env dev --yes

# Run the build
fluid apply contract.fluid.yaml --mode amend-and-build

# Check the deployment: Glue columns, Athena row count, governance
fluid verify contract.fluid.yaml --env dev --strict

# Compile advisory IAM bindings from accessPolicy
fluid policy-compile contract.fluid.yaml --env dev --out runtime/policy/bindings.json

# Generate Airflow DAG
fluid generate-airflow contract.fluid.yaml --output airflow-dags/bitcoin_aws.py

# Export standards
fluid odps export contract.fluid.yaml --out standards/product.odps.json --format json
fluid odcs export contract.fluid.yaml --output standards/product.odcs.yaml

fluid diff reads the Glue table in the region fluid apply uses and compares its columns and types with the contract. Since 0.18.1 its live check resolves {{ env.* }} placeholders in the binding before it reads the target.

fluid diff and fluid verify read AWS through boto3; install it with pip install 'data-product-forge[aws]'. Builds that land data with DuckDB need the local extra (duckdb>=1.5.0).

Where a build lands data

A pattern: acquisition build with engine: duckdb that reads one stream (source.streams lists one stream or none) writes to the destination of the expose its outputs list names. The first rule that matches decides. For a file format the build writes a file under the prefix; a table-format sink keeps the URI unchanged:

The path is bucket-relative and always treated as a prefix, with or without a trailing slash, and it is the same prefix the Glue table's location uses, so the data and the catalog agree. A path that is already a full s3:// URI is used as written. The extension follows sink.format (parquet, csv, or ndjson for json), and the credentials come from the ambient AWS credential chain, signed for the binding's location.region.

These rules apply to a build that reads one stream (source.streams lists one stream or none). A build with several streams writes each to out/<stream>.<ext>.

The bucket value has to resolve when the build runs. A {{ env.S3_BUCKET }} that is unset leaves the acquisition runner with no bucket to compose a URI from, and it writes location.path under the contract directory instead of failing. An embedded-SQL build refuses an unset bucket variable.

An embedded-SQL build (engine: sql) lands its result the same way when the first expose is an S3 binding: in s3://<bucket>/<path>/<location.table or exposeId>.<ext>, inside the Glue table's location. It writes parquet, csv or json, and refuses another format on an S3 binding.

Contract SQL on the DuckDB engine can read and write only the contract's own directory and the locations the contract declares. For S3 that means the declared s3:// locations: an s3:// URL in SQL that the contract does not declare is refused, and http(s)://, gs:// and Azure URLs cannot be read from contract SQL at all. Land remote data with an acquisition build first. The rules are in DuckDB sandbox for contract SQL; $ref composition is confined the same way, see Contract references.

For acquisition build properties, see Source-aligned acquisition; the same landing rules are in where the build lands data.

Credentials setup

Jenkins CI (recommended)

Create a Jenkins Secret File credential containing your AWS env vars:

# File contents (plain key=value, no 'export' prefix)
AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_REGION=eu-central-1
AWS_ACCOUNT_ID=123456789012
S3_BUCKET=my-fluid-data-bucket

The Universal Pipeline auto-detects this format and sources it into the pipeline's stages. No provider-specific credential logic.

Local development

# Option 1: AWS CLI profile
aws configure --profile fluid-dev

# Option 2: .env file (same format as Jenkins)
cat > .env << 'EOF'
AWS_ACCESS_KEY_ID=AKIAxxxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AWS_REGION=eu-central-1
AWS_ACCOUNT_ID=123456789012
S3_BUCKET=my-fluid-data-bucket
EOF

# Source and run
set -a; . .env; set +a
fluid apply contract.fluid.yaml --env dev --yes

Local testing against emulators (LocalStack / moto / MinIO)

Set the standard AWS SDK endpoint override — global or per-service:

export AWS_ENDPOINT_URL=http://localhost:4566        # global (LocalStack default port)
# …or per-service:
export AWS_ENDPOINT_URL_S3=http://localhost:4566
export AWS_ENDPOINT_URL_GLUE=http://localhost:4566

# LocalStack accepts any non-empty test credentials
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_REGION=us-east-1

fluid generate iac contract.fluid.yaml --provider aws --out ./review
tofu -chdir=./review init && tofu -chdir=./review apply

When AWS_ENDPOINT_URL* is set, the emitted module includes:

On real AWS (no AWS_ENDPOINT_URL* set) the provider block holds only the region, and only when the bindings pin one. The provider reads credentials from the standard credential chain.

For DuckDB builds:

• AWS_ENDPOINT_URL_S3 is used first, then AWS_ENDPOINT_URL. A value that is not an http(s):// URL naming a host, or that carries a user name or password, is ignored with a warning, and the build then reaches AWS itself.
• The store is addressed path-style unless its host ends in .amazonaws.com.
• AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true turns the override off.
• With an override set, a DuckDB CREATE SECRET that fails (for example because the aws extension cannot load) fails the run with ObjectStoreEndpointError. Without that, the build would read and write real AWS while the infrastructure was in the emulator. Make the DuckDB httpfs and aws extensions loadable, or unset the variable.

Governance on AWS

Governance parity sets these fields beside their GCP equivalents.

Forge emits no platform-native dynamic data masking on AWS. policy.privacy.rowLevelPolicy is a declaration the AWS emitter does not read; use lakeFormation.rowFilter for row-level filtering.

A policy the binding cannot apply is refused rather than dropped: fluid validate, fluid generate iac and fluid apply refuse a column restriction on an aws binding that has no Lake Formation grants or no Glue table.

Lake Formation

On AWS, access and row- and column-level governance are implemented through Lake Formation, declared under binding.governance.lakeFormation (with account-level admins and tag definitions under the top-level governance.lakeFormation block). On the OpenTofu apply path Forge emits:

The grants are on the Glue table, or on the database when the binding names no location.table. A grant's principal is an IAM principal ARN.

exposes:
  - exposeId: bitcoin_prices_table
    binding:
      platform: aws
      format: parquet
      governance:
        lakeFormation:
          registerLocation: true
          grants:
            - principal: "arn:aws:iam::123456789012:role/data-analyst"
              permissions: [SELECT]
              columns: [price_timestamp, price_usd]   # column-limited grant
          rowFilter:
            name: recent_only
            rowExpression: "price_usd > 0"

rowFilter takes a name and a rowExpression, both required. expression is not a field and fails validation. Optional columnNames, excludedColumnNames or allColumns set which columns the filter shows; with none of them every column is visible.

Column-limited grants

A grant limits columns in one of two ways. Both emit SELECT only.

grants:
  # An allow-list: the principal reads these columns.
  - principal: "arn:aws:iam::123456789012:role/data-analyst"
    permissions: [SELECT]
    columns: [price_timestamp, price_usd]

  # An exclusion list: the principal reads every column except these.
  - principal: "arn:aws:iam::123456789012:role/intern"
    permissions: [SELECT]
    excludedColumns: [market_cap_usd]

columns becomes table_with_columns.column_names. excludedColumns becomes wildcard = true with excluded_column_names; without the wildcard the provider's tofu plan fails, which was the module Forge wrote before 0.16.6.

The emitter refuses these, with the error kind lakeformation-grant-columns:

These refusals come from fluid generate iac and fluid apply. As of 0.18.1, fluid validate does not run them, so a contract with a misspelt excludedColumns entry passes fluid validate --strict and fails when the module is emitted:

CLI command error
❌ unsupported_binding  [ERR_UNSUPPORTED_BINDING]
  kind: lakeformation-grant-columns
  error: governance.lakeFormation.grants[1] names the columns ['market_cap'],
which the expose's contract.schema does not declare, so the Glue table has no
such column. A misspelt excludedColumns entry would leave the real column
readable.

Review with tofu plan, not only tofu validate. tofu validate accepts an excluded_column_names block that tofu plan rejects, because the table name is not known until plan time.

Column restrictions

policy.authz.columnRestrictions is the cloud-neutral way to say who may not read which columns. On AWS each Lake Formation SELECT or ALL grant excludes the restricted columns its principal may not read. The example at the top of the page produces, for the intern grant:

"table_with_columns": [
  {
    "excluded_column_names": ["market_cap_usd", "volume_24h_usd"],
    "wildcard": true
  }
]

and a plain table grant for data-analyst. The rules:

• deny hides the columns from the principal. allow makes the columns readable only by the principals an allow names. A deny beats an allow.

• A restriction never grants access. The readers are the principals with a Lake Formation SELECT or ALL grant on the binding. An allowed principal with no read grant is logged, not added.

• A restriction needs Lake Formation grants. On an aws binding with none, fluid validate, fluid generate iac and fluid apply refuse it, because nothing would enforce it:

   1. exposes.policy.authz.columnRestrictions restricts columns, but this aws
  binding declares no governance.lakeFormation.grants, the only column-level
  control the AWS emitter writes. Nothing would enforce the restriction. Add
  governance.lakeFormation to the aws overlay's binding: registerLocation: true
  and a grant for each reader; forge-cli then writes the restricted columns into
  each grant's excluded columns.
  

  The same error kind, column-restriction-unenforceable, covers a binding whose format has no Glue table. As of 0.18.1, fluid plan writes a plan for a contract whose aws binding has no Lake Formation grants without an error; the refusal comes at validate, generate and apply.

• A grant that already carries excludedColumns must agree with the restrictions, or the emit is refused with column-restriction-conflict. A grant's hand-written columns must not include a restricted column the principal may not read. Either way, remove the hand-written list and let the restrictions write it.

• A restriction on a column the schema does not declare is refused, and so is one that leaves a read grant with no column to read (lakeformation-grant-columns).

• A restriction's principal must be an IAM principal ARN. A logical principal such as role:intern fails validation unless binding.principals maps it:

  exposes.policy.authz.columnRestrictions: principal 'role:intern' resolves to
  'role:intern', which is not an IAM principal ARN
  (arn:aws:iam::<account>:role/<name> or :user/<name>). Map it in
  binding.principals to the IAM role or user ARN it stands for.
  

A table that still grants IAM_ALLOWED_PRINCIPALS, which Lake Formation's default settings give new tables, lets any IAM principal with S3 access read every column whatever the grants say. fluid verify holds IAM_ALLOWED_PRINCIPALS to every restricted column and fails when it can read one; see Verify on AWS.

Logical principals and binding.principals

0.7.6. The base contract can name principals as the business knows them, and each environment's binding maps them to the IAM identities they are on that cloud:

fluidVersion: "0.7.6"
exposes:
  - exposeId: bitcoin_prices_table
    binding:
      platform: aws
      principals:
        group:analysts@example.com: "arn:aws:iam::123456789012:role/data-analyst"
        group:interns@example.com: "arn:aws:iam::123456789012:role/intern"
    policy:
      authz:
        columnRestrictions:
          - principal: group:interns@example.com
            columns: [market_cap_usd, volume_24h_usd]
            access: deny

A value is one ARN, a list of them, or [] for no identity on this cloud. With binding.principals present, every principal the contract names in accessPolicy.grants and columnRestrictions for the expose must be mapped; an unmapped one is refused. lakeFormation.grants[].principal stays an ARN.

Bucket policy

0.7.6. A grant with an arn: principal can also get a statement in an aws_s3_bucket_policy (s3:ListBucket and s3:GetBucketLocation on the bucket, s3:GetObject under location.path). That statement lets the principal read the objects straight from S3, which skips Lake Formation's column filters, row filters and revocations. bucketPolicy chooses who gets one:

governance:
  lakeFormation:
    registerLocation: true
    bucketPolicy: none

The emitted bucket policy is authoritative: it replaces every other statement on the bucket. When no grantee is left in cross-account mode, the resource gets a plan-time count of zero, so the bucket's own policy is untouched. Two exposes with grants on one bucket must use the same bucketPolicy value, and their statements are merged into the bucket's one policy.

bucketPolicy exists only in schema 0.7.6. A 0.7.5 contract gets the cross-account behaviour and cannot opt back.

Masking at landing

policy.privacy.masking is applied by the DuckDB acquisition runner, inside the COPY and after the quality gates, so the S3 object and the dead-letter queue hold treated values, not cleartext. It is not applied by other engines (a python build, for example), and no AWS resource is emitted for it.

exposes:
  - exposeId: subscribers
    policy:
      privacy:
        masking:
          - column: msisdn
            strategy: hash
    contract:
      schema:
        - name: msisdn
          type: string

The build is refused, before any rows move, for k_anonymity (a property of a whole table, not of one value), for parameters the strategy does not take, for a literal salt or key in params, for an unset or too-short secret, and for a column whose declared type is not a string, because a treated column always lands as a string. Set the secret in the environment that runs the build, never in the contract. Quality gates run on the source values, before the rewrite.

fluid verify fails (CRITICAL) a masked column whose landed values lack the strategy's shape, so a table that declares masking but holds cleartext in that column, loaded by a python build or by hand, fails the check. See Verify on AWS. For the strategies in full, see masking at landing and tag PII. What the platform enforces across clouds is on Governance.

Encryption at rest and retention

0.7.6. Both fields need fluidVersion: "0.7.6" and apply to a bucket this product owns.

fluidVersion: "0.7.6"
exposes:
  - exposeId: bitcoin_prices_table
    lifecycle:
      retention: P90D
      expire: true
    binding:
      platform: aws
      format: parquet
      location:
        database: crypto_data
        table: bitcoin_prices
        bucket: "{{ env.S3_BUCKET }}"
        path: data/bitcoin/prices/
        region: eu-west-1
      encryption:
        kms: product

Added to the example's module: aws_kms_key, aws_kms_alias, aws_s3_bucket_server_side_encryption_configuration and aws_s3_bucket_lifecycle_configuration.

Encryption. binding.encryption.kms takes:

The key policy of a product key lets the account's IAM policies decide who uses it. When the binding has governance.lakeFormation, it also lets the Lake Formation service-linked role use the key, so a principal querying through Athena with credentials Lake Formation vends needs no KMS permission of its own. A grantee that the bucket policy lets read objects directly gets kms:Decrypt, through S3 only. The identity that writes the data needs kms:GenerateDataKey and kms:Decrypt from IAM. A key on a shared (pool) bucket is the pool owner's: product is refused there, and a named key is only checked.

Retention. lifecycle.retention is an ISO-8601 duration. It stays a declaration until expire: true turns it into a rule:

• One rule per expose in the bucket's lifecycle configuration, filtered to the binding's prefix (location.path, else <database>/<table>/). A binding with neither is refused, so no rule expires a whole bucket.
• It expires current objects retention after they are written. Years count as 365 days and months as 30; a part of a day rounds up to a whole one.
• It removes a noncurrent version one day after it stops being current and aborts an incomplete multipart upload after seven days, or after the retention period if that is shorter.
• A rule for fluid verify's own result prefix, .fluid/athena-results/, is added at the shortest retention declared on the bucket.
• Two exposes whose prefixes contain each other, with different schedules, are refused (retention-overlap).

The lifecycle configuration is authoritative for the whole bucket: S3 keeps one, so rules Forge did not write are replaced. Forge therefore writes it only on a bucket the product owns. On a shared pool bucket nothing is written, the pool's owner holds the rule, and fluid verify checks it. An existing bucket's rules and default encryption are imported with the bucket, so the plan shows them changing in place.

Removing expire, or the key, removes a resource that holds policy. fluid apply refuses that plan without --allow-data-loss; removing a Lake Formation grant is a revocation and is not gated.

exposes[].lifecycle.retention is separate from the top-level retention: block that sweeps run state and logs; see fluid retention.

accessPolicy on AWS

accessPolicy.grants is the cloud-neutral statement of who may read. It is a top-level block of the contract, not a field of an expose; under exposes[] it is a schema error. The AWS emitter does not write it: on AWS, access is the binding's governance.lakeFormation.grants. A contract with accessPolicy.grants and an aws binding that has no Lake Formation grants gets a validate warning, and --strict turns it into exit 1:

⚠️  1 warning(s)

Validation Warnings:
====================
 1. accessPolicy.grants are not enforced on aws binding(s) bitcoin_prices_table:
the AWS emitter does not write accessPolicy, and these bindings declare no
governance.lakeFormation.grants, the AWS form of who may read the table. Add
them to the aws overlay's binding (column restrictions then narrow them).

A pipeline that runs fluid validate --strict fails on it; stage 2 of the 11-stage pipeline does. Without --strict it is a warning. Add lakeFormation grants to the aws binding, for example in the aws overlay of a contract that also targets GCP (see per-environment overlays). The warning is raised only when no grant exists; a binding with Lake Formation grants does not trigger it.

fluid policy-compile still reads accessPolicy.grants and writes IAM bindings, which you can use as a starting point for IAM policies. Nothing applies them:

fluid policy-compile contract.fluid.yaml --out runtime/policy/bindings.json

{
  "bindings": [
    {
      "provider": "aws",
      "resource_type": "s3.bucket",
      "resource_id": "{{ env.S3_BUCKET }}",
      "bucket": "{{ env.S3_BUCKET }}",
      "region": "eu-west-1",
      "principal": "role:data-analyst",
      "actions": ["s3:GetObject", "s3:ListBucket"]
    },
    {
      "provider": "aws",
      "resource_type": "glue.table",
      "resource_id": "crypto_data.bitcoin_prices",
      "database": "crypto_data",
      "table": "bitcoin_prices",
      "region": "eu-west-1",
      "principal": "role:data-analyst",
      "actions": [
        "glue:GetTable",
        "glue:GetDatabase",
        "athena:StartQueryExecution",
        "athena:GetQueryResults"
      ]
    },
    ...
  ],
  "warnings": []
}

Each principal gets one entry for the bucket and one for the Glue table. The permissions map to two action sets: any of write, insert, update or delete selects the write set (s3:PutObject, s3:DeleteObject, s3:GetObject, s3:ListBucket, and glue:CreateTable, glue:UpdateTable, glue:DeleteTable); every other permission selects the read set shown above. Placeholders in the bucket stay literal, and the principals are written as the contract names them.

fluid policy-apply enforces nothing on AWS. As of 0.18.1 the aws provider has no standalone policy applier, and the command exits 0:

$ fluid policy-apply runtime/policy/bindings.json --mode enforce
...
⚠️  No policy bindings were enforced — the 'aws' provider has no standalone
policy applier. For cloud providers, IAM/GRANT, masking and row-access policies
are emitted and applied during `fluid apply` (stage 7), so `fluid policy-apply`
is a no-op for this provider.

Data Sovereignty

The sovereignty block enforces region restrictions before any infrastructure is deployed:

sovereignty:
  jurisdiction: "EU"
  dataResidency: true        # a boolean — "must this data stay in the jurisdiction?"
  allowedRegions: [eu-central-1, eu-west-1]
  deniedRegions: [us-east-1, us-west-2]
  crossBorderTransfer: false
  regulatoryFramework: [GDPR]
  enforcementMode: advisory  # or strict (blocks deployment)

Which key holds what (since 0.15.0)

fluid generate iac also stopped writing a module after the provider refused the contract (since 0.15.0). The native planner ran inside a best-effort except Exception that logged at DEBUG and returned no actions, and that check is the only place AWS enforces jurisdiction and residency — so a contract bound outside its declared jurisdiction logged the violation and then emitted main.tf.json anyway, exit 0. An EU-jurisdiction contract bound to us-east-1 now exits 1 and writes no module. The provider's handler also caught only the jurisdiction error and not its residency sibling, so every residency refusal skipped the sovereignty_violation audit event — the one refusal an operator most needs in the log was the one missing from it.

Sovereignty tags

fluid:data_jurisdiction, fluid:data_residency and fluid:allowed_regions are attached to the emitted resources. Get them wrong and detective controls downstream go with them — an AWS Config rule or a tag-based SCP keys on exactly these tags. The same dataResidency confusion reached them: joining the boolean raised, and joining the dict shape emitted its keys, so one tag read fluid:allowed_regions = "allowedRegions". Fixed in 0.15.0; fluid:data_residency now reads enforced or regions-pinned, and an omitted dataResidency tags as the schema says it behaves (default: true).

Region → jurisdiction (since 0.15.0)

AWS regions now resolve through botocore's shipped endpoints.json — the partitions it lists, GovCloud and the EU Sovereign Cloud (eusc-de-east-1) included — instead of a hand-kept table, so a new AWS region arrives by upgrading boto3 rather than by waiting for a FLUID release. Verdicts change on contracts nobody edited:eu-west-2 (London) resolves to UK rather than EU, and ap-southeast-1 (Singapore) and ap-northeast-2 (Seoul) resolve to SG and KR rather than a pass-anything Global. An EU-pinned contract bound to any of the three passed 0.14.1 silently and now fails; a UK-pinned contract bound to eu-west-2 stops being a false positive. The AWS provider had kept a second table that disagreed with the canonical engine and now delegates to the one table — full detail in Sovereignty enforcement modes, under "The region → jurisdiction table is derived".

Verify on AWS

The checks, the Athena flags, the result-location order and the IAM list are in fluid verify. fluid verify checks a binding with platform: aws, a format Athena can read through a Hive SerDe (parquet, as of 0.18.1), and location.database, location.table and location.bucket. It runs in the binding's region and checks the live account against the contract. Another binding is reported as having no verifier, which never fails the run.

fluid verify contract.fluid.yaml --env dev --strict --out runtime/verify-report.json

CRITICAL fails --strict. INFO fails only with --fail-on-warning. An error fails the command with or without --strict.

The column-restrictions check lists the permissions the caller can see, so it must run as a Lake Formation administrator. When the contract's own grants are missing from the listing, it reports an error, not a pass, because a partial listing cannot be trusted.

Which run the count is held to. A build writes a run record under .fluid/runs/<contract id>/<build id>/runs/<run id>.json in the contract directory. A CI pipeline that runs verify in a separate stage must carry that directory over. Verify takes the newest run that landed in this table and compares by the mode that run recorded:

Options.

The flag wins over the variable. A workgroup that enforces its own output location always wins, and a location inside the table's own prefix is refused. Each run leaves Athena's small result objects at the location used; the lifecycle rule fluid apply writes for the default prefix expires them when retention is declared, and otherwise you expire them with a rule of your own.

IAM the verifying identity needs:

• glue:GetTable and glue:GetDatabase, plus glue:GetPartitions for a partitioned table.
• athena:GetWorkGroup (optional), athena:StartQueryExecution, athena:GetQueryExecution, athena:GetQueryResults and athena:StopQueryExecution.
• s3:GetObject on the table's prefix, and read and write on the results prefix.
• s3:ListBucket, s3:GetBucketLocation and s3:ListBucketMultipartUploads on the bucket.
• With a registered location: lakeformation:GetDataAccess and a Lake Formation SELECT grant on the table.
• With declared retention: s3:GetLifecycleConfiguration. With a declared key: kms:DescribeKey, and kms:GenerateDataKey and kms:Decrypt on it for the result file.
• For the column check: lakeformation:ListPermissions, as a Lake Formation administrator.

Upgrading an existing AWS contract

The release notes for 0.16.0 and 0.17.0 and the upgrade guide have the full lists.

• 0.16.2: a Parquet Glue table gets Athena's storage classes on re-apply. The module pins the binding's region, and fluid apply refuses a move between regions; see Region.
• 0.16.3: the default bucketPolicy no longer gives same-account grantees a direct S3 read; see Bucket policy. governance.lakeFormation.admins was always authoritative, and the bundled schemas now say so.
• 0.16.5: DuckDB builds treat masked columns at landing, and fluid verify fails a cleartext one. AWS_ENDPOINT_URL[_S3] reaches DuckDB.
• 0.17.0: a column restriction on an aws binding with no Lake Formation grants is refused, and accessPolicy.grants on an aws binding with no Lake Formation grants warns; see Column restrictions and accessPolicy on AWS.
• 0.18.0: contract SQL runs in a DuckDB sandbox and $ref stays inside the contract's directory tree; see Where a build lands data.

How far this has been exercised

The forge-cli release notes record how each part was checked. The parquet SerDe fix was measured against a real AWS account, as was landing a build's rows in S3 through the Glue table (0.16.0). The region pin and the region-move guard were exercised against an emulator. The 0.16.5 retention, encryption, masking and endpoint changes ran against moto and DuckDB writing to a moto S3 server; moto enforces neither key policies nor Lake Formation.

The column-restriction grants and fluid verify's Lake Formation check were tested against moto's stored grants. A grant of the shape 0.17.0 emits (excluded columns beside wildcard) was applied and enforced on a real account from 0.16.6, written by hand in the overlay. What is not proven is the Lake Formation half against a real account as 0.17.0 derives it from columnRestrictions. Confirm that behaviour on your own account with fluid verify before you rely on it. The GCP half was measured against real Google Cloud on 4 October 2026; see Governance parity.

CI/CD pipeline

The AWS example uses the same Jenkinsfile as GCP and Snowflake, the Universal Pipeline. The stages that touch AWS:

See the 11-stage pipeline for what each stage does.

See also

• Universal Pipeline — The Jenkinsfile the providers share
• Snowflake Provider — Snowflake Data Cloud integration
• GCP Provider — Google Cloud Platform integration
• Governance parity — one contract, AWS and GCP
• One contract, two clouds — deploy the same product to both
• fluid verify and fluid apply — Command reference
• DuckDB sandbox for contract SQL — What a build's SQL can read
• CLI Reference — Full command documentation