Contract field reference, schema 0.7.5
Status: stable. Schema 0.7.5 is the newest stable schema bundled with the CLI. The newer 0.7.6 schema is a preview; see the 0.7.6 preview delta.
Generated from fluid_build/schemas/fluid-schema-0.7.5.json in the data-product-forge 0.18.1 package (sha256 763fb786cd73bc4f). It is the schema fluid validate checks a contract against, rendered without edits. To regenerate: python scripts/gen_contract_reference.py.
Each table row is one field, addressed by its path from the contract root: . descends into an object, [] marks the items of a list, and <key> stands for any key of a map. Required means required inside its parent object; the parent can itself be optional. Allowed values lists the enum, pattern and range the schema enforces. An object typed plainly object rejects keys the schema does not list; object (open) accepts extra keys and object (free-form) has no listed keys. A row that begins "Only when ..." exists only under that condition. See how to read this reference.
Top-level fields
| Field | Type | Required |
|---|---|---|
fluidVersion | string | yes |
kind | string | yes |
id | string | yes |
name | string | yes |
metadata | object | yes |
exposes | array of object | yes |
extensions | object (free-form) | no |
description | string | no |
domain | string | no |
tags | array of string | no |
labels | object (map) | no |
consumes | array of object | no |
builds | array of object | no |
build | object | no |
lineage | object | no |
schemaEvolution | object | no |
machineLearning | object | no |
environments | object (map) | no |
lifecycle | object | no |
docs | object | no |
sovereignty | object | no |
accessPolicy | object | no |
orchestration | object (open) | no |
retention | object | no |
governance | object | no |
Single-value fields
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
fluidVersion | string | yes | one of 0.7.3, 0.7.4, 0.7.5matches ^\d+\.\d+(\.\d+)?$ | Contract schema version. Accepts '0.7.3' (source-aligned data products + acquisition pattern + ingestion engines) or '0.7.4' (adds the MCP output-port gateway + runtime agentPolicy enforcement). 0.7.4 is fully backward-compatible with 0.7.3 — every 0.7.3 contract validates as 0.7.4 unchanged. | |
kind | string | yes | one of DataProduct, MLPipeline | Kind of contract. 'MLPipeline' added for basic ML support. | |
id | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Stable product identifier, e.g. 'gold.hr.employee_360_v1'. | |
name | string | yes | min length 1 | ||
extensions | object (free-form) | no | Vendor / plugin-namespaced configuration. Each plugin claims a single sub-key (e.g. customScaffold, customCatalog) and validates its own schema there via the fluid_build.extension_validators entry-point. | ||
description | string | no | |||
domain | string | no | Business domain/mesh domain (e.g., 'HR', 'Finance'). | ||
tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Product-level tags for categorization and discovery. | |
build | object | no | Single build configuration (legacy). Use 'builds' array for multi-modal. Same fields as builds[]: read that section with this path as the prefix. |
metadata
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
metadata | object | yes | |||
metadata.provenance | object (open) | no | Generation envelope injected by fluid forge / fluid init — records how, when and by which tool/command the contract was generated. Machine-written (not user-authored); additive metadata that does not affect the data product's semantic contract. | ||
metadata.provenance.schema_version | integer | no | Envelope schema version. | ||
metadata.provenance.kind | string | no | Envelope discriminator, e.g. 'ContractMetadata'. | ||
metadata.provenance.generated_at | string | no | format date-time | ISO 8601 UTC timestamp of generation. | |
metadata.provenance.generated_by | object (open) | no | Tool, version and command that generated the contract. | ||
metadata.provenance.generated_by.tool | string | no | |||
metadata.provenance.generated_by.version | string | no | |||
metadata.provenance.generated_by.command | string | no | |||
metadata.layer | string | no | Data layer label (e.g., Bronze/Silver/Gold). Free-form. When set together with productType, the two MUST be consistent under the canonical mapping (Bronze↔SDP, Silver↔ADP, Gold↔CDP). | ||
metadata.productType | string | no | one of SDP, ADP, CDP | NEW in v0.7.3: Data Mesh data-product type. SDP = Source-Aligned Data Product (raw ingestion from external systems; corresponds to Bronze). ADP = Aggregated Data Product (cross-source joined / domain-modelled; corresponds to Silver). CDP = Consumption-Aligned Data Product (analytics- or product-shaped; corresponds to Gold). Either metadata.layer or metadata.productType is sufficient — the validator infers the missing one. When both are present they MUST agree. | |
metadata.classification | string | no | one of public, internal, confidential, restricted | NEW in v0.7.3: Data classification label propagated to catalog + access policy enforcement. | |
metadata.experimental | array of string | no | unique items | NEW in v0.7.3: Feature gates the contract opts into (e.g., 'acquisition' while Bronze acquisition is in beta). | |
metadata.owner | object | yes | |||
metadata.owner.team | string | no | |||
metadata.owner.email | string | no | format email | ||
metadata.owner.slack | string | no | |||
metadata.owner.oncall | string | no | Oncall rotation or contact for operational issues. | ||
metadata.createdAt | string | no | format date-time | ||
metadata.businessContext | object | no | Business alignment information for data mesh organization. | ||
metadata.businessContext.domain | string | no | Business domain this product belongs to. | ||
metadata.businessContext.subdomain | string | no | Business subdomain for more granular organization. | ||
metadata.businessContext.businessCapability | string | no | Business capability this product supports. | ||
metadata.businessContext.valueStream | string | no | Value stream this product participates in. | ||
metadata.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Product-level tags on metadata for discovery and categorization. |
exposes
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes | array of object | yes | min items 1 | ||
exposes[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Stable interface handle (used by consumers & build.outputs). | |
exposes[].title | string | no | |||
exposes[].description | string | no | Detailed description of the exposed resource. | ||
exposes[].version | string | no | matches ^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$ | Semantic version (semver.org). | |
exposes[].kind | string | yes | one of table, view, api, file, stream, topic, feature_store, model, vector, graph, time_series, other | ||
exposes[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Expose-level tags for categorization and routing. |
exposes[].labels
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].labels | object (map) | no | Expose-level labels for metadata and automation. | ||
exposes[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].contract
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].contract | object | yes | at least one of schema or openapiRef is required | ||
exposes[].contract.schemaSignature | string | no | matches ^sha256:[0-9a-fA-F]{64}$ | Hash of canonical schema/OpenAPI. Used by contract-tests gates. | |
exposes[].contract.schemaPolicy | string | no | one of strict, discover_and_freeze, evolve_safe, evolve_all | "strict" | NEW in v0.7.3: Schema evolution policy. strict = fail on any schema change; discover_and_freeze = capture from source on first run, fail thereafter; evolve_safe = accept additive changes; evolve_all = accept all changes (cast on type narrow, route failures to DLQ). |
exposes[].contract.schema | array of object | no | Tabular schema (for table/view/file). | ||
exposes[].contract.schema[].name | string | yes | min length 1 | ||
exposes[].contract.schema[].type | string | yes | one of 79 valuesstring text varchar varchar2 nvarchar char nchar character clob int integer int2 int4 int8 int16 int32 int64 tinyint smallint mediumint bigint long longint serial bigserial float float4 float8 float32 float64 double real decimal dec numeric number bignumeric money boolean bool bit date time datetime datetime2 smalldatetime timestamp timestamptz timestamp_tz timestamp_ntz timestamp_ltz timestampntz interval year variant object array struct map record row json jsonb super binary varbinary bytes blob bytea raw geography geometry geom point uuid uniqueidentifier guid enum hllor matches a 684-character pattern(?i)^\s*(string|text|varchar|varchar2|nvarchar|char|nchar|character|clob|int|integer|int2|int4|int8|int16|int32|int64|tinyint|smallint|mediumint|bigint|long|longint|serial|bigserial|float|float4|float8|float32|float64|double|real|double\s+precision|decimal|dec|numeric|number|bignumeric|money|boolean|bool|bit|date|time|datetime|datetime2|smalldatetime|timestamp|timestamptz|timestamp_tz|timestamp_ntz|timestamp_ltz|timestampntz|timestamp\s+with(out)?\s+time\s+zone|interval|year|variant|object|array|struct|map|record|row|json|jsonb|super|binary|varbinary|bytes|blob|bytea|raw|geography|geometry|geom|point|uuid|uniqueidentifier|guid|enum|hll)\s*((\s*[0-9A-Za-z_,\s'"-]\s))?\s*$ | Column data type. Accepts a canonical FLUID/SQL type name (case-insensitive) or a parameterized form such as decimal(18,4) or varchar(255). The enum lists the canonical bare types; the pattern additionally permits case variants and parameter suffixes. Intentionally generous — it rejects obvious garbage (e.g. MYSTERY_TYPE) while admitting the cross-dialect superset. | |
exposes[].contract.schema[].required | boolean | no | |||
exposes[].contract.schema[].description | string | no | |||
exposes[].contract.schema[].sensitivity | string | no | one of 12 valuesnone internal confidential restricted pii phi cleartext treated anonymized pseudonymized tokenized encrypted | Field-level sensitivity classification for improved policy ergonomics. | |
exposes[].contract.schema[].semanticType | string | no | Semantic meaning (e.g., 'email', 'phone', 'identifier'). | ||
exposes[].contract.schema[].businessName | string | no | Business-friendly name for this field. | ||
exposes[].contract.schema[].businessDefinition | string | no | Business definition and meaning of this field. | ||
exposes[].contract.schema[].validationRules | array of object | no | Field-level validation constraints. | ||
exposes[].contract.schema[].validationRules[].type | string | no | one of range, regex, enum, custom | ||
exposes[].contract.schema[].validationRules[].constraint | string | no | |||
exposes[].contract.schema[].validationRules[].message | string | no | |||
exposes[].contract.schema[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Field-level tags for categorization and data governance. | |
exposes[].contract.schema[].labels | object (map) | no | Field-level labels for metadata and automation. | ||
exposes[].contract.schema[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].contract.openapiRef | string | no | format uri | OpenAPI document URL (for APIs). | |
exposes[].contract.guarantees | object | no | |||
exposes[].contract.guarantees.compatibility | string | no | one of none, patch, minor, backward, forward, strict | Compatibility promise for non-breaking evolution. | |
exposes[].contract.guarantees.evolution | array of string | no | each item: one of 6 valuesadditive_columns additive_fields additive_endpoints nullable_to_required_disallowed field_removal_with_notice type_widening_allowed | ||
exposes[].contract.dq | object | no | Enhanced data quality rules including anomaly detection. | ||
exposes[].contract.dq.rules | array of object | no | |||
exposes[].contract.dq.rules[].id | string | yes | |||
exposes[].contract.dq.rules[].type | string | yes | one of freshness, completeness, uniqueness, valid_values, accuracy, schema, anomaly_detection, drift_detection | ||
exposes[].contract.dq.rules[].selector | string | no | |||
exposes[].contract.dq.rules[].threshold | number | no | |||
exposes[].contract.dq.rules[].operator | string | no | one of >=, >, <=, <, ==, != | ||
exposes[].contract.dq.rules[].window | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].contract.dq.rules[].severity | string | yes | one of info, warn, error, critical | ||
exposes[].contract.dq.rules[].description | string | no | |||
exposes[].contract.dq.rules[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | DQ rule tags for categorization and automation. | |
exposes[].contract.dq.rules[].labels | object (map) | no | DQ rule labels for metadata and routing. | ||
exposes[].contract.dq.rules[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].contract.dq.monitoring | object | no | |||
exposes[].contract.dq.monitoring.enabled | boolean | no | true | ||
exposes[].contract.dq.monitoring.window | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].contract.dq.monitoring.owner | string | no | |||
exposes[].contract.dq.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Data quality specification tags. | |
exposes[].contract.dq.labels | object (map) | no | Data quality specification labels. | ||
exposes[].contract.dq.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].contract.quality | array of object | no | Simplified SQL-expression data quality rules (alternate form of dq.rules for inline assertions). | ||
exposes[].contract.quality[].rule | string | yes | Rule identifier. | ||
exposes[].contract.quality[].expression | string | yes | SQL or boolean expression that must evaluate to true. | ||
exposes[].contract.quality[].severity | string | yes | one of error, warning, info | Severity when the rule fails. | |
exposes[].contract.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Contract-level tags for data quality and governance. | |
exposes[].contract.labels | object (map) | no | Contract-level labels for automation and metadata. | ||
exposes[].contract.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].qos
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].qos | object | no | |||
exposes[].qos.availability | string | no | matches ^(100(\.0+)?|\d{2}(\.\d+)?|\d{1}\d(\.\d+)?)%$ | Availability percentage (e.g., 99.9%). | |
exposes[].qos.freshnessSLO | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].qos.dataLossSLO | string | no | e.g., '0 rows' | ||
exposes[].qos.latencyP95 | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].qos.completenessTarget | number | no | min 0 max 1 | Target completeness ratio (0.0 to 1.0). | |
exposes[].qos.errorBudget | number | no | min 0 max 1 | Acceptable error rate (0.0 to 1.0). | |
exposes[].qos.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | QoS-level tags for SLA management. | |
exposes[].qos.labels | object (map) | no | QoS-level labels for automation and reporting. | ||
exposes[].qos.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].policy
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].policy | object | no | |||
exposes[].policy.authn | string | no | one of oidc, oauth2, api_key, none, custom, iam, jwt | ||
exposes[].policy.authz | object | no | |||
exposes[].policy.authz.readers | array of string | no | |||
exposes[].policy.authz.writers | array of string | no | |||
exposes[].policy.authz.columnRestrictions | array of object | no | Column-level access control. | ||
exposes[].policy.authz.columnRestrictions[].principal | string | no | |||
exposes[].policy.authz.columnRestrictions[].columns | array of string | no | |||
exposes[].policy.authz.columnRestrictions[].access | string | no | one of allow, deny | ||
exposes[].policy.authz.columnRestrictions[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Access control rule tags. | |
exposes[].policy.authz.columnRestrictions[].labels | object (map) | no | Access control rule labels. | ||
exposes[].policy.authz.columnRestrictions[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.authz.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Authorization policy tags. | |
exposes[].policy.authz.labels | object (map) | no | Authorization policy labels. | ||
exposes[].policy.authz.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.privacy | object | no | |||
exposes[].policy.privacy.masking | array of object | no | |||
exposes[].policy.privacy.masking[].column | string | yes | |||
exposes[].policy.privacy.masking[].strategy | string | yes | one of mask, hash, tokenize, encrypt, k_anonymity | ||
exposes[].policy.privacy.masking[].params | object (free-form) | no | |||
exposes[].policy.privacy.masking[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Privacy masking rule tags. | |
exposes[].policy.privacy.masking[].labels | object (map) | no | Privacy masking rule labels. | ||
exposes[].policy.privacy.masking[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.privacy.rowLevelPolicy | object | no | |||
exposes[].policy.privacy.rowLevelPolicy.expression | string | yes | Provider-specific predicate | ||
exposes[].policy.privacy.rowLevelPolicy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Row-level policy tags. | |
exposes[].policy.privacy.rowLevelPolicy.labels | object (map) | no | Row-level policy labels. | ||
exposes[].policy.privacy.rowLevelPolicy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.privacy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Privacy policy tags. | |
exposes[].policy.privacy.labels | object (map) | no | Privacy policy labels. | ||
exposes[].policy.privacy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.classification | string | no | one of Public, Internal, Confidential, Restricted | ||
exposes[].policy.agentPolicy | object | no | NEW in v0.7.1: AI/LLM usage governance - controls which AI models can consume this data and for what purposes. | ||
exposes[].policy.agentPolicy.allowedModels | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-_.]*[a-z0-9]$|^[a-z0-9]$ | Whitelist of AI models permitted to consume this data. Examples: ['gpt-4', 'claude-3-opus', 'gemini-pro']. Empty array = no AI access. | |
exposes[].policy.agentPolicy.deniedModels | array of string | no | unique items | Blacklist of AI models explicitly prohibited from consuming this data. Takes precedence over allowedModels. | |
exposes[].policy.agentPolicy.maxTokensPerRequest | integer | no | min 1 | Maximum tokens per AI request. Prevents excessive data exposure in single queries. | |
exposes[].policy.agentPolicy.maxTokensPerDay | integer | no | min 1 | Daily token limit for AI consumption. Enforces usage quotas. | |
exposes[].policy.agentPolicy.allowedUseCases | array of string | no | unique items each item: one of 12 valuesinference reasoning analysis summarization classification embedding search qa code_generation fine_tuning training rag | Permitted AI use cases. Controls whether data can be used for training, inference, reasoning, etc. | |
exposes[].policy.agentPolicy.deniedUseCases | array of string | no | unique items each item: one of 12 valuesinference reasoning analysis summarization classification embedding search qa code_generation fine_tuning training rag | Prohibited AI use cases. Example: Allow reporting but deny model training. | |
exposes[].policy.agentPolicy.canReason | boolean | no | false | Whether AI agents can use this data for multi-step reasoning/chain-of-thought. False = simple retrieval only. | |
exposes[].policy.agentPolicy.canStore | boolean | no | false | Whether AI systems can cache/store this data. False = ephemeral processing only. | |
exposes[].policy.agentPolicy.retentionPolicy | object (open) | no | Data retention requirements for AI systems consuming this data. | ||
exposes[].policy.agentPolicy.retentionPolicy.maxRetentionDays | integer | no | min 0 | Maximum days AI system can retain this data. 0 = no retention allowed. | |
exposes[].policy.agentPolicy.retentionPolicy.requireDeletion | boolean | no | true | Whether AI system must delete data after use. | |
exposes[].policy.agentPolicy.auditRequired | boolean | no | true | Whether AI consumption must be logged/audited. | |
exposes[].policy.agentPolicy.purposeLimitation | string | no | Free-text description of allowed purposes. Example: 'Customer service chatbot only, not for marketing'. | ||
exposes[].policy.agentPolicy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Agent policy tags for categorization. | |
exposes[].policy.agentPolicy.labels | object (map) | no | Agent policy labels for automation. | ||
exposes[].policy.agentPolicy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].policy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Policy-level tags for governance. | |
exposes[].policy.labels | object (map) | no | Policy-level labels for automation. | ||
exposes[].policy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].binding
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].binding | object | yes | |||
exposes[].binding.platform | string | yes | one of gcp, aws, azure, snowflake, databricks, kafka, confluent, local, kubernetes, postgres, pgvector, other | Target platform. NEW in v0.7.5: 'pgvector' for the vector/embeddings output port (pgvector-on-PostgreSQL RAG target — emits the embeddings table + ANN index DDL for the ai-embeddable columns). NEW in v0.7.5: 'confluent' for the Confluent Cloud Tableflow managed Kafka→Iceberg emitter. NEW in v0.7.4: 'postgres' for the MCP output-port PostgreSQL driver (AWS Athena uses platform 'aws'). | |
exposes[].binding.format | string | yes | one of 22 valuesbigquery_table snowflake_table snowflake_view gcs_file s3_file http_api grpc_api pubsub_topic kafka_topic delta_table iceberg parquet csv json redshift_table redshift_serverless redshift_external_schema postgres_table athena_table glue_table pgvector_table other | Binding format. NEW in v0.7.5: 'pgvector_table' — a pgvector embeddings table target for the vector output port. NEW in v0.7.4: 'postgres_table' (PostgreSQL driver), 'athena_table' / 'glue_table' (AWS Athena driver) for the MCP output port. | |
exposes[].binding.icebergConfig | object | no | Apache Iceberg table format configuration | ||
exposes[].binding.icebergConfig.writeVersion | integer | no | one of 1, 2 | Iceberg table format version | |
exposes[].binding.icebergConfig.fileFormat | string | no | one of parquet, orc, avro | Underlying file format for data files | |
exposes[].binding.icebergConfig.partitionSpec | array of object | no | Partition transformation specifications | ||
exposes[].binding.icebergConfig.partitionSpec[].name | string | no | Partition field name | ||
exposes[].binding.icebergConfig.partitionSpec[].sourceColumn | string | yes | Source column for partitioning | ||
exposes[].binding.icebergConfig.partitionSpec[].transform | string | yes | one of identity, year, month, day, hour, bucket, truncate | Partition transform function | |
exposes[].binding.icebergConfig.partitionSpec[].numBuckets | integer | no | Number of buckets (for bucket transform) | ||
exposes[].binding.icebergConfig.partitionSpec[].width | integer | no | Truncate width (for truncate transform) | ||
exposes[].binding.icebergConfig.sortOrder | array of object | no | Sort order specifications for data files | ||
exposes[].binding.icebergConfig.sortOrder[].column | string | yes | Column to sort by | ||
exposes[].binding.icebergConfig.sortOrder[].direction | string | no | one of asc, desc | Sort direction | |
exposes[].binding.icebergConfig.sortOrder[].nullOrder | string | no | one of nulls-first, nulls-last | NULL value ordering | |
exposes[].binding.icebergConfig.properties | object (map) | no | Iceberg table properties | ||
exposes[].binding.icebergConfig.properties.<key> | string | no | |||
exposes[].binding.vectorConfig | object | no | NEW in v0.7.5: vector / embeddings output-port configuration (platform 'pgvector', format 'pgvector_table'). Consumed by fluid_build.output_ports.vector to emit the pgvector embeddings-table + ANN-index DDL and a RAG provenance manifest for the schema's ai-embeddable columns. | ||
exposes[].binding.vectorConfig.dimensions | integer | yes | min 1 max 16000 | Embedding vector width (e.g. 1536 for OpenAI text-embedding-3-small, 1024 for Cohere embed-english-light). Maps to the pgvector column type vector(<dimensions>). | |
exposes[].binding.vectorConfig.embeddingModel | string | no | Provenance: the embedding model that produces the vectors (e.g. 'text-embedding-3-small'). Recorded in the RAG manifest and the embeddings-table's embedding_model column so a retriever can verify model parity. | ||
exposes[].binding.vectorConfig.vectorType | string | no | one of vector, halfvec | "vector" | pgvector column type: 'vector' (float32, up to 2000 dims) or 'halfvec' (float16, half the storage, up to 4000 dims). |
exposes[].binding.vectorConfig.indexType | string | no | one of hnsw, ivfflat, none | "hnsw" | ANN index built on the embedding column. 'hnsw' (higher recall / lower query latency, buildable before data lands), 'ivfflat' (lower build cost, needs data first), or 'none' (no index — exact search only). |
exposes[].binding.vectorConfig.distanceMetric | string | no | one of cosine, l2, inner_product, l1 | "cosine" | Distance function the index optimises for → pgvector operator class: cosine→vector_cosine_ops, l2→vector_l2_ops, inner_product→vector_ip_ops, l1→vector_l1_ops. |
exposes[].binding.vectorConfig.table | string | no | Target embeddings table name. Defaults to '<exposeId>_embeddings' when omitted. | ||
exposes[].binding.vectorConfig.sourceKeyColumn | string | no | Provenance: the source primary-key column that ties each embedding row back to its origin row (written to the embeddings-table's source_key column). Enables update/delete propagation from the source product to the vector store. | ||
exposes[].binding.vectorConfig.hnsw | object | no | HNSW index tuning (used when indexType is 'hnsw'). | ||
exposes[].binding.vectorConfig.hnsw.m | integer | no | min 1 | Max bi-directional links per node (pgvector default 16). | |
exposes[].binding.vectorConfig.hnsw.efConstruction | integer | no | min 1 | Build-time dynamic candidate-list size (pgvector default 64). | |
exposes[].binding.vectorConfig.ivfflat | object | no | IVFFlat index tuning (used when indexType is 'ivfflat'). | ||
exposes[].binding.vectorConfig.ivfflat.lists | integer | no | min 1 | Number of inverted lists (pgvector guidance: rows/1000 for < 1M rows). | |
exposes[].binding.location | object | yes | Provider-native addressing with tightened structure for better interop. | ||
exposes[].binding.location.account | string | no | Cloud account ID | ||
exposes[].binding.location.project | string | no | GCP project ID | ||
exposes[].binding.location.dataset | string | no | BigQuery dataset or database name | ||
exposes[].binding.location.database | string | no | Database name | ||
exposes[].binding.location.schema | string | no | Schema name | ||
exposes[].binding.location.table | string | no | Table name | ||
exposes[].binding.location.bucket | string | no | Storage bucket name | ||
exposes[].binding.location.path | string | no | File or object path | ||
exposes[].binding.location.gateway | string | no | API gateway endpoint | ||
exposes[].binding.location.baseUrl | string | no | format uri | Base URL for API endpoints | |
exposes[].binding.location.topic | string | no | Pub/Sub or Kafka topic name | ||
exposes[].binding.location.subscription | string | no | Pub/Sub subscription name | ||
exposes[].binding.location.region | string | no | Cloud region | ||
exposes[].binding.location.zone | string | no | Cloud zone | ||
exposes[].binding.location.environment_id | string | no | NEW in v0.7.5: Confluent Cloud environment id (env-xxxxx) for the Tableflow emitter. | ||
exposes[].binding.location.kafka_cluster_id | string | no | NEW in v0.7.5: Confluent Cloud Kafka cluster id (lkc-xxxxx) the Tableflow topic belongs to. | ||
exposes[].binding.location.stream | string | no | NEW in v0.7.5: AWS Kinesis Data Stream name — maps to aws_kinesis_stream in the streaming IaC emitter. | ||
exposes[].binding.location.namespace | string | no | NEW in v0.7.5: Amazon Redshift Serverless namespace name — maps to aws_redshiftserverless_namespace. | ||
exposes[].binding.location.workgroup | string | no | NEW in v0.7.5: Amazon Redshift Serverless workgroup name — maps to aws_redshiftserverless_workgroup. | ||
exposes[].binding.location.iam_role_arn | string | no | NEW in v0.7.5: ARN of the IAM role attached to the Redshift Serverless namespace / used by the redshift-data CREATE EXTERNAL SCHEMA bridge. | ||
exposes[].binding.location.external_schema | string | no | NEW in v0.7.5: Redshift external (Spectrum) schema name created over a Glue database via the redshift-data CREATE EXTERNAL SCHEMA bridge. | ||
exposes[].binding.location.glue_database | string | no | NEW in v0.7.5: Glue Data Catalog database that backs the Redshift external schema. | ||
exposes[].binding.location.confluent_role_arn | string | no | NEW in v0.7.5: ARN of the pre-created AWS IAM role Confluent Tableflow assumes (byob_aws). Its trust policy is updated post-apply with the provider integration's external_id (two-phase IAM). | ||
exposes[].binding.location.catalog | string | no | NEW in v0.7.5: Iceberg catalog kind/profile selector — maps to iceberg.catalog.type (e.g. glue, rest, nessie, hive). REST-fronted catalogs (polaris / unity / snowflake open catalog) map to 'rest'. Consumed by providers/_iceberg_catalog.py and the streaming-sink validator. | ||
exposes[].binding.location.warehouse | string | no | NEW in v0.7.5: Iceberg warehouse — the canonical Iceberg REST 'warehouse' property (maps to iceberg.catalog.warehouse): an object-store URI (s3://|gs://|abfss://) for object-store catalogs, or the catalog/warehouse name for a REST catalog. | ||
exposes[].binding.location.uri | string | no | NEW in v0.7.5: REST Iceberg catalog endpoint URL — the canonical Iceberg REST 'uri' property (maps to iceberg.catalog.uri). | ||
exposes[].binding.location.partitionBy | array of string | no | NEW in v0.7.5: Iceberg partition columns — a FLUID abstraction over the Iceberg PARTITIONED BY / PartitionSpec (maps to iceberg.tables.default-partition-by). | ||
exposes[].binding.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Binding configuration tags for infrastructure. | |
exposes[].binding.labels | object (map) | no | Binding configuration labels for automation. | ||
exposes[].binding.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].binding.properties | object (free-form) | no | Provider-specific binding properties (e.g., cluster_by, comment for Snowflake). | ||
exposes[].binding.governance | object | no | Per-resource governance: registerLocation, principal grants, tag associations, row/column filters. Account-wide governance (admins, tag definitions) lives at the contract top-level governance block. | ||
exposes[].binding.governance.lakeFormation | object | no | Per-resource LF settings: location registration, principal grants, LF-tag associations, row/column filters. Applies only to formats backed by S3 + Glue catalog (iceberg/parquet/csv/json/avro/orc/delta). | ||
exposes[].binding.governance.lakeFormation.registerLocation | boolean | no | false | When true, emit aws_lakeformation_resource that registers the binding's S3 path with Lake Formation. Required before LF can manage access to objects under that path. | |
exposes[].binding.governance.lakeFormation.grants | array of object | no | Principal-based LF grants on this exposure's database/table. Each entry maps a principal to a permission set (and optional column projection). Emitted as aws_lakeformation_permissions resources. Field idiom adapted from ODCS v3 roles[] (principal + access-type), bound to IAM ARNs since LF is cloud-specific. | ||
exposes[].binding.governance.lakeFormation.grants[].principal | string | yes | matches ^arn:aws[a-z0-9-]*:iam:: | IAM principal ARN (role or user) receiving the grant. | |
exposes[].binding.governance.lakeFormation.grants[].permissions | array of string | yes | min items 1 unique items each item: one of ALL, SELECT, ALTER, DROP, DELETE, INSERT, DESCRIBE, CREATE_TABLE, CREATE_DATABASE, DATA_LOCATION_ACCESS | LF permissions granted. Valid values per the hashicorp/aws aws_lakeformation_permissions resource. | |
exposes[].binding.governance.lakeFormation.grants[].permissionsWithGrantOption | array of string | no | unique items | Subset of permissions the principal can re-grant. Maps to aws_lakeformation_permissions.permissions_with_grant_option. | |
exposes[].binding.governance.lakeFormation.grants[].columns | array of string | no | unique items | Optional column-level restriction. When set, the grant is emitted against aws_lakeformation_permissions.table_with_columns instead of .table, allowing the principal to read only the listed columns. | |
exposes[].binding.governance.lakeFormation.grants[].excludedColumns | array of string | no | unique items | Opposite of columns: every column EXCEPT these is granted. Maps to table_with_columns.excluded_column_names. Mutually exclusive with columns. | |
exposes[].binding.governance.lakeFormation.tags | object (map) | no | LF-tag (TBAC) associations applied to this table. Keys must reference tag definitions declared in the contract's top-level governance.lakeFormation.tagDefinitions. Emitted as aws_lakeformation_resource_lf_tags. | ||
exposes[].binding.governance.lakeFormation.tags.<key> | string | no | |||
exposes[].binding.governance.lakeFormation.rowFilter | object | no | Row-level (and optionally column-level) filter applied to all reads of this table via LF. Emitted as aws_lakeformation_data_cells_filter. The filter must then be granted to principals via grants[].permissions including SELECT — LF enforces both layers. | ||
exposes[].binding.governance.lakeFormation.rowFilter.name | string | yes | Filter name (must be unique per table). | ||
exposes[].binding.governance.lakeFormation.rowFilter.rowExpression | string | yes | PartiQL-flavoured row predicate. ALL lets every row through (use when only column filtering is intended). | ||
exposes[].binding.governance.lakeFormation.rowFilter.columnNames | array of string | no | unique items | Columns visible through the filter. Mutually exclusive with allColumns / excludedColumnNames. | |
exposes[].binding.governance.lakeFormation.rowFilter.excludedColumnNames | array of string | no | unique items | Columns hidden through the filter (everything else is visible). | |
exposes[].binding.governance.lakeFormation.rowFilter.allColumns | boolean | no | false | Convenience for the LF column-wildcard form (every column visible). When true, columnNames / excludedColumnNames must be omitted. |
exposes[].lifecycle
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].lifecycle | object | no | |||
exposes[].lifecycle.state | string | no | one of preview, active, deprecated, retired | Unified lifecycle state vocabulary. | |
exposes[].lifecycle.retention | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].lifecycle.deprecationPolicy | object | no | |||
exposes[].lifecycle.deprecationPolicy.noticePeriod | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
exposes[].lifecycle.deprecationPolicy.contact | string | no | |||
exposes[].lifecycle.deprecationPolicy.replacement | string | no | Reference to replacement data product or expose. | ||
exposes[].lifecycle.deprecationPolicy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Deprecation policy tags. | |
exposes[].lifecycle.deprecationPolicy.labels | object (map) | no | Deprecation policy labels. | ||
exposes[].lifecycle.deprecationPolicy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].lifecycle.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Lifecycle tags for automation. | |
exposes[].lifecycle.labels | object (map) | no | Lifecycle labels for governance. | ||
exposes[].lifecycle.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].observability
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].observability | object | no | |||
exposes[].observability.metrics | array of object | no | |||
exposes[].observability.metrics[].name | string | yes | |||
exposes[].observability.metrics[].source | string | yes | |||
exposes[].observability.metrics[].sli | string | no | Human/machine-readable SLI check description | ||
exposes[].observability.onBreach | array of object | no | |||
exposes[].observability.onBreach[].type | string | yes | one of slack, email, webhook, pagerduty, custom | ||
exposes[].observability.onBreach[].channel | string | no | |||
exposes[].observability.onBreach[].target | string | no | |||
exposes[].observability.onBreach[].url | string | no | format uri | ||
exposes[].observability.onBreach[].severity | string | no | one of info, warning, critical | ||
exposes[].observability.onBreach[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Alert configuration tags. | |
exposes[].observability.onBreach[].labels | object (map) | no | Alert configuration labels. | ||
exposes[].observability.onBreach[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].observability.defaultSLIs | object | no | Default SLI trio (freshness, completeness, latency) by kind. | ||
exposes[].observability.defaultSLIs.enabled | boolean | no | true | Enable auto-generation of default SLIs. | |
exposes[].observability.defaultSLIs.freshness | object | no | Default freshness SLI configuration. | ||
exposes[].observability.defaultSLIs.freshness.enabled | boolean | no | true | ||
exposes[].observability.defaultSLIs.freshness.threshold | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | Freshness threshold (auto-set by kind: table=PT6H, stream=PT1M, api=PT1S) | |
exposes[].observability.defaultSLIs.freshness.severity | string | no | one of info, warning, critical | "warning" | |
exposes[].observability.defaultSLIs.freshness.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Freshness SLI tags. | |
exposes[].observability.defaultSLIs.freshness.labels | object (map) | no | Freshness SLI labels. | ||
exposes[].observability.defaultSLIs.freshness.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].observability.defaultSLIs.completeness | object | no | Default completeness SLI configuration. | ||
exposes[].observability.defaultSLIs.completeness.enabled | boolean | no | true | ||
exposes[].observability.defaultSLIs.completeness.threshold | number | no | min 0 max 1 | 0.95 | Completeness threshold (auto-set by kind: table=0.95, stream=0.99) |
exposes[].observability.defaultSLIs.completeness.severity | string | no | one of info, warning, critical | "warning" | |
exposes[].observability.defaultSLIs.completeness.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Completeness SLI tags. | |
exposes[].observability.defaultSLIs.completeness.labels | object (map) | no | Completeness SLI labels. | ||
exposes[].observability.defaultSLIs.completeness.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].observability.defaultSLIs.latency | object | no | Default latency SLI configuration. | ||
exposes[].observability.defaultSLIs.latency.enabled | boolean | no | true | ||
exposes[].observability.defaultSLIs.latency.threshold | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | P95 latency threshold (auto-set by kind: api=PT200MS, table=PT5S, stream=PT100MS) | |
exposes[].observability.defaultSLIs.latency.percentile | number | no | min 0 max 100 | 95 | Latency percentile to measure. |
exposes[].observability.defaultSLIs.latency.severity | string | no | one of info, warning, critical | "warning" | |
exposes[].observability.defaultSLIs.latency.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Latency SLI tags. | |
exposes[].observability.defaultSLIs.latency.labels | object (map) | no | Latency SLI labels. | ||
exposes[].observability.defaultSLIs.latency.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].observability.defaultSLIs.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Default SLI configuration tags. | |
exposes[].observability.defaultSLIs.labels | object (map) | no | Default SLI configuration labels. | ||
exposes[].observability.defaultSLIs.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
exposes[].observability.alert | object | no | Alert dispatcher config consumed by the acquisition Alerter (DLQ overflow, schema-fingerprint changes, anomaly signals). | ||
exposes[].observability.alert.channels | array of object | no | |||
exposes[].observability.alert.channels[].kind | string | yes | one of log, file, webhook | ||
exposes[].observability.alert.channels[].path | string | no | For kind=file: NDJSON sink path (env-vars expanded). | ||
exposes[].observability.alert.channels[].url | string | no | format uri | For kind=webhook: Slack-compatible incoming webhook URL. Validated against the SSRF guard at construction time. | |
exposes[].observability.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Observability configuration tags. | |
exposes[].observability.labels | object (map) | no | Observability configuration labels. | ||
exposes[].observability.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].docs
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].docs | object | no | |||
exposes[].docs.homepage | string | no | format uri | ||
exposes[].docs.runbook | string | no | format uri | ||
exposes[].docs.dictionary | string | no | format uri | ||
exposes[].docs.changeLog | string | no | format uri | ||
exposes[].docs.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Documentation tags for organization. | |
exposes[].docs.labels | object (map) | no | Documentation labels for automation. | ||
exposes[].docs.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].semantics
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].semantics | object | no | NEW in v0.7.2: Semantic model definition mapping physical columns to business concepts. Defines entities, measures, dimensions, and metrics for agent-consumable business logic. Eliminates semantic hallucination by providing machine-readable KPI definitions. Compatible with dbt MetricFlow, Snowflake Semantic Views, and OSI-aligned formats. | ||
exposes[].semantics.name | string | no | Human-readable name for this semantic model. | ||
exposes[].semantics.description | string | no | Business context explaining what this semantic model represents. | ||
exposes[].semantics.defaultAggTimeDimension | string | no | Default time dimension used for measure aggregation. Can be overridden per-measure. | ||
exposes[].semantics.entities | array of object | no | Join keys with type annotations. Entities connect this table to others in the semantic graph. | ||
exposes[].semantics.entities[].name | string | yes | Entity name (e.g., 'order', 'customer'). | ||
exposes[].semantics.entities[].type | string | yes | one of primary, foreign, unique, natural | Entity key type: primary (unique, complete), foreign (references another table), unique (unique, possibly partial), natural (real-world identifier). | |
exposes[].semantics.entities[].expr | string | no | Column expression. Defaults to entity name if omitted. | ||
exposes[].semantics.entities[].description | string | no | |||
exposes[].semantics.measures | array of object | no | Aggregatable column expressions. Measures are the atomic building blocks from which metrics are composed. | ||
exposes[].semantics.measures[].name | string | yes | Measure name (e.g., 'total_amount', 'order_count'). | ||
exposes[].semantics.measures[].description | string | no | |||
exposes[].semantics.measures[].agg | string | yes | one of sum, avg, count, count_distinct, min, max, median, percentile | Aggregation function applied to the expression. | |
exposes[].semantics.measures[].expr | string | no | SQL expression to aggregate. Defaults to measure name if omitted. | ||
exposes[].semantics.measures[].aggTimeDimension | string | no | Override the model-level default time dimension for this measure. | ||
exposes[].semantics.measures[].nonAdditiveDimension | object | no | Dimension over which this measure cannot be naively aggregated (e.g., account balances). | ||
exposes[].semantics.measures[].nonAdditiveDimension.name | string | no | |||
exposes[].semantics.measures[].nonAdditiveDimension.windowChoice | string | no | one of min, max | Which value to pick when the non-additive dimension is present. | |
exposes[].semantics.measures[].createMetric | boolean | no | false | If true, automatically create a simple metric from this measure. | |
exposes[].semantics.dimensions | array of object | no | Categorical and time-based grouping axes. Dimensions define how data can be sliced in queries. | ||
exposes[].semantics.dimensions[].name | string | yes | Dimension name (e.g., 'order_date', 'region', 'product_category'). | ||
exposes[].semantics.dimensions[].type | string | yes | one of categorical, time | Dimension type: categorical (text/enum grouping) or time (temporal axis). | |
exposes[].semantics.dimensions[].expr | string | no | SQL expression. Defaults to dimension name if omitted. | ||
exposes[].semantics.dimensions[].description | string | no | |||
exposes[].semantics.dimensions[].typeParams | object | no | Type-specific parameters. | ||
exposes[].semantics.dimensions[].typeParams.timeGranularity | string | no | one of day, week, month, quarter, year, hour, minute | Default time granularity for this dimension. | |
exposes[].semantics.metrics | array of object | no | Composable KPI definitions built from measures. Metrics are the named business calculations that agents and tools resolve. | ||
exposes[].semantics.metrics[].name | string | yes | Metric name (e.g., 'net_revenue', 'customer_churn_rate'). | ||
exposes[].semantics.metrics[].description | string | no | Business definition of this metric. | ||
exposes[].semantics.metrics[].type | string | yes | one of simple, derived, ratio | Metric type: simple (single measure + filter), derived (expression over other metrics), ratio (numerator/denominator). | |
exposes[].semantics.metrics[].measure | string | no | Reference to a measure name (for simple metrics). | ||
exposes[].semantics.metrics[].filter | string | no | SQL WHERE clause filter (e.g., "status = 'completed' AND is_refunded = FALSE"). | ||
exposes[].semantics.metrics[].inputMetrics | array of string | no | References to other metric names (for derived/ratio metrics). | ||
exposes[].semantics.metrics[].expr | string | no | Mathematical expression combining input metrics (for derived metrics). | ||
exposes[].semantics.metrics[].numerator | string | no | Numerator metric name (for ratio metrics). | ||
exposes[].semantics.metrics[].denominator | string | no | Denominator metric name (for ratio metrics). | ||
exposes[].semantics.metrics[].owner | string | no | Business owner of this metric definition. | ||
exposes[].semantics.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Semantic model tags for discovery and categorization. | |
exposes[].semantics.labels | object (map) | no | Semantic model labels for automation and metadata. | ||
exposes[].semantics.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
exposes[].crawler
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].crawler | object (open) | no | AWS Glue crawler configuration for auto-discovery of schema from S3 paths. | ||
exposes[].crawler.name | string | no | Glue crawler name. | ||
exposes[].crawler.role | string | no | IAM role ARN for the crawler. | ||
exposes[].crawler.schedule | string | no | Cron schedule expression (e.g., 'cron(0 6 * * ? *)'). | ||
exposes[].crawler.classifiers | array of string | no | Custom classifier names to apply. | ||
exposes[].crawler.schemaChangePolicy | object (open) | no | Policy for schema changes detected by the crawler. | ||
exposes[].crawler.schemaChangePolicy.updateBehavior | string | no | one of UPDATE_IN_DATABASE, LOG | ||
exposes[].crawler.schemaChangePolicy.deleteBehavior | string | no | one of LOG, DELETE_FROM_DATABASE, DEPRECATE_IN_DATABASE |
exposes[].iceberg
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].iceberg | object (open) | no | AWS Glue Iceberg table management configuration (snapshots, compaction). | ||
exposes[].iceberg.writeFormat | string | no | one of parquet, orc, avro | Underlying file format for Iceberg data files. | |
exposes[].iceberg.snapshotRetention | object (open) | no | Snapshot retention policy. | ||
exposes[].iceberg.snapshotRetention.maxSnapshotAgeMs | integer | no | Maximum age of snapshots in milliseconds. | ||
exposes[].iceberg.snapshotRetention.minSnapshotsToKeep | integer | no | Minimum number of snapshots to retain. | ||
exposes[].iceberg.compaction | object (open) | no | Iceberg compaction settings. | ||
exposes[].iceberg.compaction.enabled | boolean | no | |||
exposes[].iceberg.compaction.targetFileSizeMb | integer | no | Target file size in MB after compaction. |
exposes[].mcp
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
exposes[].mcp | object | no | NEW in v0.7.4: Declares this expose as agent-consumable over MCP, with per-expose overrides for the consumer-side output-port server (fluid mcp output-port serve). Its presence opts the expose into the Fluid MCP gateway, where policy.agentPolicy.allowedModels / allowedUseCases are enforced at every read. | ||
exposes[].mcp.sampling | object | no | |||
exposes[].mcp.sampling.maxRows | integer | no | min 1 | Hard cap for the sample MCP tool. Defends against an over-curious agent asking for petabyte-scale row counts. Default: 100. | |
exposes[].mcp.classification | object | no | |||
exposes[].mcp.classification.dataClass | string | no | one of public, internal, confidential, restricted | Surfaced on the describe tool so consumer agents can declare downstream handling. Advisory; the real gate is policy.agentPolicy. |
labels
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
labels | object (map) | no | Product-level labels for metadata and automation. | ||
labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
consumes
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
consumes | array of object | no | |||
consumes[].productId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Upstream data product identifier (e.g., 'silver.hr.people_v2'). Renamed from 'provider' for clarity. | |
consumes[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | The upstream exposeId being consumed. | |
consumes[].versionConstraint | string | no | Semver range (e.g., ^1.2, ~1.4, >=1.0.0). Not strictly validated by regex; tooling enforces. | ||
consumes[].qosExpectations | object | no | |||
consumes[].qosExpectations.freshnessMax | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
consumes[].qosExpectations.maxStaleness | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | Maximum acceptable staleness for consumed data. | |
consumes[].qosExpectations.minCompleteness | number | no | min 0 max 1 | Minimum completeness expectation. | |
consumes[].qosExpectations.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | QoS expectation tags. | |
consumes[].qosExpectations.labels | object (map) | no | QoS expectation labels. | ||
consumes[].qosExpectations.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
consumes[].requiredPolicies | array of string | no | |||
consumes[].purpose | string | no | Business purpose for consuming this data. | ||
consumes[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Consumption relationship tags. | |
consumes[].labels | object (map) | no | Consumption relationship labels. | ||
consumes[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
builds
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds | array of object | no | NEW in v0.5.5: Multiple build configurations for multi-modal data products (e.g., batch + streaming, SQL + ML). | ||
builds[].id | string | no | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Build identifier for multi-build scenarios (e.g., 'batch-processing', 'ml-training'). | |
builds[].description | string | no | Human-readable description of this build configuration. | ||
builds[].pattern | string | no | one of hybrid-reference, embedded-logic, multi-stage, acquisition | "hybrid-reference" | Build pattern: hybrid-reference (dbt-style), embedded-logic (raw SQL/code), multi-stage (complex pipelines), or acquisition (source-aligned ingestion from external systems). |
builds[].engine | string | no | one of dbt, sql, python, spark, glue, custom, duckdb, airbyte, meltano, dlt, kafka-connect, debeziumor matches ^dbt-[a-z0-9]+([_-][a-z0-9]+)*$ | "dbt" | Build/ingestion engine. Transformation engines: dbt, sql, python, spark, glue, custom. Adapter-qualified dbt engines (dbt-<adapter>, e.g. dbt-glue, dbt-snowflake, dbt-bigquery) are accepted too and route through the dbt build runner, which derives the adapter from the suffix. Ingestion engines (acquisition pattern only): duckdb (zero-infra files/JDBC), airbyte (350+ SaaS), meltano (Singer 600+ taps), dlt (Python-native), kafka-connect (streaming), debezium (CDC). AWS Glue: glue (Spark ETL jobs managed by Glue). |
builds[].capabilities | array of string | no | unique items each item: one of 12 valuesfull_refresh incremental_append incremental_dedup incremental_merge cdc streaming schema_discovery schema_evolution dlp_scan at_most_once at_least_once exactly_once | Capabilities the build asks for (acquisition pattern). Validation enforces capabilities ⊆ runner.declared_capabilities. | |
builds[].repository | string | no | |||
builds[].outputs | array of string | no | each item: matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | List of exposeIds produced by this build. | |
builds[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects. |
builds[].properties
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds[].properties | object (free-form) | no | When pattern is hybrid-reference: Hybrid-reference pattern from v0.4.0 (dbt-style).When pattern is embedded-logic: Embedded logic pattern from v0.4.0.When pattern is multi-stage: NEW in v0.5.5: Multi-stage orchestration pattern.When pattern is acquisition: NEW in v0.7.3: Source-aligned acquisition pattern. The build ingests data from an external system (no transformation). Engine ∈ {duckdb, airbyte, meltano, dlt, kafka-connect, debezium}. | Pattern-specific build properties with conditional validation. | |
builds[].properties.model | string | yes | Only when pattern is hybrid-reference. | ||
builds[].properties.target | string | no | Only when pattern is hybrid-reference. dbt profile target to run against. Forwarded to dbt as --target <name> by the build runner (allowlisted against the profile's real targets; falls back to the profile default when the requested target is unavailable). | ||
builds[].properties.select | string or array of string | no | Only when pattern is hybrid-reference. dbt node selector(s) forwarded to dbt --select. Accepts a single selector string or an array of selector strings; the build runner drops flag-like values (those starting with '-'). | ||
builds[].properties.models | string or array of string | no | Only when pattern is hybrid-reference. Legacy alias for select (dbt's older --models flag). Same string-or-array shape; select takes precedence when both are set. | ||
builds[].properties.vars | object (free-form) | no | Only when pattern is hybrid-reference. | ||
builds[].properties.materializations | object (map) | no | Only when pattern is hybrid-reference. | ||
builds[].properties.materializations.<key> | string | no | one of table, view, incremental, ephemeral | Only when pattern is hybrid-reference. | |
builds[].properties.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Only when pattern is hybrid-reference. Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects. | |
builds[].properties.labels | object (map) | no | Only when pattern is hybrid-reference. Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects. | ||
builds[].properties.labels.<key> | string | no | Only when pattern is hybrid-reference. Label values are always strings for consistency and tooling compatibility. | ||
builds[].properties.sql | string | yes | Only when pattern is embedded-logic. | ||
builds[].properties.language | string | no | one of sql, flink_sql, pyspark, scala, python, r | Only when pattern is embedded-logic. | |
builds[].properties.parameters | object (free-form) | no | Only when pattern is embedded-logic. | ||
builds[].properties.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Only when pattern is embedded-logic. Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects. | |
builds[].properties.labels | object (map) | no | Only when pattern is embedded-logic. Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects. | ||
builds[].properties.labels.<key> | string | no | Only when pattern is embedded-logic. Label values are always strings for consistency and tooling compatibility. | ||
builds[].properties.stages | array of object | no | Only when pattern is multi-stage. | ||
builds[].properties.stages[].name | string | no | Only when pattern is multi-stage. | ||
builds[].properties.stages[].pattern | string | no | one of hybrid-reference, embedded-logic | "hybrid-reference" | Only when pattern is multi-stage. |
builds[].properties.stages[].properties | object (free-form) | no | Only when pattern is multi-stage. Base properties for all build patterns. | ||
builds[].properties.stages[].dependsOn | array of string | no | Only when pattern is multi-stage. | ||
builds[].properties.stages[].outputs | array of string | no | each item: matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Only when pattern is multi-stage. | |
builds[].properties.stages[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Only when pattern is multi-stage. Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects. | |
builds[].properties.stages[].labels | object (map) | no | Only when pattern is multi-stage. Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects. | ||
builds[].properties.stages[].labels.<key> | string | no | Only when pattern is multi-stage. Label values are always strings for consistency and tooling compatibility. | ||
builds[].properties.orchestration | object (open) | no | Only when pattern is multi-stage. | ||
builds[].properties.orchestration.parallelism | integer | no | min 1 | Only when pattern is multi-stage. | |
builds[].properties.orchestration.retryPolicy | object (open) | no | Only when pattern is multi-stage. | ||
builds[].properties.orchestration.retryPolicy.maxAttempts | integer | no | min 1 | Only when pattern is multi-stage. | |
builds[].properties.orchestration.retryPolicy.backoffStrategy | string | no | one of fixed, exponential, linear | Only when pattern is multi-stage. | |
builds[].properties.orchestration.retryPolicy.initialDelay | string | no | Only when pattern is multi-stage. | ||
builds[].properties.orchestration.retryPolicy.maxDelay | string | no | Only when pattern is multi-stage. | ||
builds[].properties.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Only when pattern is multi-stage. Simple string tags for categorization, discovery, and automation. Consistent pattern used throughout FLUID objects. | |
builds[].properties.labels | object (map) | no | Only when pattern is multi-stage. Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects. | ||
builds[].properties.labels.<key> | string | no | Only when pattern is multi-stage. Label values are always strings for consistency and tooling compatibility. | ||
builds[].properties.source | object | yes | Only when pattern is acquisition. External source system specification. | ||
builds[].properties.source.kind | string | yes | Only when pattern is acquisition. Source system kind: filesystem, postgres, mysql, sqlite, http, salesforce, stripe, github, kafka, etc. Engine-specific catalog of supported kinds. | ||
builds[].properties.source.connection | object (open) | no | Only when pattern is acquisition. Connection details. Use ${VAR} placeholders for env values; use secretRef for credentials. Inline secrets are forbidden by validator. | ||
builds[].properties.source.connection.uri | string | no | Only when pattern is acquisition. | ||
builds[].properties.source.connection.host | string | no | Only when pattern is acquisition. | ||
builds[].properties.source.connection.port | integer or string | no | Only when pattern is acquisition. string form: Allows ${ENV_VAR} placeholders that resolve to an integer at runtime. | ||
builds[].properties.source.connection.database | string | no | Only when pattern is acquisition. | ||
builds[].properties.source.connection.instance_url | string | no | Only when pattern is acquisition. | ||
builds[].properties.source.connection.secretRef | string | no | matches ^[a-z][a-z0-9_+.-]*://.+ | Only when pattern is acquisition. Reference to a secret in the configured backend (vault://, aws://, gcp://, azure://, env://). | |
builds[].properties.source.connection.user | string | no | Only when pattern is acquisition. NEW in v0.7.3: Username/principal for SQL- or REST-style sources. Required by most database drivers (Postgres/MySQL/Oracle/etc.) — without it the underlying client falls back to the OS-user identity and typically fails authentication. Goes alongside secretRef (or an inline password) for the credential pair. | ||
builds[].properties.source.connection.schema | string | no | Only when pattern is acquisition. NEW in v0.7.3: Default schema/namespace within the source database to read from. Maps to engine-specific config: dlt sql_database schema=, Meltano tap-postgres filter_schemas=[], Airbyte source-postgres schemas=[], Debezium schema.include.list, Kafka Connect JDBC schema.pattern. Use schemas (array) to scope to multiple schemas at once. | ||
builds[].properties.source.connection.schemas | array of string | no | unique items | Only when pattern is acquisition. NEW in v0.7.3: Multi-schema variant of schema. When both are set, schemas wins and schema is treated as a deprecated alias. Engines that don't natively support multi-schema reads will iterate the list. | |
builds[].properties.source.mode | string | yes | one of full_refresh, incremental_append, incremental_dedup, incremental_merge, cdc, streaming | Only when pattern is acquisition. | |
builds[].properties.source.cursor_field | string | no | Only when pattern is acquisition. Column used for incremental cursor (e.g., updated_at, SystemModstamp). | ||
builds[].properties.source.watermark | object | no | Only when pattern is acquisition. | ||
builds[].properties.source.watermark.strategy | string | no | one of high_water_mark, log_position, lsn | Only when pattern is acquisition. | |
builds[].properties.source.watermark.allowedLateness | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | Only when pattern is acquisition. ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
builds[].properties.source.streams | array of string | no | Only when pattern is acquisition. List of streams/tables/objects to ingest. | ||
builds[].properties.source.reader | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.source.reader.format | string | no | one of csv, parquet, json, ndjson, avro, orc | Only when pattern is acquisition. | |
builds[].properties.source.reader.options | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.sink | object | no | Only when pattern is acquisition. Destination format for ingested data. binding.platform on the expose remains the source of truth for where; sink.format describes how. | ||
builds[].properties.sink.format | string | no | one of iceberg, delta, parquet, csv, json, snowflake_table, bigquery_table, redshift_table, duckdb_table | Only when pattern is acquisition. | |
builds[].properties.sink.catalog | string | no | one of rest, glue, nessie, unity, snowflake-managed, hive | Only when pattern is acquisition. | |
builds[].properties.sink.partitionBy | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.delivery | object | no | Only when pattern is acquisition. Delivery semantics + idempotency + DLQ configuration. | ||
builds[].properties.delivery.guarantee | string | no | one of at_most_once, at_least_once, exactly_once | "at_least_once" | Only when pattern is acquisition. |
builds[].properties.delivery.idempotencyKey | string | no | "{run_id}:{stream}:{record_pk}" | Only when pattern is acquisition. Template for the idempotency key. Default: {run_id}:{stream}:{record_pk} | |
builds[].properties.delivery.dlq | object | no | Only when pattern is acquisition. | ||
builds[].properties.delivery.dlq.enabled | boolean | no | true | Only when pattern is acquisition. | |
builds[].properties.delivery.dlq.sink | object | no | Only when pattern is acquisition. | ||
builds[].properties.delivery.dlq.sink.format | string | no | one of parquet, json, ndjson | Only when pattern is acquisition. | |
builds[].properties.delivery.dlq.sink.location | string | no | Only when pattern is acquisition. | ||
builds[].properties.delivery.dlq.maxRecordsBeforeAbort | integer | no | min 0 | 10000 | Only when pattern is acquisition. |
builds[].properties.delivery.dlq.alertOn | array of string | no | each item: one of pii_classification_failed, schema_violation, destination_write_failed, quality_gate_failed | Only when pattern is acquisition. | |
builds[].properties.schemaEvolution | object | no | Only when pattern is acquisition. Schema evolution behavior. Stricter-wins between policy and per-change overrides. | ||
builds[].properties.schemaEvolution.policy | string | no | one of strict, discover_and_freeze, evolve_safe, evolve_all | "strict" | Only when pattern is acquisition. |
builds[].properties.schemaEvolution.onAddedColumn | string | no | one of include, warn, fail | Only when pattern is acquisition. | |
builds[].properties.schemaEvolution.onRemovedColumn | string | no | one of drop, warn, fail | Only when pattern is acquisition. | |
builds[].properties.schemaEvolution.onTypeChange | string | no | one of cast, warn, fail | Only when pattern is acquisition. | |
builds[].properties.schemaEvolution.sourceFingerprint | string | no | one of required, optional, disabled | "required" | Only when pattern is acquisition. |
builds[].properties.preLand | array of string | no | each item: one of dlp_scan, tokenize_pii, quality_gate, emit_lineage_input | Only when pattern is acquisition. Pre-land hook chain - runs on each batch before destination write. | |
builds[].properties.quality | object | no | Only when pattern is acquisition. Pre-land quality gates + anomaly signals. | ||
builds[].properties.quality.gates | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].rule | string | yes | one of not_null, unique, regex, range, row_count_anomaly, freshness | Only when pattern is acquisition. | |
builds[].properties.quality.gates[].columns | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].column | string | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].pattern | string | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].min | number | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].max | number | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].algorithm | string | no | one of ewma, iqr, exact | Only when pattern is acquisition. | |
builds[].properties.quality.gates[].sensitivity | number | no | Only when pattern is acquisition. | ||
builds[].properties.quality.gates[].severity | string | yes | one of info, warn, error | Only when pattern is acquisition. | |
builds[].properties.quality.onError | string | no | one of route_to_dlq, abort_run, best_effort | "route_to_dlq" | Only when pattern is acquisition. |
builds[].properties.quality.anomalies | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.quality.anomalies[].signal | string | yes | one of 7 valuesrecord_count_drop record_count_surge bytes_per_record_outlier latency_outlier dlq_rate_spike cursor_stalled schema_fingerprint_changed | Only when pattern is acquisition. | |
builds[].properties.quality.anomalies[].algorithm | string | no | one of ewma, iqr, exact | Only when pattern is acquisition. | |
builds[].properties.quality.anomalies[].sensitivity | number | no | Only when pattern is acquisition. | ||
builds[].properties.quality.anomalies[].severity | string | yes | one of info, warn, error | Only when pattern is acquisition. | |
builds[].properties.cost | object | no | Only when pattern is acquisition. Cost tracking + budget enforcement. | ||
builds[].properties.cost.budget | object | no | Only when pattern is acquisition. | ||
builds[].properties.cost.budget.monthly | object | no | Only when pattern is acquisition. | ||
builds[].properties.cost.budget.monthly.rows | integer | no | min 0 | Only when pattern is acquisition. | |
builds[].properties.cost.budget.monthly.bytes | string | no | Only when pattern is acquisition. Human-readable size with unit suffix (e.g., '50GB', '500MB'). | ||
builds[].properties.cost.budget.monthly.computeMinutes | integer | no | min 0 | Only when pattern is acquisition. | |
builds[].properties.cost.budget.onExceed | string | no | one of warn, abort | "warn" | Only when pattern is acquisition. |
builds[].properties.cost.chargeback | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.cost.chargeback.team | string | no | Only when pattern is acquisition. | ||
builds[].properties.cost.chargeback.project | string | no | Only when pattern is acquisition. | ||
builds[].properties.cost.chargeback.costCenter | string | no | Only when pattern is acquisition. | ||
builds[].properties.catalog | object | no | Only when pattern is acquisition. Catalog auto-registration on first apply. | ||
builds[].properties.catalog.register | array of string | no | each item: one of datahub, openmetadata, datamesh_manager, unity, glue, snowflake_horizon | Only when pattern is acquisition. | |
builds[].properties.catalog.documentation | string | no | one of auto, manual, none | "auto" | Only when pattern is acquisition. |
builds[].properties.concurrency | object | no | Only when pattern is acquisition. Single-flight concurrency control. | ||
builds[].properties.concurrency.lock | object | no | Only when pattern is acquisition. | ||
builds[].properties.concurrency.lock.scope | string | no | one of product, build | "product" | Only when pattern is acquisition. |
builds[].properties.concurrency.lock.timeout | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | Only when pattern is acquisition. ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
builds[].properties.concurrency.lock.onContended | string | no | one of abort, queue, replace | "abort" | Only when pattern is acquisition. |
builds[].properties.lineage | object | no | Only when pattern is acquisition. | ||
builds[].properties.lineage.emit | boolean | no | true | Only when pattern is acquisition. | |
builds[].properties.duckdb | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.duckdb.extensions | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.connector_image | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.version | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.normalization | boolean | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.image_signature | object | no | Only when pattern is acquisition. Connector image supply-chain verification. | ||
builds[].properties.airbyte.image_signature.verifier | string | no | one of cosign | "cosign" | Only when pattern is acquisition. |
builds[].properties.airbyte.image_signature.publicKey | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.image_signature.slsaProvenance | string | no | one of required, optional, disabled | "optional" | Only when pattern is acquisition. |
builds[].properties.airbyte.deployment | object | no | Only when pattern is acquisition. Engine deployment mode and target. | ||
builds[].properties.airbyte.deployment.mode | string | yes | one of embedded, bring-your-own, managed | "embedded" | Only when pattern is acquisition. |
builds[].properties.airbyte.deployment.server_url | string | no | Only when pattern is acquisition. URL for bring-your-own mode. | ||
builds[].properties.airbyte.deployment.auth | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.auth.secretRef | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed | object | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.target | string | yes | one of docker, kubernetes, terraform, opentofu | Only when pattern is acquisition. | |
builds[].properties.airbyte.deployment.managed.profile | string | no | one of small, medium, large | "small" | Only when pattern is acquisition. |
builds[].properties.airbyte.deployment.managed.chart | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.chart.repo | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.chart.name | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.chart.version | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.values_overlay | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.secrets | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.secrets[].name | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.secrets[].ref | string | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.network | object | no | Only when pattern is acquisition. | ||
builds[].properties.airbyte.deployment.managed.network.egressAllowList | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.tap | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.project_dir | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment | object | no | Only when pattern is acquisition. Engine deployment mode and target. | ||
builds[].properties.meltano.deployment.mode | string | yes | one of embedded, bring-your-own, managed | "embedded" | Only when pattern is acquisition. |
builds[].properties.meltano.deployment.server_url | string | no | Only when pattern is acquisition. URL for bring-your-own mode. | ||
builds[].properties.meltano.deployment.auth | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.auth.secretRef | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed | object | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.target | string | yes | one of docker, kubernetes, terraform, opentofu | Only when pattern is acquisition. | |
builds[].properties.meltano.deployment.managed.profile | string | no | one of small, medium, large | "small" | Only when pattern is acquisition. |
builds[].properties.meltano.deployment.managed.chart | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.chart.repo | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.chart.name | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.chart.version | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.values_overlay | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.secrets | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.secrets[].name | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.secrets[].ref | string | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.network | object | no | Only when pattern is acquisition. | ||
builds[].properties.meltano.deployment.managed.network.egressAllowList | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.dlt | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.dlt.source_module | string | no | Only when pattern is acquisition. | ||
builds[].properties.dlt.pipeline_name | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.connector_class | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.connector_name | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.tasks_max | integer | no | min 1 | 1 | Only when pattern is acquisition. |
builds[].properties.kafka-connect.schema_registry | object | no | Only when pattern is acquisition. Confluent-compatible Schema Registry config. When url is set, the runner injects key.converter/value.converter for Avro into the emitted connector config. | ||
builds[].properties.kafka-connect.schema_registry.url | string | no | format uri | Only when pattern is acquisition. Schema Registry HTTP endpoint, e.g. http://schema-registry:8081 | |
builds[].properties.kafka-connect.schema_registry.auth | object | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.schema_registry.auth.secretRef | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.sink_connector_name | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.sink_connector_config | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.image_signature | object | no | Only when pattern is acquisition. Connector image supply-chain verification. | ||
builds[].properties.kafka-connect.image_signature.verifier | string | no | one of cosign | "cosign" | Only when pattern is acquisition. |
builds[].properties.kafka-connect.image_signature.publicKey | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.image_signature.slsaProvenance | string | no | one of required, optional, disabled | "optional" | Only when pattern is acquisition. |
builds[].properties.kafka-connect.deployment | object | no | Only when pattern is acquisition. Engine deployment mode and target. | ||
builds[].properties.kafka-connect.deployment.mode | string | yes | one of embedded, bring-your-own, managed | "embedded" | Only when pattern is acquisition. |
builds[].properties.kafka-connect.deployment.server_url | string | no | Only when pattern is acquisition. URL for bring-your-own mode. | ||
builds[].properties.kafka-connect.deployment.auth | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.auth.secretRef | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed | object | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.target | string | yes | one of docker, kubernetes, terraform, opentofu | Only when pattern is acquisition. | |
builds[].properties.kafka-connect.deployment.managed.profile | string | no | one of small, medium, large | "small" | Only when pattern is acquisition. |
builds[].properties.kafka-connect.deployment.managed.chart | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.chart.repo | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.chart.name | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.chart.version | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.values_overlay | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.secrets | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.secrets[].name | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.secrets[].ref | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.network | object | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.deployment.managed.network.egressAllowList | array of string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.iceberg_sink_enabled | boolean | no | Only when pattern is acquisition. Opt-in: derive the Iceberg sink connector config. Defaults OFF when a hand-written sink_connector_config is present (RFC §6.2). | ||
builds[].properties.kafka-connect.sink_topics | array of string | no | Only when pattern is acquisition. Explicit topics the derived Iceberg sink consumes (else derived from source streams). | ||
builds[].properties.kafka-connect.iceberg_catalog_overrides | object (map) | no | Only when pattern is acquisition. Operator escape hatch: connector config keys merged LAST over the derived Iceberg sink config. | ||
builds[].properties.kafka-connect.iceberg_catalog_overrides.<key> | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink | object | no | Only when pattern is acquisition. Optional Iceberg streaming-sink tuning (RFC §6.2). All keys optional. | ||
builds[].properties.kafka-connect.streamingSink.commitIntervalMs | integer | no | min 1 | Only when pattern is acquisition. | |
builds[].properties.kafka-connect.streamingSink.routeField | string | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink.dynamicEnabled | boolean | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink.autoCreate | boolean | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink.evolveSchema | boolean | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink.upsertMode | boolean | no | Only when pattern is acquisition. | ||
builds[].properties.kafka-connect.streamingSink.controlTopic | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.connector_class | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.snapshot_mode | string | no | one of initial, schema_only, never, when_needed, always | Only when pattern is acquisition. | |
builds[].properties.debezium.deployment | object | no | Only when pattern is acquisition. Engine deployment mode and target. | ||
builds[].properties.debezium.deployment.mode | string | yes | one of embedded, bring-your-own, managed | "embedded" | Only when pattern is acquisition. |
builds[].properties.debezium.deployment.server_url | string | no | Only when pattern is acquisition. URL for bring-your-own mode. | ||
builds[].properties.debezium.deployment.auth | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.auth.secretRef | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed | object | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.target | string | yes | one of docker, kubernetes, terraform, opentofu | Only when pattern is acquisition. | |
builds[].properties.debezium.deployment.managed.profile | string | no | one of small, medium, large | "small" | Only when pattern is acquisition. |
builds[].properties.debezium.deployment.managed.chart | object (open) | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.chart.repo | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.chart.name | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.chart.version | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.values_overlay | object (free-form) | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.secrets | array of object | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.secrets[].name | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.secrets[].ref | string | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.network | object | no | Only when pattern is acquisition. | ||
builds[].properties.debezium.deployment.managed.network.egressAllowList | array of string | no | Only when pattern is acquisition. |
builds[].execution
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds[].execution | object | no | Build execution configuration (triggers, runtime, retries). | ||
builds[].execution.trigger | object (open) | no | |||
builds[].execution.trigger.type | string | no | one of schedule, event, manual, dependency, dataset, schedule_and_dataset, timetable | ||
builds[].execution.trigger.schedule | string | no | |||
builds[].execution.trigger.event | string | no | |||
builds[].execution.trigger.condition | string | no | |||
builds[].execution.trigger.datasets | array of string or object | no | NEW in v0.7.0: Can reference data products directly via exposeRef for strong typing and auto-URI generation. Backward compatible with raw URIs. string form: Raw dataset URI (legacy, e.g., 'dataset://gcp/project/dataset/table') object form: NEW in v0.7.0: Data product reference with strong typing | ||
builds[].execution.trigger.datasets[].productId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Data product ID to depend on | |
builds[].execution.trigger.datasets[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Expose ID within the data product | |
builds[].execution.trigger.datasets[].versionConstraint | string | no | Optional version constraint (e.g., '^2.0.0') | ||
builds[].execution.trigger.datasetsOperator | string | no | one of any, all | "all" | |
builds[].execution.trigger.cron | string | no | Cron expression for schedule trigger | ||
builds[].execution.trigger.timezone | string | no | "UTC" | ||
builds[].execution.trigger.timetable | object (free-form) | no | Custom timetable configuration | ||
builds[].execution.runtime | object (open) | no | |||
builds[].execution.runtime.resources | object (open) | no | |||
builds[].execution.runtime.resources.cpu | string | no | |||
builds[].execution.runtime.resources.memory | string | no | |||
builds[].execution.runtime.resources.disk | string | no | |||
builds[].execution.runtime.timeout | string | no | |||
builds[].execution.runtime.environment | object (map) | no | |||
builds[].execution.runtime.environment.<key> | string | no | |||
builds[].execution.runtime.platform | string | no | one of gcp, aws, azure, local, kubernetes, composer, mwaa, databricks, snowflake, athena, glue, redshift | ||
builds[].execution.runtime.executor | string | no | one of LocalExecutor, CeleryExecutor, KubernetesExecutor, SequentialExecutor | ||
builds[].execution.runtime.image | string | no | Container image for execution | ||
builds[].execution.runtime.serviceAccount | string | no | Service account for cloud execution | ||
builds[].execution.retries | object (open) | no | |||
builds[].execution.retries.maxAttempts | integer | no | min 1 | ||
builds[].execution.retries.backoffStrategy | string | no | one of fixed, exponential, linear | ||
builds[].execution.retries.initialDelay | string | no | |||
builds[].execution.retries.maxDelay | string | no | |||
builds[].execution.notifications | array of object | no | |||
builds[].execution.notifications[].type | string | no | one of email, slack, webhook, pagerduty | ||
builds[].execution.notifications[].target | string | no | |||
builds[].execution.notifications[].condition | string | no | one of success, failure, always | ||
builds[].execution.orchestration | object (open) | no | |||
builds[].execution.orchestration.engine | string | yes | one of airflow, dagster, prefect, kubeflow, custom, none | ||
builds[].execution.orchestration.mode | string | no | one of generated, manual, hybrid | "generated" | |
builds[].execution.orchestration.generateOnChange | boolean | no | true | ||
builds[].execution.orchestration.airflow | object (open) | no | |||
builds[].execution.orchestration.airflow.dagId | string | yes | matches ^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$ | ||
builds[].execution.orchestration.airflow.dagConfig | object (open) | no | |||
builds[].execution.orchestration.airflow.dagConfig.description | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.schedule | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.startDate | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.endDate | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.catchup | boolean | no | false | ||
builds[].execution.orchestration.airflow.dagConfig.maxActiveRuns | integer | no | min 1 | ||
builds[].execution.orchestration.airflow.dagConfig.dagRunTimeout | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
builds[].execution.orchestration.airflow.dagConfig.defaultArgs | object (free-form) | no | |||
builds[].execution.orchestration.airflow.dagConfig.tags | array of string | no | Airflow DAG tags (different from FLUID tags) | ||
builds[].execution.orchestration.airflow.dagConfig.onSuccessCallback | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.onFailureCallback | string | no | |||
builds[].execution.orchestration.airflow.dagConfig.slaCallback | string | no | |||
builds[].execution.orchestration.airflow.taskDefaults | object (open) | no | Default configuration for all tasks (can be overridden per task) | ||
builds[].execution.orchestration.airflow.taskDefaults.retries | integer | no | min 0 | ||
builds[].execution.orchestration.airflow.taskDefaults.retryDelay | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
builds[].execution.orchestration.airflow.taskDefaults.executionTimeout | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
builds[].execution.orchestration.airflow.taskDefaults.pool | string | no | |||
builds[].execution.orchestration.airflow.taskDefaults.queue | string | no | |||
builds[].execution.orchestration.airflow.taskDefaults.priority | integer | no | |||
builds[].execution.orchestration.airflow.taskDefaults.triggerRule | string | no | one of all_success, all_failed, all_done, one_success, one_failed, none_failed, none_skipped | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling | object (open) | no | NEW in v0.7.0: Enhanced error categorization and handling | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.strategy | string | no | one of fixed, exponential_backoff, linear_backoff | "exponential_backoff" | |
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.retryableErrors | array of string | no | Error types that should trigger retries | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.nonRetryableErrors | array of string | no | Error types that should fail immediately | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.onError | array of object | no | |||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.onError[].action | string | no | one of notify, create_incident, log, custom | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.onError[].channel | string | no | |||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.onError[].severity | string | no | one of low, medium, high, critical | ||
builds[].execution.orchestration.airflow.taskDefaults.errorHandling.onError[].when | string | no | one of always, maxAttemptsExceeded, nonRetryableError | ||
builds[].execution.orchestration.airflow.tasks | array of object | no | min items 1 Each item: when operator is set and type is not set: Operator field is deprecated but still works. Maps: FluidExecuteOperator→fluid_execute, BashOperator→bash, etc.Each item: when type is provider_action: requires provider, action, params | ||
builds[].execution.orchestration.airflow.tasks[].taskId | string | yes | matches ^[a-z0-9_][a-z0-9_.-]*[a-z0-9_]$|^[a-z0-9_]$ | ||
builds[].execution.orchestration.airflow.tasks[].type | string | no | one of 19 valuesprovider_action fluid_validate fluid_plan fluid_apply fluid_execute fluid_verify fluid_dq_check bash python branch_python sensor email http snowflake_query bigquery_query bigquery_job glue_job databricks_job custom | NEW in v0.7.0: Simplified task type taxonomy. Replaces 'operator' field (backward compatible). Types: provider_action (NEW!), fluid_validate, fluid_plan, fluid_apply, fluid_execute, fluid_verify, fluid_dq_check, bash, python, sensor, or provider-specific operators. | |
builds[].execution.orchestration.airflow.tasks[].operator | string | no | DEPRECATED in v0.7.0: Use 'type' instead. Maintained for backward compatibility with v0.6.1 contracts. Maps to 'type' automatically (FluidExecuteOperator → fluid_execute, BashOperator → bash). | ||
builds[].execution.orchestration.airflow.tasks[].provider | string | no | one of aws, gcp, azure, snowflake, databricks, kafka, kubernetes, local, custom | NEW in v0.7.0: Provider name when type=provider_action. Examples: aws, gcp, azure, snowflake, databricks. | |
builds[].execution.orchestration.airflow.tasks[].action | string | no | matches ^[a-z0-9_]+\.[a-z0-9_]+$ | NEW in v0.7.0: Provider action when type=provider_action. Format: service.operation (e.g., s3.ensure_bucket, table.ensure, kinesis.create_stream). | |
builds[].execution.orchestration.airflow.tasks[].buildStepRef | string | no | NEW in v0.7.0: Reference to transformation step in builds[].transformations[]. Links orchestration tasks to build steps for clearer mapping. | ||
builds[].execution.orchestration.airflow.tasks[].dependencies | array of string | no | |||
builds[].execution.orchestration.airflow.tasks[].params | object (free-form) | no | When type is bash or operator matches Bash: at least one of bash_command or command is requiredWhen type is python or operator matches Python: at least one of python_callable or callable is required | ||
builds[].execution.orchestration.airflow.tasks[].produces | array of object | no | NEW in v0.7.0: Declares what exposes[] this task produces. Enables automatic dataset registration and lineage tracking. | ||
builds[].execution.orchestration.airflow.tasks[].produces[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | The exposeId from exposes[] that this task produces | |
builds[].execution.orchestration.airflow.tasks[].produces[].dataset | string | no | Optional: Airflow dataset URI. Auto-generated from binding.location if not specified. | ||
builds[].execution.orchestration.airflow.tasks[].produces[].updateStrategy | string | no | one of replace, append, merge, upsert | "replace" | How this task updates the data |
builds[].execution.orchestration.airflow.tasks[].dataProductDependencies | array of object | no | NEW in v0.7.0: Tasks can declare dependencies on other data products with version constraints and freshness requirements. | ||
builds[].execution.orchestration.airflow.tasks[].dataProductDependencies[].productId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Data product ID to depend on | |
builds[].execution.orchestration.airflow.tasks[].dataProductDependencies[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Specific expose within the product | |
builds[].execution.orchestration.airflow.tasks[].dataProductDependencies[].versionConstraint | string | no | Required version (e.g., '^2.0.0', '>=1.5.0') | ||
builds[].execution.orchestration.airflow.tasks[].dataProductDependencies[].awaitFreshness | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | Maximum age of data to wait for (e.g., PT6H = 6 hours) | |
builds[].execution.orchestration.airflow.tasks[].overrides | object (free-form) | no | Task-specific overrides for taskDefaults | ||
builds[].execution.orchestration.airflow.tasks[].doc | string | no | |||
builds[].execution.orchestration.airflow.tasks[].docMd | string | no | |||
builds[].execution.orchestration.airflow.tasks[].uiColor | string | no | |||
builds[].execution.orchestration.airflow.tasks[].uiFgcolor | string | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom | object (open) | no | XCom configuration for inter-task communication (from v0.6.1) | ||
builds[].execution.orchestration.airflow.tasks[].xcom.push | array of object | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.push[].key | string | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.push[].value | any | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.push[].serializer | string | no | one of json, pickle, cloudpickle, custom | ||
builds[].execution.orchestration.airflow.tasks[].xcom.pull | array of object | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.pull[].key | string | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.pull[].task_id | string | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.pull[].dag_id | string | no | |||
builds[].execution.orchestration.airflow.tasks[].xcom.pull[].map_indexes | array | no | |||
builds[].execution.orchestration.airflow.tasks[].accessControl | object (open) | no | Task-level access control (from v0.6.1) | ||
builds[].execution.orchestration.airflow.tasks[].accessControl.requiredRoles | array of string | no | |||
builds[].execution.orchestration.airflow.tasks[].accessControl.approvalRequired | boolean | no | |||
builds[].execution.orchestration.airflow.tasks[].accessControl.approvers | array of string | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate | object (open) | no | Cost tracking (enhanced in v0.7.0) | ||
builds[].execution.orchestration.airflow.tasks[].costEstimate.compute | object (open) | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.compute.units | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.compute.pricePerUnit | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.compute.currency | string | no | "USD" | ||
builds[].execution.orchestration.airflow.tasks[].costEstimate.storage | object (open) | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.storage.units | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.storage.pricePerUnit | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.total | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.trackActual | boolean | no | false | NEW in v0.7.0: Track actual costs vs estimate | |
builds[].execution.orchestration.airflow.tasks[].costEstimate.alertThreshold | number | no | NEW in v0.7.0: Alert if actual > threshold * estimate (e.g., 1.5 = 150%) | ||
builds[].execution.orchestration.airflow.tasks[].costEstimate.budget | object (open) | no | NEW in v0.7.0: Budget enforcement | ||
builds[].execution.orchestration.airflow.tasks[].costEstimate.budget.monthly | number | no | |||
builds[].execution.orchestration.airflow.tasks[].costEstimate.budget.stopOnExceed | boolean | no | false | ||
builds[].execution.orchestration.airflow.tasks[].costEstimate.budget.alertOnExceed | boolean | no | true | ||
builds[].execution.orchestration.airflow.tasks[].params.contract_path | string | yes | Only when type is fluid_execute or operator matches FluidExecute. | ||
builds[].execution.orchestration.airflow.tasks[].params.build_id | string | no | Only when type is fluid_execute or operator matches FluidExecute. | ||
builds[].execution.orchestration.airflow.sensors | array of object | no | |||
builds[].execution.orchestration.airflow.sensors[].taskId | string | yes | |||
builds[].execution.orchestration.airflow.sensors[].sensorType | string | yes | one of FileSensor, ExternalTaskSensor, TimeSensor, HttpSensor, S3KeySensor, GCSObjectSensor, custom | ||
builds[].execution.orchestration.airflow.sensors[].params | object (free-form) | no | |||
builds[].execution.orchestration.airflow.sensors[].pokInterval | integer | no | min 1 | 60 | |
builds[].execution.orchestration.airflow.sensors[].timeout | integer | no | min 1 | 3600 | |
builds[].execution.orchestration.airflow.sensors[].mode | string | no | one of poke, reschedule | "poke" | |
builds[].execution.orchestration.airflow.sensors[].dependencies | array of string | no | |||
builds[].execution.orchestration.airflow.taskGroups | array of object | no | |||
builds[].execution.orchestration.airflow.taskGroups[].groupId | string | yes | |||
builds[].execution.orchestration.airflow.taskGroups[].description | string | no | |||
builds[].execution.orchestration.airflow.taskGroups[].tasks | array of object | no | Same fields as builds[].execution.orchestration.airflow.tasks[]: read that section with this path as the prefix. | ||
builds[].execution.orchestration.dagster | object (free-form) | no | |||
builds[].execution.orchestration.prefect | object (free-form) | no |
builds[].dependencies
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds[].dependencies | array of object | no | Dependencies on other builds (for multi-build orchestration). | ||
builds[].dependencies[].buildId | string | no | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Unified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged. | |
builds[].dependencies[].condition | string | no | one of success, completion, always |
builds[].transformations
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds[].transformations | array of object | no | Legacy transformations array for backward compatibility. | ||
builds[].transformations[].name | string | yes | |||
builds[].transformations[].model | string | no | Path/script identifier (dbt model, sql file, etc.) | ||
builds[].transformations[].outputs | array of string | no | each item: matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | List of exposeIds materialized by this step. VALIDATION: Tools must validate that every output exists in exposes[].exposeId. | |
builds[].transformations[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Transformation step tags. | |
builds[].transformations[].labels | object (map) | no | Transformation step labels. | ||
builds[].transformations[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
builds[].labels
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
builds[].labels | object (map) | no | Key-value labels for structured metadata and automation. Consistent pattern used throughout FLUID objects. | ||
builds[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
lineage
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
lineage | object | no | Optional lineage tracking for better data governance. | ||
lineage.granularity | string | no | one of table_level, field_level | "table_level" | Level of lineage tracking granularity. |
lineage.upstream | array of object | no | Upstream dependencies and their field mappings. | ||
lineage.upstream[].productId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Upstream data product ID. | |
lineage.upstream[].exposeId | string | yes | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Upstream expose ID being consumed. | |
lineage.upstream[].fieldMappings | array of object | no | Field-level lineage mappings (optional). | ||
lineage.upstream[].fieldMappings[].sourceField | string | yes | |||
lineage.upstream[].fieldMappings[].targetField | string | yes | |||
lineage.upstream[].fieldMappings[].transformation | object | no | |||
lineage.upstream[].fieldMappings[].transformation.type | string | no | one of direct_mapping, aggregation, calculation, lookup | ||
lineage.upstream[].fieldMappings[].transformation.expression | string | no | Transformation logic or SQL expression. | ||
lineage.upstream[].fieldMappings[].transformation.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Transformation tags. | |
lineage.upstream[].fieldMappings[].transformation.labels | object (map) | no | Transformation labels. | ||
lineage.upstream[].fieldMappings[].transformation.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
lineage.upstream[].fieldMappings[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Field mapping tags. | |
lineage.upstream[].fieldMappings[].labels | object (map) | no | Field mapping labels. | ||
lineage.upstream[].fieldMappings[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
lineage.upstream[].relationship | string | no | one of direct_consumption, lookup_enrichment, aggregation | Type of data relationship. | |
lineage.upstream[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Upstream relationship tags. | |
lineage.upstream[].labels | object (map) | no | Upstream relationship labels. | ||
lineage.upstream[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
lineage.downstream | array of object | no | Known downstream consumers (optional). | ||
lineage.downstream[].consumer | string | no | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Downstream data product ID. | |
lineage.downstream[].impact | string | no | one of critical, high, medium, low | Impact level if this data product changes. | |
lineage.downstream[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Downstream relationship tags. | |
lineage.downstream[].labels | object (map) | no | Downstream relationship labels. | ||
lineage.downstream[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
lineage.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Lineage configuration tags. | |
lineage.labels | object (map) | no | Lineage configuration labels. | ||
lineage.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
schemaEvolution
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
schemaEvolution | object | no | Optional schema evolution strategy for managing changes. | ||
schemaEvolution.strategy | string | no | one of semantic_versioning, date_based, sequential | Versioning strategy for schema changes. | |
schemaEvolution.compatibility | string | no | one of backward_compatible, forward_compatible, full_compatible, breaking | Compatibility approach for schema evolution. | |
schemaEvolution.changePolicy | object | no | |||
schemaEvolution.changePolicy.changeWindowDays | integer | no | min 0 | Advance notice period for breaking changes. | |
schemaEvolution.changePolicy.approvalRequired | boolean | no | Whether schema changes require approval. | ||
schemaEvolution.changePolicy.approvers | array of string | no | List of approvers for schema changes. | ||
schemaEvolution.changePolicy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Change policy tags. | |
schemaEvolution.changePolicy.labels | object (map) | no | Change policy labels. | ||
schemaEvolution.changePolicy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
schemaEvolution.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Schema evolution tags. | |
schemaEvolution.labels | object (map) | no | Schema evolution labels. | ||
schemaEvolution.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
machineLearning
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
machineLearning | object | no | Optional ML model specifications for MLPipeline kind. | ||
machineLearning.enabled | boolean | no | false | ||
machineLearning.framework | string | no | one of mlflow, kubeflow, sagemaker, vertex_ai, custom | ||
machineLearning.models | array of object | no | |||
machineLearning.models[].name | string | yes | |||
machineLearning.models[].version | string | yes | matches ^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-[0-9A-Za-z.-]+)?(?:\+[0-9A-Za-z.-]+)?$ | Semantic version (semver.org). | |
machineLearning.models[].type | string | yes | one of classification, regression, clustering, recommendation, other | ||
machineLearning.models[].algorithm | string | no | |||
machineLearning.models[].features | array of string | no | List of feature names used by this model. | ||
machineLearning.models[].target | string | no | Target variable for supervised learning. | ||
machineLearning.models[].validation | object | no | |||
machineLearning.models[].validation.metrics | array of string | no | Validation metrics (e.g., accuracy, auc_roc). | ||
machineLearning.models[].validation.thresholds | object (map) | no | Minimum acceptable metric values. | ||
machineLearning.models[].validation.thresholds.<key> | number | no | |||
machineLearning.models[].validation.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Model validation tags. | |
machineLearning.models[].validation.labels | object (map) | no | Model validation labels. | ||
machineLearning.models[].validation.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
machineLearning.models[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | ML model tags. | |
machineLearning.models[].labels | object (map) | no | ML model labels. | ||
machineLearning.models[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
machineLearning.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | ML configuration tags. | |
machineLearning.labels | object (map) | no | ML configuration labels. | ||
machineLearning.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
environments
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
environments | object (map) | no | Optional environment-specific overrides (dev, staging, prod). | ||
environments.<key> | object | no | |||
environments.<key>.metadata | object | no | Environment-specific metadata overrides. | ||
environments.<key>.metadata.owner | object | no | |||
environments.<key>.metadata.owner.team | string | no | |||
environments.<key>.metadata.owner.email | string | no | |||
environments.<key>.metadata.owner.slack | string | no | |||
environments.<key>.metadata.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Environment-specific metadata tags. | |
environments.<key>.metadata.labels | object (map) | no | Environment-specific metadata labels. | ||
environments.<key>.metadata.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
environments.<key>.exposes | array of object | no | Environment-specific expose overrides. | ||
environments.<key>.exposes[].exposeId | string | no | matches ^[A-Za-z0-9_][A-Za-z0-9_.-]*[A-Za-z0-9_]$|^[A-Za-z0-9_]$ | Unified identifier pattern: starts/ends with alphanumeric or underscore, allows dots and hyphens in middle. Both casing conventions accepted (snake_case / dotted lowercase / PascalCase / UPPER) so existing catalog ids round-trip unchanged. | |
environments.<key>.exposes[].binding | object | no | Same fields as exposes[].binding: read that section with this path as the prefix. | ||
environments.<key>.exposes[].qos | object | no | |||
environments.<key>.exposes[].qos.availability | string | no | matches ^(100(\.0+)?|\d{2}(\.\d+)?|\d{1}\d(\.\d+)?)%$ | Availability percentage (e.g., 99.9%). | |
environments.<key>.exposes[].qos.freshnessSLO | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
environments.<key>.exposes[].qos.dataLossSLO | string | no | e.g., '0 rows' | ||
environments.<key>.exposes[].qos.latencyP95 | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
environments.<key>.exposes[].qos.completenessTarget | number | no | min 0 max 1 | Target completeness ratio (0.0 to 1.0). | |
environments.<key>.exposes[].qos.errorBudget | number | no | min 0 max 1 | Acceptable error rate (0.0 to 1.0). | |
environments.<key>.exposes[].qos.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | QoS-level tags for SLA management. | |
environments.<key>.exposes[].qos.labels | object (map) | no | QoS-level labels for automation and reporting. | ||
environments.<key>.exposes[].qos.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
environments.<key>.exposes[].tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Environment-specific expose tags. | |
environments.<key>.exposes[].labels | object (map) | no | Environment-specific expose labels. | ||
environments.<key>.exposes[].labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
environments.<key>.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Environment configuration tags. | |
environments.<key>.labels | object (map) | no | Environment configuration labels. | ||
environments.<key>.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
lifecycle
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
lifecycle | object | no | |||
lifecycle.state | string | no | one of preview, active, deprecated, retired | Unified lifecycle state vocabulary. | |
lifecycle.retention | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
lifecycle.deprecationPolicy | object | no | |||
lifecycle.deprecationPolicy.noticePeriod | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
lifecycle.deprecationPolicy.contact | string | no | |||
lifecycle.deprecationPolicy.replacement | string | no | Reference to replacement data product or expose. | ||
lifecycle.deprecationPolicy.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Deprecation policy tags. | |
lifecycle.deprecationPolicy.labels | object (map) | no | Deprecation policy labels. | ||
lifecycle.deprecationPolicy.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. | ||
lifecycle.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Lifecycle tags for automation. | |
lifecycle.labels | object (map) | no | Lifecycle labels for governance. | ||
lifecycle.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
docs
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
docs | object | no | |||
docs.homepage | string | no | format uri | ||
docs.runbook | string | no | format uri | ||
docs.dictionary | string | no | format uri | ||
docs.changeLog | string | no | format uri | ||
docs.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Documentation tags for organization. | |
docs.labels | object (map) | no | Documentation labels for automation. | ||
docs.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
sovereignty
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
sovereignty | object | no | NEW in v0.7.1: Jurisdiction and data residency requirements for compliance enforcement. | ||
sovereignty.jurisdiction | string | no | one of EU, US, UK, CA, AU, JP, CN, IN, BR, Global, Multi-Region | Required legal jurisdiction for data storage and processing. Used to validate binding.location matches sovereignty intent. | |
sovereignty.allowedRegions | array of string | no | unique items | Explicit list of allowed cloud regions. Example: ['eu-west-1', 'eu-central-1'] for EU-only. | |
sovereignty.deniedRegions | array of string | no | unique items | Explicit list of prohibited cloud regions. Takes precedence over allowedRegions. | |
sovereignty.dataResidency | boolean | no | true | Whether data must remain within jurisdiction boundaries at rest and in transit. True = strict residency. | |
sovereignty.crossBorderTransfer | boolean | no | false | Whether cross-border data transfer is permitted. False = data never leaves jurisdiction. | |
sovereignty.transferMechanisms | array of string | no | unique items each item: one of SCCs, BCRs, Adequacy, DPF, Consent, Derogation | Allowed legal mechanisms for cross-border transfer (if permitted). Example: ['SCCs', 'BCRs', 'Adequacy']. | |
sovereignty.regulatoryFramework | array of string | no | unique items each item: one of GDPR, CCPA, CPRA, HIPAA, PIPEDA, LGPD, PDPA, POPIA, DPA, APPI | Applicable regulatory frameworks. Example: ['GDPR', 'CCPA', 'HIPAA']. | |
sovereignty.enforcementMode | string | no | one of strict, advisory, audit | "strict" | How sovereignty violations are handled. strict = block deployment, advisory = warn, audit = log only. |
sovereignty.validationRequired | boolean | no | true | Whether binding.location must be validated against sovereignty rules before deployment. | |
sovereignty.tags | array of string | no | unique items each item: matches ^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$ | Sovereignty policy tags. | |
sovereignty.labels | object (map) | no | Sovereignty policy labels. | ||
sovereignty.labels.<key> | string | no | Label values are always strings for consistency and tooling compatibility. |
accessPolicy
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
accessPolicy | object | no | Root-level access policy for automated IAM binding generation. Defines principals, permissions, and resources for data product access control. | ||
accessPolicy.grants | array of object | no | List of access grants to principals (users, groups, service accounts). | ||
accessPolicy.grants[].principal | string | yes | Identity receiving access. Format: 'user:email@domain.com', 'group:name@domain.com', 'serviceAccount:name@project.iam.gserviceaccount.com' | ||
accessPolicy.grants[].permissions | array of string | no | unique items each item: one of read, select, query, write, insert, update, delete, create, admin, manage | Permissions to grant. Will be mapped to provider-specific roles (e.g., BigQuery roles). | |
accessPolicy.grants[].resources | array of string | no | Resources this grant applies to. JSONPath expressions referencing exposes[] items. If omitted, applies to all exposed resources. | ||
accessPolicy.grants[].conditions | object (free-form) | no | Optional conditions for conditional access (e.g., IP restrictions, time windows). |
orchestration
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
orchestration | object (open) | no | Top-level orchestration configuration for scheduling and workflow engines. | ||
orchestration.engine | string | yes | one of airflow, dagster, prefect, kubeflow, custom, none | ||
orchestration.mode | string | no | one of generated, manual, hybrid | "generated" | |
orchestration.generateOnChange | boolean | no | true | ||
orchestration.airflow | object (open) | no | Same fields as builds[].execution.orchestration.airflow: read that section with this path as the prefix. | ||
orchestration.dagster | object (free-form) | no | |||
orchestration.prefect | object (free-form) | no |
retention
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
retention | object | no | NEW in v0.7.3: Retention durations for run state, run logs, lineage events, and DLQ records. Honored by the retention sweeper invoked from policy-apply. | ||
retention.runState | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
retention.runLogs | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
retention.lineage | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). | |
retention.dlq | string | no | matches ^P(?!$)(\d+Y)?(\d+M)?(\d+W)?(\d+D)?(T(\d+H)?(\d+M)?(\d+S)?)?$ | ISO-8601 duration (e.g., P1D, PT15M, P2Y6M). |
governance
| Path | Type | Required | Allowed values | Default | Description |
|---|---|---|---|---|---|
governance | object | no | Contract-level governance: account/project-wide settings (e.g. AWS Lake Formation admins and LF-tag definitions). Per-resource grants and tag associations live under binding.governance. | ||
governance.lakeFormation | object | no | AWS Lake Formation account-wide settings: data-lake admins and LF-tag (TBAC) definitions. Applies to the AWS account the contract's IaC apply targets. | ||
governance.lakeFormation.admins | array of string | no | unique items each item: matches ^arn:aws[a-z0-9-]*:iam:: | IAM principal ARNs (roles or users) granted Lake Formation admin privileges. Emitted as aws_lakeformation_data_lake_settings.admins. AUTHORITATIVE, not additive: Lake Formation's PutDataLakeSettings replaces the account's admin list with the one sent, so an admin NOT listed here is REMOVED on apply. List every principal that must stay a data lake admin, including the role or user that runs the apply: left out, it loses Lake Formation admin on that apply, and the Lake Formation grants and registrations that need an admin can then fail. The terraform-provider-aws resource also clears the data lake settings its configuration omits, and this block sets only the admins, so the account's create-database and create-table default permissions (the IAMAllowedPrincipals default for new databases and tables), trusted resource owners and parameters are cleared too. Destroying the resource empties the admin list and resets CROSS_ACCOUNT_VERSION to 1. Re-applying the same list is a no-op. | |
governance.lakeFormation.tagDefinitions | object (map) | no | LF-TBAC tag-key definitions. Each entry is a tag key mapped to its allowed string values. Emitted as one aws_lakeformation_lf_tag per key; values are the tag's allowed-values list. Field idiom borrowed from OpenMetadata's classification-label model — see ODCS roles + OpenMetadata data contracts spec. | ||
governance.lakeFormation.tagDefinitions.<key> | array of string | no | min items 1 unique items |