Composing a contract with $ref

A contract can pull any object from another YAML or JSON file with $ref. fluid validate, fluid plan, fluid apply and fluid bundle resolve every ref before they do anything else, so the rest of the pipeline sees one document.

This page covers how a single ref is read and which files it may name. For working with a split product (the layout fluid split writes, which commands read it, overlays, CI and digests), see Contract fragments.

Example

orders/
├── contract.fluid.yaml
├── owner.yaml
├── data/orders.csv
└── parts/
    ├── build.yaml
    └── policy.yaml

# orders/contract.fluid.yaml
fluidVersion: 0.7.3
kind: DataProduct
id: sales.orders_v1
name: Orders
domain: sales
metadata:
  layer: Silver
  owner:
    $ref: ./owner.yaml                 # the whole file
builds:
  - $ref: parts/build.yaml             # refs work inside lists
exposes:
  - exposeId: orders_summary
    kind: table
    binding:
      platform: local
      format: csv
      location:
        path: out/summary.csv
    policy:
      $ref: parts/policy.yaml#/gold    # file + JSON pointer
    contract:
      schema:
        - name: id
          type: INTEGER
        - name: doubled
          type: INTEGER

# orders/parts/policy.yaml
gold:
  classification: Internal
  authz:
    readers:
      - group:sales-analysts

$ fluid validate contract.fluid.yaml
✅ Valid FLUID contract (schema v0.7.3)
Validation completed in 0.001s

fluid bundle writes the single resolved document. The ref nodes are replaced by their targets:

$ fluid bundle contract.fluid.yaml
fluidVersion: 0.7.3
kind: DataProduct
id: sales.orders_v1
name: Orders
domain: sales
metadata:
  layer: Silver
  owner:
    team: sales-analytics
    email: sales-analytics@example.com
builds:
- id: summarise
  pattern: embedded-logic
  engine: sql
  properties:
    sql: SELECT id, amount * 2 AS doubled FROM read_csv('data/orders.csv')
exposes:
- exposeId: orders_summary
  ...
  policy:
    classification: Internal
    authz:
      readers:
      - group:sales-analysts
  ...

How a ref is read

• A $ref node is an object whose only key is $ref. An object with other keys next to $ref is not resolved, and fails the schema with Additional properties are not allowed ('$ref', 'description' were unexpected). There is no sibling-override; change a field of a fragment with an overlay instead.
• A ref is resolved relative to the file that contains it, so parts/build.yaml may itself say $ref: ./policy.yaml.
• file.yaml#/a/b selects the object at JSON pointer/a/b inside the file. Without a #, the whole file is used.
• Same-document refs ($ref: "#/definitions/x") are left in place as written.
• A ref can stand for a whole list: builds: {$ref: lib/lib.yaml#/builds}. A YAML file used as a fragment must have a mapping at its root (YAML root must be an object/dict), so put a list under a key and point at it. A JSON fragment may be a list.
• JSON pointer escapes (~0 for ~, ~1 for /) are not decoded, so a key containing / cannot be addressed.
• A ref that leads back to itself fails with Circular $ref detected, and resolution stops past a fixed depth with $ref nesting depth exceeded 20.
• Relative data paths inside a fragment, such as a read_csv('data/orders.csv') in a build or a binding.location.path, are resolved from the root contract's directory, not the fragment's.
• Errors found after resolution name a path in the resolved document, such as exposes[1].binding.format. Count the entries in the root's list, or run fluid bundle, to find the fragment it came from.

The ref root

Every ref must name a file inside the ref root. By default the ref root is the directory that holds the root contract file (orders/ above). It applies to every ref, including refs inside fragments: a fragment in orders/parts/ can reach anything under orders/ and nothing outside it.

The check runs after .. segments and symlinks are resolved, so neither can be used to get out. It also runs before the target is opened, so the error is the same whether or not the target exists.

What a refused ref looks like

fluid validate reports it as contract_load_failed and exits 1:

$ fluid validate contract.fluid.yaml
❌ Validation error: contract_load_failed
   error: $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in
   /work/orders/contract.fluid.yaml escapes the ref root /work/orders (after resolving
   '..' and symlinks); refs may only name files inside it. To compose fragments from a
   wider tree (e.g. a monorepo's shared/ directory), set FLUID_REF_ROOT to that
   directory or pass ref_root= to the loader; see docs/contract-refs.md.
   [ERR_CONTRACT_LOAD_FAILED]

fluid bundle prints the same message and exits 2. The other refusals name their reason:

$ fluid bundle contract.fluid.yaml       # with $ref: /etc/hosts
❌ $ref resolution error: $ref '/etc/hosts' at JSON pointer '/exposes/0/policy' in
/work/orders/contract.fluid.yaml must be a relative path, got absolute — use a path
relative to the file that contains the ref

$ fluid bundle contract.fluid.yaml       # with $ref: file:///etc/hosts
❌ $ref resolution error: $ref 'file:///etc/hosts' at JSON pointer '/exposes/0/policy'
in /work/orders/contract.fluid.yaml is a URL; remote refs (including file://) are not
supported — use a relative path to a file inside the ref root

Each of these confinement refusals (a URL, an absolute path, an escape from the ref root) names the ref, the JSON pointer of the $ref node, and the file it was written in. Other ref errors, such as a ref blocked because it resolves into a system directory, a missing target file, or a JSON pointer that does not resolve, do not carry all three.

Widening the root for a monorepo

To share fragments between products, set the ref root to a directory that contains both the contracts and the shared fragments:

repo/
├── shared/policy.yaml
└── products/orders/contract.fluid.yaml   # $ref: ../../shared/policy.yaml#/gold

$ cd repo
$ FLUID_REF_ROOT=. fluid validate products/orders/contract.fluid.yaml
✅ Valid FLUID contract (schema v0.7.3)
Validation completed in 0.001s

Without FLUID_REF_ROOT, the same command fails with escapes the ref root.

From Python, pass ref_root= to the engine's loader. load_contract, load_with_overlay and compile_contract in fluid_build.loader all accept it, and it wins over FLUID_REF_ROOT:

from fluid_build.loader import load_contract

contract = load_contract("repo/products/orders/contract.fluid.yaml", ref_root="repo")
print(contract["exposes"][0]["policy"]["classification"])
# Internal

The public contract-loading API (fluid_build.api.load_contract) has no ref_root argument. It goes through the same loader, so it honours FLUID_REF_ROOT. Its in-memory forms (load_contract_from_text, load_contract_from_dict) do not: their ref root is always the base_dir you pass.

Rules for the wider root

• It widens the root only for a contract inside it, and only if it is an existing directory.

• FLUID_REF_ROOT is ignored for a contract outside it, and when it is not a directory or cannot be resolved. That contract gets the default root, its own directory, and a ref_root_env_ignored warning says so. Refs that leave the directory then fail, and the error says the variable was ignored and why:

  $ cd repo
  $ FLUID_REF_ROOT=. fluid validate /work/upload-1234/contract.fluid.yaml
  ref_root_env_ignored: contract /work/upload-1234/contract.fluid.yaml is outside
  FLUID_REF_ROOT='.' (resolved to /work/repo); the ref root must contain the contract.
  Ignoring it for this contract: its $refs are confined to the contract's own directory
  /work/upload-1234 (the default). Logged once per value in this process; a later
  contract it is ignored for gets no warning, but its escape errors say the variable
  was ignored and why.
  ❌ Validation error: contract_load_failed
     error: $ref '../shared/policy.yaml#/gold' at JSON pointer '/exposes/0/policy' in
     /work/upload-1234/contract.fluid.yaml escapes the ref root /work/upload-1234 (after
     resolving '..' and symlinks); refs may only name files inside it. FLUID_REF_ROOT is
     set but was ignored for this contract: ...
  

• ref_root= is strict. The caller sets it for one contract, so a ref_root that cannot be resolved, is not a directory, or does not contain the contract raises RefResolutionError:

  load_contract("repo/products/orders/contract.fluid.yaml", ref_root="orders")
  # RefResolutionError: contract repo/products/orders/contract.fluid.yaml is outside
  # ref_root='orders' (resolved to /work/orders); the ref root must contain the contract
  

• A blank FLUID_REF_ROOT counts as unset.

• It is only consulted when the contract has a ref to another file.

• It widens the root and nothing else. URLs and absolute paths are still refused, and refs that resolve into system directories (/etc, /proc, /private/etc on macOS, and so on) are still blocked.

Upgrading monorepos that share fragments

Before 0.18.0 a relative ref could climb out of the contract's directory, so monorepos shared fragments with $ref: ../other-product/…. Those refs now fail with escapes the ref root until the root is widened. Set FLUID_REF_ROOT (or ref_root=) to the repository root, or to the narrowest directory that holds the products and their shared fragments:

FLUID_REF_ROOT="$(git rev-parse --show-toplevel)" \
  fluid validate products/orders/contract.fluid.yaml

In CI, set it on the step that runs fluid, not for the whole job.

Contracts whose refs stay inside their own directory need no change.

Catching the error in Python

RefConfinementError is a subclass of RefResolutionError, so existing except RefResolutionError handlers keep working. It carries the details as attributes:

from fluid_build.loader import RefConfinementError, load_contract

try:
    load_contract("repo/products/orders/contract.fluid.yaml")
except RefConfinementError as err:
    print(err.ref)                   # ../../shared/policy.yaml#/gold
    print(err.pointer)               # /exposes/0/policy
    print(err.source)                # /work/repo/products/orders/contract.fluid.yaml
    print(err.root)                  # /work/repo/products/orders
    print(err.ignored_ref_root_env)  # None

ignored_ref_root_env holds the FLUID_REF_ROOT value when it was set but did not apply to this contract.

The public API raises ContractLoadError with event == "contract_ref_unresolved" for the same refusal; see Contract loading API.

OpenAPI fragments inside a bundle

When fluid validate checks a fluid bundle --format tgz archive, OpenAPI documents extracted into sources/openapi/ have no directory of their own, so the same check applies with no root: only same-document refs ($ref: "#/components/schemas/Order") are allowed. Any other $ref is reported as an OAS-REF-EXTERNAL error, and openapi-spec-validator is not run on that fragment (it would follow file:// and http(s):// refs). Inline the referenced schemas under components instead.

This applies to a $ref key anywhere in the fragment, including inside example, examples.*.value and x-* payloads. If an example payload has to contain a $ref, rename the key in the payload (for example ref) or drop the example.

The check runs where openapi-spec-validator would: when it is not installed, the fragment is reported as not validated instead.

Related

• Contract fragments: working with a product split into fragment files
• fluid bundle: write the resolved document or a .tgz bundle
• Contract loading API: load a contract from Python exactly as fluid plan sees it
• DuckDB sandbox: the matching confinement for SQL inside a contract
• Upgrading to 0.18.0