Catalog Integration: Architecture

This page is the technical companion to the catalogs index. The user-facing walkthrough lives there. This page documents why the architecture looks the way it does, what invariants it maintains across the seven catalogs, and where the extension points live for contributors. The release it arrived in is described in the release notes; earlier docs called it "V1.5".

Audience: contributors adding a new catalog adapter, operators debugging an unexpected forge result, security teams auditing the read-only contract.

What it ships

The catalog adapters sit behind one ABC and are callable through two surfaces:

                                ┌─ CLI: fluid forge data-model from-source
                                │
      ┌─────────────────────────┤
      │                         │
      │                         └─ MCP: forge_from_source tool
      │
      ▼
┌──────────────────────────────────────────────────────────────────┐
│   Catalog dispatch                                               │
│   _build_catalog_adapter / _SOURCE_ADAPTERS                      │
└────────────────────────────────┬─────────────────────────────────┘
                                 │
       ┌─────────┬───────────┬───┴───────┬──────────┬────────┐
       ▼         ▼           ▼           ▼          ▼        ▼
   Snowflake  Unity      BigQuery    Dataplex    Glue     DataHub  DMM
   adapter   adapter      adapter     adapter   adapter   adapter  adapter
       │         │           │           │          │        │      │
       └────┬────┴───────────┴───┬───────┴──────────┴────────┴──────┘
            ▼                    ▼
   CatalogAdapter ABC     CatalogTable, CatalogColumn,
   (abstract: list_tables, get_table,
   get_lineage, list_glossary_terms)   CatalogLineage, GlossaryTerm,
                          CatalogScope (Pydantic)
            │
            ▼
   LogicalAgent.from_catalog
   (translates CatalogTable[] → TableDefinition[] → LogicalDraft)
            │
            ▼
   Staged pipeline (Logical → Builder → Readme → Transformation → Validator)
            │
            ▼
   Fluid contract + .model.json sidecar + OSI v0.1.1 standalone

The four invariants

Every catalog adapter is written to hold these:

1. Read-only on metadata only

No SELECT * against any data table. Adapters request metadata privileges (INFORMATION_SCHEMA-equivalent on each catalog) and nothing more.

2. Lazy SDK import

The adapter module imports cleanly even when its underlying SDK isn't installed. The actual SDK import happens on first method call, raising CatalogConfigError with the exact pip install suggestion if the dep is missing. This keeps fluid --help fast (no 500ms boto3 import on cold start) and lets users pip install data-product-forge without forcing every catalog SDK.

def _client(self):
    try:
        import snowflake.connector  # type: ignore[import-not-found]
    except ImportError as exc:
        raise CatalogConfigError(
            "snowflake-connector-python missing. Install with: "
            'pip install "data-product-forge[snowflake]"',
            suggestions=['pip install "data-product-forge[snowflake]"'],
        ) from exc
    ...

3. Per-call client lifecycle

The adapter constructs / closes its SDK client per logical operation. No long-lived connection pool spans tool calls inside fluid mcp serve — defends against an upstream agent escalating "list one schema" into "dump everything" by replaying old credentials in a long-running session.

4. Soft-fail on optional reads

Lineage / glossary / sensitivity-tag reads return empty results when the user lacks the privilege OR the API isn't enabled, instead of erroring. The forge isn't blocked by missing optional metadata — you get a working contract, just without the optional signal.

These four invariants are documented in fluid_build/copilot/catalog/_patterns.py.

The nine reusable patterns

_patterns.py exports three helpers and documents nine patterns:

A new adapter follows the same nine patterns. The walkthrough at CONTRIBUTING.md → Adding a Catalog Adapter walks through each.

Three-stage catalog metadata flow

Catalog signal isn't just an input to the Logical stage; it shapes every stage of the pipeline:

Stage 1 — Logical (Conceptual + IR + OSI)

Entry point: LogicalAgent.from_catalog(...).

Stage 2 — Contract (BuilderAgent emits the latest stable schema)

The BuilderAgent prompt directive: every metadata field the catalog supplied must appear verbatim in the contract; do not re-invent.

Stage 3 — Transformation (TransformationAgent emits dbt builds[])

System roles never become owners

A specific design decision worth calling out: catalog adapters that surface a "creating role" or "table owner" field (Snowflake OWNER, Glue Owner, etc.) do not promote Snowflake-style system roles into metadata.owner.team.

_SYSTEM_ROLE_NAMES: frozenset[str] = frozenset({
    "ACCOUNTADMIN", "SYSADMIN", "SECURITYADMIN", "USERADMIN",
    "ORGADMIN", "PUBLIC", "ADMIN", "ADMINS", "ROOT", "DBO",
    "POSTGRES",
})

These names land in labels.catalogCreatingRoles (audit only) instead. The contract's owner field reflects the business team, not the role that ran the DDL — preventing every Snowflake table from being attributed to ACCOUNTADMIN.

Industry auto-detection

When a catalog scope is forged, the adapter's catalog tags are matched against INDUSTRY_DOMAIN_HINTS to auto-pick an industry pack. The hints cover four packs: telecommunications, healthcare, finance and retail. Each pack has a set of tag words, and a word can sit in more than one set: in 0.18.1 customer is a telecommunications hint and consumer and product are retail hints, for example. The sets live in fluid_build/copilot/industry/compiler.py.

The match wins per-table; the most common hit per scope wins as the chosen industry. Operators can always override with --industry.

Three-layer security model

The catalog tools inherit the MCP server's three-layer access control and add catalog-specific defenses:

Layer 1: Tool allow/deny list

The --allow-tools / --deny-tools flags on fluid mcp serve control which tools an MCP client can call. Catalog tools are opt-out: removing forge_from_source (the only mutating catalog tool) leaves the read-only catalog tools available without ever risking writes.

Layer 2: Read-only mode

fluid mcp serve --read-only blocks every tool that mutates files or writes store namespaces. forge_from_source is blocked; list_source_tables / inspect_source_table / list_source_lineage / list_source_glossary / list_source_adapters all run.

Layer 3: Sandboxes

--writable-paths pins the filesystem roots forge_from_source may write to; --writable-namespaces pins the conceptual store namespaces. Defaults are safe: writable_paths=cwd, writable_namespaces={history, audit}.

Catalog-specific defense: per-call credential lookup

The MCP server itself never holds catalog credentials. Each tool call passes credentials.credential_id — a string pointer into ~/.fluid/sources.yaml. The server resolves the pointer at call time:

1. Inline credentials (only via direct CLI; never over MCP wire).
2. OS keyring (macOS Keychain / Windows Credential Manager / Linux secret-service).
3. ~/.fluid/sources.yaml (non-sensitive fields only — secrets live in keyring).
4. Environment variables.
5. Cloud metadata service (opt-in via allow_metadata_service=true, which an MCP client can set on each call; see Credential resolver).
6. Fail-closed.

Defends against an upstream agent escalating "list tables" into "dump every secret" by replaying old credentials in a long-running MCP session.

Audit trail

Every catalog tool call writes an audit event via copilot/store/audit_trail.py. The record has this shape (timestamp, event name, scope, outcome):

{
  "event": "mcp_list_source_tables",
  "timestamp_utc": "<ISO-8601 time>",
  "payload": {
    "catalog_name": "<adapter name>",
    // plus the adapter's non-secret context; Snowflake adds account, user, role and auth_method
    "scope": { "database": "<database>", "schema": "<schema>" },
    "result_count": <number of tables>
  }
}

Credentials are scrubbed before write — audit_context() returns non-sensitive fields only.

The trail's canonical read surface is the on-disk store. Each event is written to ~/.fluid/store/audit/ as {timestamp}_{suffix}_{event}.json, so it is read with ordinary file tools and filtered by the event name in the filename:

ls ~/.fluid/store/audit/
cat ~/.fluid/store/audit/*_mcp_list_source_tables.json
fluid memory status   # record count for the 'audit' namespace

There is no fluid memory show audit: that subcommand's scopes are project, team, personal, episodic, semantic and history only, and history is version history, not the audit trail. fluid memory search --ns audit does not reach these records either.

Determinism guarantees

Catalog reads are deterministic — same catalog state, same scope, same JSON shape. The forge as a whole is near-deterministic:

1. Cache hit — byte-identical output (cache key includes capability_matrix, so flipping a flag like extended_thinking invalidates cleanly).
2. Cache miss, temp=0 — near-identical entity names, relationships, hash keys. Minor wording in description / ai_context.instructions may vary by 1-2 tokens.
3. --deterministic flag — turns the staged cache off and tiering off and records the run as deterministic in its audit metadata. The request temperature is 0.0 by default and the CLI sets no seed.

The capability_matrix segment in the cache key is what makes "flipping a capability flag invalidates the cache cleanly" actually true:

generate_cache_key(
    model="claude-sonnet-4-6",
    prompt=prompt_blob,
    params=stage_params,
    capability_matrix={"extended_thinking": True},  # 4th hash segment
)

Without this, two runs with identical model/prompt/params but different capability matrices would collide on the same cached response — leaking a "no-thinking" answer into a "with-thinking" run. The hash segment makes them hash distinct.

See also

• Catalogs index — user-facing walkthrough
• Credential resolver — security model details
• Cost tracking: the cost summary and its footers
• MCP server — full MCP walkthrough