Data Lineage

Ontul automatically tracks data lineage — which tables and columns a table was derived from — at both table and column granularity. Lineage is extracted from the actual Calcite query plan, not from heuristic SQL parsing, so joins, sub-queries, CTEs, and aggregations are resolved accurately.

Automatic Capture

When a table is created or populated, Ontul records its lineage — no configuration, no annotations. Capture happens on:

CREATE TABLE ... AS SELECT (CTAS)
INSERT INTO ... SELECT
MERGE INTO
CREATE VIEW
SUBMIT STREAMING — an Ontul Flow: the flow's source(s) → sink edge is recorded when the flow starts

Capture is best-effort: it runs after the statement succeeds and never blocks or fails the query.

For a streaming flow, Ontul-catalog tables appear under their catalog.schema.table name (so a flow's edge lines up with SQL-DML lineage), while external systems are recorded as scheme-prefixed datasets — kafka://<topic>, cdc://<connector>/<table>, jdbc://<table>, es://<index>, http://<endpoint>, neorunbase://<table>, or the S3 path for a file source.

How Repeated Writes Accumulate

A table is often fed by more than one statement — a CTAS from one engine, then INSERTs from another. Table-level lineage keeps one entry per target table, and each new edge is combined with what is already there according to what the statement means for the target:

Operation (as recorded)	Produced by	Effect on the target's sources
`CTAS`	`CREATE TABLE … AS SELECT` (Ontul or an engine plugin)	Replaces them — the table is defined afresh
`VIEW`	`CREATE [OR REPLACE] VIEW`	Replaces them — a redefined view drops its old sources
`semantic-view`	Semantic-layer view registration	Replaces them
`INSERT_INTO`, `INSERT`	`INSERT INTO … SELECT` (Ontul / engine plugin)	Adds the new sources
`MERGE_INTO`	`MERGE INTO`	Adds the new sources
`STREAMING`	`SUBMIT STREAMING` (Ontul Flow start)	Adds the flow's sources
`ACTION`	Ontology action invocation	Adds
`EXTERNAL` (default) or any other value sent	`POST /v1/api/lineage`	Adds — only `CTAS`, `VIEW` and `semantic-view` replace

Two further rules keep the graph honest:

A table reading itself is not an edge. INSERT INTO t SELECT … FROM t (de-duplication, backfill from its own history) does not record t ← t.
The label belongs to the statement that introduced the source. The stored operation labels the edges in the graph. A statement that adds no new source — such as the self-INSERT above — leaves the existing label (for example CTAS) in place instead of relabelling every edge as INSERT_INTO.

For example:

-- Trino (via the chango plugin):  CREATE TABLE ice.s.orders AS SELECT … FROM tpch.tiny.orders
-- Ontul:                          INSERT INTO ice.s.orders SELECT … FROM pg.public.orders_backfill
-- Ontul:                          INSERT INTO ice.s.orders SELECT … FROM ice.s.orders WHERE …
SHOW LINEAGE ice.s.orders;
-- tpch.tiny.orders
-- pg.public.orders_backfill

Column-level lineage is stored per target and reflects the most recent statement that wrote the table.

Table-Level Lineage

Table-level lineage records the set of source tables a table was built from — every table scanned by the defining query, across joins and unions.

CREATE TABLE iceberg_catalog.demo.cust_orders AS
SELECT c.c_name,
       count(o.o_orderkey) AS order_count,
       sum(o.o_totalprice) AS total_spend
FROM tpch.tiny.customer c
JOIN tpch.tiny.orders o ON c.c_custkey = o.o_custkey
GROUP BY c.c_name;

SHOW LINEAGE iceberg_catalog.demo.cust_orders;
-- tpch.tiny.customer
-- tpch.tiny.orders

Column-Level Lineage

For each output column, Ontul records the source (table, column) it derives from. A column that passed through an expression — SUM(...), CAST(...), an aggregate — is flagged derived; a straight projection of a base column is not. Column origins come from Calcite column-origin metadata, so they are tracked correctly through joins and GROUP BY.

SHOW LINEAGE COLUMNS iceberg_catalog.demo.cust_orders;
-- c_name       ←  tpch.tiny.customer.c_name
-- order_count  ←  tpch.tiny.orders.o_orderkey      [derived]
-- total_spend  ←  tpch.tiny.orders.o_totalprice    [derived]

SQL Commands

Command	Result
`SHOW LINEAGE <table>`	Upstream source tables
`SHOW LINEAGE COLUMNS <table>`	Column-level lineage edges
`SHOW LINEAGE DOWNSTREAM <table>`	Tables built from this one

REST API

Read-only lineage endpoints (any authenticated user):

Endpoint	Returns
`GET /admin/lineage/graph?root=<table>&depth=<n>&direction=<up\\|down\\|both>`	Lineage graph `{nodes, edges}`
`GET /admin/lineage/columns?table=<table>`	Column-level lineage
`GET /admin/lineage/impact?table=<table>`	Transitive downstream tables
`GET /admin/lineage/pii-flow?table=<table>&column=<col>`	Downstream flow of a column

External-Engine Lineage Ingestion

Ontul is not only the source of lineage — it is also a collection point for other engines. Trino, Spark, and Flink authorization plugins (as deployed by Chango) already route their access-control decisions through Ontul IAM; the same plugins push the lineage edges they observe into Ontul over a REST endpoint, so lineage from every engine lands in one replicated store.

The endpoint lives at the server root, not under /admin:

POST /v1/api/lineage
Authorization: Token <ontul-token>        # or: Bearer <jwt> / AccessKey
Content-Type: application/json

{
  "target":    "lake.curated.orders_daily",
  "sources":   ["hive.sales.orders", "hive.sales.customers"],
  "operation": "INSERT"
}

Response: {"status":"ok"}; target is required (400 otherwise). operation defaults to EXTERNAL. The plugins send CTAS for CREATE TABLE … AS and INSERT for inserts, so the accumulation rules apply to their edges exactly as to Ontul's.

The ingested edge is stored and queried exactly like a natively-captured one — it appears in SHOW LINEAGE, the graph/impact/PII-flow endpoints, and the Admin UI graph, regardless of which engine produced it. This is the lineage half of Ontul's cross-engine Audit Log; the audit half records who / which engine / which query for the same operation.

Admin UI

The Data Lineage page renders an interactive graph (sources on the left, derived tables on the right). Enter a table, choose direction and depth, and trace its lineage. Click any node to open a side panel showing its column-level lineage — with the derived flag — and its downstream impact.

Each edge is labelled with the operation that introduced it (CTAS, INSERT_INTO, MERGE_INTO, VIEW, …). The page can be opened directly by URL (/admin/lineage) or reloaded.

The Audit → Overview & Graph page shows a different, complementary graph — user → engine → table built from audit events — see Audit Log.

PII Flow Detection

Given a sensitive column, Ontul traces every downstream (table, column) it transitively flows into, via column-level lineage:

GET /admin/lineage/pii-flow?table=pg.public.users&column=ssn

This answers "where did this PII go?" — for example, finding that users.ssn was copied by an ETL job into a table that has no access policy. Ontul surfaces the flow for an operator to review; it does not automatically modify IAM policies, since lineage is not exhaustive (opaque UDFs, dynamic SQL) and silent policy changes would be surprising.

Cross-Catalog Lineage

Because every catalog's tables — Iceberg, JDBC, Kafka, TPC-H — are first-class scan sources, lineage crosses systems automatically. A table built FROM jdbc_catalog.public.users JOIN kafka_catalog.events.clicks records both as upstream sources, giving cross-system lineage with no extra configuration.

OpenLineage Emission

Ontul can emit OpenLineage-spec events so external catalogs (Marquez, DataHub, Atlan, …) ingest Ontul lineage through the industry-standard wire format.

Set ontul.lineage.openlineage.url to an HTTP endpoint; each CTAS/INSERT/MERGE/VIEW — and each streaming Ontul Flow start — then emits a COMPLETE RunEvent with inputs, outputs, and a standard columnLineage facet. When unset, emission is disabled.

Storage

Lineage is persisted in the leader's metadata store and replicated to followers — the same mechanism used for catalogs and IAM — so it survives restarts and leadership changes. An edge recorded on a follower master (the statement or Flow start was routed there) is forwarded to the leader and replicated back, the same way as audit records.