Configuration

Ontul reads its configuration from conf/ontul.properties. Every property can also be overridden at runtime, and the same keys apply to both the Master and the Worker (each node simply ignores the sections that do not concern it).

Configuration Precedence

A value is resolved in the following order, highest priority first:

1. System Property — passed with -D, e.g. -Dontul.master.admin.port=8081
2. Environment Variable — see the naming convention below
3. Properties file — conf/ontul.properties

If a key is set in more than one place, the higher-priority source wins.

Environment Variable Naming

To set any property through an environment variable, take the property key, replace dots with underscores, and uppercase it:

Property	Environment Variable
ontul.master.admin.port	ONTUL_MASTER_ADMIN_PORT
ontul.zk.serverList	ONTUL_ZK_SERVERLIST
ontul.base.data.dir	ONTUL_BASE_DATA_DIR

Many path properties default to a subdirectory of ${ontul.base.data.dir} (for example kms/, iam/, metadata/, exchange/, job-history/). Changing the base data directory moves all of them at once unless they are overridden individually.

Persist the data directory

In containers ontul.base.data.dir MUST resolve to a mounted volume (e.g. /app/data). Otherwise all cluster state — KMS keys, IAM, metadata — is lost on restart.

Base

Property	Default	Description
ontul.base.data.dir	./data	Root directory for all node-local persistent state. Most other path properties default to a subdirectory of this via ${ontul.base.data.dir}. Relative paths resolve against the process working directory.

ZooKeeper

Property	Default	Description
ontul.zk.serverList	localhost:2181	Comma-separated ZooKeeper connect string (host:port[,host:port...]) used for cluster coordination: master leader election, live worker registry, and shared config watches. All masters and workers must point at the same ensemble.
ontul.zk.rootPath	/ontul	Base znode namespace under which Ontul creates all its coordination nodes. Change this to run multiple independent clusters on one ZK ensemble.
ontul.zk.sessionTimeoutMs	30000	ZooKeeper session timeout (ms). If a node cannot heartbeat within this window the session expires, the leader loses leadership, and workers are deregistered. Larger values tolerate GC/network blips but slow failover.
ontul.zk.connectionTimeoutMs	10000	Maximum time (ms) to wait for the initial TCP connection to ZooKeeper before giving up on a connect attempt. Affects startup and reconnect latency.
ontul.zk.admin.probeTimeoutMs	2000	Admin UI topology: socket timeout (ms) for asking each ZooKeeper server its status with the srvr command. A server that does not answer in time is shown Offline.
ontul.zk.admin.probeCacheTtlMs	5000	How long (ms) one probe of the ensemble is reused for topology requests, so open admin pages do not add load to ZooKeeper. 0 probes on every request.

Master

Property	Default	Description
ontul.master.host	0.0.0.0	Interface the Master binds to. 0.0.0.0 binds all interfaces. Also advertised to clients as the JDBC/Flight SQL host (unless ontul.flight.sql.advertised.host overrides it), so behind NAT or a proxy a routable address may be required.
ontul.master.admin.port	8080	TCP port for the Master's Admin HTTP server (REST API + Admin UI) — the main management/control-plane endpoint.
ontul.master.internal.port	19999	TCP port for the Master's internal NIO control-plane server, used for master↔worker control messaging (not bulk data, which goes over Arrow Flight).
ontul.master.admin.worker.threads	8	Size of the thread pool serving Admin HTTP requests. Raise for higher concurrent admin/REST load; each thread handles one request at a time.
ontul.master.admin.ui.static.path	./admin-ui	Filesystem directory containing the pre-built Admin UI static assets (HTML/JS/CSS) served by the Admin HTTP server. Relative to the process working directory.
ontul.master.flight.sql.port	47470	TCP port for the Master's Arrow Flight SQL server — the unified JDBC endpoint (jdbc:arrow-flight-sql://host:port) and the SDK data endpoint.
ontul.master.broadcast.timeout.ms	5000	Timeout (ms) for broadcasting a control message from the Master to all workers (e.g. config/key sync). A worker that does not respond within this window is treated as not acknowledging.
ontul.master.leader.deference.window.ms	3000	Sticky-incumbent leader election: on restart, defer this many ms before joining the election if a previous leader is still alive, so it can reclaim leadership without a leadership toggle. Set to 0 to disable.

Worker

Property	Default	Description
ontul.worker.host	0.0.0.0	Interface this Worker binds to and the host it registers under in the worker registry. The registered host:port (using the internal port) becomes the worker's nodeId.
ontul.worker.internal.port	19998	TCP port for this Worker's internal NIO control-plane server (master↔worker and worker↔worker control messaging). Also forms the worker's nodeId (host:port); each worker on the same host needs a distinct value.
ontul.worker.flight.port.offset	1000	Offset added to ontul.worker.internal.port to derive the Worker's Arrow Flight data port (flightPort = internal.port + offset). With the defaults the Flight port is 19998 + 1000 = 20998.
ontul.worker.shutdown.drain.timeout.seconds	60	Graceful-shutdown drain timeout (seconds). On shutdown the Worker stops accepting new work and waits up to this long for in-flight tasks to finish before forcing termination.

KMS

Property	Default	Description
ontul.kms.enabled	true	Master switch for the built-in Key Management Service. When true the KMS provides envelope-encryption keys for metadata, spill data, internal protocol payloads, etc. Disabling turns off at-rest and in-transit envelope encryption that depends on KMS keys.
ontul.kms.rocksdb.path	${ontul.base.data.dir}/kms	Filesystem path to the embedded RocksDB store holding wrapped (encrypted) data encryption keys. Must persist across restarts; losing it makes all KMS-encrypted data unrecoverable.
ontul.kms.master.key.env	ONTUL_MASTER_KEY	Name of the environment variable (not the key value) that holds the KMS master passphrase/key. The master key unwraps all data encryption keys and seeds JWT signing. The named env var must be set identically on every node.
ontul.kms.pbkdf2.iterations	200000	PBKDF2 iteration count used to derive the KMS master key from the passphrase. Higher is stronger against brute force but slower at startup. Must be identical on all nodes or derived keys will not match.
ontul.kms.encrypt.internal.protocol	true	When true, internal NIO control-protocol message payloads are KMS envelope-encrypted in transit between master and workers using the key id below.
ontul.kms.internal.protocol.key.id	internal-protocol	KMS key id used to encrypt internal NIO protocol payloads. Only meaningful when ontul.kms.encrypt.internal.protocol=true. The Master ensures this key exists.
ontul.kms.fetch.max.retries	60	Maximum retries a Worker performs when fetching KMS keys from the Master at startup (the Master may not have finished key init). Combined with the retry interval this bounds total wait (~60s default).
ontul.kms.fetch.retry.interval.ms	1000	Delay (ms) between successive KMS key-fetch retry attempts (see ontul.kms.fetch.max.retries).

IAM

Property	Default	Description
ontul.iam.rocksdb.path	${ontul.base.data.dir}/iam	Filesystem path to the embedded RocksDB store for IAM state (users, access/secret keys, roles, policies). Must persist across restarts.
ontul.iam.admin.user	admin	Username of the bootstrap admin account, created on first startup if no users exist. Used to log into the Admin UI/REST API initially.
ontul.iam.admin.password	admin	Initial password for the bootstrap admin account (used only when the account is first created). Change this in production; afterwards rotate via the Admin UI or bin/ontul-cli.sh iam:reset-password.
ontul.iam.audit.dir	${ontul.base.data.dir}/iam-audit	Directory for the append-only IAM/security audit log (authentication attempts, access-key and password changes, role/policy edits). Entries are pruned per ontul.audit.log.retention.days. Must persist across restarts.
ontul.admin.socket.enabled	true	Whether the Master exposes the out-of-band Unix domain socket used for local admin recovery (bin/ontul-cli.sh iam:reset-password). The socket file's OS permission (mode 600) is the only authentication. Set false to remove the local recovery path. See Admin Password Recovery.
ontul.admin.socket.path	${ontul.base.data.dir}/admin.sock	Filesystem path of the admin recovery Unix domain socket. Must live on a local filesystem that supports Unix domain sockets (not a networked mount); recreated on each Master start.
ontul.admin.socket.marker.file	master.socket	Name of the file under <ontul.home>/bin where the Master writes the socket path it actually bound to. bin/ontul-cli.sh prefers that published path over re-deriving one from this file, since ontul.base.data.dir can be overridden with -D at launch. Removed on shutdown.

Metadata

Property	Default	Description
ontul.metadata.rocksdb.path	${ontul.base.data.dir}/metadata	Filesystem path to the embedded RocksDB store holding cluster metadata: catalog definitions, connections, table maintenance config, storage settings, etc. Must persist across restarts.
ontul.metadata.kms.key.id	metadata-encryption	KMS key id used to envelope-encrypt metadata at rest in the RocksDB store. Requires KMS to be enabled.
ontul.metadata.forward.queue.capacity	10000	Metadata replicates leader → followers only, so a write made on a follower (audit record, lineage edge, Flow status, …) is also forwarded to the leader, asynchronously and in order. This is the most writes that may wait to be forwarded; beyond it, a write stays on that follower only and a warning is logged.
ontul.metadata.forward.max.attempts	5	Attempts to forward one follower write while the leader is unreachable or an election is in progress (pause 1 s, 2 s, … between attempts) before giving up on it.

RPC Timeouts

Property	Default	Description
ontul.query.rpc.timeout.ms	30000	Timeout (ms) for query-related RPCs the Master issues to Workers (e.g. dispatching a query stage and awaiting its result). Raise for long-running stages over slow links.
ontul.internal.rpc.timeout.ms	10000	Timeout (ms) for internal (non-query) control-plane RPCs, e.g. small master↔worker coordination calls. Keep relatively short.

Semantic Layer / Retrievers / Ontology

Property	Default	Description
ontul.retriever.max.rows.ceiling	1000	Hard cluster-wide cap on the number of rows a single retriever invocation (POST /api/v1/retrievers/{fqn}/invoke) may return. Applied on top of each retriever's own maxRowsCeiling, so the effective cap is the smaller of the two. Lets an operator bound retriever result size globally regardless of per-retriever definitions.
ontul.rerank.enabled	true	Master kill switch for retriever re-ranking. false makes every retriever's rerank block report applied=false, reason=disabled-cluster-wide — an operator can drop re-ranking during an incident without editing any retriever.
ontul.rerank.max.timeout.ms	5000	Upper bound on a retriever's rerank.timeoutMs. The re-rank call sits inside the synchronous invoke, so this is also the worst-case latency a caller pays before falling back to the backend's ordering. Keeping it low is what stops a generative LLM — seconds per call, prose output, needs retries — from being wired into a retriever; that belongs in an agent loop.
ontul.rerank.min.timeout.ms	50	Lower bound on rerank.timeoutMs. A budget below this is almost always a typo (a 5 meant as seconds), and it would make every call time out while looking like the endpoint is at fault.
ontul.rerank.max.candidates	200	Upper bound on rerank.maxCandidates: how many rows may be sent to the endpoint in one request. Bounds endpoint load and payload size regardless of the maxRows a caller asks for.
ontul.rerank.max.text.chars	4000	Upper bound on rerank.maxTextChars, the per-document truncation. Cross-encoders take a fixed window (commonly 512 tokens) and silently drop the tail, so this caps how much text may be sent per row before that happens invisibly.
ontul.rerank.http.connect.timeout.ms	1000	TCP connect timeout for the re-ranker endpoint, separate from the per-call budget: an unreachable host fails fast instead of burning the whole request timeout.
ontul.rerank.default.path	/rerank	Request path appended to a RERANK connection's baseUrl when that connection does not set its own path. Change it only if every endpoint in the deployment mounts the API elsewhere — per-endpoint differences belong on the connection.
ontul.ontology.read.max.rows.ceiling	10000	Hard cluster-wide cap on rows returned by an ontology read — an ObjectSet query (POST /api/v1/object-types/{fqn}/query) or a link traversal (POST /api/v1/link-types/{fqn}/traverse). Applied on top of each request's own limit, so the effective cap is the smaller of the two. Mirrors the retriever ceiling: it bounds what an agent can pull through the ontology in one call regardless of what it asks for.
ontul.action.exec.timeout.ms	600000 (10 min)	Time budget for a single action type invocation in DML mode — the governed write-back executed through the query engine. A write that exceeds it is abandoned rather than holding the invoke connection open indefinitely.
ontul.action.http.timeout.seconds	60	Per-request timeout for an action in OPERATION mode, i.e. the outbound HTTP call to the rest-operation connector (ERP/SaaS write-back). Keep it well under ontul.action.exec.timeout.ms; a slow ERP should surface as a failed operation, not as a stuck action.

Worker Health Check

Property	Default	Description
ontul.worker.health.check.interval.ms	10000	Interval (ms) between rounds of the Master's worker health checker. Each round pings every registered worker. Lower means faster failure detection but more background traffic.
ontul.worker.health.check.timeout.ms	5000	Per-worker health-check ping timeout (ms). A ping that does not return within this window counts as one failure for that worker.
ontul.worker.health.max.consecutive.failures	3	Number of consecutive failed health checks before a worker is declared down and removed from the cluster (its in-flight work is rescheduled/failed). With the defaults a worker is evicted after ~3 missed 10s rounds.
ontul.worker.health.check.future.grace.ms	2000	Extra grace (ms) added on top of the per-check timeout when waiting for each Future result during a health-check round. Prevents premature TimeoutException for checks about to return.

Catalogs

Property	Default	Description
ontul.catalog.reload.retry.seconds	30	A catalog that cannot be loaded (its REST catalog or that catalog's database is down when a master starts or becomes leader) is kept as unavailable: queries on it fail with Catalog '<name>' is unavailable: <reason> and GET /admin/catalogs shows available: false. Every master retries it at this interval until it loads. POST /admin/catalogs/<name>/reload retries at once. 0 turns the timer off. See Catalog availability.

Query

Property	Default	Description
ontul.query.history.max.size	200	Maximum number of finished queries retained in the in-memory query history (shown in the Admin UI). Oldest entries are evicted once the cap is reached.
ontul.query.max.global.concurrency	100	Cluster-wide admission limit: max queries allowed to run concurrently across all users. Enforced by a global semaphore; excess queries wait (see acquire timeout) then are rejected.
ontul.query.max.per.user.concurrency	10	Per-user admission limit: max concurrent queries a single IAM identity may run. Prevents one user from starving others within the global limit.
ontul.query.default.timeout.ms	300000	Default per-query execution timeout (ms, 5 minutes) when a query/job does not specify its own. Long-running queries exceeding this are cancelled.
ontul.query.max.memory.bytes	1073741824	Soft memory budget (bytes) per query for in-memory (Arrow) execution before operators spill intermediate data to disk via the exchange manager. Default is 1 GiB.
ontul.query.semaphore.acquire.timeout.seconds	30	Maximum time (seconds) a query waits to acquire a concurrency admission slot before it is rejected as overloaded.
ontul.iam.operation.permission.enforced	false	Whether a missing ontology:InvokeOperation grant BLOCKS an ontology action that writes to an EXTERNAL system, or is only reported. The grant's resource namespace is operation:<operationCatalog>:<operation> — separate from data:table:… used for reads, so broad read access cannot imply the right to post into an ERP. Report-only by default so upgrading does not revoke working actions: the missing grant is logged and audited as action:invoke:grant-missing with the exact resource, then set this to true to enforce. Applies to both the REST invoke endpoint and SQL CALL.
ontul.sql.strict	true	SQL capability gate. The master validates every planned query and rejects constructs the execution engine cannot evaluate correctly (window functions, subqueries, INTERSECT/EXCEPT/de-duplicating UNION, COUNT(DISTINCT), OFFSET, composite/non-equi join conditions, unimplemented scalar functions, …) with an explicit error. Set to false ONLY to triage a regression: it restores the previous behaviour where those queries run and return wrong results. DELETE/UPDATE/MERGE predicates are never gated. See SQL Capability Gate.

Connector

Property	Default	Description
ontul.connector.batch.size	4096	Number of rows per Arrow record batch when connectors read from / write to external sources (the vector batch size). Larger batches improve throughput and vectorization but use more memory per in-flight batch.

Iceberg Catalog (REST)

Server-wide defaults for Iceberg REST catalogs. Every value is a default: a catalog registered through the Admin UI / POST /admin/catalogs can override it with its own catalog.rest.* config. These apply on both the Master (interactive query) and Worker (job source/sink) code paths. See Iceberg Integration for per-catalog config.

Property	Default	Description
ontul.iceberg.polaris.default.scope	PRINCIPAL_ROLE:ALL	Default OAuth2 scope requested from a Polaris REST catalog when a catalog does not set catalog.rest.scope. Ignored for the glue flavor (not OAuth2).
ontul.iceberg.glue.signing.name	glue	AWS SigV4 signing service name for the AWS Glue Iceberg REST endpoint. Use s3tables when pointing at the Amazon S3 Tables REST endpoint.
ontul.iceberg.default.region	us-east-1	Default AWS region used whenever none is otherwise specified — for S3 data access (when s3.region is absent) and for SigV4-signing Glue REST requests (when neither catalog.rest.signing.region nor s3.region is set). The single fallback for every Iceberg AWS region.
ontul.iceberg.rest.credential.vending.bypass	true	Suppress catalog-vended S3 credentials (Polaris STS-subscoped tokens / Glue Lake Formation) so Ontul reads and writes S3 directly with the configured static keys, keeping Ontul IAM the authoritative access boundary. Leave true (the only tested path; S3-compatible servers reject the STS temp keys Polaris would otherwise vend).
ontul.iceberg.default.format.version	2	Default Iceberg table format version for newly created tables. 2 = position-delete files; 3 = deletion vectors (Puffin), row lineage, variant. Opt-in: existing tables keep their own version and the library reads/writes all versions. On 3, DELETE/UPDATE/MERGE write deletion vectors instead of position-delete files. See Iceberg Integration.

Cluster Maintenance Mode

Unrelated to Iceberg Maintenance below, which is table compaction and keeps running during a window. See Cluster Maintenance Mode.

Property	Default	Description
ontul.cluster.maintenance.retry.after.seconds	30	How long a client is told to wait (Retry-After, seconds) before retrying a write that a maintenance window refused. Set it to roughly how long a window usually lasts: too low and clients hammer a cluster that is deliberately closed, too high and they stay away well after it reopens. The window itself is a runtime switch stored in the replicated metadata store, not a property — toggle it from Cluster Topology in the Admin UI or POST /admin/cluster/maintenance.

Iceberg Maintenance

Property	Default	Description
ontul.maintenance.interval.hours	6	Cluster-wide default interval (hours) between automatic Iceberg table maintenance runs (expire snapshots + rewrite/compact data files). Runs on the Master leader. Individual tables can override this.
ontul.maintenance.snapshot.retention.hours	168	Default snapshot retention (hours) for expire-snapshots: snapshots older than this are removed during maintenance (default 7 days). Expired snapshots can no longer be time-travelled to.
ontul.maintenance.target.file.size.mb	128	Default target output file size (MB) for the data-file rewrite/compaction step (passed as file_size_threshold_mb to OPTIMIZE). Larger means fewer, bigger files and fewer small-file reads.
ontul.maintenance.initial.delay.ms	60000	Initial delay (ms) after Master startup before the first maintenance round runs, so the cluster can finish coming up first (default 60s).
ontul.maintenance.history.retention.days	30	Retention (days) for the maintenance job history (records of past expire/compaction runs). Older entries are pruned.

Iceberg Observability (Table Health)

Continuous collection of Iceberg table health behind the Admin UI's Iceberg Health page, the /admin/iceberg/health REST surface, and the ontul_iceberg_table_* Prometheus gauges. Collection is a cluster singleton — only the Master leader walks the catalogs, because table health is a property of the table rather than of the master that read it. See Iceberg Observability.

Property	Default	Description
ontul.iceberg.observability.enabled	true	Master collects Iceberg table health. Set false to disable the Iceberg Health page, the REST surface, and the per-table gauges.
ontul.iceberg.observability.fast.interval.seconds	60	Fast-tier period. The fast tier reads only the already-loaded metadata.json — current snapshot summary, snapshot list, metadata log, manifest list — so it performs no manifest I/O and is safe to run often.
ontul.iceberg.observability.deep.interval.seconds	1800	Deep-tier period, per table. The deep tier plans a file scan to build the per-file size distribution (small-file ratio, percentiles, histogram); it reads every manifest, so it runs rarely and its result is cached and carried forward.
ontul.iceberg.observability.deep.max.per.cycle	3	Deep scans started per collection cycle. Bounds the manifest I/O a freshly elected leader performs when it holds no cached distribution for any table yet — remaining tables are picked up on later cycles. A scan requested from the UI bypasses this cap.
ontul.iceberg.observability.series.retention.seconds	86400	How long the per-table time series is kept in memory (default 24h). It backs the trend chart only; it is not persisted and does not survive a leader failover.
ontul.iceberg.observability.small.file.divisor	4	A data file smaller than targetFileSizeMB / N counts as "small". The divisor applies to each table's own maintenance target, so a table configured for 512 MB files is judged against 512 MB, not a global constant.
ontul.iceberg.observability.effects.max	500	How many maintenance before/after records to keep in memory. Every maintenance run is bracketed with a health sample so the UI can show what the run actually changed.

Audit Log

Property	Default	Description
ontul.audit.log.retention.days	90	Retention (days) for the query/access audit log and IAM/security entries — the absolute local cap; records older than this are pruned from the local store whether or not tiering is on. Editable at runtime from the Admin UI.
ontul.audit.read.enabled	true	Whether SELECT/read queries are audited (who ran which query). Volume is bounded by the sample rate.
ontul.audit.read.sample.rate	1.0	Fraction (0.0–1.0) of read queries recorded when read auditing is on.
ontul.audit.tier.mode	none	Audit tiering target: none (local only), iceberg (recommended), or parquet (fallback). Older records are offloaded to durable, queryable S3.
ontul.audit.tier.iceberg.table	ice.ops.audit_events	Target Iceberg table (catalog.schema.table) for tier.mode=iceberg. Written as a format-v2 table partitioned by day(event_time) via Ontul's in-process Iceberg writer; the namespace and table are created if missing. A target set from the Admin UI is validated on save (HTTP 400 if unusable).
ontul.audit.tier.parquet.path	(empty)	Target s3:// prefix for the tier.mode=parquet fallback.
ontul.audit.tier.connectionId	(empty)	S3 connection id (from the Connection registry) used by the tier.mode=parquet fallback.
ontul.audit.tier.retain.local.days	7	Days a tiered record is also kept locally as a hot copy (durable history lives in S3). Only records already tiered and older than this are pruned; 0 keeps only untiered records locally. The Admin UI still shows Iceberg-tiered history.
ontul.audit.tier.interval.ms	300000	How often the tiering sweep runs (ms).
ontul.audit.tier.batch.size	5000	Max records tiered per sweep. The sweep runs on the leader only and selects records by ingest time.
ontul.audit.tier.ingest.settle.ms	2000	Records ingested within this many ms of a sweep wait for the next one, so a record stored in the same millisecond as the scan is not stranded behind the watermark.
ontul.audit.viewer.max.page	5000	Max events one GET /admin/audit request (Admin UI Audit pages) may return; each request merges the local store with a scan of the Iceberg-tiered tables.
ontul.audit.tier.maintenance.enabled	true	When tiering→iceberg, auto-register the audit table for periodic Iceberg Table Maintenance (compaction + expire-snapshots + orphan cleanup).
ontul.audit.tier.maintenance.cron	0 * * * *	5-field UNIX cron for the audit table's maintenance (default hourly).
ontul.audit.tier.maintenance.snapshot.retain.last	24	expire_snapshots retain-last for the audit table — always keep at least this many recent commits.
ontul.audit.tier.maintenance.remove.orphan.files	true	Run remove_orphan_files on the audit table (reclaims files left by compaction/expiry).

See Audit Log for the full feature description.

Job

Property	Default	Description
ontul.job.history.dir	${ontul.base.data.dir}/job-history	Directory where submitted-job history and job logs are stored. Accepts a local path or an S3 path (e.g. s3://bucket/ontul/job-history). If a jobLogs S3 storage connection is configured in the Admin UI, that S3 location overrides this value.
ontul.job.large.row.threshold	100000	Row-count threshold above which a job's result set is treated as "large" and handled via the streaming/spill path rather than buffered fully in memory.
ontul.driver.sync.chunk.size.bytes	4194304 (4 MiB)	Chunk size used when the master streams an uploaded JDBC driver JAR out to the workers. Larger chunks mean fewer round-trips on a fast network; smaller chunks keep each NIO frame (and its retry cost) small on a lossy one.
ontul.deps.parent.first.prefixes	(empty)	Extra package prefixes kept parent-first in the child-first classloader used for UDFs and job dependencies, beyond the built-in JDK / Arrow / Ontul / slf4j set. Comma-separated. Use it when a job JAR ships its own copy of a library that must stay identical to the engine's (otherwise you get ClassCastException between two copies of the same class).

Flight SQL

Property	Default	Description
ontul.flight.sql.advertised.host	(commented / optional)	Specific hostname advertised for nginx-safe result streaming. When behind nginx, set to this Master's direct IP/hostname. Leave empty/unset to reuse the same connection (works without nginx).

Logging

Property	Default	Description
ontul.log.path	${user.dir}/logs	Directory where log files are written (defaults to a logs folder under the process working directory). Used by the logback configuration to place per-node log files.
ontul.log.output.name	(commented / optional)	Base name used when naming this node's log/stdout file (e.g. master or worker). When unset, workers fall back to a name derived from their internal port (worker-<port>.out).

Exchange Manager

The Exchange Manager handles spill of intermediate shuffle/exchange data and execution-state checkpoints (streaming snapshots). It has two storage modes, selected by ontul.exchange.s3.enabled, and all payloads are KMS envelope-encrypted at rest in both modes.

• Local-disk mode (default). Data is written to — and read from — a worker-local directory. This is fast, but the data is tied to that worker's disk: a stage that is retried on a different worker cannot see it.
• S3-primary mode (ontul.exchange.s3.enabled=true). Data is read and written directly to S3 (the object store is the primary exchange storage, not a background copy). Because the spill/snapshot objects live in S3, a task that is reassigned to a different worker reads the same data from S3 — this is what enables worker-independent, fault-tolerant recovery. Object keys are worker-independent ({prefix}spill/{stageId}/…, {prefix}checkpoints/{stateId}/chk-{id}/w{worker}), so any worker configured with the same bucket and prefix resolves them.

Streaming checkpoints are stored per job under checkpoints/{stateId}/ (one part per worker plus a _latest pointer to the last globally completed checkpoint) and pruned per job; the startup orphan sweep never touches them. Payloads are encrypted with the KMS key exchange-spill, which the leader master creates at startup and sends to the workers. A worker that does not have the key refuses to write exchange data (the query fails and says why) rather than writing plaintext, unless ontul.exchange.encryption.enabled=false. Every stored object carries a format marker (OXE1 envelope-encrypted, OXP1 plaintext, written only with encryption turned off), so a reader never has to guess. A streaming job keeps its window state and offsets across a worker failure only in S3-primary mode — in local-disk mode a checkpoint lives on the disk of the worker that wrote it.

Note

S3-primary mode is analogous to a filesystem/object-store-backed exchange manager (exchange.base-directories=s3://…) in engines that support fault-tolerant execution: the object store replaces local disk for exchange data — it is not merely a backup of it.

Property	Default	Description
ontul.exchange.base.dir	${ontul.base.data.dir}/exchange	Worker-local directory used in local-disk mode for spilling intermediate shuffle/exchange data and execution-state checkpoints. Scratch/spill space (safe to be node-local); KMS envelope-encrypted at rest. Ignored in S3-primary mode.
ontul.exchange.encryption.enabled	true	true: exchange data (both modes) is always KMS envelope-encrypted with the key exchange-spill; a worker without the key fails the write instead of storing plaintext. false: plaintext is written when the key is missing, with a WARN in the worker log.
ontul.exchange.s3.enabled	false	false selects local-disk mode. true selects S3-primary mode, where spill and snapshot data is read and written directly to S3 (not local disk, and not a background backup).
ontul.exchange.s3.endpoint	(commented / optional)	S3 endpoint URL for the exchange bucket. Set for S3-compatible stores (e.g. MinIO http://localhost:9000); leave unset to use the AWS default endpoint.
ontul.exchange.s3.region	(commented / optional, default us-east-1)	AWS region for the exchange bucket.
ontul.exchange.s3.bucket	(commented / optional, default ontul-exchange)	S3 bucket name used to store exchange objects.
ontul.exchange.s3.prefix	exchange/	Key prefix under which exchange spill/snapshot objects are stored inside the bucket, so multiple clusters can share one bucket. Trailing slash optional.
ontul.exchange.s3.path.style	(commented / optional, default true)	Use path-style S3 addressing (bucket in the URL path) instead of virtual-hosted style. Required by most S3-compatible servers like MinIO.
ontul.exchange.s3.access.key	(commented / optional)	Static access key for the exchange S3 bucket. Leave empty to use the default AWS credential provider chain (env/instance profile/etc).
ontul.exchange.s3.secret.key	(commented / optional)	Static secret key paired with the access key above.
ontul.exchange.writer.threads	2	Number of background writer threads the exchange manager uses for spill and state-snapshot writes in local-disk mode. More threads means more parallel disk I/O for spill-heavy or fault-tolerant workloads.
ontul.exchange.s3.orphan.ttl.ms	86400000	S3-primary mode only. On worker startup, sweep exchange objects older than this TTL (ms) — orphans left by crashed jobs whose stage cleanup never ran. 0 disables the sweep. Default 24h.
ontul.exchange.s3.multipart.threshold.bytes	67108864	S3-primary mode only. Object size (bytes) above which uploads switch to multipart — avoids the 5 GB single-PUT limit and improves throughput for large spills. Default 64 MiB.
ontul.exchange.s3.multipart.part.size.bytes	16777216	S3-primary mode only. Part size (bytes) for multipart uploads (S3 minimum is 5 MiB). Default 16 MiB.

Streaming

Server-wide defaults for streaming jobs (Streaming Guide). Where a row names a "job override", that key in the SUBMIT STREAMING config wins for that job; the other rows are cluster-wide only.

Property	Default	Description
ontul.streaming.checkpoint.interval.ms	10000	Interval (ms) between checkpoints — the commit point of exactly-once sinks and the restore point after a failure. Job override: checkpointIntervalMs (an older job that sets only commitIntervalMs uses that).
ontul.streaming.checkpoint.timeout.ms	30000	A checkpoint not acknowledged by every worker within this time is aborted: it is never a restore point, and Iceberg files that did arrive are committed with the next checkpoint. Job override: checkpointTimeoutMs.
ontul.streaming.checkpoint.send.timeout.ms	5000	Timeout (ms) for delivering a checkpoint barrier to one worker.
ontul.streaming.checkpoint.tolerable.failures	3	Consecutive failed commits of a checkpoint to the Iceberg sink (by the master's single committer) tolerated before the job fails and restarts from the last completed checkpoint. A transient catalog or S3 error is retried with the next checkpoint; a persistent one fails the job instead of stalling it.
ontul.streaming.checkpoint.max.empty.commits	10	On an idle stream the coordinator still stamps an empty Iceberg commit (checkpoint id + source offsets, no data files) after this many consecutive empty checkpoints, so the restore point recorded in the table keeps up. 1 stamps every checkpoint.
ontul.streaming.max.retries	3	Automatic restarts before the job is marked FAILED. Job override: maxRetries.
ontul.streaming.retry.reset.checkpoints	5	A restarted job that completes this many checkpoints gets its retry budget back, so failures weeks apart do not add up.
ontul.streaming.source.poll.timeout.ms	100	Source poll timeout (ms). The executor polls continuously — there is no micro-batch trigger.
ontul.streaming.commit.interval.ms	10000	Flush interval (ms) for non-transactional sinks (Elasticsearch, HTTP, non-transactional Kafka) and for an Iceberg sink with ontul.streaming.exactly.once=false. Source offsets are never committed on this timer — only at a checkpoint. Job override: commitIntervalMs.
ontul.streaming.commit.row.threshold	10000	Row-count flush threshold for the same sinks (whichever of interval/rows comes first). Job override: commitRowThreshold.
ontul.streaming.log.interval.ms	30000	Interval (ms) between streaming progress log lines.
ontul.streaming.exactly.once (JVM system property)	true	Exactly-once Iceberg sink: the master commits every worker's files of a checkpoint in one Iceberg commit stamped with the checkpoint id and offsets. false restores per-worker commits (at-least-once). Job override: ontul.streaming.exactly.once in the job config.

Ontul Flow

Tuning for the named streaming-pipeline registry (Ontul Flow). A running flow's live status + run history are replicated across masters via the metadata store so the Flow list and History tab are consistent regardless of which master serves the request.

Property	Default	Description
ontul.flow.status.persist.interval.ms	5000	How often the owning master persists a running flow's live status + records to the replicated metadata store (flow.status.<name> / flow.history.<name>). Lower = fresher Flow list at the cost of more metadata writes/broadcasts.
ontul.flow.status.freshness.ms	15000	A persisted flow.status snapshot older than this is treated as stale (the flow is shown as not-running until the owner refreshes it). Keep it a few multiples of the persist interval.
ontul.flow.history.max.runs	20	Maximum run records retained per flow in the replicated run history (History tab).

Rejected records, schema drift and lag

A record a flow cannot decode (malformed JSON, undecodable Avro, an empty payload) is rejected: it never reaches the sink, and it is counted. The count is reported on the job status as rejectedRows / rejectsByReason and as the Rejected column in the Admin UI Flow list, so a decode failure is never silent. See Ontul Flow → Data quality.

Property	Default	Description
ontul.flow.reject.log.first	10	Log the first N rejected records of a run in full (reason, error, topic-partition@offset). A poison topic can reject every record, so beyond this the log is sampled. The count is always exact regardless of how much is logged.
ontul.flow.reject.log.every	1000	After the first N above, log every Nth rejected record. Set both to 0 to silence the per-record lines entirely and rely on the counters.
ontul.flow.error.sink.buffer.max.rows	10000	Maximum rejected records buffered in memory for the quarantine (errorSink) table between flushes. Quarantine must never be the reason a worker runs out of memory: past this the records are still counted, but not written, and an error is logged once.
ontul.flow.error.sink.payload.max.chars	65536	Truncate a quarantined payload at this many characters (a …[truncated] marker is appended). Quarantine exists to diagnose and replay the offending record, not to mirror arbitrarily large source messages.
ontul.flow.lag.refresh.interval.ms	5000	How often a source refreshes its lag reading. For Kafka this is a broker round-trip (endOffsets), so it is rate-limited rather than measured on every poll. Lag is reported as sourceLag (records behind) and sourceLagMs (age of the newest record processed).
ontul.flow.schema.evolution	add	Default schema-drift policy for an Iceberg sink when the flow spec does not set sink.schemaEvolution. add — a column the source starts sending is added to the target table, and int→long / float→double are promoted, before the batch is written. none — the table schema wins and the new column is ignored. strict — drift fails the flow instead of guessing. Only additive, read-compatible changes are ever applied; a column the source dropped is left in the table (later rows carry NULL), because one absent field in one batch is not evidence the column is gone.

Job Logs

Streaming and batch job logs can be flushed to append-only rolling gzip segments on S3 ({prefix}jobLogs/<jobId>/<seq>-<ts>.log.gz), so long-running jobs' logs stay durable and queryable by time range after the in-memory log is gone. By default these settings inherit the Exchange Manager's S3 target, so turning on S3-primary exchange also puts job logs on S3.

Property	Default	Description
ontul.job.logs.s3.enabled	(inherits ontul.exchange.s3.enabled)	Enable flushing job logs to S3 rolling segments. Defaults to whatever the exchange S3 mode is.
ontul.job.logs.memory.tail	1000	Number of most-recent log lines kept in memory per job (the rest are flushed to S3 segments).
ontul.job.logs.flush.interval.ms	10000	How often the worker/master drains buffered job-log lines to an S3 segment (time-based roll).

Each of the following overrides the corresponding ontul.exchange.s3.* value for job logs only — useful when logs belong in a different bucket (or a different account) from the exchange state. Leave them unset to inherit.

Property	Default	Description
ontul.job.logs.s3.endpoint	(inherits ontul.exchange.s3.endpoint)	S3 endpoint for job-log segments (e.g. a ShannonStore/MinIO URL).
ontul.job.logs.s3.region	(inherits)	S3 region.
ontul.job.logs.s3.bucket	(inherits)	Bucket holding the job-log segments.
ontul.job.logs.s3.prefix	(inherits)	Key prefix; segments are written under {prefix}jobLogs/<jobId>/<seq>-<ts>.log.gz.
ontul.job.logs.s3.access.key	(inherits)	Access key. Prefer configuring an S3 connection from the Admin UI Storage page so the secret is not in a properties file.
ontul.job.logs.s3.secret.key	(inherits)	Secret key. Same caveat as above.
ontul.job.logs.s3.path.style	(inherits)	true forces path-style addressing (required by most S3-compatible stores); false uses virtual-host style.

The full set can also be overridden with ontul.job.logs.s3.*. The backend can also be switched from the Admin UI Storage page at runtime (no restart), which persists to the replicated metadata store as storage.jobLogs.* / storage.exchange.* referencing an S3 connection.

Admin

Property	Default	Description
ontul.admin.token.access.expiry.ms	900000	Lifetime (ms) of short-lived JWT access tokens issued after login. After expiry the client must use its refresh token to obtain a new access token (default 15 minutes).
ontul.admin.token.refresh.expiry.ms	86400000	Lifetime (ms) of JWT refresh tokens, which let a client mint new access tokens without re-authenticating (default 24 hours). Effectively the maximum idle session length.
ontul.admin.http.max.content.size.bytes	10485760	Maximum accepted Admin HTTP request body size (bytes); larger requests are rejected (default 10 MiB). Raise if uploading large dependency JARs/drivers via the Admin API.

Authentication — Password Storage

Property	Default	Description
ontul.auth.password.hash.iterations	600000	PBKDF2-HMAC-SHA256 iterations used when a local password is written. The count travels with each stored hash, so raising this does not invalidate existing passwords — they are rewritten at the new cost on their owner's next successful login. Values written under the previous unsalted SHA-256 verify the same way and are upgraded identically, so nobody is locked out.

Single Sign-On (OIDC / SAML / LDAP)

Every setting below can also be managed from the admin console under Single Sign-On, which stores it in the replicated metadata store and applies it on every master with no restart. Stored settings win over this file: the file brings a cluster up, and the console is how it is changed afterwards — if the file won, a console change would be reverted by the next restart, silently. See Single Sign-On for how the console and the REST surface differ.

Identity mapping (all three providers)

Property	Default	Description
ontul.sso.group.mappings	(empty)	idpGroup:ontulGroup pairs, comma-separated. Empty means provider group names are used as they are. Once set the mapping is exhaustive — a group not named here is dropped, so creating a group at the provider cannot grant access on this cluster by itself.
ontul.sso.allow.unmapped.groups	false	Whether an identity whose groups all map to nothing may still sign in. Off deliberately: such a session has no policies and is denied every action, so admitting it produces a caller who is signed in and can do nothing. The directory login endpoint reports that case as 403, separately from a wrong password's 401.
ontul.sso.federated.session.seconds	3600	Lifetime of a federated session — the record that lets /v1/api/authz/check resolve a federated caller's groups. It bounds how long access outlives a revocation at the provider, which this cluster is not told about. No refresh token is issued for a federated session for the same reason.

OpenID Connect

Property	Default	Description
ontul.sso.oidc.enabled	false	Enable the OIDC provider.
ontul.sso.oidc.issuer	(empty)	Issuer URL. Endpoints and the signing key set are read from its discovery document, so they are not configured individually.
ontul.sso.oidc.client.id	(empty)	Client id registered at the provider.
ontul.sso.oidc.client.secret	(empty)	Client secret. Credential — never read back by the console.
ontul.sso.oidc.redirect.uri	http://localhost:8090/admin/auth/sso/oidc/callback	Must match the redirect URI registered at the provider exactly, and must be the address browsers reach — the load balancer's, not one master's.
ontul.sso.oidc.scopes	openid profile email	Scopes requested. Deliberately excludes groups: it is not a standard scope, and a provider that does not define it rejects the whole authorization request with invalid_scope.
ontul.sso.oidc.username.claim	preferred_username	Claim holding the login name.
ontul.sso.oidc.groups.claim	groups	Claim holding group membership. The provider must be configured to include it.
ontul.sso.oidc.audience	(empty)	Expected audience. Empty falls back to the client id. A token issued for another application is refused even though it is genuine and correctly signed.

SAML 2.0

Property	Default	Description
ontul.sso.saml.enabled	false	Enable the SAML provider.
ontul.sso.saml.idp.entity.id	(empty)	Identity provider entity ID. Read automatically when the provider's metadata is imported from the console.
ontul.sso.saml.idp.sso.url	(empty)	IdP single sign-on URL.
ontul.sso.saml.idp.certificate	(empty)	Base64 IdP signing certificate. Every assertion's signature is verified against it.
ontul.sso.saml.sp.entity.id	ontul	This cluster's entity ID, as it appears in the SP metadata the provider imports.
ontul.sso.saml.sp.acs.url	http://localhost:8090/admin/auth/sso/saml/acs	Assertion consumer URL. As with the OIDC redirect, this must be the address browsers reach.
ontul.sso.saml.nameid.format	(empty)	Requested NameID format. Empty omits the request entirely and lets the provider issue what it is configured for — naming one breaks more integrations than it fixes.
ontul.sso.saml.sign.requests	false	Sign authentication requests. Needs an SP keypair, generated from the console; re-import the SP metadata at the provider afterwards so it picks up the certificate.
ontul.sso.saml.username.attribute	uid	Assertion attribute holding the login name.
ontul.sso.saml.groups.attribute	groups	Assertion attribute holding group membership.

LDAP / Active Directory

Property	Default	Description
ontul.sso.ldap.enabled	false	Enable the directory provider. With it on, a directory password also works on the ordinary console login form.
ontul.sso.ldap.url	ldap://ldap.example.com:389	Directory URL. Use ldaps:// or enable StartTLS — otherwise the bind password crosses the network in the clear.
ontul.sso.ldap.bind.dn	(empty)	Service account that searches for user entries. Authentication is search then bind: the user's DN cannot be constructed, since Active Directory puts people under CN=John Doe,OU=Staff,… where neither component is the login name.
ontul.sso.ldap.bind.password	(empty)	Service account password. Credential.
ontul.sso.ldap.user.base.dn	(empty)	Subtree searched for user entries.
ontul.sso.ldap.user.filter	(uid={0})	Filter locating the user; {0} is the login name, escaped per RFC 4515 before substitution. Active Directory usually wants (sAMAccountName={0}).
ontul.sso.ldap.group.base.dn	(empty)	Subtree searched for groups.
ontul.sso.ldap.group.filter	(member={0})	Filter locating groups containing the user; {0} is the user's DN. Membership is read both from this search and from the user's memberOf, because directories disagree about which side records it.
ontul.sso.ldap.group.name.attribute	cn	Attribute holding the group name.
ontul.sso.ldap.starttls	false	Upgrade a plain ldap:// connection with StartTLS.

Python

Property	Default	Description
ontul.python.path	python3	Python interpreter used by Workers to run Python UDFs and Python-based exports (PDF/XLSX/PPTX). Either a command on PATH or an absolute path. Must exist on every worker host with the required packages.

Cache

Property	Default	Description
ontul.cache.max.bytes	268435456	Maximum size (bytes) of the in-memory Arrow LRU result/data cache. Once full, least-recently-used entries are evicted (default 256 MiB). Larger means more cache hits but higher memory use.
ontul.result.cache.enabled	true	Cache small query results so a repeated identical query is answered without re-planning or re-scanning.
ontul.result.cache.max	1000	Maximum number of cached result entries.
ontul.result.cache.max.bytes	8388608	Largest single result that may be cached (8 MiB). Bigger results stream through uncached rather than evicting everything else.
ontul.result.cache.ttl.ms	60000	How long a cached result stays valid (60s). Kept short because the underlying tables can commit at any time.
ontul.cache.data.enabled	false	Opt-in on-disk cache of Iceberg data files, so repeated scans of the same files avoid the object store. See Query Performance.
ontul.cache.data.dir	<ontul.base.data.dir>/…	Directory holding the on-disk data cache.
ontul.cache.data.max.bytes	5368709120	Cache capacity on disk (5 GiB). Least-recently-used files are evicted past this.
ontul.cache.data.max.file.bytes	67108864	Largest single data file that may be cached (64 MiB).
ontul.cache.data.ttl.ms	0	Time-to-live for a cached file; 0 disables expiry and relies on LRU eviction alone.
ontul.cache.data.kms.key.id	metadata-encryption	KMS key used to encrypt cached data files at rest, so the cache does not weaken the encryption guarantees of the source.

Data Lineage

Property	Default	Description
ontul.lineage.openlineage.url	(commented / optional)	OpenLineage-compatible HTTP endpoint (Marquez, DataHub, ...) to emit a RunEvent on every CTAS/INSERT/MERGE/CREATE VIEW. Leave empty to disable emission (lineage is still captured and stored locally).
ontul.lineage.openlineage.namespace	ontul	Namespace string reported in emitted OpenLineage events, used by the lineage backend to group datasets/jobs under this Ontul instance.
ontul.lineage.openlineage.timeout.ms	5000	HTTP timeout (ms) for the POST that emits a single OpenLineage event. Kept short so lineage emission never blocks query completion for long.

Client / SDK Properties (ontul.user.*)

These properties are not used by the server directly. They are set on the client side — by the SDK, JDBC driver, or job submission — via system properties (-D) or environment variables. They are listed in ontul.properties for reference only.

Property	Default	Description
ontul.job.driver.mode	(commented / optional, default master)	Job driver delegation mode: master (interactive) or worker (submit). Set per-job in the job config, not as a server property.
ontul.master.flight.port	(commented / optional, default 47470)	Flight SQL port used by the SDK to connect to the Master. The server uses ontul.master.flight.sql.port; this is the client-side equivalent passed as a system property when launching jobs.
ontul.user.accessKey	(commented / optional)	Client-side credential: IAM access key id used by the SDK / submitted jobs to authenticate to the Master. Set via -D or env var on the client. Paired with ontul.user.secretKey.
ontul.user.secretKey	(commented / optional)	Client-side credential: IAM secret key paired with ontul.user.accessKey. Treat as a secret; set on the client side only.
ontul.user.token	(commented / optional)	Client-side credential: pre-obtained JWT bearer token for SDK/JDBC auth, as an alternative to access/secret key login. Set on the client side only.
ontul.user.stsToken	(commented / optional)	Client-side credential: short-lived STS temporary token for SDK auth (temporary session credentials). Set on the client side only.