Run kinedb

kinedb is one program, kinedb. The same program is a server, a node of a cluster, a SQL shell, and an embedded engine for one-off SQL. This page tells how to start it, and it lists every command-line argument and every environment variable.

Get kinedb

There are two distributions: the binaries, and a container image. Both need no login.

Download the binaries

Every build of main publishes an archive for each platform. The archive holds the kinedb binary, the kd-tool binary (backup, restore and import, see Backup, restore and import) and REVISION.txt, which names the git commit. kinedb --version prints the same commit.

Platform	File
Linux x86-64, any distribution (static)	`kinedb-x86_64-unknown-linux-musl.tar.gz`
Linux x86-64, glibc 2.34 or newer	`kinedb-x86_64-unknown-linux-gnu.tar.gz`
macOS, Apple silicon	`kinedb-aarch64-apple-darwin.tar.gz`
macOS, Intel	`kinedb-x86_64-apple-darwin.tar.gz`
The web console and these docs (all platforms)	`kinedb-web.tar.gz`
The styled login pages (all platforms)	`kinedb-auth-pages.tar.gz`

The newest build is always at this address:

https://git.kinedb.com/api/packages/kinedb/generic/kinedb/latest/<file>

For example, on Linux:

base=https://git.kinedb.com/api/packages/kinedb/generic/kinedb/latest
curl -fLO "$base/kinedb-x86_64-unknown-linux-musl.tar.gz"
curl -fLO "$base/SHA256SUMS-linux.txt"
sha256sum -c --ignore-missing SHA256SUMS-linux.txt
tar xzf kinedb-x86_64-unknown-linux-musl.tar.gz
cd kinedb-x86_64-unknown-linux-musl
./kinedb --help

# optional: the web console and these docs, as dist/ next to the binary
curl -fLO "$base/kinedb-web.tar.gz"
tar xzf kinedb-web.tar.gz

# optional: the styled login pages, as dist-auth/
curl -fLO "$base/kinedb-auth-pages.tar.gz"
tar xzf kinedb-auth-pages.tar.gz

On macOS, use the file of your platform and its checksum file, for example kinedb-aarch64-apple-darwin.tar.gz and SHA256SUMS-aarch64-apple-darwin.txt, and check the file with shasum -a 256 -c SHA256SUMS-aarch64-apple-darwin.txt.

The web console. The node serves the web console and these docs from a dist directory next to the binary. Unpack kinedb-web.tar.gz in the same directory as kinedb, as in the example; it makes dist/. Without it, the node serves no console. The flag --webroot DIR or the environment variable KINEDB_WEB_ROOT names another directory (see Command-line arguments).
The styled login pages. kinedb-auth-pages.tar.gz makes dist-auth/. Start the node with --auth-webroot dist-auth, or copy the content of dist-auth/ to auth/ in the dist directory, and restart the node. See Download the pages for the steps and the check.
Versions. Each build is also a version of its own, named by the first 12 characters of its git commit, the same name as the image tag. Replace latest in the address with that name. A change that a newer change replaces before its build starts gets no build, so not every commit has a version. The list of versions is on the package page. The newest 10 builds stay available.
glibc or static. The static build runs on any x86-64 Linux, also on Alpine and on hosts with an old glibc. The glibc build needs glibc 2.34 or newer (for example Debian 12, Ubuntu 22.04, RHEL 9).

The container image

docker pull git.kinedb.com/kinedb/kinedb:latest

Tags. latest, and one tag per build: the first 12 characters of its git commit.
Content. The static kinedb binary at /app/kinedb, the static kd-tool binary at /app/kd-tool, and the web console with these docs at /app/dist. The image is built for linux/amd64.
Defaults. The image runs as the user 65532. Its default command is --port 4820 --data-dir /data, and /data is a volume.

One server

docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
  -e KINEDB_ROOT_PASSWORD='choose-a-password' \
  git.kinedb.com/kinedb/kinedb:latest

Or with the binary:

KINEDB_ROOT_PASSWORD='choose-a-password' ./kinedb --port 4820 --data-dir ./data

Without --data-dir, the data stays in memory, and it is lost when the process stops.
Without --gc-retention, the server never reclaims disk space. Each commit writes new tree nodes and keeps the nodes it replaced, so the disk grows with the number of commits, not with the size of the data. Many small commits into a table with a wide index can fill a disk with a small amount of data. A server that serves only the current state needs --gc-retention 0commits. See Command-line arguments.
One port carries everything: the clients, the peers, the web console at /ui/, and these docs at /ui/docs/.
The server raises its soft limit of open files to the hard limit when it starts. On macOS the limit is also capped by kern.maxfilesperproc. The log shows one line, for example [RLIMIT_NOFILE] soft 256 -> 92160 (hard unlimited). A node uses about 13 files, plus 1 for each shard, 4 for each peer and 2 for each client connection.
When the node reaches the limit, it does not stop. A new shard waits and its node tries again, with the log line [RAFT_BIRTH_PARKED]. New connections wait too ([ACCEPT_BACKOFF]). A node that cannot open its shard logs at start stops with exit code 1 and a FATAL: [RAFT_LOG_OPEN_BOOT] line that names the limits. Raise the hard limit (ulimit -Hn, or LimitNOFILE= in a systemd unit) and start the node again.

Stop the server

docker stop kinedb, Ctrl-C or kill <pid> sends SIGTERM or SIGINT. The server then does these steps:

It hands off the leadership of each Raft group that it leads to a live voter. The writes of that group continue on the new leader with no wait for an election. A voter is live when it answered the leader recently. When no voter answered yet (for example, a stop right after the cluster formed or a voter joined), the server sends a heartbeat round each 100 ms and waits for an answer until the end of KINEDB_STOP_HANDOFF_MS. A voter that tells the leader that it stops does not get the leadership. So when you stop a second node while the first one stops, each group moves to a node that stays up. A group with a split, a merge or a retire in progress is handed off too; the new leader continues it. A group whose other voters all stop is not handed off: it elects a leader after the stop.
It refuses each new client request with a retryable error: node <id> is stopping; the request was not applied; retry on another node. It also refuses each statement that another node forwards to it, with the same error. GET /health answers 503.
It waits for the open requests to answer. Then it half-closes every inbound connection (the client connections and the links of the other nodes) after its last answer, and waits up to 200 ms for these connections to close. When every open request answered, a node that forwarded a statement to it gets "not applied" (retryable), never "outcome unknown". When the wait ends with a request still open, a forwarded statement can get "outcome unknown".
It writes one [CLEAN_STOP] hand-off summary: … line.
It does one final sync of its storage, writes one [CLEAN_STOP] … final sync done, exit 0 line and exits with code 0. This is also true when the server is PID 1 in a container.

KINEDB_STOP_HANDOFF_MS sets the time of steps 1 to 3 without the 200 ms wait: 3000 ms by default, 20000 ms at most. 0 skips steps 1 to 4. So a stop takes up to KINEDB_STOP_HANDOFF_MS + 200 ms longer than a stop with 0, before the final sync. A node that leads no Raft group and has no open request uses less of this time.

A watchdog ends the process with code 1 and no final sync when steps 1 to 4 did not end 5 s after KINEDB_STOP_HANDOFF_MS. The watchdog does not cover the final sync: the server turns the watchdog off before the sync starts. Set the grace period of the host (docker stop waits 10 s, Kubernetes 30 s) to more than KINEDB_STOP_HANDOFF_MS + 5 s, plus the time of the final sync. A final sync that does not end before the grace period ends is stopped by the SIGKILL of the host, as with KINEDB_STOP_HANDOFF_MS=0.

What happens	Exit code
The final sync is done.	0
The final sync failed. The server writes one `[CLEAN_STOP] … the final sync failed` line.	1
The watchdog fired. The server writes one `[CLEAN_STOP] watchdog: …` line.	1
A second signal, or a signal during the start. The process ends at once, with no final sync.	128 + the signal number: 143 for SIGTERM, 130 for SIGINT

With KINEDB_STOP_HANDOFF_MS=0 the stop has no timeout. The final sync waits for the storage lock. A request that holds this lock and does not end also stops the sync. Then a second signal, or the SIGKILL of docker stop after its timeout, ends the process. A SIGKILL or a power cut does no final sync.

Log in

Clients must log in by default (KINEDB_CLIENT_AUTH=required).

The first boot creates the user root. With KINEDB_ROOT_PASSWORD, root gets that password. Without it, the server makes a password and writes it once to its log:

docker logs kinedb 2>&1 | grep 'GENERATED password'

A single process writes the password to its log BEFORE it stores the root record. When the store write fails, the node does not start: it logs [B2861_ROOT_SEED_FAILED] and exits with code 3. Then the password above that line is not valid. Repair the store and start the node again; the next start makes a new password.

On a node with a metadata Raft group (--calvin, or a cluster), the first boot opens the port some seconds before root exists. In that window GET /health answers 503, and a login answers the user registry is not ready: the root user is not seeded yet; retry shortly. Wait until GET /health answers 200, then log in. See Troubleshooting.

The SQL shell reads the password from KINEDB_ROOT_PASSWORD and logs in as root. Use --user and --password-env for another user. There is no password prompt.

docker exec -it kinedb /app/kinedb --connect 127.0.0.1:4820
docker exec kinedb /app/kinedb --connect 127.0.0.1:4820 -e "SHOW DATABASES"

Over HTTP, POST /login gives a token, and every other request carries it:

curl -s -X POST http://localhost:4820/login \
  -d '{"username": "root", "password": "choose-a-password"}'
# the answer is a JSON object with a "token"

curl -s -X POST http://localhost:4820/sql \
  -H "Authorization: Bearer <token>" \
  --data-binary 'SHOW DATABASES'

POST /sql also takes parameters. Send content-type: application/vnd.kinedb.sql-params; the body is then the binary value [sql, params] of the wire protocol, not text. The client libraries build it for sql(text, params). See Parameters.

POST /sql also takes ONE complete transaction. Send content-type: application/vnd.kinedb.sql-txn; the body is then the binary payload of the Transaction frame of the wire protocol. The client libraries build it for transaction(statements). A db of null in the payload takes ?db=. A transaction on a branch other than main is refused. The answer is {"type": "created"}, or the error, and then nothing of the transaction is written. BEGIN, COMMIT and ROLLBACK as text are refused. See Transactions.

An application uses a client library: JavaScript or Python.

The other ways to log in (an API secret, a key pair), the guest, the bearer token, its refresh and its limits, and the KINEDB_CLIENT_AUTH modes are on the page Authentication.

A cluster

A cluster is any number of kinedb nodes. To start one:

Choose a join secret. Give every node the same KINEDB_JOIN_SECRET. The first node makes the cluster key; a joining node uses the join secret to get the cluster key from a running node (The cluster key). Peer authentication is on by default, and its key comes from the cluster key: a joining node with no cluster key and no join secret refuses to start.
Start the first node with --calvin. To convert a standalone node, start its data directory here, as the first node. The first node compacts its metadata log above the tables of that directory before a second node can join, so each joiner gets them in a metadata point. A first node that is the only voter, whose metadata log starts at index 1 and whose store holds tables, does the same once at its next start (Troubleshooting).
Start every other node with --calvin --join <host:port of a running node>. Give a new node an empty data directory. A joiner gets every table from the cluster, so it refuses to start on a directory that a node without --calvin wrote with tables in it (Troubleshooting).
Set KINEDB_ADVERTISE_ADDR on every node to the address that the other nodes use to reach it. A node that listens on all interfaces, which includes every container, otherwise tells its peers 127.0.0.1.

A data directory of a cluster of more than one node does not start without --calvin: a server without --calvin, -e and -f refuse it with exit code 3 and write nothing. Such a start runs no Raft group, so the other replicas never get its writes. To use the directory without a cluster, start it once with KINEDB_DETACH_FROM_CLUSTER=1. After that the directory never starts with --calvin again (Troubleshooting).

Three hosts, 10.0.0.1 to 10.0.0.3, with Docker:

# on 10.0.0.1: the first node
docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
  -e KINEDB_JOIN_SECRET='the-join-secret' \
  -e KINEDB_ROOT_PASSWORD='choose-a-password' \
  -e KINEDB_ADVERTISE_ADDR=10.0.0.1 \
  git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin

# on 10.0.0.2 (and the same on 10.0.0.3, with its own address)
docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
  -e KINEDB_JOIN_SECRET='the-join-secret' \
  -e KINEDB_ROOT_PASSWORD='choose-a-password' \
  -e KINEDB_ADVERTISE_ADDR=10.0.0.2 \
  git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin --join 10.0.0.1:4820

Arguments after the image name replace the default command, so they repeat --port and --data-dir.

Ports. The nodes talk to each other on the same TCP port as the clients. There is no other port to open.
Disk space. These commands start the nodes without --gc-retention, so no node ever reclaims disk space (see One server). Add the same --gc-retention to every node when the cluster does not need all of its history.
Replicas. Each shard has 3 replicas by default. KINEDB_REPLICATION_FACTOR sets the first value on a new cluster; after that, SET CLUSTER cluster.replication_factor = <n> changes it.
Joining. A joining node waits 10 seconds for the leader to accept it, and then it exits with an error (KINEDB_JOINER_BOOT_TIMEOUT_MS).
Durability. A write is durable when the cluster acknowledges it: each replica writes the entry to disk before it counts toward the quorum. Each node keeps its Raft log in <data-dir>/raft_wal.
Restarting. A node writes the addresses of its peers to <data-dir>/peer_addrs each time the membership changes. When it restarts, it reads that file and dials every peer at once, so it does not wait for gossip to find them. A cluster that went down entirely reforms in the same way: restart every node with --join <a peer>, and the nodes elect a leader from the membership they persisted. When every node holds all the data (no more nodes than the replication factor), each data shard also elects a leader from that membership, however long the cluster ran before it went down. With more nodes than the replication factor, a node that restarts takes its part again in each data shard where the newest voter set that it holds names it and whose Raft state it still holds, so a shard elects a leader again also when most or all of its voters restarted (B2442, B2435, B2464). The newest voter set is the last voter change in its Raft log, else the voter set of its snapshot point, else the last voter set that it applied. A node that this voter set does not name, or whose newest voter set is not known (a snapshot point written by an older version), waits passive until a leader adds it. A node that restarts without --join keeps its part only when its Raft log holds no voter change that it did not apply, other than a change to the same voter set (B2463), and when the voter set of its snapshot point, if it has one above the last voter set that it applied, is its boot voter set (B2511). A snapshot point written by an older version keeps such a node passive. When a new leader removes that voter change from the log, the node checks this rule again and takes its part (B2446). A peer that moved to a new address is found through gossip, as before. A node that was a voter of a larger cluster never leads alone: without --join, and even when <data-dir>/last_peer_count is missing, it waits for a majority of the membership it persisted. Two operator overrides change this: KINEDB_FORCE_QUORUM=<n> sets the quorum of that node, and KINEDB_DISABLE_AUTO_REFORM=1 makes a node without last_peer_count boot as a new seed while the other nodes wait passive for it to lead.
Upgrading past B933. A node now sends a read that another node serves with the identity of the client, and the other node checks the permissions of that client. A node from before this change cannot read that request, and a new node refuses the old request. So while old and new nodes run together, a read that another node must serve fails with an error. Reads never run without a permission check. Restart every node with the new image at the same time. A cluster where each node holds all the data is not affected.

Upgrade kinedb

kinedb has no release versions yet, and it keeps no compatibility between builds. Read Breaking changes before each upgrade.

The server and the clients move together. Update the server, every SDK and kd-tool in one change. Use the client packages of the same commit as the server image. The image kinedb:<sha12> goes with the Python packages 0.0.0.dev<N>+g<sha12> (kinedb-client, kinedb-embedded) and the npm tag @kinedb/client@<short sha> (see Python client and JavaScript client).
A mismatch fails in both directions. A client that is older than the server can get the error client protocol version none is not supported at login. A client that is newer than the server can get no error: the login waits until its timeout, for example WS login timed out after 10000ms. After an upgrade, treat a login timeout as a version mismatch first.
kd-tool is a client too. A kd-tool of a new build can fail to log in to an old server, and the reverse. Use the kd-tool of the same build as the server that it connects to. A manual upgrade through a dump uses two tools: run kd-tool backup of the OLD build against the old server. Then run kd-tool restore of the NEW build into the new server. A dump of an older build restores into a newer build (see Backup, restore and import).
An old kd-tool does not dump files. A kd-tool before dump format version 6 writes the rows of a file table, but not the files of put_file or INSERT … FILE (see Files). Do not use the dump route for a database with files when the old build has such a kd-tool.
POST /sql refusals. A refused statement answers HTTP 200 OK with "type": "error". A bad query string or body answers 400, a missing login 401. So check the "type" of each reply, not only the status (see the table in Databases).
The packages need no login. The kinedb organisation is public, so the binaries, the image and the package indexes answer without a credential. A token that has no package scope gets 401 Unauthorized. Use no token, or a token with the package read scope.

Embedded: SQL without a server

With -e or -f and without --connect, kinedb runs the SQL in its own process and exits. No server starts, and no port opens. With --data-dir, the data stays for the next call.

./kinedb --data-dir ./data -e "CREATE TABLE t (id INT PRIMARY KEY, name TEXT)"
./kinedb --data-dir ./data -e "INSERT INTO t (id, name) VALUES (1, 'a'); SELECT * FROM t"
./kinedb --data-dir ./data -f setup.sql

-e splits the SQL on ; and runs the statements in order. -f file.sql does the same with a file, and -f file.kd runs a kd script.
-e stops at the first statement that fails. It prints a line that starts with ERROR:, and it exits with 1.
When the reader of the output stops early (| head, | grep -q), kinedb stops quietly and runs no more statements. It exits with 0 when every statement ran, and with 141 when statements did not run.
Some statements need a server; the SQL reference marks them.

In Rust, the kinedb crate is the same engine as a library (Db::open(path), db.execute(sql)).

Command-line arguments

Argument	Default	What it does
`--port PORT`	`4820`	The TCP port for the clients, the peers, the web console and these docs.
`--listen IP`	`0.0.0.0`	The IP address that the server listens on, with the port of `--port`. Without it, kinedb uses `KINEDB_LISTEN_ADDR`; without both, all interfaces. Give an IPv4 or IPv6 address, for example `127.0.0.1`, `::1` or `[::1]`. A host name or a bad value stops the start with one error (exit 1). Use `127.0.0.1` for a node behind a reverse proxy on the same host: then only that host can reach the port.
`--trusted-proxy LIST`	not set	The reverse proxies that the node trusts: a comma list of IP addresses and CIDR ranges, for example `127.0.0.1,10.0.0.0/8`. Without it, kinedb uses `KINEDB_TRUSTED_PROXIES`; without both, the node trusts no proxy. When a connection comes from a trusted proxy, the node takes the client address from `X-Forwarded-For` (TLS for the clients). An empty value means no proxy. A host name, a port or another bad entry stops the start with one error (exit 1): `Error: --trusted-proxy: "<entry>" is not an IP address or a CIDR range; …`. `SHOW SETTINGS` shows the value as `boot.trusted_proxies`.
`--data-dir DIR`	not set	The directory of the data. Without it, the data stays in memory.
`--memory SIZE`	not set	The total memory budget of the process, for example `4GB` or `3800MB`. kinedb enforces it with its own accounting. A value that is not a size, or whose byte count is too large, stops the start with one error (exit 1). Under pressure the node holds back client writes, including a bulk load from `kd-tool import`, and answers a held write with a retryable refusal; the clients retry, so a load slows down and never kills the node. It needs `--data-dir`, and it does not work with `--s3` or `--connect`. On Linux the process turns off transparent huge pages for itself, so the host's huge-page setting does not change the memory use.
`--cache-slab SIZE`	not set	A block cache that is allocated at start. Under `--memory`, it must fit the budget. Alone, it is the older way to size the cache. The cache takes the largest size at or below SIZE on the steps 1, 1.25, 1.5 and 1.75 times a power of two (for example 300MB gives 256 MiB, 3GB stays 3 GiB), so the allocator holds exactly the bytes that kinedb counts.
`--gc-retention SPEC`	not set	Turns on the garbage collection of dead blocks, and sets the history to keep: `<N>commits` (the head and N older commits; `0commits` keeps only the head) or `<N>s`, `<N>m`, `<N>h`, `<N>d` (for example `7d`). Without it, nothing is reclaimed. It does not work with `--s3`: the node refuses the combination and exits with code 1 (S3 storage).
`--calvin`	off	Cluster mode: Raft and the deterministic sequencer. Every node of a cluster needs it.
`--join HOST:PORT`	not set	Join a cluster through a running node. Use it with `--calvin`.
`--peer-auth MODE`	`required`	`required`: the nodes must prove that they hold the cluster key. `permissive`: they need not.
`--cluster-key KEY`	`kinedb`	The name of the cluster. A node ignores the gossip of a node with another key, and it refuses the peer connection of such a node. It is not a secret. Give each cluster its own key when two clusters share hosts.
`--realm NAME`	not set	The realm of the node. Leave it unset for one realm.
`--s3 URI`	not set	Store the blocks in S3: `s3://KEY:SECRET@HOST/BUCKET/PREFIX?region=R`. `--data-dir` is then the local cache; without it, a temporary directory. The released binaries include S3. A build from source needs `cargo build --release --features kinedb-bin/s3`; without the feature, kinedb refuses `--s3` and exits. See S3 storage.
`--webroot DIR`	found	The directory of the web console. Without it, kinedb uses `KINEDB_WEB_ROOT` when it is set and not empty. Without both, kinedb looks for `dist` next to the binary, then for `../../web/dist` from the binary, then for `dist` and `web/dist` in the working directory. With no match, the node serves no console.
`--no-web`	off	Serve no web console and no docs. It wins over `--webroot` and `KINEDB_WEB_ROOT`.
`--auth-webroot DIR`	not set	The directory of the custom login pages. Without it, kinedb uses `KINEDB_AUTH_WEBROOT` when it is set and not empty. Without both, kinedb uses `auth/` in the web console directory when that directory exists. Without all three, the built-in login pages apply. A path that is not a directory gives a warning at the start, and the built-in pages apply. `--no-web` does not turn it off. See Custom login pages. `SHOW SETTINGS` shows the full path as `boot.auth_webroot`.
`--connect HOST:PORT`, `-c`	not set	Client mode: open a SQL shell on a running server, or run `-e` or `-f` there.
`--user NAME`	`root`	Client mode: the user to log in as.
`--password-env VAR`	`KINEDB_ROOT_PASSWORD`	Client mode: the environment variable that holds the password.
`-e SQL`	not set	Run SQL and exit. Without `--connect`, the SQL runs in this process.
`-f FILE`	not set	Run a file and exit: `.kd` is a kd script; any other file is SQL.
`--show-types`	off	Client mode: put the column type in the header of a result, for example `price:Float`.
`--help`, `-h`		Print the usage and exit.
`--version`, `-V`		Print `kinedb <revision>` and exit 0. The revision is the git commit of the build, or `unknown`: the same value as `revision` in `GET /health` (Monitoring). Compare it on the client and the server before an upgrade.

The smallest `--memory` budget

kinedb divides the --memory budget into its caches, the write pipeline and a reserve. Each part has a floor, so a budget has a floor too. Today the floor is about 1680 MB when kinedb derives the cache slab itself, and about 1029 MB with --cache-slab 256MB. A smaller budget is refused at start: kinedb prints a line that names the floor in bytes, for example --memory 1048576 (the total budget) is below the minimum plannable floor 1761607664 …, and it exits with 1. To get the exact value of your binary, start it with --memory 1MB --data-dir <a new directory> and read that line. In a checkout of the repository, sh scripts/mem_floor.sh does this and prints the floor in whole MB.

Under --memory, three limits of each shard also come from the budget. They are the write buffer cap (cluster.storage.memtable_max_bytes), the sequencer queue cap (cluster.calvin.sequencer_pending_bytes_limit) and the flush generation size (cluster.storage.flush_generation_max_bytes). Each one is the smaller of its setting and its share of the budget. The write buffer cap is never less than 32 MiB. SHOW METRICS gives the value in force (…_effective) and the setting (…_configured). Without --memory, the settings apply unchanged.

Two raft log limits of each shard also come from the budget. The node raft log share W is one eighth of the shed ceiling, and never less than 32 MiB (about 84 MiB at the smallest budget). The compaction trigger (cluster.calvin.raft_log_compaction_bytes) is the smaller of its setting and W/2, and never less than 4 MiB. The replication queue cap (cluster.calvin.raft_log_pending_bytes_limit) is the smaller of its setting and W. A setting of 0 still disables the limit. SHOW METRICS gives both values, as for the three limits above.

The budget is not yet a hard limit on the resident memory of the process. In the nightly low-RAM test (--memory 1680MB, 10 M rows, the A-J load), the peak RSS of a node was 0.71 to 0.92 times the budget (B458: 30 node samples, 2026-10-12 to 2026-10-21). That test samples only the load phase, every 5 seconds. It does not sample the bulk load before it (B3311). Until that phase is measured, keep the margin: for --memory 1680MB, provision about 2.5 GB of RAM per node.

The connection and peer buffers of a memory budget

With --memory, kinedb also takes five limits from the reserve of the budget. They do not change the floor of the budget. Without --memory, none of them applies.

Limit	Value	What happens at the limit
Peer buffers	an eighth of the reserve, divided by the number of peers	The per-peer cap `cluster.calvin.peer_broadcast_queue_bytes_limit`. A value that you set (the environment variable or `SET CLUSTER`) wins.
Client inbound messages	a thirty-second of the reserve, for all the client connections together	WebSocket messages and HTTP request bodies count. The server stops reading a client socket until earlier messages are done, so the client waits. One message larger than this limit is always read when no other message holds bytes.
Client connections	an eighth of the reserve, divided by the fixed cost of one connection	WebSocket and HTTP connections count. The server refuses the next WebSocket connection or HTTP request with HTTP 503 and `Retry-After: 1`. The clients report a retryable error.
Write executor threads	a sixteenth of the reserve, divided by the fixed cost of one thread (64 KB), at most 256	A pipelined write runs on the thread of its connection, so it waits for the earlier statements of that connection. The forward pool to each leader has the same number of connections.
Inbound forward links	a sixteenth of the reserve, divided by the fixed cost of one link (128 KB); each peer always gets 4	A peer forwards its writes to the leader over a pool of links. Over the limit the leader refuses a new link with HTTP 503, and the peer sends its writes over the links it has. A write waits; it is not refused.

The reserve is 35 % of the budget, at least 256 MB and at most 2 GB. For example, at --memory 4GB the peer buffers are 179 MB, the client inbound messages 44 MB, and the node accepts about 1 400 client connections (2 048 from about 6 GB up) and 716 inbound forward links (1 024 from about 6 GB up). Every budget that the server accepts gives 256 write executor threads, the same number as a node without --memory. The connections between the nodes of the cluster do not count against the client limits. SHOW METRICS shows the values in force, in the rows that start with client_admission., in peer_broadcast.queue_bytes_limit, in write_pool.thread_limit, in leader_forward.slots_per_leader and in leader_forward.inbound_link_limit.

Environment variables

Setup

Variable	Default	What it does
`KINEDB_ROOT_PASSWORD`	not set	The password of `root`, set at the first boot. Without it, the first boot makes a password and logs it once. The SQL shell also reads it to log in.
`KINEDB_CLIENT_AUTH`	`required`	`required`: clients must log in, except the guest (`KINEDB_ALLOW_GUEST`, under Tuning). `permissive`: anyone can query.
`KINEDB_PEER_AUTH`	`required`	The same as `--peer-auth`. The flag wins.
`KINEDB_JOIN_SECRET`	not set	The join secret of the cluster. You choose the value. It is only an environment variable, and the node never writes it to disk. A node that starts with `--join` and has no `cluster.key` uses it to get the cluster key from a running node before it joins, and each running node uses it to answer. Give every node the same value. A `--join` node with no `cluster.key` and no join secret refuses to start with exit 3 (Troubleshooting); a joiner that gets no key exits with `FATAL: no running node gave the cluster key` (Troubleshooting).
`KINEDB_LISTEN_ADDR`	not set	The same as `--listen`. The flag wins. An empty value or a bad value stops the start with one error (exit 1).
`KINEDB_TRUSTED_PROXIES`	not set	The same as `--trusted-proxy`. The flag wins. An empty value means no proxy. A bad entry stops the start with one error (exit 1).
`KINEDB_ADVERTISE_ADDR`	not set	The address that the peers use to reach this node, `host` or `host:port` (the port part is ignored). Without it, a node that listens on all interfaces or on loopback advertises `127.0.0.1`, and a node that listens on one other address advertises that address. A node that listens on loopback and advertises another address logs one warning: the peers cannot reach it directly.
`KINEDB_GC_RETENTION`	not set	The same as `--gc-retention`. The flag wins. With `--s3`, the node refuses it and exits with code 1.
`KINEDB_WEB_ROOT`	not set	The same as `--webroot`. The flag wins, and `--no-web` wins over both. An empty value is the same as not set.
`KINEDB_AUTH_WEBROOT`	not set	The same as `--auth-webroot`. The flag wins. An empty value is the same as not set. `--no-web` has no effect on it.
`KINEDB_VACUUM_DEAD_FRACTION`	`0.5`	How dead one sealed segment must be before a sweep rewrites it, as a fraction of its bytes. A value outside `[0, 1]` is ignored.
`KINEDB_VACUUM_DEAD_SHARE`	`0.125`	How much of what a sweep examines may stay dead. Above this share, the sweep rewrites the deadest segments until the share of what is left reaches it. This is the rule that reclaims garbage spread thin over many segments. A value outside `[0, 1]` is ignored.
`KINEDB_RAFT_WAL_DIR`	`<data-dir>/raft_wal`	The directory of the Raft logs. Without a data directory and without this variable, the Raft log stays in memory. A cluster member that does not find its Raft log here refuses to start. See Troubleshooting.
`KINEDB_RAFT_ELECTION_US`	`3000000` (3 s)	With `--calvin`, the Raft election timeout in microseconds. Raise it when commits are slow on big tables. It must stay above the read lease window plus one heartbeat (`KINEDB_RAFT_HEARTBEAT_US`); for a heartbeat of 8 ms or more, that is more than 3.5 heartbeats. A pair that breaks this bound, or a value that is not a whole number, stops the start with exit 3. See Troubleshooting.
`KINEDB_RAFT_HEARTBEAT_US`	`500000` (500 ms)	With `--calvin`, the Raft heartbeat interval of a leader in microseconds. It also sets the read lease window: 2.5 heartbeats for a heartbeat of 8 ms or more. It must be above 0, and the window plus one heartbeat must stay under `KINEDB_RAFT_ELECTION_US`. A bad value stops the start with exit 3. See Troubleshooting.
`KINEDB_LOG_SINK`	not set	`ws://host:port` of a kinedb server that collects logs and metrics. With it, the node sends its logs and metrics there. See Monitoring.
`KINEDB_LOG_SINK_USER`, `KINEDB_LOG_SINK_PASSWORD`	not set	The login at the log sink. For the rights of this user, see The collector user.
`KINEDB_METRICS_RETENTION_SECS`	`604800` (7 days)	The age in seconds after which the node deletes the rows of the `metrics` table of the collector. `0` keeps all rows. A value that is not a whole number also keeps all rows, and the node logs one warning. See Monitoring.
`KINEDB_STOP_HANDOFF_MS`	`3000`	The time in ms that a stopping node uses to hand off its Raft leadership and to finish its open writes, before its final sync. `0` skips the hand-off. A whole number above `20000` is `20000`. A value that is not a whole number is the default, and at the stop the server writes one warning line that names the value. A stop takes up to this value + 200 ms longer. See Stop the server.
`KINEDB_OPEN_WITH_UNUSABLE_INDEX`	not set	`1`: the node starts although an index is UNUSABLE, so you can run the repair. Set it for one start only. See Troubleshooting.
`KINEDB_OPEN_WITH_UNREPLICATED_ADOPTION`	not set	`1`: with `--calvin`, a node whose metadata log re-floors a table that the log never created (a cluster that a version before B3201 made from an adopted directory) starts anyway, and logs each table. The joiners still miss these tables. Use it only to read the data before the repair. See Troubleshooting.
`KINEDB_REPAIR_SHARD_LAYOUT`	not set	`1`: the node repairs a shard layout that does not cover every key exactly once, then starts. Set it for one start only. See Troubleshooting.
`KINEDB_CONVERT_UNSHARDED_TABLES`	not set	`1`: the node marks each table of the removed unsplittable table mode of an older version as a normal table, then starts. No data moves. Stop every node first, and set it for one start of every node. See Troubleshooting.
`KINEDB_REKEY_BRANCH_FILE_TREES`	not set	`1`: the node re-keys the file rows of each branch table that a version between B599 and B1898 forked with an index, then starts. Only the key prefix of each file row changes. Stop every node first, and set it for one start of every node. See Troubleshooting.
`KINEDB_REPAIR_RETIRED_PREFIX_INDEX_ENTRIES`	not set	`1`: the server rebuilds each index slice that holds entries of a retired shard (a version before B1921 wrote them) from the rows of its base shard, then starts. Only index entries change. Stop every node first, and set it for one start of every node. An embedded engine cannot run this repair. See Troubleshooting.
`KINEDB_SET_HISTORY_FLOOR_AT_HEAD`	not set	`1`: a server on a data directory of a version before B3817 (commits and no `HISTORY_GAP` file) sets a floor of every table at the commit before HEAD, writes an empty `HISTORY_GAP`, then starts. On that node, every history read older than HEAD then refuses. Rows and commits do not change. Set it for the first start of each node on this version. See Troubleshooting.
`KINEDB_OPEN_DROPPING_UNREGISTERED_INDEX_ROWS`	not set	`1`: an OLDER version wrote the storage WAL, and a batch of it holds rows of an index that has no layout (B934). The open drops those index rows, keeps every other row, then starts. Set it for one start only. See Troubleshooting.
`KINEDB_OPEN_WITH_DAMAGED_BLOCKS`	not set	`1`: the node starts although a segment file holds a damaged block. Every good block is served, and every read of a damaged block is an error that names it. Set it for one start only. See Troubleshooting. With `--s3`, the S3 HEAD recovery also skips each damaged item that its `B3466:` refusal names: see Troubleshooting.
`KINEDB_ALLOW_HEAD_WITHOUT_TXN_ID`	not set	`1`: an OLDER version wrote the `HEAD` file with no txn id, and the WAL holds no commit marker (B2640). The open takes the txn id from the timestamp of the HEAD commit, writes the `HEAD` file again with it, then starts. Set it for one start only. See Troubleshooting.
`KINEDB_OPEN_CUTTING_DAMAGED_WAL`	not set	`1`: a storage WAL file holds valid commit markers after an entry that does not decode (B2952). The open replays the file up to that entry, drops every commit after it, and logs one warning per damaged file. B2975: the last commit marker of `wal/wal.log` names a block that the store does not hold; the open starts with the last marker whose block it holds (or the `HEAD` file) and logs one warning per dropped marker. Set it for one start only. See Troubleshooting.
`KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP`	not set	`1`: with `--calvin`, an OLDER version (before B2931) wrote this data directory, and it grew the data shards outside their Raft logs. When the cluster never had more nodes than the replication factor, the node writes the metadata voters as the voter record of each of its data shards, writes the membership marker, then starts, and each shard commits one ConfigChange of its voters: stop every node first, and set it for one start of every node. When the cluster once had more nodes than the replication factor, the node writes no record and boots each data shard passive until the live leader commits one ConfigChange of the live voters (keep a quorum of each shard up), or it takes the voters from `KINEDB_RFN_MIRROR_VOTERS_FILE`. See Troubleshooting.
`KINEDB_RFN_MIRROR_VOTERS_FILE`	not set	A path. With `KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP=1`, when the cluster once had more nodes than the replication factor: a file with one line `<shard id>=<voter id>,<voter id>,...` for each data shard that the error names, with the live voters that `SHOW RAFTGROUP <shard id>` printed on the older version. A set that names this node becomes its voter record; a set that does not name it boots the shard passive. A file that does not name every shard, or that does not parse, stops the node with exit code 3. See Troubleshooting.
`KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS`	not set	`1`: with `--s3`, the node starts on the newest commit that the segments hold, when the S3 `HEAD` names a lost block. The commits after it are lost. Set it for one start only. See Troubleshooting.
`KINEDB_S3_TAKE_OVER_PREFIX`	not set	`1`: with `--s3`, the node takes the prefix over from another node: it writes its own id to `PREFIX/OWNER`, then starts. The next sync of the other node uploads nothing. Set it only when the other node is stopped for good, for example to restore a lost node on a new directory. Set it for one start only. See Troubleshooting.
`KINEDB_RESTAMP_CATALOG_SLOTS`	not set	`1`: with `--calvin`, the node re-stamps each catalog entry that a node without `--calvin` wrote, and each metadata row (a cluster setting, a user, a function and others) that holds an index of an earlier metadata log, then starts. Set it for one start only. See Troubleshooting.
`KINEDB_REFLOOR_FOREIGN_STAMPS`	not set	`1`: with `--calvin`, the node repairs each table whose rows or index entries have stamps above the Raft log of their data group (a node without `--calvin` wrote them), then starts. Each table gets new Raft groups above its highest stamp. A write that was lost before the repair does not come back. Set it for one start only. See Troubleshooting. It does not start a node that refuses with `B2452:` (see Troubleshooting).
`KINEDB_REPLAY_OVER_FOREIGN_STAMPS`	not set	`1`: with `--calvin`, the node replays the Raft log of each data group although a table of the group has rows with stamps above that log (a node without `--calvin` wrote them), then repairs each such table as `KINEDB_REFLOOR_FOREIGN_STAMPS` does. The replay can skip a committed write that the disk does not hold; such a write does not come back. Prefer the manual fix. Set it for one start only. See Troubleshooting.
`KINEDB_RECORD_LONE_METADATA_VOTER`	not set	`1`: with `--calvin`, a store of an older version whose metadata log names this node the only voter, but that has no `<data-dir>/shard_configs/1`, writes that file and starts. Without it, the node refuses with exit 3. Set it for one start only. See Troubleshooting.
`KINEDB_REJOIN_PASSIVE_AFTER_LOST_RAFT_WAL`	not set	`1`: with `--calvin`, a cluster member whose Raft log is lost starts, and each Raft group of the lost log is a passive follower that does not vote until the leader adds it again. Set it for one start only. It does not start the only voter of the metadata group (for example, a 1-node cluster): no other member holds that log. See Troubleshooting.
`KINEDB_RAFT_WAL_CUT_AT_HOLE`	not set	`1`: with `--calvin`, a cluster member whose Raft log has a hole (a power loss in a version before B2117) cuts each such log at its first missing index, then starts. The leader sends the removed entries again. Set it for one start only. See Troubleshooting.
`KINEDB_RAFT_WAL_CUT_AT_DAMAGE`	not set	`1`: with `--calvin`, a cluster member whose last Raft log file holds a valid record after a record that does not decode (media damage, B2971) cuts each such log at the bad record, then starts. The bad record and every later one are removed; the leader sends them again. Set it for one start only. See Troubleshooting.
`KINEDB_SKIP_UNBOUND_SHARD0_WRITES`	not set	`1`: with `--calvin`, the node starts although the Raft log of shard 0 holds writes of a version before B2114 that name no table generation. The apply of shard 0 skips each such write and logs it. A skip can lose a write that was not yet on the disk. That start writes a durable record of the skipped writes, so set it for one start only: the next start reads the record and needs no variable. With a record that does not decode, the start writes a new record in its place. See Troubleshooting and the record that does not decode.
`KINEDB_START_WITH_UNRETIRED_MERGES`	not set	`1`: with `--calvin`, the node starts although a shard merge of a version before B2301 has no retired-source row in the global transaction ledger. A cross-shard transaction on a source of such a merge is not refused. The node writes the merges into an acknowledgement file in the data directory. Set it for one start only. See Troubleshooting.
`KINEDB_START_WITH_UNRECORDED_NON_VOTER_SHARDS`	not set	`1`: with `--calvin`, a node of a cluster starts although a shard has no voter record and no Raft log on the node (an older version wrote no record on a node outside the voters of a shard). The node builds no Raft group of such a shard at boot, and writes the shards into an acknowledgement file in the data directory. Set it for one start only. See Troubleshooting.
`KINEDB_DROP_LOCAL_CLUSTER_SETTINGS`	not set	`1`: with `--calvin`, on a data directory of a cluster of more than one node or on a start with `--join`, the node writes the `SET CLUSTER` statement of each cluster setting that a standalone server set on this node only into the file `b3279_dropped_cluster_settings` of the data directory, removes the setting, then starts. Run each logged statement again on the cluster. Set it for one start only. See Troubleshooting.
`KINEDB_DETACH_FROM_CLUSTER`	not set	`1`: a start without `--calvin` (a server, `-e`, `-f`) on a data directory of a cluster of more than one node writes the file `b3290_detached_from_cluster` into the data directory and opens the store. Without it such a start refuses with exit 3. Each later start without `--calvin` opens with no variable, and the directory never starts with `--calvin` again. See Troubleshooting.
`KINEDB_DROP_LOCAL_TICKERS`	not set	`1`: with `--calvin`, the node removes each ticker that only this node holds (a version before B2332 wrote it with no metadata log entry), logs its `CREATE TICKER` statement, then starts. Run each logged statement again. Set it for one start only. See Troubleshooting.
`KINEDB_PURGE_SUPERSEDED_POINT_LAYOUTS`	not set	`1`: with `--calvin`, the node removes each index tree of a dropped index that a version before B2972 kept at a metadata point install, with its rows, then starts. An index tree of an older build or a table with an older id still refuses, and then the node writes nothing: rebuild the node. Set it for one start only. See Troubleshooting.
`KINEDB_OPEN_WITH_UNREGISTERED_DEFAULT_COLLECTIONS`	not set	`1`: the node starts although `default` holds a collection with no `CREATE` while another database exists (a version before B2040 stored such a collection of another database there). It changes nothing in the store and writes an acknowledgement file in the data directory. Set it for one start only. See Troubleshooting.
`KINEDB_ADOPT_IMPLICIT_TABLES`	not set	`1`: the node starts although a table has rows and no catalog entry (a version before B2251 with `--calvin`, or before B2763 without `--calvin` and with `-e` or `-f`, made such a table at its first write). This applies in every mode. The node adopts each such table: it writes the catalog entry with the existing table id and keeps the rows. Set it for one start only; a later start needs no variable. See Troubleshooting.
`KINEDB_ACCEPT_V1_SLICE_DIGESTS`	not set	`1`: the node renames each retired-shard slice file that has the v1 digest name of a version before B1429, then starts. The bytes stay the same. Set it for one start only. See Troubleshooting.
`KINEDB_SYNC_MOVE_WINDOW_ROW`	not set	`1`: an embedded sync client (`kinedb-sync`) moves a window that a build from source between B2202 and B2204 stored in the unnamed subscription row to its own row, then opens. Set it for one open only. See Troubleshooting.

Tuning

These variables belong to the settings of the server; SHOW SETTINGS lists the same settings with the values in force. For a boot. setting, its source column names the flag or the variable that gave the value, or auto-detect or default. A variable of a cluster. setting sets the first value on a new cluster; after that, SET CLUSTER <setting> = <value> changes it. A node reads the variable of a node. setting when it starts. The table lists only a variable that the server reads; a setting whose variable only a test sets is not in it.

Variable	Setting	Default	What it does
`KINEDB_ALLOW_GUEST`	`cluster.auth.allow_guest`	true	Under KINEDB_CLIENT_AUTH=required, a client with no login runs a SELECT, INSERT, UPSERT, UPDATE or DELETE as the guest, which gets only GRANT ...
`KINEDB_CALVIN_APPLY_TIMEOUT_SECS`	`cluster.calvin.apply_timeout_secs`	60 secs	How long a Calvin write waits for local apply before returning an error.
`KINEDB_SEQUENCER_BATCH_AGE_MS`	`cluster.calvin.sequencer_batch_age_ms`	2 ms	Maximum age (ms) a sequencer batch waits before flushing.
`KINEDB_SEQUENCER_BATCH_SIZE`	`cluster.calvin.sequencer_batch_size`	1024	Maximum number of operations in a single Calvin sequencer batch.
`KINEDB_FORWARD_BUDGET_MS`	`cluster.forward.budget_ms`	30000 ms	How long (ms) a node waits for the reply to a request it forwards to another node, when the request it serves has no deadline of its own.
`KINEDB_MERGE_DRAIN_WAIT_MS`	`cluster.merge.drain_wait_ms`	5000 ms	How long (ms) MERGE SHARDS waits for its two source shards to drain their cross-shard transaction work when that work refuses the merge.
`KINEDB_PLACEMENT`	`cluster.placement.enabled`	true	At N>RF a node acquires only the shards that the placement draw (pick_voters_for_shard) names.
`KINEDB_FORCE_QUORUM`	`cluster.quorum`	not set	Recovery only: forces `cluster.quorum` on this node for the life of the process.
`KINEDB_REPLICATION_FACTOR`	`cluster.replication_factor`	3	Number of Raft voter replicas per shard (RF<N mode).
`KINEDB_FORCE_REPLICATION_FACTOR`	`cluster.replication_factor`	not set	Recovery only: forces `cluster.replication_factor` on this node for the life of the process.
`KINEDB_SPLIT_COOLDOWN_SECS`	`cluster.split.cooldown_secs`	60 secs	How long (secs) after a successful split before the same shard can be split again.
`KINEDB_SPLIT_DRAIN_AFTER_TICKS`	`cluster.split.drain_after_ticks`	3	How many consecutive SplitManager ticks a size-eligible shard may be refused for cross-shard transaction work (the gtxn-in-flight or gtxn-holds refusal) before its leader arms a drain gate on the metadata leader.
`KINEDB_SPLIT_INTERVAL_MS`	`cluster.split.interval_ms`	10000 ms	How often (ms) the SplitManager wakes to check if any shard has crossed the split threshold.
`KINEDB_SPLIT_MAX_CONCURRENT_APPLIES`	`cluster.split.max_concurrent_applies`	2	How many split applies this node may have in flight at once.
`KINEDB_MAX_SHARDS`	`cluster.split.max_shards`	256	Global cap on the total number of shards.
`KINEDB_SPLIT_THRESHOLD_BYTES`	`cluster.split.threshold_bytes`	67108864 bytes (64 MiB)	Shard size (bytes) at which the SplitManager proposes an automatic split.
`KINEDB_FLUSH_GENERATION_BYTES`	`cluster.storage.flush_generation_max_bytes`	67108864 bytes (64 MiB)	Payload-byte bound on ONE flush generation.
`KINEDB_FLUSH_GENERATION_KEYS`	`cluster.storage.flush_generation_max_keys`	512000	Key-count bound on ONE flush generation, the companion to flush_generation_max_bytes — tree-mutate cost tracks KEYS while block-write cost tracks BYTES, and neither rate alone predicts cycle time across row shapes, so both are checked and whichever binds first rotates.
`KINEDB_PER_SHARD_MEMTABLE_BYTES_LIMIT`	`cluster.storage.memtable_max_bytes`	268435456 bytes (256 MiB)	Per-shard memtable byte cap.
`KINEDB_SEQUENCER_PENDING_BYTES_LIMIT`	`cluster.calvin.sequencer_pending_bytes_limit`	536870912 bytes (512 MiB)	Per-shard Calvin sequencer staging-buffer byte cap.
`KINEDB_RAFT_LOG_PENDING_BYTES_LIMIT`	`cluster.calvin.raft_log_pending_bytes_limit`	536870912 bytes (512 MiB)	Per-shard cap on bytes the leader has proposed but not yet replicated to min(match_index) across peers.
`KINEDB_RAFT_LOG_COMPACTION_BYTES`	`cluster.calvin.raft_log_compaction_bytes`	67108864 bytes (64 MiB)	Per-shard byte budget for the byte-aware raft-log compaction trigger.
`KINEDB_RAFT_LOG_MARGIN_BYTES`	`cluster.calvin.raft_log_margin_bytes`	4194304 bytes (4 MiB)	Byte budget for the raft-log compaction SAFETY MARGIN — the trailing band each shard holds ABOVE effective_floor so a briefly lagging peer can still be served by AE retransmit.
`KINEDB_RAFT_LOG_COMPACTION_NODE_BYTES`	`cluster.calvin.raft_log_compaction_node_bytes`	536870912 bytes (512 MiB)	Node-wide ceiling on the sum of every local shard's retained raft-log body.
`KINEDB_PEER_BROADCAST_QUEUE_BYTES_LIMIT`	`cluster.calvin.peer_broadcast_queue_bytes_limit`	1073741824 bytes (1 GiB)	Per-peer outbound broadcast-queue byte cap.
`KINEDB_WATCH_MAX_FILL_ROWS`	`cluster.watch.max_fill_rows`	100000000	The most changed rows that ONE watch fill walks: a WATCH … SINCE resume, a SINCE START fill, or the set-change diff of ALTER WATCH.
`KINEDB_WATCH_MAX_PREDICATES_PER_CLIENT`	`cluster.watch.max_predicates_per_client`	100000	The most watch predicates that ONE connection holds: the sum, over the watches of the connection, of the predicates of each watch (a watch with a bare WHERE or no WHERE counts 1).
`KINE_DIAG`	`node.diag.diag`	false	Enable diagnostic logging mode (presence of KINE_DIAG enables).
`KINE_TRANSFER_LOG`	`node.diag.transfer_log`	false	Enable verbose logging of shard transfer operations (KINE_TRANSFER_LOG).
`KINEDB_RAFT_ELECTION_US`	`node.raft.election_us`	3000000 us	Raft election timeout in microseconds.
`KINEDB_RAFT_HEARTBEAT_US`	`node.raft.heartbeat_us`	500000 us	Raft leader heartbeat interval in microseconds.
`KINEDB_JOINER_BOOT_TIMEOUT_MS`	`node.raft.joiner_boot_timeout_ms`	10000 ms	How long (ms) a joining node waits to receive its first AppendEntries before giving up and exiting with an error.
`KINEDB_JOIN_PIN_BUDGET_MS`	`node.raft.join_pin_budget_ms`	60000 ms	How long (ms) a shard leader holds the Raft-log compaction floor pinned for a reserved joiner.
`KINEDB_JOIN_PIN_WAIT_MS`	`node.raft.join_pin_wait_ms`	5000 ms	How long (ms) ONE shard acquisition waits for a peer that can serve a snapshot at or above the reserved pin.
`KINEDB_VOTER_EVICT_MS`	`node.raft.voter_evict_ms`	not set (an operator evicts a dead voter)	How long (ms) a voter must be continuously dead in SWIM before the Raft observer removes it from the voting set.
`KINEDB_RAFT_SWIM_POLL_MS`	`node.raft.swim_poll_ms`	500 ms	How often (ms) the Raft membership observer polls SWIM state to detect peer additions and removals.
`KINEDB_FILE_UPLOAD_SWEEP_INTERVAL_MS`	`node.files.upload_sweep_interval_ms`	3600000 ms	How often (ms) this node sweeps the upload ledger: it marks the rows of uploads that sent no manifest, and deletes them one TTL later.
`KINEDB_FILE_UPLOAD_TTL_MS`	`node.files.upload_ttl_ms`	86400000 ms	How long (ms) the chunks of an upload with no manifest stay, counted from the first sweep that sees them.

The server reads more KINEDB_* variables for diagnostics, tests and recovery. They are not a stable interface, and this page does not list them.

Ports and endpoints

Everything uses one TCP port, 4820 by default. Monitoring tells how to watch a running node.

Path	What it is
`GET /health`	Health check. It needs no login. It answers `503 Service Unavailable` while an index is UNUSABLE, and the body's `unusable_indexes` array carries the repair for each one. See Troubleshooting. It also answers `503` after the first boot until the user `root` exists. See A login fails right after the first start. It also answers `503` after each start until the Raft groups replayed their log. See A request answers "the node is not ready" right after a start.
`POST /login`	`{"username": ..., "password": ...}` in, a token out. An API secret logs in with `{"username": ..., "api_secret": "kdsk_..."}` in place of the password (API secrets).
`POST /token/refresh`	`Authorization: Bearer <token>` in, a new token out, like `/login`. The old token ends at once. See Refresh a token.
`POST /sql`	SQL text in, JSON out. Send `Authorization: Bearer <token>`. One statement per request. `?db=<name>` selects the database of the request; it is the only query parameter, and an other one gets `400 Bad Request`. `USE` is refused here: write `db.table`, or use `?db=`. A refused statement (also an unknown `?db=`) gets `200 OK` with `"type": "error"` in the body, so read the body, not only the status. See Databases for each status. With `content-type: application/vnd.kinedb.sql-params` the body is the binary `[sql, params]` of `sql(text, params)`; a body that does not decode gets `400 Bad Request`. With `content-type: application/vnd.kinedb.sql-txn` the body is one complete transaction of `transaction(statements)`; a body that does not decode gets `400 Bad Request`. `BEGIN`, `COMMIT` and `ROLLBACK` are refused.
`/ws`	A WebSocket with the binary protocol of the client libraries. Live queries (`WATCH`) use it.
`/ui/`	The web console.
`/ui/docs/`	These docs, for the version of this node. `/ui/docs/llms-full.txt` holds all pages in one file.
`/ui/demos/`	Five browser demonstrations of the embedded engine. See Embedded database.
`GET /auth/files/<path>`	The files of the custom login pages. It needs no login. Without a directory of custom login pages it answers `404 Not Found`. See Custom login pages.

How the node serves `/ui/`

The node serves static files from the webroot, and it applies these rules in this order:

A path that names a file serves that file. /ui/docs/index.html is an example.
A path that names a directory with an index.html serves that page. /ui/demos/ serves <webroot>/demos/index.html.
A path without an extension and without a trailing slash serves the file with that name plus .html, when that file exists. /ui/docs/run serves <webroot>/docs/run.html. So a docs URL on a node is the same as on https://docs.kinedb.com.
A directory path without the trailing slash answers 302 Found and sends the browser to the slashed form. /ui/demos redirects to /ui/demos/. The pages in a directory use relative links, so the trailing slash matters.

Any other path answers 404 Not Found. The node never lists a directory, and it never answers one path with the page of another: a 404 means the file is not in the webroot of this node. The console keeps its own routes in the URL fragment (/ui/#/query), so a console route never reaches the server.

You can replace the built-in login pages with your own static files. The node finds the directory one time, at the start, in this order:

--auth-webroot DIR.
KINEDB_AUTH_WEBROOT, when it is set and not empty.
auth/ in the web console directory, when that directory exists.
No directory: the built-in pages apply.

A path that is not a directory gives a warning at the start, and the built-in pages apply. --no-web turns off the console only, so steps 1 and 2 still apply.

Put these files in the directory:

login.html replaces the answer of GET /auth/login. It also replaces the 401 Unauthorized answer of POST /auth/password after a failed password.
error.html replaces each error page of /auth/. It gets the same HTTP status as the built-in error page. The node replaces each in the file with the fixed text of the built-in page, HTML-escaped. A file with no does not change. The node also replaces each with the reason of the failure, when it has one: after a failed code exchange, the answer of the token endpoint of the provider (for example error="invalid_client"). The text has no control character and at most 300 characters. The built-in page never shows it. The node log holds the same reason. A 429 answer keeps its Retry-After header.
link.html replaces the answer of GET /auth/link: the prompt to link a second provider. The node checks the link cookie before it selects the page, so an error gives the error page, never link.html. The page reads its values from GET /auth/link/prompt. See A custom link page.
Each other file is served at /auth/files/<path>. A page at /auth/login links files/style.css. A missing page file gives the built-in page.

The node checks the return address before it selects the page. A refused address gives the error page, never the login page.

The node reads each file again on each request, so a changed file needs no restart. A path that contains .., an absolute path and a symbolic link out of the directory each answer 404 Not Found. The node decodes no % escape in the path.

The headers of a custom page are fixed. You cannot change them:

Content-Security-Policy: default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self'; connect-src 'self'; form-action 'self'; base-uri 'none'; frame-ancestors 'none'
X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: same-origin, Cache-Control: no-store. With same-origin, a form POST of the page carries its origin, and the node accepts it. A request to another site carries no referrer.

Each file under /auth/files/ gets X-Content-Type-Options: nosniff and the same Content-Security-Policy. So a page can load a script, a style sheet, an image or a font from this node only, and no other site can show the page in a frame.

A custom page is static: the node fills in no value. The login page reads the return and app parameters from its own address, and its script calls the same public routes as the built-in page:

GET /auth/providers gives the login providers as JSON.
GET /auth/start?provider=<name>&return=<address>[&app=<name>] starts the login at a provider. The page links to this address for each provider.
POST /auth/password with the form fields username, password, return and app logs in with a password. A correct password answers 303 See Other to the return address, with the session cookie.

A failed password answers 401 Unauthorized with login.html again, so the page cannot see the failure from its own body. To show a message, the page can send the form with fetch and read the 401 status.

For a set of login pages with the style of the web console, see Styled login pages.

A node behind a reverse proxy on the same host can listen on loopback only: --listen 127.0.0.1. Then the proxy reaches the node, and other hosts do not.

An HTTP/1.1 connection is persistent by default, so a reverse proxy (Caddy, nginx) can pool its connections to the node. When the node closes a connection after a response, the response says Connection: close. Send Connection: close on a request when you want the socket closed after the answer.

TLS and encryption

kinedb has no TLS of its own. Every connection to a node is plaintext: HTTP and WebSocket from the clients, and the traffic between the nodes. Nothing is encrypted at rest.

TLS for the clients

Put a reverse proxy that terminates TLS in front of the client port. The proxy must pass the WebSocket upgrade of /ws: it forwards the Upgrade: websocket and Connection: upgrade headers. One proxy hostname carries /sql, /login, /ws and /ui/.

Tell the node the address of the proxy with --trusted-proxy (or KINEDB_TRUSTED_PROXIES). Without it, every client has the address of the proxy. Then the limit of failed password checks counts all clients as one source, and one client with a wrong password delays the login of all of them. SHOW AUDIT also shows the proxy address in place of the client address. With --trusted-proxy, the node reads X-Forwarded-For from the right. It skips each address that is a trusted proxy, and takes the first address that is not. The node ignores the header of a connection that does not come from a trusted proxy. The proxy must set the header. Caddy sets it by itself. nginx sets it with the X-Forwarded-For line below. For the two examples below, start the node with --listen 127.0.0.1 --trusted-proxy 127.0.0.1.

With Caddy, reverse_proxy passes the upgrade by itself, and Caddy gets the certificate:

db.example.com {
    reverse_proxy 127.0.0.1:4820
}

With nginx:

nginx

server {
    listen 443 ssl;
    server_name db.example.com;
    ssl_certificate     /etc/ssl/db.example.com.crt;
    ssl_certificate_key /etc/ssl/db.example.com.key;

    location / {
        proxy_pass http://127.0.0.1:4820;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 1h;   # a WATCH socket can be idle for a long time
    }
}

Do not let a client go around the proxy. On one server, start the node with --listen 127.0.0.1. A cluster node cannot do that, because its peers use the same port: let the firewall accept that port only from the proxy and from the other nodes.

The web interface under a path prefix

A proxy can serve the web interface under another path, for example /admin/. The other paths of the host can go to another site.

The web interface asks the server for two groups of paths only:

The paths under its own folder. The files of the page (assets/, fonts/, favicon.svg, kinedb-icon.svg), and the API: sql, login, login/challenge, logout, health and ws. A page at /admin/ asks /admin/sql and wss://<host>/admin/ws. The node answers these API paths under /ui/ too, so /ui/sql is the same as /sql.
The provider login at the root of the host: /auth/*. These paths are not under the folder. The cookies of the login have the paths /auth/ and /auth/session, and the provider sends the browser back to https://<host>/auth/callback.

So the proxy needs two rules: the folder, and /auth/*. With Caddy:

db.example.com {
    redir /admin /admin/
    handle_path /admin/* {
        rewrite * /ui{path}
        reverse_proxy 127.0.0.1:4820
    }
    handle /auth/* {
        reverse_proxy 127.0.0.1:4820
    }
    # The other paths of the host go to another site.
    reverse_proxy 127.0.0.1:8080
}

handle_path removes /admin from the path, and rewrite adds /ui. So /admin/sql arrives at the node as /ui/sql. The redir line sends /admin to /admin/: the page must end with /, because it finds its files relative to its folder. The proxy must keep the Host header of the browser: the node chooses the providers of the login page by this host. Caddy keeps it by itself. With nginx, set proxy_set_header Host $host;.

The built-in docs pages use the fixed folder /ui/docs/. Behind these two rules, the link Docs of the web interface opens /admin/docs/, but the pages of the docs ask /ui/docs/ for their files. Read the docs at https://docs.kinedb.com, or add a third rule for /ui/docs/*.

Do these steps for a login with a provider (see Log in with an identity provider):

Give the redirect address to the provider. The address is at the ROOT of the host, never under the folder:
```
https://<host>/auth/callback
```
The page Identity providers of the web interface shows this address for the host of the page, with a Copy button. Add it to the redirect URLs of the client at the provider (Google: "Authorized redirect URIs").
Use the same text in Redirect URIs of the provider in kinedb. Open the page Identity providers, click the name of the provider (or New provider), and put the address in Redirect URIs. Then click Save (or Create). The address at the provider and the address in Redirect URIs must be the same text. kinedb sends the stored text to the provider, and the provider compares the two. The create form uses the address of the host as the default of this field.
Enter the host name in Host names. In the same form, enter the host name of the page, for example cvr.example.com, in Host names. The hint under the field shows the host name of the page, and the create form uses it as the default. Without a host name, the provider accepts a login on any host name that reaches the node, and the page Identity providers shows a warning on the provider (see Limit the host names of a provider).
Allow the return to the folder. After the login, the server sends the browser back to the folder of the page, here /admin/. The server accepts a return only when a pattern of the app kinedb-admin allows it. The default list holds only /ui/*, so add the folder to the list in the web interface:
1. Open the page Apps.
2. Click Edit on the row kinedb-admin.
3. In Return addresses, add /admin/* on a new line. Keep /ui/*. The hint under the field shows the pattern of the page, for example "Example for this page: /admin/*".
4. Click Save.
Without this step, a click on the provider button shows the page "Login failed" with the text "The return address is not allowed." (HTTP 400). See Apps and return addresses.
Trust the proxy. This step is a start option of the node, not a setting of the web interface. Start the node with --trusted-proxy <address of the proxy> (or KINEDB_TRUSTED_PROXIES), for example --trusted-proxy 127.0.0.1. The proxy must send X-Forwarded-Proto: https. Caddy sends it by itself. With nginx, set proxy_set_header X-Forwarded-Proto $scheme;. The node gives the cookies of the login the Secure flag (and the names __Host-kinedb_login, __Host-kinedb_session and __Host-kinedb_sso) only when the connection comes from a trusted proxy and the last value of X-Forwarded-Proto is https. Without a trusted proxy, the cookies have no Secure flag, and every client has the address of the proxy (see TLS for the clients).

Test the setup:

curl https://<host>/auth/providers gives HTTP 200, and the JSON lists the provider. The list holds a provider only when the edit form shows Enabled and Login page on, Host names is empty or holds the host, and one entry of Redirect URIs is for the host.
The page https://<host>/admin/ shows the logo in the top-left corner.
The user menu of the web interface shows Link an account.
A login with the provider ends on https://<host>/admin/.

The client URLs behind such a proxy:

Python. KineDB("https://db.example.com"). The client changes https to wss for the socket, so ws_url is not necessary.
JavaScript in a browser. A page that the proxy serves needs no URL: the client reads https: from the page and uses wss://.
JavaScript in Node.js. Give both URLs: new KineDB('https://db.example.com', 'wss://db.example.com/ws'). Without the second argument, the client uses ws://127.0.0.1:4820/ws.
kd-tool and the kinedb command-line client have no TLS option. They connect in plaintext (kd-tool --host H --port P, kinedb --connect HOST:PORT). Run them on the host of the node, or on a network that you trust.
Rust (kinedb::remote::Remote, feature remote). Give a wss:// URL and a connector. See The Rust client over wss://.

The Rust client over wss://

The Rust client has no TLS code. Your program gives a connector: a function from a host and a port to a duplex stream (kinedb_protocol::stream::Connector). The connector opens the TCP connection, does the TLS handshake and checks the certificate, for example with rustls. The client then sends the WebSocket upgrade on that stream.

rust

use kinedb::remote::Remote;

// `my_tls_connector()` gives a `Connector` that wraps rustls (or another TLS library).
let mut remote = Remote::connect_url_with("wss://db.example.com/ws", my_tls_connector())?;
remote.login_blocking("alice", "secret")?;

A wss:// URL with no port uses port 443. A ws:// URL with no port uses port 80.
Remote::connect_url("ws://127.0.0.1:4820/ws") connects in plaintext, with no connector. Remote::connect(host, port) works as before.
Remote::connect_url with a wss:// URL fails at once, and the error names the connector. The client opens no connection, so nothing goes out in plaintext.
An error of the connector, for example an unknown certificate, comes back in the error of the connect.
Remote::reconnect dials the same URL with the same connector.
The connector must give a stream that implements kinedb_protocol::stream::WsStream. The documentation of Connector holds an example with rustls. The client turns on TCP keepalive only for its own plaintext connection. A connector turns it on for its own socket.

The traffic between the nodes

The nodes of a cluster talk to each other on the client port, in plaintext. A node dials the address of a peer (KINEDB_ADVERTISE_ADDR) directly, so a proxy in front of the client port does not cover this traffic. --calvin does not change the listener.

The cluster key never crosses the wire: a node proves that it holds the cluster key with a keyed hash of its link key (a key derived from the cluster key) that expires in about 3 minutes. That is authentication, not encryption. The rows, the Raft log entries and the replies cross the wire in plaintext. Put the nodes on a private network, or connect them through an encrypted tunnel (for example WireGuard or an IPsec VPN) when the network is not trusted.

Encryption at rest

kinedb encrypts nothing on disk or in a bucket:

the data files, the write-ahead log and the Raft log (<data-dir>/raft_wal) under --data-dir;
the audit spool (<data-dir>/audit_spool): the audit events of this node (logins, refusals) that the node did not yet send to the cluster. It holds no password, no hash and no token. The node deletes a spool file when every event in it is in the cluster;
the objects that a node writes to S3. The node asks for no server-side encryption; the default encryption of the bucket, if it has one, applies;
the dumps of kd-tool backup.

When a policy requires encryption at rest (for example ISO/IEC 27001:2022 control 8.24), encrypt below kinedb: an encrypted file system or volume (LUKS, an encrypted cloud disk), the default encryption of the bucket, and an encrypted place for the dumps.

The key of a node

A server with --data-dir has a lock key and a key pair (ECDSA P-256). It makes both at its first start and keeps them in the data directory:

the lock key: in a TPM 2.0 on Linux when the server can open /dev/tpmrm0 (the file lock.tpm holds the sealed blob). Else in the file lock.key. A copy of the data directory with lock.key is a copy of the key. At each start the node logs B3963: the lock key is in the TPM or B3963: the lock key is in a key file: <why>;
the private half of the key pair: in the file node.key, sealed by the lock key. Each start logs B3963: the node key is ready: fingerprint <hex>. The fingerprint does not change over a restart.

Both files are private to the owner (mode 0600). The node never makes a new key over an existing file. A node.key that does not open refuses the start: see Troubleshooting. -e, -f and the REPL make no key.

The cluster key

The cluster has one cluster key pair (ECDSA P-256, for signing). Each node keeps its own copy in the file cluster.key of its data directory, sealed by its own lock key. The copy is local: it is not replicated, and a copy of one node does not open on another node. (The cluster key pair is not the cluster name that --cluster-key gives.)

A server with no cluster.key, no --join and no multi-node history is the first node of a new cluster (or a single node). It makes the pair at its first start.
A server that starts with --join never makes a pair. It reads its copy when it has one.
A new server (no multi-node history) that starts with --join and has no copy gets the pair from a running node BEFORE it joins the cluster. It asks the --join node, then each node that it knew before. Both nodes prove KINEDB_JOIN_SECRET (an environment variable that the node never writes to disk), so the joining node and the running nodes need the same value. Each try waits at most 2 s for a node. A node that gets no key does not join, and it exits after KINEDB_JOINER_BOOT_TIMEOUT_MS with FATAL: no running node gave the cluster key: check KINEDB_JOIN_SECRET on both sides. A running node with no join secret answers 503; a joining node with another join secret gets 403.
A new server that starts with --join, has no copy and has no KINEDB_JOIN_SECRET refuses to start (exit code 3), and the text names the variable.
A member of a cluster from before B3964 has multi-node history and no copy, and its cluster may have no key. The join secret is for the first start of a node only: such a member starts with no join secret and joins first, as before.
Each start logs B3964: the cluster key is ready: version <n>, fingerprint <hex>, or B3964: this node has no cluster key yet on a joining node with no copy.

The file is private to the owner (mode 0600). The node never makes a new pair over an existing file. A cluster.key that does not open refuses the start: see Troubleshooting.

The node takes three keys from the cluster key pair, each with its own fixed context:

the link key: the key of the peer authentication. Two nodes with another cluster key refuse each other's peer connection;
the token key material: a part of the key that signs the bearer tokens, the SSO cookies, the login challenges and the codes of the identity provider (Authentication);
the seal key: the key of the sealed client secrets and of the private key that signs the ID tokens.

A node with peers and no cluster key yet (a joiner before its key arrives) refuses a peer credential, and it answers 503 to a login and to a token check. A copy of cluster.key together with the lock key of its node (lock.key on a node with no TPM) holds all three keys: keep both as private as the root password.

The upgrade from the cluster secret

A cluster of an older version authenticated its nodes with the cluster secret (an environment variable that this version does not read). The old and the new peer authentication refuse each other, so the upgrade is not rolling. It has two steps: the build before this one (items 1 and 2), then this build (item 3):

Install the build before this one on every node: it makes the cluster key and agrees it between the nodes, and it seals the client secrets and the ID token keys again under the cluster key. Keep the cluster secret of the older version on every node, and set KINEDB_JOIN_SECRET.
Wait until each node logs the cluster key is ready and the log shows no [B3971_RESEAL_SKIPPED] line (no value is still sealed under the cluster secret).
Stop all nodes, install this build on each node, and start all nodes with KINEDB_JOIN_SECRET.

A direct jump from a build before the cluster key refuses at boot (exit code 3): a node with multi-node history, no cluster.key and no KINEDB_JOIN_SECRET can never get the cluster key, so it can open no peer link. A node with no cluster.key and with KINEDB_JOIN_SECRET starts, joins, and asks a running node for the key; while no node has it, it logs a warning on each try that names the first step. A value that is still sealed under the cluster secret also refuses the start (exit code 3); the text names each value, and KINEDB_ALLOW_LOST_V1_SECRETS=1 accepts the loss (then set each client secret again and run ALTER CLUSTER ROTATE ID TOKEN KEY).

A node whose cluster.key is not the key of the cluster is refused by its peers. The peer logs [B3971_PEER_KEY_MISMATCH] with both keys (this node <version>/<fingerprint>, the peer <version>/<fingerprint>); compare them with the line the cluster key is ready of each node.

A node that holds the cluster key also refuses a peer that names a cluster but sends no credential. Such a peer has no cluster.key: a node of another cluster with the same --cluster-key name, a returning member before it gets the key, or a node in quarantine. The node logs [B1411_UNPROVEN_PEER] (rate-limited) and counts peer_auth.unproven_peer_refused_total. A returning member gets the key over HTTP with KINEDB_JOIN_SECRET, then its peers accept it. The name --cluster-key alone never separates two clusters: the cluster key pair does.

Every bearer token, SSO cookie and code of the older build gets 401 after the upgrade: users log in again. The first login makes the new token key.

Run kinedb ​

Get kinedb ​

Download the binaries ​

The container image ​

One server ​

Stop the server ​

Log in ​

A cluster ​

Upgrade kinedb ​

Embedded: SQL without a server ​

Command-line arguments ​

The smallest --memory budget ​

The connection and peer buffers of a memory budget ​

Environment variables ​

Setup ​

Tuning ​

Ports and endpoints ​

How the node serves /ui/ ​

Custom login pages ​

TLS and encryption ​

TLS for the clients ​

The web interface under a path prefix ​

A provider login under a path prefix ​

The Rust client over wss:// ​

The traffic between the nodes ​

Encryption at rest ​

The key of a node ​

The cluster key ​

The upgrade from the cluster secret ​