# KineDB documentation > KineDB is one database engine that runs as one server, as a cluster of servers, or embedded in an application. These docs tell how to run it, how to connect to it, and which SQL it accepts. Revision: ca3bd1f6f170c7a9b1b0341a23c2553c708e32d2 --- Source: https://docs.kinedb.com/index.md # kinedb kinedb is a database engine. The same engine runs as one server, as a cluster of servers, or inside your application. In all three cases, your application sends SQL. kinedb stores tables, document collections (JSON documents with an `_id`) and key-value collections. Every commit makes a new version of the data, and the old versions stay readable. So a branch of a whole database costs one pointer, not a copy. ## The three ways it runs ### One server The `kinedb` binary starts a server on port 4820. Clients connect over HTTP (`POST /sql`, JSON) or over a WebSocket (`/ws`, a binary protocol). The same port serves a web console at `/ui/` and these docs at `/ui/docs/`. **Status:** works today. See [Run kinedb](./run.md). ### A cluster of servers A cluster is any number of `kinedb` servers that act as one database. You start every node with `--calvin`, and every node after the first with `--join
` of a running node. The nodes find each other with a gossip protocol. The cluster splits each table into shards by key range, and it splits a shard again when the shard grows. Each shard has replicas on several nodes. The replicas agree on one order of the writes with Raft, and every replica applies the writes in that order. A write is durable when the cluster acknowledges it: each replica writes the entry to disk before it counts toward the quorum. **Status:** works today. See [Architecture](./architecture.md). ### Embedded in your application The engine is also a library. In Rust, the `kinedb` crate opens a database with `Db::open(path)` or `Db::open_in_memory()`, and runs SQL with `db.execute(sql)`. No server runs. The `kinedb` binary gives you the same engine from a shell: ```sh kinedb --data-dir ./data -e "SHOW TABLES" ``` **Status:** the library works on disk and in memory, and it compiles to WebAssembly. ## For AI agents These docs are written for coding agents first. Every page is also plain Markdown: - The index of all pages: - All pages in one file: - One page: replace `.html` with `.md` in its URL, for example . Every kinedb node also serves the docs of its own build. Use these when you work against a specific server, because they match that server exactly: `http://:4820/ui/docs/llms-full.txt`. To give an agent the docs, put this in the `CLAUDE.md` (or `AGENTS.md`) of your project: ```md ## KineDB This project uses KineDB. Before you write KineDB SQL or client code, read https://docs.kinedb.com/llms-full.txt. The server of this project also serves the docs of its own version at http://:4820/ui/docs/llms-full.txt; when the two differ, the server's copy is correct. ``` ## Where to go next - [Run kinedb](./run.md): Docker or the binary, every argument and every environment variable. - [SQL reference](./sql.md): every statement that kinedb runs today. - [Architecture](./architecture.md): how it works today, and where it is going. - [JavaScript client](./clients/javascript.md) and [Python client](./clients/python.md): connect from an application. --- Source: https://docs.kinedb.com/run.md # Run kinedb kinedb is one program, `kinedb`. The same program is a server, a node of a cluster, a SQL shell, and an embedded engine for one-off SQL. This page tells how to start it, and it lists every command-line argument and every environment variable. ## Get kinedb Today the only distribution is a container image. The pull needs no login: ```sh docker pull git.kinedb.com/kinedb/kinedb:latest ``` - **Tags.** `latest`, and one tag per build: the first 12 characters of its git commit. - **Content.** The static `kinedb` binary at `/app/kinedb`, and the web console with these docs at `/app/dist`. The image is built for `linux/amd64`. - **Defaults.** The image runs as the user 65532. Its default command is `--port 4820 --data-dir /data`, and `/data` is a volume. The binary in the image is static, so you can copy it out and run it on any x86-64 Linux: ```sh id=$(docker create git.kinedb.com/kinedb/kinedb:latest) docker cp "$id":/app/kinedb ./kinedb docker rm "$id" ``` ## One server ```sh docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \ -e KINEDB_ROOT_PASSWORD='choose-a-password' \ git.kinedb.com/kinedb/kinedb:latest ``` Or with the binary: ```sh KINEDB_ROOT_PASSWORD='choose-a-password' ./kinedb --port 4820 --data-dir ./data ``` - Without `--data-dir`, the data stays in memory, and it is lost when the process stops. - One port carries everything: the clients, the peers, the web console at `/ui/`, and these docs at `/ui/docs/`. ### Log in Clients must log in by default (`KINEDB_CLIENT_AUTH=required`). The first boot creates the user `root`. With `KINEDB_ROOT_PASSWORD`, `root` gets that password. Without it, the server makes a password and writes it once to its log: ```sh docker logs kinedb 2>&1 | grep 'GENERATED password' ``` The SQL shell reads the password from `KINEDB_ROOT_PASSWORD` and logs in as `root`. Use `--user` and `--password-env` for another user. There is no password prompt. ```sh docker exec -it kinedb /app/kinedb --connect 127.0.0.1:4820 docker exec kinedb /app/kinedb --connect 127.0.0.1:4820 -e "SHOW DATABASES" ``` Over HTTP, `POST /login` gives a token, and every other request carries it: ```sh curl -s -X POST http://localhost:4820/login \ -d '{"username": "root", "password": "choose-a-password"}' # the answer is a JSON object with a "token" curl -s -X POST http://localhost:4820/sql \ -H "Authorization: Bearer " \ --data-binary 'SHOW DATABASES' ``` An application uses a client library: [JavaScript](./clients/javascript.md) or [Python](./clients/python.md). `KINEDB_CLIENT_AUTH=permissive` turns the login off: then anyone who reaches the port can read and write. Use it only on a network that you trust. ## A cluster A cluster is any number of `kinedb` nodes. To start one: 1. Choose a cluster secret. Give every node the same `KINEDB_CLUSTER_SECRET`. Peer authentication is on by default, and a cluster node without a secret refuses to start. 2. Start the first node with `--calvin`. 3. Start every other node with `--calvin --join `. 4. Set `KINEDB_ADVERTISE_ADDR` on every node to the address that the other nodes use to reach it. A node that listens on all interfaces, which includes every container, otherwise tells its peers `127.0.0.1`. Three hosts, `10.0.0.1` to `10.0.0.3`, with Docker: ```sh # on 10.0.0.1: the first node docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \ -e KINEDB_CLUSTER_SECRET='a-long-random-secret' \ -e KINEDB_ROOT_PASSWORD='choose-a-password' \ -e KINEDB_ADVERTISE_ADDR=10.0.0.1 \ git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin # on 10.0.0.2 (and the same on 10.0.0.3, with its own address) docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \ -e KINEDB_CLUSTER_SECRET='a-long-random-secret' \ -e KINEDB_ROOT_PASSWORD='choose-a-password' \ -e KINEDB_ADVERTISE_ADDR=10.0.0.2 \ git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin --join 10.0.0.1:4820 ``` Arguments after the image name replace the default command, so they repeat `--port` and `--data-dir`. - **Ports.** The nodes talk to each other on the same TCP port as the clients. There is no other port to open. - **Replicas.** Each shard has 3 replicas by default. `KINEDB_REPLICATION_FACTOR` sets the first value on a new cluster; after that, `SET CLUSTER cluster.replication_factor = ` changes it. - **Joining.** A joining node waits 10 seconds for the leader to accept it, and then it exits with an error (`KINEDB_JOINER_BOOT_TIMEOUT_MS`). - **Durability.** A write is durable when the cluster acknowledges it: each replica writes the entry to disk before it counts toward the quorum. Each node keeps its Raft log in `/raft_wal`. ## Embedded: SQL without a server With `-e` or `-f` and without `--connect`, `kinedb` runs the SQL in its own process and exits. No server starts, and no port opens. With `--data-dir`, the data stays for the next call. ```sh ./kinedb --data-dir ./data -e "CREATE TABLE t (id INT, name TEXT, PRIMARY KEY (id))" ./kinedb --data-dir ./data -e "INSERT INTO t (id, name) VALUES (1, 'a'); SELECT * FROM t" ./kinedb --data-dir ./data -f setup.sql ``` - `-e` splits the SQL on `;` and runs the statements in order. `-f file.sql` does the same with a file, and `-f file.kd` runs a kd script. - `-e` stops at the first statement that fails. It prints a line that starts with `ERROR:`, and it exits with 1. - Some statements need a server; the [SQL reference](./sql.md) marks them. In Rust, the `kinedb` crate is the same engine as a library (`Db::open(path)`, `db.execute(sql)`). ## Command-line arguments | Argument | Default | What it does | |---|---|---| | `--port PORT` | `4820` | The TCP port for the clients, the peers, the web console and these docs. | | `--data-dir DIR` | not set | The directory of the data. Without it, the data stays in memory. | | `--memory SIZE` | not set | The total memory budget of the process, for example `4GB` or `3800MB`. kinedb enforces it with its own accounting. It needs `--data-dir`, and it does not work with `--s3` or `--connect`. | | `--cache-slab SIZE` | not set | A block cache that is allocated at start. Under `--memory`, it must fit the budget. Alone, it is the older way to size the cache. | | `--gc-retention SPEC` | not set | Turns on the garbage collection of dead blocks, and sets the history to keep: `commits` (the head and N older commits; `0commits` keeps only the head) or `s`, `m`, `h`, `d` (for example `7d`). Without it, nothing is reclaimed. | | `--calvin` | off | Cluster mode: Raft and the deterministic sequencer. Every node of a cluster needs it. | | `--join HOST:PORT` | not set | Join a cluster through a running node. Use it with `--calvin`. | | `--peer-auth MODE` | `required` | `required`: the nodes must prove that they know the cluster secret. `permissive`: they need not. | | `--cluster-key KEY` | `kinedb` | The name of the cluster. A node ignores the gossip of a node with another key. It is not a secret. | | `--realm NAME` | not set | The realm of the node. Leave it unset for one realm. | | `--s3 URI` | not set | Store the blocks in S3: `s3://KEY:SECRET@HOST/BUCKET/PREFIX?region=R`. `--data-dir` is then the local cache; without it, a temporary directory. | | `--webroot DIR` | found | The directory of the web console. Without it, kinedb looks for `dist` next to the binary, then for `dist` and `web/dist` in the working directory. | | `--no-web` | off | Serve no web console and no docs. | | `--connect HOST:PORT`, `-c` | not set | Client mode: open a SQL shell on a running server, or run `-e` or `-f` there. | | `--user NAME` | `root` | Client mode: the user to log in as. | | `--password-env VAR` | `KINEDB_ROOT_PASSWORD` | Client mode: the environment variable that holds the password. | | `-e SQL` | not set | Run SQL and exit. Without `--connect`, the SQL runs in this process. | | `-f FILE` | not set | Run a file and exit: `.kd` is a kd script; any other file is SQL. | | `--show-types` | off | Client mode: put the column type in the header of a result, for example `price:Float`. | | `--help`, `-h` | | Print the usage and exit. | ## Environment variables ### Setup | Variable | Default | What it does | |---|---|---| | `KINEDB_ROOT_PASSWORD` | not set | The password of `root`, set at the first boot. Without it, the first boot makes a password and logs it once. The SQL shell also reads it to log in. | | `KINEDB_CLIENT_AUTH` | `required` | `required`: clients must log in. `permissive`: anyone can query. | | `KINEDB_PEER_AUTH` | `required` | The same as `--peer-auth`. The flag wins. | | `KINEDB_CLUSTER_SECRET` | not set | The secret that the nodes of a cluster share. It is only an environment variable, so it does not show in a process list. | | `KINEDB_ADVERTISE_ADDR` | not set | The address that the peers use to reach this node, `host` or `host:port` (the port part is ignored). Without it, a node that listens on all interfaces advertises `127.0.0.1`. | | `KINEDB_GC_RETENTION` | not set | The same as `--gc-retention`. The flag wins. | | `KINEDB_RAFT_WAL_DIR` | `/raft_wal` | The directory of the Raft logs. Without a data directory and without this variable, the Raft log stays in memory. | | `KINEDB_LOG_SINK` | not set | `ws://host:port` of a kinedb server that collects logs and metrics. With it, the node sends its logs and metrics there. | | `KINEDB_LOG_SINK_USER`, `KINEDB_LOG_SINK_PASSWORD` | not set | The login at the log sink. | ### Tuning These variables belong to the settings of the server; `SHOW SETTINGS` lists the same settings with the values in force. A variable of a `cluster.` setting sets the first value on a new cluster; after that, `SET CLUSTER = ` changes it. A node reads the variable of a `node.` setting when it starts. | Variable | Setting | Default | What it does | |---|---|---|---| | `KINEDB_CALVIN_APPLY_TIMEOUT_SECS` | `cluster.calvin.apply_timeout_secs` | 60 secs | How long a Calvin write waits for local apply before returning an error. | | `KINEDB_SEQUENCER_BATCH_AGE_MS` | `cluster.calvin.sequencer_batch_age_ms` | 2 ms | Maximum age (ms) a sequencer batch waits before flushing. | | `KINEDB_SEQUENCER_BATCH_SIZE` | `cluster.calvin.sequencer_batch_size` | 1024 | Maximum number of operations in a single Calvin sequencer batch. | | `KINEDB_FORCE_QUORUM` | `cluster.quorum` | not set | Recovery only: forces `cluster.quorum` on this node for the life of the process. | | `KINEDB_REPLICATION_FACTOR` | `cluster.replication_factor` | 3 | Number of Raft voter replicas per shard (RF`. | | `/ws` | A WebSocket with the binary protocol of the client libraries. Live queries (`WATCH`) use it. | | `/ui/` | The web console. | | `/ui/docs/` | These docs, for the version of this node. `/ui/docs/llms-full.txt` holds all pages in one file. | --- Source: https://docs.kinedb.com/sql.md # SQL reference This page lists the SQL that kinedb runs today. Every example in a plain `sql` block ran on the current build, in the order of this page, in one database. So a later example can use a table that an earlier example created. A `-- expect:` comment shows a part of the output. The examples use the embedded mode of the binary, which starts no server: ```sh kinedb --data-dir ./data -e "SHOW TABLES" ``` Some statements need a running server (`kinedb --connect host:port`, `POST /sql`, or a client library). The text before such an example says so. The embedded mode refuses them with `requires a running server`. ## Data types - `INT`: a 64-bit signed integer. - `FLOAT`: a 64-bit floating-point number (IEEE 754). - `BOOL`: `true` or `false`. - `TEXT`: a UTF-8 string. - `BLOB`: any bytes. - `TIMESTAMP`: microseconds since the Unix epoch. - `VECTOR`: a list of numbers. ```sql -- expect: OK CREATE TABLE demo ( id INT, amount FLOAT, is_active BOOL, name TEXT, data BLOB, ts TIMESTAMP, embedding VECTOR, PRIMARY KEY (id) ); ``` ## Tables A table has typed columns and a primary key. ```sql -- expect: OK CREATE TABLE accounts ( account_id INT, owner TEXT, balance FLOAT, created_at TIMESTAMP, PRIMARY KEY (account_id) ); ``` List the tables, and show the columns of one table: ```sql -- expect: 2 rows SHOW TABLES; -- expect: account_id DESCRIBE accounts; -- expect: account_id SHOW COLUMNS FROM accounts; ``` Drop a table and its data: ```sql CREATE TABLE temp_table (id INT, PRIMARY KEY (id)); DROP TABLE temp_table; ``` ## INSERT Insert one row, or several rows in one statement: ```sql INSERT INTO accounts (account_id, owner, balance, created_at) VALUES (1, 'Alice', 1000.0, 1000000); INSERT INTO accounts (account_id, owner, balance, created_at) VALUES (2, 'Bob', 2000.0, 2000000), (3, 'Charlie', 3000.0, 3000000), (4, 'Diana', 4000.0, 4000000); -- expect: 4 SELECT COUNT(*) FROM accounts; ``` ## SELECT ```sql -- expect: Alice SELECT * FROM accounts WHERE account_id = 1; -- expect: Bob SELECT owner, balance FROM accounts WHERE owner = 'Bob'; ``` ### WHERE The comparison operators are `=`, `!=`, `<`, `<=`, `>` and `>=`. `AND`, `OR`, `NOT` and parentheses combine conditions. ```sql -- expect: 1 rows SELECT * FROM accounts WHERE balance > 2000.0 AND owner != 'Diana'; -- expect: 3 rows SELECT owner FROM accounts WHERE NOT account_id = 1; -- expect: Bob SELECT owner FROM accounts WHERE (account_id = 1 OR account_id = 2) AND balance > 1500.0; ``` `IN`, `BETWEEN`, `LIKE`, `ILIKE` (not case-sensitive) and `IS NULL`: ```sql -- expect: 2 rows SELECT owner FROM accounts WHERE account_id IN (1, 2); -- expect: 2 rows SELECT owner FROM accounts WHERE balance BETWEEN 1000.0 AND 2000.0; -- expect: Alice SELECT owner FROM accounts WHERE owner LIKE 'Al%'; -- expect: Alice SELECT owner FROM accounts WHERE owner ILIKE 'al%'; -- expect: 0 rows SELECT owner FROM accounts WHERE owner IS NULL; ``` ### ORDER BY, LIMIT and OFFSET `ORDER BY` sorts by one or more columns, `ASC` (the default) or `DESC`. `OFFSET` comes after `LIMIT`. ```sql -- expect: Diana SELECT owner FROM accounts ORDER BY balance DESC LIMIT 1; -- expect: 2 rows SELECT * FROM accounts LIMIT 2; -- expect: Charlie SELECT owner FROM accounts ORDER BY account_id LIMIT 1 OFFSET 2; ``` ### Aggregates `COUNT(*)`, `SUM`, `AVG`, `MIN` and `MAX`: ```sql -- expect: 4 SELECT COUNT(*) FROM accounts; -- expect: 10000 SELECT SUM(balance) FROM accounts; -- expect: 2500 SELECT AVG(balance) FROM accounts; -- expect: 1000 SELECT MIN(balance) FROM accounts; -- expect: 4000 SELECT MAX(balance) FROM accounts; ``` ### GROUP BY and HAVING ```sql CREATE TABLE orders ( order_id INT, account_id INT, amount FLOAT, PRIMARY KEY (order_id) ); INSERT INTO orders (order_id, account_id, amount) VALUES (101, 1, 100.0), (102, 1, 200.0), (103, 2, 150.0), (104, 3, 300.0), (105, 3, 250.0); -- expect: 3 rows SELECT account_id, COUNT(*) as order_count, SUM(amount) as total FROM orders GROUP BY account_id; -- expect: 2 rows SELECT account_id, SUM(amount) as total FROM orders GROUP BY account_id HAVING SUM(amount) > 150.0; ``` ### JOIN `JOIN` (inner), `LEFT JOIN` and `RIGHT JOIN` join two tables on a condition. ```sql -- expect: Alice SELECT accounts.owner, orders.order_id FROM accounts JOIN orders ON accounts.account_id = orders.account_id WHERE accounts.account_id = 1; -- expect: 6 rows SELECT accounts.owner, orders.order_id FROM accounts LEFT JOIN orders ON accounts.account_id = orders.account_id; -- expect: 5 rows SELECT accounts.owner, orders.order_id FROM accounts RIGHT JOIN orders ON accounts.account_id = orders.account_id; ``` ## UPDATE ```sql UPDATE accounts SET balance = 1500.0 WHERE account_id = 1; UPDATE accounts SET balance = balance + 100.0; -- expect: 1600 SELECT balance FROM accounts WHERE account_id = 1; ``` An UPDATE without `WHERE` changes every row. ## DELETE ```sql DELETE FROM accounts WHERE account_id = 4; -- expect: 3 SELECT COUNT(*) FROM accounts; ``` ## Document collections A document collection holds JSON documents. Each document has a key, `_id`. ```sql -- expect: OK CREATE COLLECTION documents TYPE DOCUMENT; ``` `INSERT` without an `_id` makes one. `PUT '' {...}` writes the document at that key, and it replaces a document that is there. `GET` reads one document by its key. ```sql -- expect: OK INSERT INTO documents {"name": "Alice", "age": 30}; -- expect: 1 rows SELECT * FROM documents; -- expect: OK PUT 'doc1' {"name": "Bob", "title": "Engineer"} INTO documents; -- expect: doc1 GET 'doc1' FROM documents; ``` A `WHERE` reads nested fields with dots. On an array field, `=` is true when one element of the array is equal to the value. ```sql -- expect: OK INSERT INTO documents {"_id": "doc2", "name": "Carol", "address": {"city": "Oslo"}, "tags": ["red", "blue"]}; -- expect: Carol SELECT name FROM documents WHERE address.city = 'Oslo'; -- expect: Carol SELECT name FROM documents WHERE tags = 'red'; ``` `UPDATE` and `DELETE` work on documents with any `WHERE`: ```sql UPDATE documents SET age = 31 WHERE name = 'Alice'; -- expect: 31 SELECT age FROM documents WHERE name = 'Alice'; DELETE FROM documents WHERE _id = 'doc2'; -- expect: 2 SELECT COUNT(*) FROM documents; ``` ## Key-value collections A key-value collection holds one value per key. ```sql -- expect: OK CREATE COLLECTION cache TYPE KV; -- expect: OK PUT 'key1' 1000 INTO cache; -- expect: OK PUT 'key2' 2000 INTO cache; -- expect: key1 GET 'key1' FROM cache; -- expect: OK DELETE 'key1' FROM cache; ``` ## Indexes A secondary index speeds up a `WHERE` on its column. `DESCRIBE SELECT` shows the plan of a query. ```sql CREATE INDEX idx_owner ON accounts (owner); -- expect: 1 rows SHOW INDEXES; -- expect: 1 rows SHOW INDEXES FROM accounts; -- expect: index scan DESCRIBE SELECT * FROM accounts WHERE owner = 'Alice'; DROP INDEX idx_owner; ``` ## Transactions `BEGIN` and `COMMIT` group statements. ```sql BEGIN; INSERT INTO accounts (account_id, owner, balance, created_at) VALUES (5, 'Eve', 500.0, 5000000); COMMIT; -- expect: 4 SELECT COUNT(*) FROM accounts; ``` A `COMMIT` can carry an idempotency token. A client that lost its connection asks `GET TRANSACTION STATUS` whether the commit happened, and it retries only when it did not. ```sql BEGIN; INSERT INTO accounts (account_id, owner, balance, created_at) VALUES (6, 'Frank', 600.0, 6000000); COMMIT WITH TOKEN 'token-unique-001'; -- expect: OK GET TRANSACTION STATUS 'token-unique-001'; ``` `TRANSFER` moves an amount from one row to another in one atomic step. It needs a server connection: ```sql server TRANSFER 10 OF balance ON accounts FROM 1 TO 2; ``` ## Time travel and history Every commit keeps the old data readable. `AS OF` reads a table at a commit (`HEAD` is the newest), and `SHOW COMMITS` lists the commits. `SHOW COMMIT ` shows one commit. ```sql -- expect: 5 SELECT COUNT(*) FROM accounts AS OF HEAD; -- expect: hash SHOW COMMITS; ``` ## Branches A branch is a named root of the whole database. It costs no copy. ```sql -- expect: OK CREATE BRANCH dev_branch; -- expect: 2 rows SHOW BRANCHES; -- expect: OK USE BRANCH dev_branch; -- expect: OK USE BRANCH main; -- expect: OK DROP BRANCH dev_branch; ``` ## WATCH: live queries `WATCH [FAST] [VERBOSE] [WHERE ]` streams the changes of a table to the client, over the WebSocket of a server connection, until `UNWATCH`. The embedded mode answers `OK`, but it has no connection to stream on. ```sql server WATCH accounts WHERE balance > 100; UNWATCH accounts; ``` ## Databases A database holds its own tables and collections. ```sql -- expect: OK CREATE DATABASE mydb; -- expect: 2 rows SHOW DATABASES; DROP DATABASE mydb; ``` `USE ` switches the database of a session. It needs a server connection: ```sql server USE mydb; ``` ## Functions and tickers A function is a script in kd, the scripting language of kinedb. Its parameters are variables in the body, and `return` gives the result. Functions and tickers need a server. ```sql server CREATE FUNCTION add_one(x) RETURNS VALUE LANGUAGE kd AS $$ return x + 1; $$; CALL add_one(41); SHOW FUNCTIONS; DROP FUNCTION add_one; ``` A ticker calls a function on a schedule. `OVERLAP SKIP` (the default) skips a run while the last run still works; `OVERLAP WAIT` waits for it. ```sql server CREATE FUNCTION ping() RETURNS VALUE LANGUAGE kd AS $$ return 1; $$; CREATE TICKER ping_every_minute EVERY 60s CALL ping OVERLAP SKIP; SHOW TICKERS; DROP TICKER ping_every_minute; ``` ## Users, roles and permissions These statements need a server. ```sql server CREATE USER 'alice' PASSWORD 'secret123'; ALTER USER 'alice' SET PASSWORD 'newsecret'; ALTER USER 'alice' REVOKE SESSIONS; DROP USER 'alice'; ``` A permission goes to a target: `ANYONE`, `AUTHENTICATED`, `ROLE `, or `MATCH ` (a rule that matches the user against a field of the row). A `WHERE` limits the permission to the matching rows. A user gets permissions through roles. ```sql server CREATE ROLE analysts; GRANT ROLE analysts TO 'alice'; GRANT SELECT ON documents TO ROLE analysts; GRANT SELECT ON documents TO ANYONE WHERE _type = 'public'; GRANT ALL ON orders TO ROLE analysts; REVOKE SELECT ON documents FROM ROLE analysts; REVOKE ROLE analysts FROM 'alice'; DROP ROLE analysts; ``` `ALTER COLLECTION SET POLICY ''` replaces the whole policy list of a collection: ```sql server ALTER COLLECTION documents SET POLICY '[]'; ``` ## Settings and cluster operations These statements are for operators, and most of them need a server. ```sql server SHOW SETTINGS; SET CLUSTER cluster.slow_query_threshold_ms = 100; SHOW CLUSTER STATUS; SHOW CLUSTER QUORUM; SET CLUSTER QUORUM = 1; SHOW NODES; SHOW PEERS; SHOW RAFTGROUPS; SHOW RAFTGROUP 0; SHOW SHARDS; SHOW PLACEMENT; SHOW TABLE STATS; SHOW METRICS; SHOW SLOW QUERIES; SHOW MEMORY; SHOW DISK STATUS; SHOW SYNC STATUS; SHOW S3 STATUS; SHOW ROOTS; SHOW LEAVES OF accounts LIMIT 100; CLUSTER VERIFY; SPLIT SHARD 0 OF accounts AT 'm'; MERGE SHARDS 0 1 OF accounts; ALTER CLUSTER EVICT NODE 'abc123'; ALTER CLUSTER READMIT NODE 'abc123'; VACUUM; ``` - `SHOW SETTINGS` lists every setting with its value; `SET CLUSTER = ` changes a cluster setting on every node. See [Run kinedb](./run.md). - `SHOW NODES`, `SHOW PEERS`, `SHOW RAFTGROUPS` and `SHOW SHARDS` show the members, the Raft groups and the shards. `CLUSTER VERIFY` checks that the replicas agree. - `SPLIT SHARD` and `MERGE SHARDS` split and merge shards by hand. The cluster also splits a shard by itself when it grows. - `ALTER CLUSTER EVICT NODE` removes a node from the cluster, and `READMIT` lets it back. - `VACUUM` reclaims the space of dead blocks. --- Source: https://docs.kinedb.com/kd.md # kd: the scripting language kd is the scripting language of kinedb. A kd script runs in three places: - **A file:** `kinedb --data-dir ./data -f script.kd` runs the script in the process, with no server. - **A stored function:** `CREATE FUNCTION ... LANGUAGE kd` stores a function in the database, and `CALL` runs it on the server. - **A ticker:** `CREATE TICKER ... CALL ` runs a stored function on a schedule. Every kd value is a JSON value: a number, a string, a boolean, `null`, an array or an object. Every example on this page ran on the current build with `-f`, in the order of this page, in one database. A `// expect:` comment shows a part of the output. ## Run a script ```sh kinedb --data-dir ./data -f script.kd ``` `print(...)` writes its arguments on one line, with a space between them. After the last statement, kinedb prints the value of that statement: a string as it is, any other value as JSON, and nothing for `null`. ```kd // expect: hello kd 42 print("hello", "kd", 42); ``` ## Values `typeof` gives the type of a value. A string takes double or single quotes, and the escapes `\n`, `\t`, `\r`, `\\`, `\'` and `\"`. ```kd // expect: number string bool null array object print(typeof(1), typeof("a"), typeof(true), typeof(null), typeof([1]), typeof({a: 1})); ``` ## Variables `let` declares a variable. `=`, `+=` and `-=` assign to it. ```kd // expect: 35 let x = 10; x = 30; x += 10; x -= 5; print(x); ``` ## Operators The arithmetic operators are `+`, `-`, `*`, `/` and `%`. `*`, `/` and `%` bind stronger than `+` and `-`, and parentheses group. ```kd // expect: 17 2 1 2.5 25 print(2 + 3 * 5, 17 % 5, 7 - 6, 5 / 2, (2 + 3) * 5); ``` `+` also joins strings and arrays: ```kd // expect: hello world [1,2,3,4] print("hello" + " " + "world", [1, 2] + [3, 4]); ``` The comparison operators are `==`, `!=`, `<`, `<=`, `>` and `>=`. The logical operators are `&&`, `||` and `!`. ```kd // expect: true true false true print(1 == 1.0, "a" != "b", true && false, !false || false); ``` In a condition, `null`, `false`, `0` and `""` are false. Every other value is true, empty arrays and empty objects too. ```kd // expect: no no no no yes yes let out = ""; for v in [null, false, 0, "", [], {}] { if v { out = out + "yes "; } else { out = out + "no "; } } print(out); ``` ## Arrays and objects An index reads an array element (from 0) or an object field. A dot also reads a field. An assignment to an index or a field changes the value. ```kd // expect: 2 Alice let a = [1, 2, 3]; let p = {name: "Alice", age: 30}; print(a[1], p.name); ``` ```kd // expect: [1,99,3] 31 let a = [1, 2, 3]; a[1] = 99; let p = {name: "Alice", age: 30}; p.age = 31; print(a, p["age"]); ``` ## Control flow `if`, `else if` and `else` take a condition without parentheses and a block in braces. ```kd // expect: medium let n = 15; if n > 20 { print("large"); } else if n > 10 { print("medium"); } else { print("small"); } ``` `for in ` walks the elements of an array, or the keys of an object. `continue` goes to the next element, and `break` leaves the loop. ```kd // expect: 7 let sum = 0; for x in [1, 2, 3, 4, 5, 6] { if x == 3 { continue; } if x == 5 { break; } sum += x; } print(sum); ``` ```kd // expect: a=1 b=2 let o = {a: 1, b: 2}; let s = ""; for k in o { s = s + k + "=" + to_string(o[k]) + " "; } print(s); ``` ## Functions `fn` defines a function. `return` gives its result; a function without `return` gives `null`. A function can call itself. ```kd // expect: 120 fn fact(n) { if n <= 1 { return 1; } return n * fact(n - 1); } print(fact(5)); ``` ## Comments ```kd // expect: 3 // a line comment /* a block comment */ print(1 + 2); ``` ## Built-in functions | Function | What it gives | |---|---| | `print(a, b, ...)` | Writes the values on one line. Gives `null`. | | `typeof(v)` | `"number"`, `"string"`, `"bool"`, `"null"`, `"array"` or `"object"`. | | `length(v)` | The number of characters of a string, or of elements of an array or an object. | | `keys(o)`, `values(o)` | The keys, or the values, of an object, as an array. | | `to_string(v)` | The value as a string. | | `to_number(s)` | The string as a number, or `null` when it is not a number. | | `to_int(n)` | The number without its fraction. | | `floor(n)`, `ceil(n)`, `round(n)` | The number rounded down, up, or to the nearest integer. | ```kd // expect: 3 5 ["a","b"] [1,2] print(length([1, 2, 3]), length("hello"), keys({a: 1, b: 2}), values({a: 1, b: 2})); ``` ```kd // expect: 42 42 3 3 -4 4 4 print(to_number("42"), to_string(42), to_int(3.7), floor(3.9), floor(-3.2), ceil(3.1), round(3.5)); ``` ## The database A script reads and writes the database with these functions. The same functions work in a file (`-f`) and in a stored function. | Function | What it does | |---|---| | `query(sql)` | Runs the SQL. A `SELECT` gives an array of objects, one per row. | | `find(name)`, `find(name, where)` | `SELECT * FROM `, with an optional `WHERE `. | | `insert(name, object)` | `INSERT INTO `, with the keys of the object as the columns. | | `update(name, object, where)` | `UPDATE SET = , ... WHERE `. | | `delete(name, where)`, `remove(name, where)` | `DELETE FROM `, with an optional `WHERE `. | | `log(a, b, ...)` | Writes the values to the log output. | The examples use this table: ```sql CREATE TABLE players (id INT, name TEXT, score INT, PRIMARY KEY (id)); INSERT INTO players (id, name, score) VALUES (1, 'ann', 10), (2, 'bob', 20); ``` ```kd // expect: 2 ann 20 let rows = query("SELECT name, score FROM players ORDER BY id"); print(length(rows), rows[0].name, rows[1].score); ``` ```kd // expect: [{"id":2,"name":"bob","score":20},{"id":3,"name":"cid","score":30}] insert("players", {id: 3, name: "cid", score: 30}); print(find("players", "score > 15")); ``` ```kd // expect: 99 2 update("players", {score: 99}, "id = 1"); delete("players", "id = 2"); print(query("SELECT score FROM players WHERE id = 1")[0].score, length(find("players"))); ``` The last statement of this script is a value, so kinedb prints it: ```kd // expect: 129 let total = 0; for r in find("players") { total += r.score; } total ``` ## Stored functions and tickers `CREATE FUNCTION () [RETURNS VALUE | TABLE] LANGUAGE kd AS $$ $$` stores a function, and `CALL ()` runs it. The arguments bind to the parameter names in order, and a parameter without an argument is `null`. The array `args` also holds all the arguments. The body uses the built-in functions and the database functions above. Stored functions, `CALL` and tickers need a server connection. ```sql server CREATE FUNCTION add_points(player, points) RETURNS VALUE LANGUAGE kd AS $$ update("players", {score: points}, "id = " + to_string(player)); return points; $$; CALL add_points(1, 50); CREATE FUNCTION decay() RETURNS VALUE LANGUAGE kd AS $$ query("UPDATE players SET score = score - 1"); return null; $$; CREATE TICKER decay_every_minute EVERY 60s CALL decay OVERLAP SKIP; ``` See [the SQL reference](./sql.md) for `SHOW FUNCTIONS`, `DROP FUNCTION` and the ticker statements. --- Source: https://docs.kinedb.com/architecture.md # Architecture This page tells how kinedb works today, and where it is going. It is written for the people and the agents who use kinedb, not for the people who change it. ```text application ── SQL ──► kinedb ◄── SQL ── application (Rust library, one engine (HTTP /sql, WebSocket /ws, in-process) JS and Python clients) │ ┌──────────────────┼───────────────────┐ embedded one server a cluster Db::open(path) kinedb --port kinedb --calvin --join ... shards ─ Raft ─ replicas ``` ## Today ### Storage: a tree of content-addressed blocks kinedb keeps every table and collection in a prolly tree. A prolly tree is a search tree whose node boundaries come from the content, so two trees with the same data have the same shape and the same root hash. Each block is stored under the hash of its bytes, so two equal blocks are stored once. A commit writes new blocks and a new root. The old roots stay readable. This is why a branch is cheap: `CREATE BRANCH` records one more root, and `USE BRANCH` moves the session to it. Old versions stay on disk until garbage collection removes them. Garbage collection is off by default; `--gc-retention` turns it on and sets how much history to keep. The blocks live in memory, on local disk, or in an S3 bucket (`--s3`). ### Durability in a cluster A write is durable when the cluster acknowledges it. The leader of a shard writes the entry to disk before the entry counts toward the quorum, and every follower does the same before it answers. A node that restarts applies its committed entries again, so an acknowledged write survives a crash. ### Transactions and the order of writes `BEGIN`, `COMMIT` and `ROLLBACK` group statements. A `COMMIT` can carry an idempotency token, and `GET TRANSACTION STATUS` tells a client that lost its connection whether the commit happened. See the [SQL reference](./sql.md). In a cluster (`--calvin`), a sequencer collects the writes into small batches. Raft gives the batches one order, and every replica applies the same batches in the same order. So the replicas of a shard reach the same state, and they need no locks to agree. This design comes from the Calvin paper on deterministic databases. ### The cluster - **Membership.** The nodes find each other with SWIM, a gossip protocol that also detects a node that stops answering. - **Shards.** The cluster splits each table into shards by key range. It splits a shard again when the shard grows past a size threshold. - **Replicas.** Each shard has a Raft group of replicas on different nodes. The replication factor is 3 by default. - **Peer security.** Nodes must prove to each other that they know the cluster secret (`KINEDB_CLUSTER_SECRET`). This check is on by default. See [Run kinedb](./run.md). ### The data model - **Tables** have typed columns and a primary key. - **Document collections** hold JSON documents. Each document has an `_id`. - **Key-value collections** hold one value per key. - **Files** go into a table with `INSERT ... FILE` and come out with `SELECT FILE`. - **Indexes** speed up a `WHERE`. `DESCRIBE SELECT ...` shows which plan a query uses. ### Queries and live queries kinedb speaks its own SQL dialect: tables and documents in one language, with `WHERE`, `ORDER BY`, `LIMIT`, aggregates, `GROUP BY` and joins. `WATCH
` streams the changes of a table to a client over the WebSocket, until `UNWATCH`. ### Logic in the server `CREATE FUNCTION ... LANGUAGE kd` stores a function in kd, the scripting language of kinedb, and `CALL` runs it. `CREATE TICKER ... EVERY CALL ` runs a function on a schedule. ### Access control A client logs in with a user name and a password and gets a token. The SQL has users, roles, `GRANT` and `REVOKE`, and a grant can carry a row filter (`WHERE`). The [SQL reference](./sql.md) tells which of these statements run today. ### The protocol and the clients One port (4820 by default) carries everything: - `POST /sql` takes SQL and answers JSON. - `/ws` is a WebSocket with a binary protocol. A result tells the column types once, not once per cell. Live queries use this path. - `POST /login` gives a token. `GET /health` answers without a login. - `/ui/` is the web console, and `/ui/docs/` is this documentation. Put a TLS proxy in front of kinedb when the traffic crosses a network that you do not trust. The [JavaScript client](./clients/javascript.md) and the [Python client](./clients/python.md) speak this protocol. ## Where it is going - **Sync between an embedded copy and a cluster.** The design: each client gets a change feed with a cursor, catches up after it was offline, and receives only the rows it may read. Conflicts fall back to last-writer-wins or a merge function. Live subscriptions and partial fetch are part of it. - **The browser.** The engine compiles to WebAssembly with in-memory storage today. Next: storage in IndexedDB, and an npm package with the engine inside (`@kinedb/embedded`; the name is reserved). - **Joins and transactions across shards** for every query plan and every statement shape. - **Branch merge.** Merge one branch into another. - **Placement.** Place replicas by data center and rack, and keep hot and cold data on different machines. - **More SQL.** Computed columns, CHECK constraints, a vector index, columnar storage for analytics, and the Postgres and MySQL wire protocols. - **More clients.** An embedded and a direct client in each language. --- Source: https://docs.kinedb.com/clients/javascript.md # JavaScript client `@kinedb/client` is the direct client of kinedb. It sends SQL to a running kinedb server over HTTP (JSON) and over a WebSocket (a binary protocol). It works in a browser, in Bun, and in Node 22 or newer. ## Install The packages live on the forge, under the `kinedb` organisation: . There is no npmjs.com copy. `.npmrc`: ``` @kinedb:registry=https://git.kinedb.com/api/packages/kinedb/npm/ //git.kinedb.com/api/packages/kinedb/npm/:_authToken= ``` Then install the package: ```sh npm install @kinedb/client ``` ## Use ```js import { KineDB } from '@kinedb/client'; const db = new KineDB('http://localhost:4820'); const r = await db.sql('SELECT 1'); // one-shot over HTTP const h = await db.health(); const ws = await db.connect(); // a persistent WebSocket const rows = await ws.sql('SELECT * FROM users'); const sub = await ws.watch('users', (n) => console.log(n)); sub.cancel(); ws.close(); ``` With no argument the client derives its URLs from the page (`defaultBaseUrl()`, `defaultWsUrl()`), which is what a browser app behind a reverse proxy wants. ### Credentials The client holds no session. An application installs two hooks once, and every `new KineDB(...)` in that application picks them up: ```js import { setAuthHooks } from '@kinedb/client'; setAuthHooks({ getToken: () => myStore.token, // read on EVERY request, never cached onUnauthorized: () => myStore.logout(), // runs on a 401, BEFORE the throw }); ``` One client can override them: ```js new KineDB(url, undefined, { getToken: () => otherToken }); ``` With neither, the client sends no bearer and bounces nobody — the right default for a consumer that never logs in. `GET /health` stays bare in every case, because it is open by design and a liveness probe has no credentials. `setAuthHooks` returns the hooks it replaced, so a test or a one-off task can put them back. ### Retrying Off by default: a server-flagged transient rejection throws at once, with `err.retryable` set so a caller can build its own loop. ```js const db = new KineDB(url, undefined, { retry: true }); ``` With it on, the client absorbs those rejections behind a random full-jitter wait, and honours the server's own `retry_ms` pacing hint on a backoff response. --- Source: https://docs.kinedb.com/clients/python.md # Python client `kinedb-client` is the direct client of kinedb for Python. It sends SQL to a running kinedb server over HTTP and over a WebSocket, with the same API as the JavaScript client. ## Install The packages live on the forge, under the `kinedb` organisation: . There is no pypi.org copy. pip: ``` pip install --index-url https://git.kinedb.com/api/packages/kinedb/pypi/simple kinedb-client ``` uv, in the `pyproject.toml` of your application: ```toml [[tool.uv.index]] name = "kinedb" url = "https://git.kinedb.com/api/packages/kinedb/pypi/simple" explicit = true [tool.uv.sources] kinedb-client = { index = "kinedb" } ``` **Use the forge as an EXPLICIT index, never `--extra-index-url` alone.** All four kinedb names are free on pypi.org, so a second index may answer first and hand your application a stranger's package. The index needs a Gitea token when the organisation is private. `pip` takes it in the URL (`https://:@git.kinedb.com/...`); `uv` takes `UV_INDEX_KINEDB_USERNAME` and `UV_INDEX_KINEDB_PASSWORD`. ## Use ```python import asyncio from kinedb.client import KineDB, rows_as_objects async def main(): db = KineDB("http://localhost:4820") print(await db.health()) print(await db.sql("SHOW DATABASES")) # one statement over HTTP sock = await db.connect() # a persistent WebSocket await sock.authenticate("root", "hunter2") rows = await sock.sql("SELECT * FROM users") print(rows_as_objects(rows)) sub = await sock.watch("users", lambda notify: print(notify)) await sub.cancel() sock.close() asyncio.run(main()) ``` `base_url` is required. Unlike the browser client, this one has no page to derive a URL from. `ws_url` is derived from `base_url` when it is not given: `http` becomes `ws`, `https` becomes `wss`, and `/ws` is appended to the path. A result is a plain dict, tagged the way the server tags its own JSON: `resp["type"]` is `rows`, `created`, `inserted`, `backoff`, `health` and so on. A typed result set also carries `resp["types"]` and `resp["schema_id"]`. **On integers.** The JavaScript client hands back a BigInt above 2^53, because a JavaScript number cannot hold an integer exactly past that point. A Python `int` is unbounded, so that distinction has no Python half: an `Int` column decodes to a plain `int` and keeps every bit, always. ### Credentials The client holds no session. An application installs two hooks once, and every `KineDB(...)` in that application picks them up: ```python from kinedb.client import set_auth_hooks set_auth_hooks( get_token=lambda: store.token, # read on EVERY request, never cached on_unauthorized=lambda: store.logout(), # runs on a 401, BEFORE the raise ) ``` Either hook may be a coroutine function; the client awaits the result. One client can override them, which is what a program that talks to a second server with a different credential needs: ```python KineDB(url, get_token=lambda: other_token) ``` With neither, the client sends no bearer and bounces nobody. That is the right default for a consumer that never logs in. `GET /health` stays bare in every case, because it is open by design and a liveness probe has no credentials. `set_auth_hooks` returns the hooks it replaced, so a test or a one-off task can put them back: ```python previous = set_auth_hooks(get_token=borrowed) ... set_auth_hooks(**previous) ``` ### Retrying Off by default: a server-flagged transient rejection raises at once, with `err.retryable` set so a caller can build its own loop. ```python db = KineDB(url, retry=True) db = KineDB(url, retry={"max_retries": 3, "base_delay_ms": 20, "max_delay_ms": 500}) ``` With it on, the client absorbs those rejections behind a random full-jitter wait, and honours the server's own `retry_ms` pacing hint on a backoff response. The CLIENT retries; the server never does. ### No runtime dependencies The package installs nothing else. The HTTP transport is `urllib.request` on a worker thread, and the WebSocket client is our own RFC 6455 codec over `asyncio` streams (`kinedb.client.ws`). An SDK that pulled `websockets` or `httpx` would force a version range on every application that installs it. Pass `http_request=` or `ws_connect=` to `KineDB(...)` to drive it from a test with no server.