# KineDB documentation
> KineDB is one database engine that runs as one server, as a cluster of servers, or embedded in an application. These docs tell how to run it, how to connect to it, and which SQL it accepts.
Revision: ca3bd1f6f170c7a9b1b0341a23c2553c708e32d2
---
Source: https://docs.kinedb.com/index.md
# kinedb
kinedb is a database engine. The same engine runs as one server, as a cluster of
servers, or inside your application. In all three cases, your application sends SQL.
kinedb stores tables, document collections (JSON documents with an `_id`) and
key-value collections. Every commit makes a new version of the data, and the old
versions stay readable. So a branch of a whole database costs one pointer, not a copy.
## The three ways it runs
### One server
The `kinedb` binary starts a server on port 4820. Clients connect over HTTP
(`POST /sql`, JSON) or over a WebSocket (`/ws`, a binary protocol). The same port
serves a web console at `/ui/` and these docs at `/ui/docs/`.
**Status:** works today. See [Run kinedb](./run.md).
### A cluster of servers
A cluster is any number of `kinedb` servers that act as one database. You start every
node with `--calvin`, and every node after the first with `--join
` of a
running node. The nodes find each other with a gossip protocol.
The cluster splits each table into shards by key range, and it splits a shard again
when the shard grows. Each shard has replicas on several nodes. The replicas agree on
one order of the writes with Raft, and every replica applies the writes in that order.
A write is durable when the cluster acknowledges it: each replica writes the entry to
disk before it counts toward the quorum.
**Status:** works today. See [Architecture](./architecture.md).
### Embedded in your application
The engine is also a library. In Rust, the `kinedb` crate opens a database with
`Db::open(path)` or `Db::open_in_memory()`, and runs SQL with `db.execute(sql)`. No
server runs. The `kinedb` binary gives you the same engine from a shell:
```sh
kinedb --data-dir ./data -e "SHOW TABLES"
```
**Status:** the library works on disk and in memory, and it compiles to WebAssembly.
## For AI agents
These docs are written for coding agents first. Every page is also plain Markdown:
- The index of all pages:
- All pages in one file:
- One page: replace `.html` with `.md` in its URL, for example
.
Every kinedb node also serves the docs of its own build. Use these when you work
against a specific server, because they match that server exactly:
`http://:4820/ui/docs/llms-full.txt`.
To give an agent the docs, put this in the `CLAUDE.md` (or `AGENTS.md`) of your
project:
```md
## KineDB
This project uses KineDB. Before you write KineDB SQL or client code, read
https://docs.kinedb.com/llms-full.txt. The server of this project also serves the docs
of its own version at http://:4820/ui/docs/llms-full.txt; when the two differ,
the server's copy is correct.
```
## Where to go next
- [Run kinedb](./run.md): Docker or the binary, every argument and every environment
variable.
- [SQL reference](./sql.md): every statement that kinedb runs today.
- [Architecture](./architecture.md): how it works today, and where it is going.
- [JavaScript client](./clients/javascript.md) and
[Python client](./clients/python.md): connect from an application.
---
Source: https://docs.kinedb.com/run.md
# Run kinedb
kinedb is one program, `kinedb`. The same program is a server, a node of a cluster, a SQL
shell, and an embedded engine for one-off SQL. This page tells how to start it, and it
lists every command-line argument and every environment variable.
## Get kinedb
Today the only distribution is a container image. The pull needs no login:
```sh
docker pull git.kinedb.com/kinedb/kinedb:latest
```
- **Tags.** `latest`, and one tag per build: the first 12 characters of its git commit.
- **Content.** The static `kinedb` binary at `/app/kinedb`, and the web console with
these docs at `/app/dist`. The image is built for `linux/amd64`.
- **Defaults.** The image runs as the user 65532. Its default command is
`--port 4820 --data-dir /data`, and `/data` is a volume.
The binary in the image is static, so you can copy it out and run it on any x86-64
Linux:
```sh
id=$(docker create git.kinedb.com/kinedb/kinedb:latest)
docker cp "$id":/app/kinedb ./kinedb
docker rm "$id"
```
## One server
```sh
docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
-e KINEDB_ROOT_PASSWORD='choose-a-password' \
git.kinedb.com/kinedb/kinedb:latest
```
Or with the binary:
```sh
KINEDB_ROOT_PASSWORD='choose-a-password' ./kinedb --port 4820 --data-dir ./data
```
- Without `--data-dir`, the data stays in memory, and it is lost when the process stops.
- One port carries everything: the clients, the peers, the web console at `/ui/`, and
these docs at `/ui/docs/`.
### Log in
Clients must log in by default (`KINEDB_CLIENT_AUTH=required`).
The first boot creates the user `root`. With `KINEDB_ROOT_PASSWORD`, `root` gets that
password. Without it, the server makes a password and writes it once to its log:
```sh
docker logs kinedb 2>&1 | grep 'GENERATED password'
```
The SQL shell reads the password from `KINEDB_ROOT_PASSWORD` and logs in as `root`. Use
`--user` and `--password-env` for another user. There is no password prompt.
```sh
docker exec -it kinedb /app/kinedb --connect 127.0.0.1:4820
docker exec kinedb /app/kinedb --connect 127.0.0.1:4820 -e "SHOW DATABASES"
```
Over HTTP, `POST /login` gives a token, and every other request carries it:
```sh
curl -s -X POST http://localhost:4820/login \
-d '{"username": "root", "password": "choose-a-password"}'
# the answer is a JSON object with a "token"
curl -s -X POST http://localhost:4820/sql \
-H "Authorization: Bearer " \
--data-binary 'SHOW DATABASES'
```
An application uses a client library: [JavaScript](./clients/javascript.md) or
[Python](./clients/python.md).
`KINEDB_CLIENT_AUTH=permissive` turns the login off: then anyone who reaches the port can
read and write. Use it only on a network that you trust.
## A cluster
A cluster is any number of `kinedb` nodes. To start one:
1. Choose a cluster secret. Give every node the same `KINEDB_CLUSTER_SECRET`. Peer
authentication is on by default, and a cluster node without a secret refuses to start.
2. Start the first node with `--calvin`.
3. Start every other node with `--calvin --join `.
4. Set `KINEDB_ADVERTISE_ADDR` on every node to the address that the other nodes use to
reach it. A node that listens on all interfaces, which includes every container,
otherwise tells its peers `127.0.0.1`.
Three hosts, `10.0.0.1` to `10.0.0.3`, with Docker:
```sh
# on 10.0.0.1: the first node
docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
-e KINEDB_CLUSTER_SECRET='a-long-random-secret' \
-e KINEDB_ROOT_PASSWORD='choose-a-password' \
-e KINEDB_ADVERTISE_ADDR=10.0.0.1 \
git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin
# on 10.0.0.2 (and the same on 10.0.0.3, with its own address)
docker run -d --name kinedb -p 4820:4820 -v kinedb-data:/data \
-e KINEDB_CLUSTER_SECRET='a-long-random-secret' \
-e KINEDB_ROOT_PASSWORD='choose-a-password' \
-e KINEDB_ADVERTISE_ADDR=10.0.0.2 \
git.kinedb.com/kinedb/kinedb:latest --port 4820 --data-dir /data --calvin --join 10.0.0.1:4820
```
Arguments after the image name replace the default command, so they repeat `--port` and
`--data-dir`.
- **Ports.** The nodes talk to each other on the same TCP port as the clients. There is
no other port to open.
- **Replicas.** Each shard has 3 replicas by default. `KINEDB_REPLICATION_FACTOR` sets the
first value on a new cluster; after that, `SET CLUSTER cluster.replication_factor = `
changes it.
- **Joining.** A joining node waits 10 seconds for the leader to accept it, and then it
exits with an error (`KINEDB_JOINER_BOOT_TIMEOUT_MS`).
- **Durability.** A write is durable when the cluster acknowledges it: each replica writes
the entry to disk before it counts toward the quorum. Each node keeps its Raft log in
`/raft_wal`.
## Embedded: SQL without a server
With `-e` or `-f` and without `--connect`, `kinedb` runs the SQL in its own process and
exits. No server starts, and no port opens. With `--data-dir`, the data stays for the
next call.
```sh
./kinedb --data-dir ./data -e "CREATE TABLE t (id INT, name TEXT, PRIMARY KEY (id))"
./kinedb --data-dir ./data -e "INSERT INTO t (id, name) VALUES (1, 'a'); SELECT * FROM t"
./kinedb --data-dir ./data -f setup.sql
```
- `-e` splits the SQL on `;` and runs the statements in order. `-f file.sql` does the same
with a file, and `-f file.kd` runs a kd script.
- `-e` stops at the first statement that fails. It prints a line that starts with
`ERROR:`, and it exits with 1.
- Some statements need a server; the [SQL reference](./sql.md) marks them.
In Rust, the `kinedb` crate is the same engine as a library (`Db::open(path)`,
`db.execute(sql)`).
## Command-line arguments
| Argument | Default | What it does |
|---|---|---|
| `--port PORT` | `4820` | The TCP port for the clients, the peers, the web console and these docs. |
| `--data-dir DIR` | not set | The directory of the data. Without it, the data stays in memory. |
| `--memory SIZE` | not set | The total memory budget of the process, for example `4GB` or `3800MB`. kinedb enforces it with its own accounting. It needs `--data-dir`, and it does not work with `--s3` or `--connect`. |
| `--cache-slab SIZE` | not set | A block cache that is allocated at start. Under `--memory`, it must fit the budget. Alone, it is the older way to size the cache. |
| `--gc-retention SPEC` | not set | Turns on the garbage collection of dead blocks, and sets the history to keep: `commits` (the head and N older commits; `0commits` keeps only the head) or `s`, `m`, `h`, `d` (for example `7d`). Without it, nothing is reclaimed. |
| `--calvin` | off | Cluster mode: Raft and the deterministic sequencer. Every node of a cluster needs it. |
| `--join HOST:PORT` | not set | Join a cluster through a running node. Use it with `--calvin`. |
| `--peer-auth MODE` | `required` | `required`: the nodes must prove that they know the cluster secret. `permissive`: they need not. |
| `--cluster-key KEY` | `kinedb` | The name of the cluster. A node ignores the gossip of a node with another key. It is not a secret. |
| `--realm NAME` | not set | The realm of the node. Leave it unset for one realm. |
| `--s3 URI` | not set | Store the blocks in S3: `s3://KEY:SECRET@HOST/BUCKET/PREFIX?region=R`. `--data-dir` is then the local cache; without it, a temporary directory. |
| `--webroot DIR` | found | The directory of the web console. Without it, kinedb looks for `dist` next to the binary, then for `dist` and `web/dist` in the working directory. |
| `--no-web` | off | Serve no web console and no docs. |
| `--connect HOST:PORT`, `-c` | not set | Client mode: open a SQL shell on a running server, or run `-e` or `-f` there. |
| `--user NAME` | `root` | Client mode: the user to log in as. |
| `--password-env VAR` | `KINEDB_ROOT_PASSWORD` | Client mode: the environment variable that holds the password. |
| `-e SQL` | not set | Run SQL and exit. Without `--connect`, the SQL runs in this process. |
| `-f FILE` | not set | Run a file and exit: `.kd` is a kd script; any other file is SQL. |
| `--show-types` | off | Client mode: put the column type in the header of a result, for example `price:Float`. |
| `--help`, `-h` | | Print the usage and exit. |
## Environment variables
### Setup
| Variable | Default | What it does |
|---|---|---|
| `KINEDB_ROOT_PASSWORD` | not set | The password of `root`, set at the first boot. Without it, the first boot makes a password and logs it once. The SQL shell also reads it to log in. |
| `KINEDB_CLIENT_AUTH` | `required` | `required`: clients must log in. `permissive`: anyone can query. |
| `KINEDB_PEER_AUTH` | `required` | The same as `--peer-auth`. The flag wins. |
| `KINEDB_CLUSTER_SECRET` | not set | The secret that the nodes of a cluster share. It is only an environment variable, so it does not show in a process list. |
| `KINEDB_ADVERTISE_ADDR` | not set | The address that the peers use to reach this node, `host` or `host:port` (the port part is ignored). Without it, a node that listens on all interfaces advertises `127.0.0.1`. |
| `KINEDB_GC_RETENTION` | not set | The same as `--gc-retention`. The flag wins. |
| `KINEDB_RAFT_WAL_DIR` | `/raft_wal` | The directory of the Raft logs. Without a data directory and without this variable, the Raft log stays in memory. |
| `KINEDB_LOG_SINK` | not set | `ws://host:port` of a kinedb server that collects logs and metrics. With it, the node sends its logs and metrics there. |
| `KINEDB_LOG_SINK_USER`, `KINEDB_LOG_SINK_PASSWORD` | not set | The login at the log sink. |
### Tuning
These variables belong to the settings of the server; `SHOW SETTINGS` lists the same
settings with the values in force. A variable of a `cluster.` setting sets the first value
on a new cluster; after that, `SET CLUSTER = ` changes it. A node reads
the variable of a `node.` setting when it starts.
| Variable | Setting | Default | What it does |
|---|---|---|---|
| `KINEDB_CALVIN_APPLY_TIMEOUT_SECS` | `cluster.calvin.apply_timeout_secs` | 60 secs | How long a Calvin write waits for local apply before returning an error. |
| `KINEDB_SEQUENCER_BATCH_AGE_MS` | `cluster.calvin.sequencer_batch_age_ms` | 2 ms | Maximum age (ms) a sequencer batch waits before flushing. |
| `KINEDB_SEQUENCER_BATCH_SIZE` | `cluster.calvin.sequencer_batch_size` | 1024 | Maximum number of operations in a single Calvin sequencer batch. |
| `KINEDB_FORCE_QUORUM` | `cluster.quorum` | not set | Recovery only: forces `cluster.quorum` on this node for the life of the process. |
| `KINEDB_REPLICATION_FACTOR` | `cluster.replication_factor` | 3 | Number of Raft voter replicas per shard (RF`. |
| `/ws` | A WebSocket with the binary protocol of the client libraries. Live queries (`WATCH`) use it. |
| `/ui/` | The web console. |
| `/ui/docs/` | These docs, for the version of this node. `/ui/docs/llms-full.txt` holds all pages in one file. |
---
Source: https://docs.kinedb.com/sql.md
# SQL reference
This page lists the SQL that kinedb runs today. Every example in a plain `sql` block ran
on the current build, in the order of this page, in one database. So a later example can
use a table that an earlier example created. A `-- expect:` comment shows a part of the
output.
The examples use the embedded mode of the binary, which starts no server:
```sh
kinedb --data-dir ./data -e "SHOW TABLES"
```
Some statements need a running server (`kinedb --connect host:port`, `POST /sql`, or a
client library). The text before such an example says so. The embedded mode refuses
them with `requires a running server`.
## Data types
- `INT`: a 64-bit signed integer.
- `FLOAT`: a 64-bit floating-point number (IEEE 754).
- `BOOL`: `true` or `false`.
- `TEXT`: a UTF-8 string.
- `BLOB`: any bytes.
- `TIMESTAMP`: microseconds since the Unix epoch.
- `VECTOR`: a list of numbers.
```sql
-- expect: OK
CREATE TABLE demo (
id INT,
amount FLOAT,
is_active BOOL,
name TEXT,
data BLOB,
ts TIMESTAMP,
embedding VECTOR,
PRIMARY KEY (id)
);
```
## Tables
A table has typed columns and a primary key.
```sql
-- expect: OK
CREATE TABLE accounts (
account_id INT,
owner TEXT,
balance FLOAT,
created_at TIMESTAMP,
PRIMARY KEY (account_id)
);
```
List the tables, and show the columns of one table:
```sql
-- expect: 2 rows
SHOW TABLES;
-- expect: account_id
DESCRIBE accounts;
-- expect: account_id
SHOW COLUMNS FROM accounts;
```
Drop a table and its data:
```sql
CREATE TABLE temp_table (id INT, PRIMARY KEY (id));
DROP TABLE temp_table;
```
## INSERT
Insert one row, or several rows in one statement:
```sql
INSERT INTO accounts (account_id, owner, balance, created_at)
VALUES (1, 'Alice', 1000.0, 1000000);
INSERT INTO accounts (account_id, owner, balance, created_at)
VALUES
(2, 'Bob', 2000.0, 2000000),
(3, 'Charlie', 3000.0, 3000000),
(4, 'Diana', 4000.0, 4000000);
-- expect: 4
SELECT COUNT(*) FROM accounts;
```
## SELECT
```sql
-- expect: Alice
SELECT * FROM accounts WHERE account_id = 1;
-- expect: Bob
SELECT owner, balance FROM accounts WHERE owner = 'Bob';
```
### WHERE
The comparison operators are `=`, `!=`, `<`, `<=`, `>` and `>=`. `AND`, `OR`, `NOT` and
parentheses combine conditions.
```sql
-- expect: 1 rows
SELECT * FROM accounts WHERE balance > 2000.0 AND owner != 'Diana';
-- expect: 3 rows
SELECT owner FROM accounts WHERE NOT account_id = 1;
-- expect: Bob
SELECT owner FROM accounts WHERE (account_id = 1 OR account_id = 2) AND balance > 1500.0;
```
`IN`, `BETWEEN`, `LIKE`, `ILIKE` (not case-sensitive) and `IS NULL`:
```sql
-- expect: 2 rows
SELECT owner FROM accounts WHERE account_id IN (1, 2);
-- expect: 2 rows
SELECT owner FROM accounts WHERE balance BETWEEN 1000.0 AND 2000.0;
-- expect: Alice
SELECT owner FROM accounts WHERE owner LIKE 'Al%';
-- expect: Alice
SELECT owner FROM accounts WHERE owner ILIKE 'al%';
-- expect: 0 rows
SELECT owner FROM accounts WHERE owner IS NULL;
```
### ORDER BY, LIMIT and OFFSET
`ORDER BY` sorts by one or more columns, `ASC` (the default) or `DESC`. `OFFSET` comes
after `LIMIT`.
```sql
-- expect: Diana
SELECT owner FROM accounts ORDER BY balance DESC LIMIT 1;
-- expect: 2 rows
SELECT * FROM accounts LIMIT 2;
-- expect: Charlie
SELECT owner FROM accounts ORDER BY account_id LIMIT 1 OFFSET 2;
```
### Aggregates
`COUNT(*)`, `SUM`, `AVG`, `MIN` and `MAX`:
```sql
-- expect: 4
SELECT COUNT(*) FROM accounts;
-- expect: 10000
SELECT SUM(balance) FROM accounts;
-- expect: 2500
SELECT AVG(balance) FROM accounts;
-- expect: 1000
SELECT MIN(balance) FROM accounts;
-- expect: 4000
SELECT MAX(balance) FROM accounts;
```
### GROUP BY and HAVING
```sql
CREATE TABLE orders (
order_id INT,
account_id INT,
amount FLOAT,
PRIMARY KEY (order_id)
);
INSERT INTO orders (order_id, account_id, amount)
VALUES
(101, 1, 100.0),
(102, 1, 200.0),
(103, 2, 150.0),
(104, 3, 300.0),
(105, 3, 250.0);
-- expect: 3 rows
SELECT account_id, COUNT(*) as order_count, SUM(amount) as total
FROM orders
GROUP BY account_id;
-- expect: 2 rows
SELECT account_id, SUM(amount) as total
FROM orders
GROUP BY account_id
HAVING SUM(amount) > 150.0;
```
### JOIN
`JOIN` (inner), `LEFT JOIN` and `RIGHT JOIN` join two tables on a condition.
```sql
-- expect: Alice
SELECT accounts.owner, orders.order_id
FROM accounts
JOIN orders ON accounts.account_id = orders.account_id
WHERE accounts.account_id = 1;
-- expect: 6 rows
SELECT accounts.owner, orders.order_id
FROM accounts
LEFT JOIN orders ON accounts.account_id = orders.account_id;
-- expect: 5 rows
SELECT accounts.owner, orders.order_id
FROM accounts
RIGHT JOIN orders ON accounts.account_id = orders.account_id;
```
## UPDATE
```sql
UPDATE accounts SET balance = 1500.0 WHERE account_id = 1;
UPDATE accounts SET balance = balance + 100.0;
-- expect: 1600
SELECT balance FROM accounts WHERE account_id = 1;
```
An UPDATE without `WHERE` changes every row.
## DELETE
```sql
DELETE FROM accounts WHERE account_id = 4;
-- expect: 3
SELECT COUNT(*) FROM accounts;
```
## Document collections
A document collection holds JSON documents. Each document has a key, `_id`.
```sql
-- expect: OK
CREATE COLLECTION documents TYPE DOCUMENT;
```
`INSERT` without an `_id` makes one. `PUT '' {...}` writes the document at that key,
and it replaces a document that is there. `GET` reads one document by its key.
```sql
-- expect: OK
INSERT INTO documents {"name": "Alice", "age": 30};
-- expect: 1 rows
SELECT * FROM documents;
-- expect: OK
PUT 'doc1' {"name": "Bob", "title": "Engineer"} INTO documents;
-- expect: doc1
GET 'doc1' FROM documents;
```
A `WHERE` reads nested fields with dots. On an array field, `=` is true when one element
of the array is equal to the value.
```sql
-- expect: OK
INSERT INTO documents {"_id": "doc2", "name": "Carol", "address": {"city": "Oslo"}, "tags": ["red", "blue"]};
-- expect: Carol
SELECT name FROM documents WHERE address.city = 'Oslo';
-- expect: Carol
SELECT name FROM documents WHERE tags = 'red';
```
`UPDATE` and `DELETE` work on documents with any `WHERE`:
```sql
UPDATE documents SET age = 31 WHERE name = 'Alice';
-- expect: 31
SELECT age FROM documents WHERE name = 'Alice';
DELETE FROM documents WHERE _id = 'doc2';
-- expect: 2
SELECT COUNT(*) FROM documents;
```
## Key-value collections
A key-value collection holds one value per key.
```sql
-- expect: OK
CREATE COLLECTION cache TYPE KV;
-- expect: OK
PUT 'key1' 1000 INTO cache;
-- expect: OK
PUT 'key2' 2000 INTO cache;
-- expect: key1
GET 'key1' FROM cache;
-- expect: OK
DELETE 'key1' FROM cache;
```
## Indexes
A secondary index speeds up a `WHERE` on its column. `DESCRIBE SELECT` shows the plan of a
query.
```sql
CREATE INDEX idx_owner ON accounts (owner);
-- expect: 1 rows
SHOW INDEXES;
-- expect: 1 rows
SHOW INDEXES FROM accounts;
-- expect: index scan
DESCRIBE SELECT * FROM accounts WHERE owner = 'Alice';
DROP INDEX idx_owner;
```
## Transactions
`BEGIN` and `COMMIT` group statements.
```sql
BEGIN;
INSERT INTO accounts (account_id, owner, balance, created_at)
VALUES (5, 'Eve', 500.0, 5000000);
COMMIT;
-- expect: 4
SELECT COUNT(*) FROM accounts;
```
A `COMMIT` can carry an idempotency token. A client that lost its connection asks
`GET TRANSACTION STATUS` whether the commit happened, and it retries only when it did not.
```sql
BEGIN;
INSERT INTO accounts (account_id, owner, balance, created_at)
VALUES (6, 'Frank', 600.0, 6000000);
COMMIT WITH TOKEN 'token-unique-001';
-- expect: OK
GET TRANSACTION STATUS 'token-unique-001';
```
`TRANSFER` moves an amount from one row to another in one atomic step. It needs a server
connection:
```sql server
TRANSFER 10 OF balance ON accounts FROM 1 TO 2;
```
## Time travel and history
Every commit keeps the old data readable. `AS OF` reads a table at a commit (`HEAD` is the
newest), and `SHOW COMMITS` lists the commits. `SHOW COMMIT ` shows one commit.
```sql
-- expect: 5
SELECT COUNT(*) FROM accounts AS OF HEAD;
-- expect: hash
SHOW COMMITS;
```
## Branches
A branch is a named root of the whole database. It costs no copy.
```sql
-- expect: OK
CREATE BRANCH dev_branch;
-- expect: 2 rows
SHOW BRANCHES;
-- expect: OK
USE BRANCH dev_branch;
-- expect: OK
USE BRANCH main;
-- expect: OK
DROP BRANCH dev_branch;
```
## WATCH: live queries
`WATCH [FAST] [VERBOSE] [WHERE ]` streams the changes
of a table to the client, over the WebSocket of a server connection, until `UNWATCH`. The
embedded mode answers `OK`, but it has no connection to stream on.
```sql server
WATCH accounts WHERE balance > 100;
UNWATCH accounts;
```
## Databases
A database holds its own tables and collections.
```sql
-- expect: OK
CREATE DATABASE mydb;
-- expect: 2 rows
SHOW DATABASES;
DROP DATABASE mydb;
```
`USE ` switches the database of a session. It needs a server connection:
```sql server
USE mydb;
```
## Functions and tickers
A function is a script in kd, the scripting language of kinedb. Its parameters are
variables in the body, and `return` gives the result. Functions and tickers need a server.
```sql server
CREATE FUNCTION add_one(x) RETURNS VALUE LANGUAGE kd AS $$ return x + 1; $$;
CALL add_one(41);
SHOW FUNCTIONS;
DROP FUNCTION add_one;
```
A ticker calls a function on a schedule. `OVERLAP SKIP` (the default) skips a run while
the last run still works; `OVERLAP WAIT` waits for it.
```sql server
CREATE FUNCTION ping() RETURNS VALUE LANGUAGE kd AS $$ return 1; $$;
CREATE TICKER ping_every_minute EVERY 60s CALL ping OVERLAP SKIP;
SHOW TICKERS;
DROP TICKER ping_every_minute;
```
## Users, roles and permissions
These statements need a server.
```sql server
CREATE USER 'alice' PASSWORD 'secret123';
ALTER USER 'alice' SET PASSWORD 'newsecret';
ALTER USER 'alice' REVOKE SESSIONS;
DROP USER 'alice';
```
A permission goes to a target: `ANYONE`, `AUTHENTICATED`, `ROLE `, or
`MATCH ` (a rule that matches the user against a field of the row). A `WHERE`
limits the permission to the matching rows. A user gets permissions through roles.
```sql server
CREATE ROLE analysts;
GRANT ROLE analysts TO 'alice';
GRANT SELECT ON documents TO ROLE analysts;
GRANT SELECT ON documents TO ANYONE WHERE _type = 'public';
GRANT ALL ON orders TO ROLE analysts;
REVOKE SELECT ON documents FROM ROLE analysts;
REVOKE ROLE analysts FROM 'alice';
DROP ROLE analysts;
```
`ALTER COLLECTION SET POLICY ''` replaces the whole policy list of a
collection:
```sql server
ALTER COLLECTION documents SET POLICY '[]';
```
## Settings and cluster operations
These statements are for operators, and most of them need a server.
```sql server
SHOW SETTINGS;
SET CLUSTER cluster.slow_query_threshold_ms = 100;
SHOW CLUSTER STATUS;
SHOW CLUSTER QUORUM;
SET CLUSTER QUORUM = 1;
SHOW NODES;
SHOW PEERS;
SHOW RAFTGROUPS;
SHOW RAFTGROUP 0;
SHOW SHARDS;
SHOW PLACEMENT;
SHOW TABLE STATS;
SHOW METRICS;
SHOW SLOW QUERIES;
SHOW MEMORY;
SHOW DISK STATUS;
SHOW SYNC STATUS;
SHOW S3 STATUS;
SHOW ROOTS;
SHOW LEAVES OF accounts LIMIT 100;
CLUSTER VERIFY;
SPLIT SHARD 0 OF accounts AT 'm';
MERGE SHARDS 0 1 OF accounts;
ALTER CLUSTER EVICT NODE 'abc123';
ALTER CLUSTER READMIT NODE 'abc123';
VACUUM;
```
- `SHOW SETTINGS` lists every setting with its value; `SET CLUSTER = `
changes a cluster setting on every node. See [Run kinedb](./run.md).
- `SHOW NODES`, `SHOW PEERS`, `SHOW RAFTGROUPS` and `SHOW SHARDS` show the members, the
Raft groups and the shards. `CLUSTER VERIFY` checks that the replicas agree.
- `SPLIT SHARD` and `MERGE SHARDS` split and merge shards by hand. The cluster also
splits a shard by itself when it grows.
- `ALTER CLUSTER EVICT NODE` removes a node from the cluster, and `READMIT` lets it back.
- `VACUUM` reclaims the space of dead blocks.
---
Source: https://docs.kinedb.com/kd.md
# kd: the scripting language
kd is the scripting language of kinedb. A kd script runs in three places:
- **A file:** `kinedb --data-dir ./data -f script.kd` runs the script in the process,
with no server.
- **A stored function:** `CREATE FUNCTION ... LANGUAGE kd` stores a function in the
database, and `CALL` runs it on the server.
- **A ticker:** `CREATE TICKER ... CALL ` runs a stored function on a schedule.
Every kd value is a JSON value: a number, a string, a boolean, `null`, an array or an
object. Every example on this page ran on the current build with `-f`, in the order of
this page, in one database. A `// expect:` comment shows a part of the output.
## Run a script
```sh
kinedb --data-dir ./data -f script.kd
```
`print(...)` writes its arguments on one line, with a space between them. After the last
statement, kinedb prints the value of that statement: a string as it is, any other value
as JSON, and nothing for `null`.
```kd
// expect: hello kd 42
print("hello", "kd", 42);
```
## Values
`typeof` gives the type of a value. A string takes double or single quotes, and the
escapes `\n`, `\t`, `\r`, `\\`, `\'` and `\"`.
```kd
// expect: number string bool null array object
print(typeof(1), typeof("a"), typeof(true), typeof(null), typeof([1]), typeof({a: 1}));
```
## Variables
`let` declares a variable. `=`, `+=` and `-=` assign to it.
```kd
// expect: 35
let x = 10;
x = 30;
x += 10;
x -= 5;
print(x);
```
## Operators
The arithmetic operators are `+`, `-`, `*`, `/` and `%`. `*`, `/` and `%` bind stronger
than `+` and `-`, and parentheses group.
```kd
// expect: 17 2 1 2.5 25
print(2 + 3 * 5, 17 % 5, 7 - 6, 5 / 2, (2 + 3) * 5);
```
`+` also joins strings and arrays:
```kd
// expect: hello world [1,2,3,4]
print("hello" + " " + "world", [1, 2] + [3, 4]);
```
The comparison operators are `==`, `!=`, `<`, `<=`, `>` and `>=`. The logical operators
are `&&`, `||` and `!`.
```kd
// expect: true true false true
print(1 == 1.0, "a" != "b", true && false, !false || false);
```
In a condition, `null`, `false`, `0` and `""` are false. Every other value is true,
empty arrays and empty objects too.
```kd
// expect: no no no no yes yes
let out = "";
for v in [null, false, 0, "", [], {}] {
if v { out = out + "yes "; } else { out = out + "no "; }
}
print(out);
```
## Arrays and objects
An index reads an array element (from 0) or an object field. A dot also reads a field.
An assignment to an index or a field changes the value.
```kd
// expect: 2 Alice
let a = [1, 2, 3];
let p = {name: "Alice", age: 30};
print(a[1], p.name);
```
```kd
// expect: [1,99,3] 31
let a = [1, 2, 3];
a[1] = 99;
let p = {name: "Alice", age: 30};
p.age = 31;
print(a, p["age"]);
```
## Control flow
`if`, `else if` and `else` take a condition without parentheses and a block in braces.
```kd
// expect: medium
let n = 15;
if n > 20 { print("large"); } else if n > 10 { print("medium"); } else { print("small"); }
```
`for in ` walks the elements of an array, or the keys of an object.
`continue` goes to the next element, and `break` leaves the loop.
```kd
// expect: 7
let sum = 0;
for x in [1, 2, 3, 4, 5, 6] {
if x == 3 { continue; }
if x == 5 { break; }
sum += x;
}
print(sum);
```
```kd
// expect: a=1 b=2
let o = {a: 1, b: 2};
let s = "";
for k in o { s = s + k + "=" + to_string(o[k]) + " "; }
print(s);
```
## Functions
`fn` defines a function. `return` gives its result; a function without `return` gives
`null`. A function can call itself.
```kd
// expect: 120
fn fact(n) {
if n <= 1 { return 1; }
return n * fact(n - 1);
}
print(fact(5));
```
## Comments
```kd
// expect: 3
// a line comment
/* a block
comment */
print(1 + 2);
```
## Built-in functions
| Function | What it gives |
|---|---|
| `print(a, b, ...)` | Writes the values on one line. Gives `null`. |
| `typeof(v)` | `"number"`, `"string"`, `"bool"`, `"null"`, `"array"` or `"object"`. |
| `length(v)` | The number of characters of a string, or of elements of an array or an object. |
| `keys(o)`, `values(o)` | The keys, or the values, of an object, as an array. |
| `to_string(v)` | The value as a string. |
| `to_number(s)` | The string as a number, or `null` when it is not a number. |
| `to_int(n)` | The number without its fraction. |
| `floor(n)`, `ceil(n)`, `round(n)` | The number rounded down, up, or to the nearest integer. |
```kd
// expect: 3 5 ["a","b"] [1,2]
print(length([1, 2, 3]), length("hello"), keys({a: 1, b: 2}), values({a: 1, b: 2}));
```
```kd
// expect: 42 42 3 3 -4 4 4
print(to_number("42"), to_string(42), to_int(3.7), floor(3.9), floor(-3.2), ceil(3.1), round(3.5));
```
## The database
A script reads and writes the database with these functions. The same functions work in
a file (`-f`) and in a stored function.
| Function | What it does |
|---|---|
| `query(sql)` | Runs the SQL. A `SELECT` gives an array of objects, one per row. |
| `find(name)`, `find(name, where)` | `SELECT * FROM `, with an optional `WHERE `. |
| `insert(name, object)` | `INSERT INTO `, with the keys of the object as the columns. |
| `update(name, object, where)` | `UPDATE SET = , ... WHERE `. |
| `delete(name, where)`, `remove(name, where)` | `DELETE FROM `, with an optional `WHERE `. |
| `log(a, b, ...)` | Writes the values to the log output. |
The examples use this table:
```sql
CREATE TABLE players (id INT, name TEXT, score INT, PRIMARY KEY (id));
INSERT INTO players (id, name, score) VALUES (1, 'ann', 10), (2, 'bob', 20);
```
```kd
// expect: 2 ann 20
let rows = query("SELECT name, score FROM players ORDER BY id");
print(length(rows), rows[0].name, rows[1].score);
```
```kd
// expect: [{"id":2,"name":"bob","score":20},{"id":3,"name":"cid","score":30}]
insert("players", {id: 3, name: "cid", score: 30});
print(find("players", "score > 15"));
```
```kd
// expect: 99 2
update("players", {score: 99}, "id = 1");
delete("players", "id = 2");
print(query("SELECT score FROM players WHERE id = 1")[0].score, length(find("players")));
```
The last statement of this script is a value, so kinedb prints it:
```kd
// expect: 129
let total = 0;
for r in find("players") { total += r.score; }
total
```
## Stored functions and tickers
`CREATE FUNCTION () [RETURNS VALUE | TABLE] LANGUAGE kd AS $$ $$`
stores a function, and `CALL ()` runs it. The arguments bind to the
parameter names in order, and a parameter without an argument is `null`. The array `args`
also holds all the arguments. The body uses the built-in functions and the database
functions above. Stored functions, `CALL` and tickers need a server connection.
```sql server
CREATE FUNCTION add_points(player, points) RETURNS VALUE LANGUAGE kd AS $$ update("players", {score: points}, "id = " + to_string(player)); return points; $$;
CALL add_points(1, 50);
CREATE FUNCTION decay() RETURNS VALUE LANGUAGE kd AS $$ query("UPDATE players SET score = score - 1"); return null; $$;
CREATE TICKER decay_every_minute EVERY 60s CALL decay OVERLAP SKIP;
```
See [the SQL reference](./sql.md) for `SHOW FUNCTIONS`, `DROP FUNCTION` and the ticker
statements.
---
Source: https://docs.kinedb.com/architecture.md
# Architecture
This page tells how kinedb works today, and where it is going. It is written for the
people and the agents who use kinedb, not for the people who change it.
```text
application ── SQL ──► kinedb ◄── SQL ── application
(Rust library, one engine (HTTP /sql, WebSocket /ws,
in-process) JS and Python clients)
│
┌──────────────────┼───────────────────┐
embedded one server a cluster
Db::open(path) kinedb --port kinedb --calvin --join ...
shards ─ Raft ─ replicas
```
## Today
### Storage: a tree of content-addressed blocks
kinedb keeps every table and collection in a prolly tree. A prolly tree is a search
tree whose node boundaries come from the content, so two trees with the same data have
the same shape and the same root hash. Each block is stored under the hash of its
bytes, so two equal blocks are stored once.
A commit writes new blocks and a new root. The old roots stay readable. This is why a
branch is cheap: `CREATE BRANCH` records one more root, and `USE BRANCH` moves the
session to it. Old versions stay on disk until garbage collection removes them.
Garbage collection is off by default; `--gc-retention` turns it on and sets how much
history to keep.
The blocks live in memory, on local disk, or in an S3 bucket (`--s3`).
### Durability in a cluster
A write is durable when the cluster acknowledges it. The leader of a shard writes the
entry to disk before the entry counts toward the quorum, and every follower does the
same before it answers. A node that restarts applies its committed entries again, so an
acknowledged write survives a crash.
### Transactions and the order of writes
`BEGIN`, `COMMIT` and `ROLLBACK` group statements. A `COMMIT` can carry an idempotency
token, and `GET TRANSACTION STATUS` tells a client that lost its connection whether
the commit happened. See the [SQL reference](./sql.md).
In a cluster (`--calvin`), a sequencer collects the writes into small batches. Raft
gives the batches one order, and every replica applies the same batches in the same
order. So the replicas of a shard reach the same state, and they need no locks to
agree. This design comes from the Calvin paper on deterministic databases.
### The cluster
- **Membership.** The nodes find each other with SWIM, a gossip protocol that also
detects a node that stops answering.
- **Shards.** The cluster splits each table into shards by key range. It splits a
shard again when the shard grows past a size threshold.
- **Replicas.** Each shard has a Raft group of replicas on different nodes. The
replication factor is 3 by default.
- **Peer security.** Nodes must prove to each other that they know the cluster secret
(`KINEDB_CLUSTER_SECRET`). This check is on by default. See [Run kinedb](./run.md).
### The data model
- **Tables** have typed columns and a primary key.
- **Document collections** hold JSON documents. Each document has an `_id`.
- **Key-value collections** hold one value per key.
- **Files** go into a table with `INSERT ... FILE` and come out with `SELECT FILE`.
- **Indexes** speed up a `WHERE`. `DESCRIBE SELECT ...` shows which plan a query uses.
### Queries and live queries
kinedb speaks its own SQL dialect: tables and documents in one language, with
`WHERE`, `ORDER BY`, `LIMIT`, aggregates, `GROUP BY` and joins.
`WATCH ` streams the changes of a table to a client over the WebSocket, until
`UNWATCH`.
### Logic in the server
`CREATE FUNCTION ... LANGUAGE kd` stores a function in kd, the scripting language of
kinedb, and `CALL` runs it. `CREATE TICKER ... EVERY CALL ` runs a
function on a schedule.
### Access control
A client logs in with a user name and a password and gets a token. The SQL has users,
roles, `GRANT` and `REVOKE`, and a grant can carry a row filter (`WHERE`). The
[SQL reference](./sql.md) tells which of these statements run today.
### The protocol and the clients
One port (4820 by default) carries everything:
- `POST /sql` takes SQL and answers JSON.
- `/ws` is a WebSocket with a binary protocol. A result tells the column types once,
not once per cell. Live queries use this path.
- `POST /login` gives a token. `GET /health` answers without a login.
- `/ui/` is the web console, and `/ui/docs/` is this documentation.
Put a TLS proxy in front of kinedb when the traffic crosses a network that you do not
trust.
The [JavaScript client](./clients/javascript.md) and the
[Python client](./clients/python.md) speak this protocol.
## Where it is going
- **Sync between an embedded copy and a cluster.** The design: each client gets a
change feed with a cursor, catches up after it was offline, and receives only the
rows it may read. Conflicts fall back to last-writer-wins or a merge function. Live
subscriptions and partial fetch are part of it.
- **The browser.** The engine compiles to WebAssembly with in-memory storage today.
Next: storage in IndexedDB, and an npm package with the engine inside
(`@kinedb/embedded`; the name is reserved).
- **Joins and transactions across shards** for every query plan and every statement
shape.
- **Branch merge.** Merge one branch into another.
- **Placement.** Place replicas by data center and rack, and keep hot and cold data on
different machines.
- **More SQL.** Computed columns, CHECK constraints, a vector index, columnar storage
for analytics, and the Postgres and MySQL wire protocols.
- **More clients.** An embedded and a direct client in each language.
---
Source: https://docs.kinedb.com/clients/javascript.md
# JavaScript client
`@kinedb/client` is the direct client of kinedb. It sends SQL to a running kinedb server
over HTTP (JSON) and over a WebSocket (a binary protocol). It works in a browser, in Bun,
and in Node 22 or newer.
## Install
The packages live on the forge, under the `kinedb` organisation:
. There is no npmjs.com copy.
`.npmrc`:
```
@kinedb:registry=https://git.kinedb.com/api/packages/kinedb/npm/
//git.kinedb.com/api/packages/kinedb/npm/:_authToken=
```
Then install the package:
```sh
npm install @kinedb/client
```
## Use
```js
import { KineDB } from '@kinedb/client';
const db = new KineDB('http://localhost:4820');
const r = await db.sql('SELECT 1'); // one-shot over HTTP
const h = await db.health();
const ws = await db.connect(); // a persistent WebSocket
const rows = await ws.sql('SELECT * FROM users');
const sub = await ws.watch('users', (n) => console.log(n));
sub.cancel();
ws.close();
```
With no argument the client derives its URLs from the page (`defaultBaseUrl()`,
`defaultWsUrl()`), which is what a browser app behind a reverse proxy wants.
### Credentials
The client holds no session. An application installs two hooks once, and every
`new KineDB(...)` in that application picks them up:
```js
import { setAuthHooks } from '@kinedb/client';
setAuthHooks({
getToken: () => myStore.token, // read on EVERY request, never cached
onUnauthorized: () => myStore.logout(), // runs on a 401, BEFORE the throw
});
```
One client can override them:
```js
new KineDB(url, undefined, { getToken: () => otherToken });
```
With neither, the client sends no bearer and bounces nobody — the right default
for a consumer that never logs in. `GET /health` stays bare in every case,
because it is open by design and a liveness probe has no credentials.
`setAuthHooks` returns the hooks it replaced, so a test or a one-off task can put
them back.
### Retrying
Off by default: a server-flagged transient rejection throws at once, with
`err.retryable` set so a caller can build its own loop.
```js
const db = new KineDB(url, undefined, { retry: true });
```
With it on, the client absorbs those rejections behind a random full-jitter
wait, and honours the server's own `retry_ms` pacing hint on a backoff response.
---
Source: https://docs.kinedb.com/clients/python.md
# Python client
`kinedb-client` is the direct client of kinedb for Python. It sends SQL to a running
kinedb server over HTTP and over a WebSocket, with the same API as the JavaScript client.
## Install
The packages live on the forge, under the `kinedb` organisation:
. There is no pypi.org copy.
pip:
```
pip install --index-url https://git.kinedb.com/api/packages/kinedb/pypi/simple kinedb-client
```
uv, in the `pyproject.toml` of your application:
```toml
[[tool.uv.index]]
name = "kinedb"
url = "https://git.kinedb.com/api/packages/kinedb/pypi/simple"
explicit = true
[tool.uv.sources]
kinedb-client = { index = "kinedb" }
```
**Use the forge as an EXPLICIT index, never `--extra-index-url` alone.** All four
kinedb names are free on pypi.org, so a second index may answer first and hand
your application a stranger's package.
The index needs a Gitea token when the organisation is private. `pip` takes it
in the URL (`https://:@git.kinedb.com/...`); `uv` takes
`UV_INDEX_KINEDB_USERNAME` and `UV_INDEX_KINEDB_PASSWORD`.
## Use
```python
import asyncio
from kinedb.client import KineDB, rows_as_objects
async def main():
db = KineDB("http://localhost:4820")
print(await db.health())
print(await db.sql("SHOW DATABASES")) # one statement over HTTP
sock = await db.connect() # a persistent WebSocket
await sock.authenticate("root", "hunter2")
rows = await sock.sql("SELECT * FROM users")
print(rows_as_objects(rows))
sub = await sock.watch("users", lambda notify: print(notify))
await sub.cancel()
sock.close()
asyncio.run(main())
```
`base_url` is required. Unlike the browser client, this one has no page to
derive a URL from. `ws_url` is derived from `base_url` when it is not given:
`http` becomes `ws`, `https` becomes `wss`, and `/ws` is appended to the path.
A result is a plain dict, tagged the way the server tags its own JSON:
`resp["type"]` is `rows`, `created`, `inserted`, `backoff`, `health` and so on.
A typed result set also carries `resp["types"]` and `resp["schema_id"]`.
**On integers.** The JavaScript client hands back a BigInt above 2^53, because a
JavaScript number cannot hold an integer exactly past that point. A Python `int`
is unbounded, so that distinction has no Python half: an `Int` column decodes to
a plain `int` and keeps every bit, always.
### Credentials
The client holds no session. An application installs two hooks once, and every
`KineDB(...)` in that application picks them up:
```python
from kinedb.client import set_auth_hooks
set_auth_hooks(
get_token=lambda: store.token, # read on EVERY request, never cached
on_unauthorized=lambda: store.logout(), # runs on a 401, BEFORE the raise
)
```
Either hook may be a coroutine function; the client awaits the result.
One client can override them, which is what a program that talks to a second
server with a different credential needs:
```python
KineDB(url, get_token=lambda: other_token)
```
With neither, the client sends no bearer and bounces nobody. That is the right
default for a consumer that never logs in. `GET /health` stays bare in every
case, because it is open by design and a liveness probe has no credentials.
`set_auth_hooks` returns the hooks it replaced, so a test or a one-off task can
put them back:
```python
previous = set_auth_hooks(get_token=borrowed)
...
set_auth_hooks(**previous)
```
### Retrying
Off by default: a server-flagged transient rejection raises at once, with
`err.retryable` set so a caller can build its own loop.
```python
db = KineDB(url, retry=True)
db = KineDB(url, retry={"max_retries": 3, "base_delay_ms": 20, "max_delay_ms": 500})
```
With it on, the client absorbs those rejections behind a random full-jitter
wait, and honours the server's own `retry_ms` pacing hint on a backoff response.
The CLIENT retries; the server never does.
### No runtime dependencies
The package installs nothing else. The HTTP transport is `urllib.request` on a
worker thread, and the WebSocket client is our own RFC 6455 codec over
`asyncio` streams (`kinedb.client.ws`). An SDK that pulled `websockets` or
`httpx` would force a version range on every application that installs it.
Pass `http_request=` or `ws_connect=` to `KineDB(...)` to drive it from a test
with no server.