S3 storage

With --s3, a node keeps a copy of its data in an S3 bucket. The node writes to its local data directory first. Every 5 seconds, a background task uploads the new data to the bucket. When the local directory is empty, the node downloads the data from the bucket before it opens.

Check the binary

The released binaries and the container image include S3 support. A binary without it refuses --s3. It prints this line and exits with code 1:

$ kinedb --s3 's3://KEY:SECRET@s3.example.com/my-bucket/db1'
Error: --s3: this kinedb build has no S3 support; rebuild with `--features s3` (the released binaries include it)

Start a node with S3

KINEDB_ROOT_PASSWORD='choose-a-password' \
  ./kinedb --port 4820 --data-dir /var/lib/kinedb \
  --s3 's3://ACCESS_KEY:SECRET_KEY@s3.eu-west-1.amazonaws.com/my-bucket/db1?region=eu-west-1'

Put the URI in single quotes, so that the shell does not read ? or &.

The URI is a command-line argument, so the secret shows in a process list. There is no environment variable for it. Limit who can read the process list of the host.

The URI

The form is s3://KEY:SECRET@HOST/BUCKET/PREFIX?region=R. The node reads it in this order:

Part	Rule
s3://	Necessary. Another scheme stops the start.
?region=R	Everything after the first ?. The node reads only region. Without it, the region is us-east-1. The region is part of the request signature, so it must be the region of the bucket.
KEY:SECRET@	Everything before the first @. The key ends at the first :. The node decodes %XX in the secret, not in the key.
HOST	Everything up to the next /. It can have a port: HOST:PORT.
BUCKET	The next path segment. Necessary.
PREFIX	The rest of the path. It can be empty. The node removes a / at the end.

Percent-encode these characters in the secret: / as %2F, + as %2B, @ as %40, ? as %3F and % as %25.

A URI that does not parse stops the start with exit code 1, before the node opens any file:

$ kinedb --s3 's3://KEY:SECRET@s3.example.com' -e 'SELECT 1'
Invalid --s3 URI: Internal: S3 URI missing bucket: expected HOST/BUCKET/PREFIX

$ kinedb --s3 'http://KEY:SECRET@s3.example.com/my-bucket' -e 'SELECT 1'
Invalid --s3 URI: Internal: S3 URI must start with s3://

The endpoint

• TLS only. The node always connects to https://HOST. There is no plain HTTP mode. An endpoint that does not speak TLS fails at the start.
• The certificate. The node checks the certificate of the endpoint against the Mozilla root certificates that are built into the binary. It does not read the trust store of the operating system. So a self-signed certificate, or a certificate from a private certificate authority, fails.
• Path style. Each request goes to https://HOST/BUCKET/KEY, with an AWS Signature Version 4. The bucket is in the path, not in the host name.

So the endpoint must accept path-style requests and have a certificate from a public certificate authority. B1809 tracks a plain-HTTP mode and a private certificate authority.

Service	HOST	Note
AWS S3	s3.<region>.amazonaws.com	Set ?region= to the region of the bucket.
Google Cloud Storage	storage.googleapis.com	Use an HMAC key as KEY:SECRET.
MinIO or another S3-compatible server	its host name, with :PORT if it is not 443	The server needs TLS with a certificate from a public certificate authority.

What goes to the bucket

The node writes these objects under PREFIX:

Object	What it is
PREFIX/segments/seg-*.dat	The segment files: the blocks of the data.
PREFIX/DICTS	The list of the compression dictionaries.
PREFIX/HEAD	The newest commit: its block hash and its stamp.
PREFIX/OWNER	The owner marker: the id, the host name and the data directory of the ONE node that writes the prefix (B1810).

Each sync uploads in this order:

1. The node reads its local HEAD, then writes its local data to disk.
2. Before its first upload, the node reads PREFIX/OWNER. When the marker names another node, the sync uploads nothing and logs the B1810: error. See One node owns one prefix.
3. It uploads each segment file that is not in the bucket yet, from the lowest number to the highest. The newest segment grows, so the node uploads it again when it is larger.
4. It uploads DICTS when the list changed.
5. It uploads HEAD last.

So the commit that HEAD names is always in an uploaded segment. When the bucket loses a segment, the node refuses to start. See Troubleshooting.

At the start of each sync, the node lists the segments in the bucket. When a segment that it uploaded is missing, it uploads that segment again.

The node never deletes an object from the bucket. A segment that the garbage collection removes from the local directory stays in the bucket.

One node owns one prefix

Two nodes on one prefix write over each other's HEAD, DICTS and segment objects, and the data of one node is lost. So one node owns one prefix (B1810):

• The node keeps a random id in the file S3_OWNER of its data directory. A new data directory gets a new id.
• At the start, before it downloads or writes anything, the node reads PREFIX/OWNER.
  • No marker (a new prefix, or a prefix that an older kinedb wrote): the node writes its marker, reads it again and starts. Then it writes its S3_OWNER file.
  • The marker of this node: the node starts.
  • The marker of another node: the node refuses to start with exit code 3. It writes nothing to the bucket and nothing to its data directory.
• Before the first upload of each sync, the node reads PREFIX/OWNER again. When the marker names another node, the sync uploads nothing.

The refusal names --s3, the prefix, the owner and the fix:

NeedsOperator: B1810: the open refuses: the --s3 prefix s3://my-bucket/db1 belongs to another node: node id <id>, host <host>, data dir <dir>. This node is node id <id>, host <host>, data dir <dir>. Two nodes on one prefix write over each other's HEAD, DICTS and segment objects, and the data of one node is lost. Fix: give this node its own PREFIX in --s3. If the other node is stopped for good (for example, it lost its disk), set KINEDB_S3_TAKE_OVER_PREFIX=1 to take the prefix over.

A restore of a lost node on a new directory is a new node. Start it once with KINEDB_S3_TAKE_OVER_PREFIX=1. See Troubleshooting.

The check is not a lock. Two nodes that start at the same time can both read no marker. Each node writes its marker and reads it again, and the node whose marker S3 does not keep refuses. A take-over stops the next sync of the old owner, but an upload that runs at that time can still complete.

The local data directory

With --s3, --data-dir is the local copy of the data:

• An empty directory (a cold start). An empty directory is a new node. On a prefix that another node owns, it needs KINEDB_S3_TAKE_OVER_PREFIX=1 (see One node owns one prefix). The node downloads HEAD, DICTS and every segment of the bucket before it opens. The directory must have space for all of the data, and the start takes longer when the data is large. The node writes the local HEAD file last, after every segment is complete on the disk. When a cold start stops before the end (an error, a crash or a power loss), the directory has no HEAD file. The next start is a cold start again and downloads the missing segments.
• A directory that has data (a warm start). The node lists the segments of the bucket and downloads nothing. The local data wins.
• Without --data-dir. The node uses kinedb-s3-cache in the temporary directory of the system, for example /tmp/kinedb-s3-cache. This path is the same for every node on the host. Always set --data-dir.

The node reads only from the local directory, never from the bucket. The write-ahead log and the Raft logs stay in the local directory; they do not go to the bucket.

When the data reaches the bucket

• A server uploads every 5 seconds. A commit goes to the local directory first. It is in the bucket after the next sync.
• A stop. The node has no shutdown step that uploads. A commit from the last 5 seconds before a stop is only in the local directory. Start the node again on the same directory, and the next sync uploads it. If you delete the local directory first, that commit is lost, and the next start needs KINEDB_S3_TAKE_OVER_PREFIX=1.
• -e and -f. The process uploads once before it exits.

Watch the sync

SHOW S3 STATUS returns one row:

Column	What it is
local_txn	The stamp of the local HEAD.
synced_txn	The stamp of the last HEAD that the node uploaded.
txn_lag	local_txn minus synced_txn. It is a difference of two stamps, not a number of transactions. 0 means that the bucket has the local HEAD.
pending_segments	The number of local segment files that the node has not recorded as uploaded.

A node without --s3 answers this:

SHOW S3 STATUS;

s3
--------
disabled
(1 rows)

On a node with --s3, a txn_lag that is not 0 for longer than a few syncs, or a pending_segments that grows, tells you that the uploads fail.

When S3 is not available

• At the start. The node needs the bucket to start. When it cannot list the segments, it stops with exit code 1. This is the result of a wrong key:

  Failed to open S3-backed database: Internal: S3 LIST failed: HTTP 403

  An endpoint without TLS gives S3 write: unexpected end of file. A closed port gives S3 connect to HOST:PORT: Connection refused.

• While the node runs. A failed sync does not stop the node. It writes an error line with the text durable_sync: on the console, and it tries again at the next sync. Reads and writes continue on the local directory. While the syncs fail, the node does not shorten its write-ahead log, so the local directory grows.

For the refusals at the start, see Troubleshooting.

Limits

• --memory does not work with --s3. The node refuses the combination and exits with code 1:

  Error: --memory (the total budget) requires --data-dir and is not supported with --s3 or --connect

• --gc-retention and KINEDB_GC_RETENTION do not work with --s3. An S3 node does not reclaim disk space, in the local directory or in S3. The node refuses the combination before it opens the data directory and exits with code 1:

  Error: --gc-retention (or KINEDB_GC_RETENTION) is not supported with --s3: an S3 node runs no garbage collection (B2995). Remove one of the two.

  The local directory and the S3 prefix grow with the number of commits. B3071 tracks the garbage collection of an S3 node.

• Give each node its own PREFIX. The object names under a prefix are fixed (HEAD, DICTS, OWNER, segments/seg-<n>.dat). A second node on a prefix that a node owns refuses to start with exit code 3 and the B1810: error. See One node owns one prefix.

• --s3 is an argument of one node. It copies the data of that node only. It is not shared storage for a cluster.