Troubleshooting

This page tells what to do when the node does not start, or when it reports a state that needs an operator. Each section names the message you see, what it means, and the steps that repair it.

A refusal that names more than 10 items prints one line for each item first, for example B1403: catalog slot 1/3165: _schemas default.main.t stamp=302 live. The refusal line comes LAST. It gives the count, the first 10 items, the fix and the variable. Thus the terminal shows the fix at the bottom, and the log keeps the full list. A refusal with 10 items or fewer is one line that names every item.

The node does not start: an index is UNUSABLE

What you see

The node prints one boot error per damaged index, then one summary line, and exits:

B979: index "idx_code" on t is UNUSABLE (an older build wrote its tree: its layout carries no base shard, or the index has no layout) — fix: DROP INDEX idx_code; CREATE INDEX idx_code ON t (code)
B979: 1 unusable index(es); the node does not start. Run the DROP INDEX / CREATE INDEX lines above on a controlled start: set KINEDB_OPEN_WITH_UNUSABLE_INDEX=1, start the node, run them, then unset it. Exit code 3 (needs an operator): a unit file may set RestartPreventExitStatus=3.

The exit code is 3, not 1.

The fix line rebuilds the index with its kind. A UNIQUE index gets CREATE UNIQUE INDEX, so the repair keeps the constraint. A FULLTEXT index gets CREATE FULLTEXT INDEX.

What it means

The data directory holds a secondary index whose on-disk layout is older than the one this build reads, or an index that the catalog lists with no layout (an older build lost the layout in a crash). The documents of the table are intact. Only the index tree is unusable, and the node cannot rebuild it by itself, because a rebuild is a DROP INDEX and a CREATE INDEX that an operator must run.

The node refuses to start by default. A node that served with a broken index would answer some queries slowly and wrongly, so the safe state is "stopped, with the repair printed".

Every committed document is safe. Before the node decides, it replays the write-ahead log and makes each document of the log durable in the store. The log can also hold entries for the unusable index. The node does not replay those entries, and it prints one warning that names each index and the number of its entries:

[B1152_RECOVERY_UNUSABLE_INDEX_ROWS] 777 WAL record(s) of UNUSABLE index tree(s) idx_code not replayed: each layout carries no base shard, so the tree predates B934 and no slice owns its entries. No document is affected — an index is derived state. Per tree: [("idx_code", 777)]. Fix: for each name run DROP INDEX <name>, then CREATE INDEX <name> again.

The CREATE INDEX of the repair builds every entry again from the documents.

The repair

1. Start the node in the controlled mode. Set KINEDB_OPEN_WITH_UNUSABLE_INDEX=1 for this one start:

   KINEDB_OPEN_WITH_UNUSABLE_INDEX=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   With Docker, pass the variable with -e:

   docker run -d --name kinedb -p 4820:4820 \
     -v kinedb-data:/data \
     -e KINEDB_OPEN_WITH_UNUSABLE_INDEX=1 \
     git.kinedb.com/kinedb/kinedb:latest

   The node starts and serves. It prints the same per-index errors, and repeats them every ten minutes. Every query that would use an unusable index fails with an error that carries the fix. The node never falls back to a silent full scan. Every write to the table of an unusable index (INSERT, UPDATE, DELETE, PUT, UPSERT) fails with an error that carries the fix too, on one node and through a cluster.

2. Log in and run the two statements from the error, for each index:

   DROP INDEX idx_code;
   CREATE INDEX idx_code ON t (code);

   CREATE INDEX builds the tree again from the documents, in the layout of this build. The node stays up while it builds.

3. Unset KINEDB_OPEN_WITH_UNUSABLE_INDEX and restart the node normally.

The setting is an opt-in for one start. Do not leave it set in a unit file or a compose file. A node that always starts in the controlled mode serves with the next damaged index too, and only /health and the log tell you.

How to see the state

• GET /health answers 503 Service Unavailable while any index is UNUSABLE. The body's unusable_indexes array carries one line per index. A superuser token (Authorization: Bearer <token>) gets "<name> on <table>: <fix>" lines; a caller with no login gets lines with no name (B1885). It answers 200 after the last DROP INDEX, without a restart.
• SHOW INDEXES has a status column, OK, WRITE-ONLY or UNUSABLE, and a fix column with the DROP INDEX / CREATE INDEX pair for an UNUSABLE index. A WRITE-ONLY index (a build that runs or that stopped) does not change GET /health. See Indexes in the SQL reference.

The exit code and a supervisor

Exit code 3 means "the node did not start, and an operator has to act". A supervisor must not restart the node on it, because every restart prints the same error and exits again. With systemd, add one line to the unit:

# /etc/systemd/system/kinedb.service
[Service]
RestartPreventExitStatus=3

Every other start failure exits with code 1. A restart may fix those.

The node does not start: a shard layout does not partition the keyspace

What you see

The node prints one boot error and exits with code 3. The error starts with B1358:, names every table whose layout is wrong with the shards and ranges concerned, and ends with the fix:

B1358: 1 table layout(s) in this store do not partition the keyspace (sorted, gapless, non-overlapping, from Unbounded to Unbounded): table "t" — ... Fix: set KINEDB_REPAIR_SHARD_LAYOUT=1 once on this node; the node then rewrites each named layout so that it matches the routing the node already does, and refuses if a shard holds a row outside its new range. Start every node of a cluster that reports this error with the variable.

What it means

A table is split into shards, and each shard owns one key range. The ranges must cover every key exactly once. The data directory holds a layout from an older build that does not follow this rule. The node finds it when it opens the data directory and refuses to start. The refusal changes nothing on disk.

The repair

1. Start the node once with KINEDB_REPAIR_SHARD_LAYOUT=1:

   KINEDB_REPAIR_SHARD_LAYOUT=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   The node makes each range end where the next shard starts. That is the shard to which the node already sends each key, so no read gives a different answer. The node moves no row. It prints one [B1358_LAYOUT_REPAIR] line per table, with the old and the new ranges, and then starts.

2. Unset the variable. The next starts need nothing.

In a cluster, start every node that prints the error with the variable. The new ranges come from the old ranges only, so every node computes the same layout.

When the repair refuses

The repair also exits with code 3, and changes nothing on disk, when:

• a shard holds a row outside its new range. The node cannot decide which shard must own that row, so a human must look at the table.
• two shards start at the same key. One of them would get an empty range, and the repair never deletes a shard.

Keep the data directory as it is, and report the full error text.

The node does not start: a segment file holds a damaged block

What you see

The node prints one boot error and exits with code 3. The error carries the kind NeedsOperator and starts with B1428:. It names every damaged block record with its segment file, its offset, its hash and the number of good records after it, and it ends with the fix. This line comes from a test boot:

[kdtrace ts=1790984619919 boot error] Failed to open data directory /tmp/nix-shell-16287-1493497883/.tmpeRF7bE/data: NeedsOperator: B1428: 1 damaged block record(s) in the segment files of /tmp/nix-shell-16287-1493497883/.tmpeRF7bE/data: segments/seg-00000001.dat offset 16: block db0b01dcf8037076b15e21cf069bffd27f46ccd5c92005df01b4e075734cf436 (170 bytes fail the check; 409 good record(s) after it). A damaged record fails its CRC32C check, or its content does not hash to its stored hash, so its block can no longer be served, and the store does not start. Fix: stop every process on this data directory, restore each named segment file from a replica or a backup, then start again. To start without the damaged blocks, set KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 — a controlled start: every good record is served, and every read of a named block is an ERROR that names it.

What it means

The node stores blocks in segment files under <data-dir>/segments. Each block record carries a checksum. A record that fails its checksum while valid records follow it was damaged on the disk: a bad sector, a copy that went wrong, or a changed byte. The node cannot serve that block, so it refuses to start. The refusal changes nothing on disk.

A record can also pass its checksum while its content does not hash to the hash that the record stores. Its content is then not known to be that block. The open checks this for each LZ4 record of a segment file without an index footer. The error names such a record in this form:

segments/seg-00000001.dat offset 426: block e89f6e4592a5babf74b057c06929d058be944a3874642c44cd91f118d66e2f5b (its content hashes to 8d8daa4ea0e150a1e87413b804443400ed7e576ad10241a993c93341f47e45a3: the stored hash is false)

The repair is the same. A record of another kind, or a record in a segment file with an index footer, is checked when it is read: the read fails with an error that names it.

The repair

1. Stop every process that uses the data directory.
2. Restore each named segment file from a replica or a backup. A segment file never changes after it is complete, so a copy of the same file from another node or a backup is correct.
3. Start the node normally.

When no copy exists, start the node once with KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1:

KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 kinedb --port 4820 --data-dir /var/lib/kinedb

The node prints one [B1428_DAMAGED_BLOCK] line per damaged record and starts. Every good block is served. Every read that needs a damaged block fails with an error that names the block, its segment file and its offset; it never answers as if the data did not exist. The GC sweep leaves a segment file with a damaged record untouched, so the evidence stays on disk. Copy the data you can read to a new store, then unset the variable.

When a damaged block is part of the catalog, the node also starts with the variable. It prints one [B2763_CATALOG_UNREADABLE] line, adopts no table (also with KINEDB_ADOPT_IMPLICIT_TABLES=1), and refuses every first write that would create a collection, with an error that names the catalog read error.

The node then refuses each statement that reads or writes a table or the catalog: a SELECT, an INSERT, a DDL statement (CREATE TABLE, DROP TABLE, CREATE INDEX), and SHOW TABLES or DESCRIBE. The error starts with [B2778_CATALOG_UNREADABLE] and names the damaged block. The node never answers "table not found" or an empty list for a table that it cannot read. A statement that needs no table entry still runs, for example SHOW COMMIT COUNT, the user statements and the operator commands. On a cluster, a replicated DDL statement on this node is a local fault: the node retries it or stops, and it never applies the statement differently from its replicas. Restore the damaged block to serve the tables again.

The database registry and the branch ledger are catalogs too. When a damaged block is part of the registry, the node prints one [B2808_DATABASES_UNREADABLE] line at the start. It then refuses SHOW DATABASES, USE <db>, CREATE DATABASE, DROP DATABASE and each statement on a database other than default. When a damaged block is part of the ledger, the node prints one [B2808_BRANCHES_UNREADABLE] line. It then refuses SHOW BRANCHES, USE BRANCH, CREATE BRANCH, DROP BRANCH, MERGE BRANCH and each statement on a branch other than main. The error starts with the same marker and names the damaged block. The node never answers "database not found" or "branch not found" for an entry that it cannot read. A statement on the default database and the main branch still runs. On a cluster, a replicated statement of the registry or the ledger is a local fault on this node, as above.

The node does not start: the disk does not hold the bytes of a file

What you see

The node prints one boot error and exits. The error holds B2966:. For most texts it is the first word. Two other boot checks put it inside their own line (see the exit codes below). Search the log for one of these texts:

B2966: the open refuses <path>: its sync_data failed. The disk does not hold some bytes of the file (a failed writeback, also of a process that stopped). The next open reads the bytes from the disk: <inner error>
B2966: the open refuses <path>: the page cache drop failed: <inner error>
B2966: the open refuses <seg path>: the migration footer append failed: <inner error>
B2966: the B2971 boot check cannot prove the Raft WAL durable: B2966: the open refused: ...
B2966: the B2117 boot check cannot prove the Raft WAL durable: B2966: the open refused: ...
B2966: the B2114 boot check cannot prove the Raft WAL durable: B2966: the open refused: ...
B2966: the open refused: open of WAL segment <path>: ...
B2966: the open refused: sync_data of WAL segment <path>: ...
B2966: the open refused: drop of the page cache of WAL segment <path>: ...

The audit spool can wrap the first text as read the spool segment <name>: <text>.

What it means

After a failed writeback, Linux can keep the lost pages in the page cache. The pages stay clean and readable, and the fsync of a new process can return Ok. A read then shows bytes that the disk does not hold. To prevent this, each open that trusts the bytes of a file first runs sync_data on the file. Then it drops the page cache of the file, and then it reads the file. The files are the block segments (segments/seg-*.dat), wal/wal.log, the database WAL (wal.d/*.wal and the old wal.log), the Raft WAL segments and the audit spool segments. On tmpfs and ramfs the node skips the check, because these have no disk.

A refusal can come after a write to the disk. The sync_data of the open writes the dirty pages of the file. The migration footer append of a segment that has no footer runs before the sync and the drop of wal/wal.log. The refusal itself writes nothing more.

The texts have these meanings:

• its sync_data failed: the sync failed, so the node cannot trust the bytes of the file. The cause can be a write error, a full disk or a read-only file system. The error after the colon is the error of the disk.
• the page cache drop failed: the node could not remove the pages of the file from the page cache. The error can say that pages stay in the page cache after posix_fadvise(DONTNEED). The cause can be another process that reads the file, for example an rsync or a backup of the data directory, or a map of the file. The error can also name an errno of posix_fadvise or cachestat.
• the migration footer append failed: the node could not append the footer to a segment during the open. Before B2966 the node ignored this error.
• WAL segment texts: the same checks, for a segment of the Raft WAL.
• the B2971 boot check / the B2117 boot check / the B2114 boot check: the first boot scans of the Raft WAL could not open, sync or drop a segment. The B2971 check runs first and reads only the last segment of each group. Linux reports a writeback error to one fsync only. So these scans must refuse, because a later open of the same file can see no error. An open can also fail because of the owner or the mode of a file.

The exit code depends on the place of the refusal:

• The open of the store (the block segments, wal/wal.log, wal.d): exit code 1.
• The B2971, B2117 and B2114 boot checks of the Raft WAL: exit code 3 (needs an operator).
• The B2931 boot check (only for a data directory of a version before B2931): exit code 3. The line starts with B2931: the boot check of the data directory of a version before B2931 failed to read the store: the Raft WAL of shard <n> in <dir> cannot be read: B2966: the open refused: ....
• The foreign-stamp check (B2452): exit code 3. The line starts with the check of the row stamps against the Raft log of each data group could not read the store:, and then the read of the Raft WAL of shard <n> failed: B2966: the open refused: ....
• A refusal of the Raft driver open after these checks: the exit code is not determined.

A restart can fix the refusal when the disk is healthy.

A warning: the node cannot prove the drop

On Linux, the proof of the drop uses cachestat(2). The node has no proof in these cases:

• The kernel is older than 6.5 (ENOSYS).
• A seccomp profile of a container blocks cachestat (EPERM).
• The build is for an architecture other than x86_64 and aarch64.

Then the node logs one warning and drops the cache with no proof. To get the proof, use a kernel 6.5 or newer, allow the cachestat syscall (number 451) in the seccomp profile, or run a build for x86_64 or aarch64. The warning starts with this text:

B2966: the node has no cachestat(2) count, so it cannot prove that the page cache drop before an open worked.

On an operating system that is not Linux, the node drops no cache and logs no warning.

The repair

1. Stop every process that uses the data directory.
2. Check the disk and the kernel log (dmesg) for I/O errors.
3. When the disk is healthy, start the node again. The next open reads the bytes from the disk only when no other process holds the pages and the drop is proven (see the warning above). It cuts a torn tail, as after a crash. A damaged record in the middle of a segment gives the B1428 refusal above.
4. When the disk has errors, replace the disk or restore the node from its replicas.

For the page cache drop failed, stop the process that reads the data directory. A backup tool that reads the data directory while kinedb starts can hold pages. Stop it, then start kinedb. No KINEDB_* variable overrides these refusals.

A change in behavior

A store on a failing disk that the open accepted before now refuses the open. One example is a store with no footer, whose migration footer sync fails.

The boot is also slower. These numbers are from ext4 on NVMe:

• Block segments: the worst case is a restart in the same boot with every segment in the page cache. It costs about 6 s at 1,400 segments of 64 MiB.
• After a reboot the cache is empty. The block segments then cost about 0.7 s.
• Raft WAL segments: each boot check that reads a segment syncs it and drops its cache first. A segment of shard 0 can be read 5 times (B2117, B2114, B2931, B2452 and the driver open). A segment of another data shard can be read 4 times, and a segment of the metadata shard 2 times. The B2971 check reads the last segment of each group one more time. B2931 and B2452 run only when their markers ask for them. These counts come from the code; a boot was not instrumented. Each read costs about 4 ms for a segment of 8 MiB. At 100 to 290 segments, 4 reads cost about 1.6 to 4.8 s.

macOS and Windows are not measured.

The node does not start: a compression dictionary is lost or DICTS is damaged

What you see

The node prints one boot error and exits with code 3. It is the same B1428: error as above, after Failed to open data directory <path>:. The error names every lost dictionary block, or the damaged DICTS file, next to any damaged block record. These texts come from the store tests (the data directory there is /db). Two dictionary blocks that no segment holds give this error:

NeedsOperator: B1428: 2 damaged block structure(s) in /db: dictionary block abababababababababababababababababababababababababababababababab: DICTS names it, and no segment holds it; every record coded with it is unreadable; dictionary block cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd: DICTS names it, and no segment holds it; every record coded with it is unreadable. A lost dictionary block or a damaged DICTS (B887) makes the records coded with it unreadable, and the store does not start. Fix: stop every process on this data directory, restore the segment files and DICTS from a backup of THIS data directory; a replica trains its own dictionaries, so its files do not fit. On a cluster node with no backup, the data of this node is lost, but its replicas still hold the shards: replace the node (B1544). Then start again. To start without the damaged blocks, set KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 — a controlled start: every good record is served, and every read of a named block is an ERROR that names it; with a damaged DICTS the store never trains a dictionary over it, never rewrites it, and refuses every GC sweep.

A DICTS file that fails its checksum gives this error:

NeedsOperator: B1428: 1 damaged block structure(s) in /db: DICTS: the dictionary list is damaged (DICTS: CRC32C mismatch); it names the dictionary blocks the coded records need and pins them against GC. A lost dictionary block or a damaged DICTS (B887) makes the records coded with it unreadable, and the store does not start. Fix: stop every process on this data directory, restore the segment files and DICTS from a backup of THIS data directory; a replica trains its own dictionaries, so its files do not fit. On a cluster node with no backup, the data of this node is lost, but its replicas still hold the shards: replace the node (B1544). Then start again. To start without the damaged blocks, set KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 — a controlled start: every good record is served, and every read of a named block is an ERROR that names it; with a damaged DICTS the store never trains a dictionary over it, never rewrites it, and refuses every GC sweep.

What it means

The node compresses most blocks with a dictionary that it trains from its own data. The dictionary is an ordinary block in a segment file. The file <data-dir>/DICTS names every dictionary block. A record that is coded with a dictionary cannot be read without that dictionary block. A lost dictionary block makes every record coded with it unreadable. A damaged DICTS file hides which blocks are dictionaries, so the GC sweep could delete one. The refusal changes nothing on disk.

The repair

1. Stop every process that uses the data directory.
2. Restore the segment files and the DICTS file from the newest backup of THIS data directory. A dictionary block never changes after it is written. A replica trains its own dictionaries, so its DICTS and its segment files do not fit this node.
3. Start the node normally.

On a cluster node with no backup, the data of this node is lost. Its replicas still hold the shards, so replace the node.

When no copy exists, start the node once with KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1:

• A lost dictionary block. The node prints one warning per lost dictionary and starts:

  [kdtrace ts=… storage warn] [B1428_DAMAGED_BLOCK] dictionary block cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd: DICTS names it, and no segment holds it; every record coded with it is unreadable — KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1: the store opens, and every read of a record coded with it is an error

  Every record coded with a dictionary that the node still holds is served. Every read of a record coded with a lost dictionary fails with an error that names the dictionary:

  Corruption: seg-00000090.dat: segment: block 818436767759a38b40190047e9eafa54cced2e97001fe27b6c210f16ff3379e8 at offset 16 is coded with dictionary cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd, and the store does not hold that dictionary block

• A damaged DICTS file. The node prints this warning and starts:

  [kdtrace ts=… storage warn] [B1428_DAMAGED_BLOCK] DICTS: the dictionary list is damaged (DICTS: CRC32C mismatch); it names the dictionary blocks the coded records need and pins them against GC — KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1: the store opens with its dictionaries frozen: it trains no dictionary, never rewrites DICTS, and runs no GC sweep

  The node serves every block that it holds. A record names its dictionary block by its hash, so the node still finds it. The node never trains a new dictionary and never rewrites DICTS. Every GC sweep refuses with this error:

  NeedsOperator: B887: the GC sweep does not run: DICTS in /db is damaged, so the store cannot pin its dictionary blocks. Restore DICTS from a backup of THIS data directory (a replica trains its own dictionaries, so its files do not fit), then start again. On a cluster node with no backup, the data of this node is lost, but its replicas still hold the shards: replace the node (B1544).

Copy the data you can read to a new store, then unset the variable.

The node does not start with --s3: the S3 HEAD names a lost block

What you see

The node prints one boot error and exits with code 3. The error carries the kind NeedsOperator and starts with B1575:. It has this form; the parts in <…> are the values of your store:

NeedsOperator: B1575: the S3 HEAD names block <hash> (txn <n>), and no segment holds it. The HEAD goes to S3 after its segments, so S3 lost a segment object. The newest commit the segments hold is <hash>. To open on it, the commits after it are lost. Fix: restore the missing segment object to the bucket from a backup or a bucket version. Then stop this process, delete the local cache directory <data-dir>, and start again. To open on the older commit instead, set KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1. Then the next sync moves the S3 HEAD to that commit.

The error can also name commit blocks that do not decode (B3379). Then the sentence about the newest commit has this form instead:

B3379: the newest commit <hash> does not decode, so the election refuses: its parent is older, and to elect the parent rolls HEAD back. The commit blocks that do not decode: <hash> (<decode error>). The setting KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1 also refuses this case.

A commit that does not decode comes from a newer kinedb (an unknown commit format version) or from a damaged block. Do not set KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1 for this case: the node refuses with it too. Restore the segment, or start the kinedb version that wrote the commit.

What it means

With --s3, the node uploads the segment files first and the HEAD object last. So the block that HEAD names is always in an uploaded segment. When no segment holds it, a segment object was deleted from the bucket: by an operator, a lifecycle rule or a fault. The node could open on an older commit that the segments still hold, but then the commits after it are lost. So the node refuses to start, and it writes no HEAD.

The repair

1. Restore the missing segment object (<prefix>/segments/seg-*.dat) from a backup or from a bucket version.
2. Delete the local cache directory (--data-dir), so the next start downloads the segments again.
3. Start the node normally.

When no copy of the segment exists, start the node once with KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1:

KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --s3 s3://…

The node prints one B1575: warning with both hashes and opens on the older commit. The next commit chains on that commit, and the next sync moves the S3 HEAD to it. The commits after it are lost for good. Then unset the variable.

When the newest commit that the segments hold does not decode, the node does not open with this variable (B3379). It prints one boot error of the kind NeedsOperator that starts with B1575:, names the missing block, its txn and up to 8 commit blocks that do not decode (the newest first, then a count of the rest), and exits with code 3. The fix in the error is: restore the missing segment object, or start a kinedb binary that decodes the newest commit. Before B3379, the node opened on the parent of that commit, and the commits after the parent were lost. It printed the B1575: warning, but no line named the commit that does not decode.

The check has a limit. A block that does not decode counts as a commit only when one of its parents is a commit that the segments hold, or another block that counts as a commit. When no segment holds such a parent, the node does not detect the block: it prints one B3379: warning with the hash and the error, and the election ignores the block. The node prints at most 8 such warnings, then one line with the count of the rest.

The recovery scan cannot read a segment item (B3466)

The recovery reads every record of the local segment files. When it cannot read an item, the node refuses to start, with or without KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1. The error has the kind NeedsOperator, starts with B3466: and has this form:

NeedsOperator: B3466: the S3 HEAD names block <hash> (txn <n>), and no segment holds it. The S3 HEAD recovery scan of the segment files of <data-dir> cannot read <n> item(s): segments/<seg-file> offset <offset>: block <hash> (<reason>); segments/<seg-file>: the file is not a readable segment (<reason>). A damaged item can hold the newest commit, so the recovery does not elect HEAD: to elect an older commit can roll HEAD back. Fix: stop this process, restore each named segment object to the bucket from a backup or a bucket version, delete the local cache directory <data-dir>, and start again. To start without the damaged items, set KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 — a controlled start: the recovery skips each named item with one warning and elects HEAD among the other commits (with KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1 too), and a commit in a named item is lost.

An item is one of these:

• a record that fails its CRC32C check (<n> bytes fail the check);
• an LZ4 record whose content does not hash to its stored hash (its content hashes to <hash>: the stored hash is false);
• a record whose bytes do not hash to its stored hash, a record whose LZ4 or zstd data does not decode, or a record that names a dictionary that the store does not hold (it does not read: <error>);
• the record of the block that the S3 HEAD names, when it lies in a torn tail (the bytes from the first bad record to the end of the file hold its hash) (the record of the S3 HEAD block is torn: the segments were synced before the HEAD upload, so a crash cannot tear it);
• a file that is not a segment (the file is not a readable segment (<error>)).

A block that another record holds in a good copy is not an item. A record whose content does not hash to its stored hash is always an item. A record that decodes but is not a commit is a value block, not an item. With more than 10 items, one numbered line per item comes first, and the summary line comes last. Any other torn tail (a crash during an append) is not an item. When the hash bytes of the HEAD record are damaged too, the node cannot find that record in a torn tail, and the torn tail stays silent. An I/O error of the disk is not an item either: the node fails with that error.

The repair:

1. Stop the node.
2. Restore each named segment object (<prefix>/segments/seg-*.dat) from a backup or from a bucket version.
3. Delete the local cache directory (--data-dir), so the next start downloads the segments again.
4. Start the node normally.

When no good copy exists, start the node once with both variables:

KINEDB_OPEN_WITH_DAMAGED_BLOCKS=1 KINEDB_RECOVER_S3_HEAD_FROM_SEGMENTS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --s3 s3://…

The node prints one [B1428_DAMAGED_BLOCK] B3466: warning per item and elects HEAD among the other commits. A commit in a named item is lost. Then unset both variables. Before B3466, the recovery skipped each such item with no log line. When the item held the newest commit, the node opened on its parent, and the commits after the parent were lost.

The node does not start with --s3: another node owns the prefix

What you see

The node prints one boot error and exits with code 3. The error carries the kind NeedsOperator and starts with B1810:. It has this form; the parts in <…> are the values of your nodes:

NeedsOperator: B1810: the open refuses: the --s3 prefix s3://<bucket>/<prefix> belongs to another node: node id <id>, host <host>, data dir <dir>. This node is node id <id>, host <host>, data dir <dir>. Two nodes on one prefix write over each other's HEAD, DICTS and segment objects, and the data of one node is lost. Fix: give this node its own PREFIX in --s3. If the other node is stopped for good (for example, it lost its disk), set KINEDB_S3_TAKE_OVER_PREFIX=1 to take the prefix over.

A node that runs can also log this error, with the sync uploads nothing in place of the open refuses, on each sync. Then another node took its prefix over. The node continues to serve its local directory, but it uploads nothing to the bucket.

What it means

One node owns one --s3 prefix. The object PREFIX/OWNER holds the id of the owner. The node keeps its id in the file S3_OWNER of its data directory. A node with another id, or a node with an empty data directory (a new id), refuses to start on the prefix. It writes nothing to the bucket and nothing to its data directory.

The repair

• Two nodes that must both run. Give this node its own PREFIX in --s3. Start it again.

• The owner is stopped for good (for example, it lost its disk, and you restore it on a new directory or a new host). Make sure that the old node does not run. Then start the new node once with KINEDB_S3_TAKE_OVER_PREFIX=1:

  KINEDB_S3_TAKE_OVER_PREFIX=1 kinedb --port 4820 --data-dir /var/lib/kinedb --s3 s3://…

  The node prints one B1810: warning that names the old owner, writes its own id to PREFIX/OWNER, and starts. An empty data directory downloads the data of the bucket. Then unset the variable. If the old node runs, its next sync uploads nothing and logs the B1810: error. Stop the old node: the commits that it did not upload are not in the bucket.

The node does not start with --calvin: the catalog has local stamps

What you see

You start a data directory with --calvin, and earlier this directory was written without --calvin: by kinedb --data-dir DIR -e …, by -f, by the embedded library or by a server without --calvin. The node prints one boot error and exits with code 3. The error starts with B1403: and names every catalog slot with a local stamp:

B1403: 4 catalog slot(s) carry a stamp ABOVE the last index 0 of the local metadata log: [_schemas default.main.gone stamp=3 tombstone; _schemas default.main.t stamp=1 live; _databases d2 stamp=4 live; _branches default.b1 stamp=5 live]. A node without a metadata log (the embedded engine, -e, or a standalone server) wrote them, and the metadata path skips or refuses every later DDL on them. The node does not start; nothing was written. Fix: start the node once with KINEDB_RESTAMP_CATALOG_SLOTS=1; it re-stamps each slot with index 0 and keeps the value or tombstone. Manual fix: start this directory without --calvin, run kd-tool backup against it, and kd-tool restore the dump into a new, empty --calvin node. Exit code 3 (needs an operator).

With more than 10 slots, the node first prints one line for each slot, B1403: catalog slot <i>/<n>: <slot>. The B1403: refusal line above comes last and shows the first 10 slots.

A directory that a standalone server of an older version wrote can also hold an index with a local build. Then the error also names each such index, with the B3179: lead:

B3179: 1 index(es) carry a registration or a layout build ABOVE the last index 0 of the local metadata log: [default.main.t.ix_v registration=117470720591790086 layout_build=none servable]. …

A data directory that keeps its metadata rows under a NEW metadata log (for example, a single node that lost its raft_wal directory and its shard_configs directory) also holds rows with an old log index. Then the error also names each such row, with the B3341: lead:

B3341: 1 metadata row(s) carry a changed_at stamp that skips each new entry of the local metadata log (last index 0): [cluster setting _cluster_settings cluster.split.max_shards changed_at=42]. …

The rows are cluster settings, users and roles, the role depth limit, functions, tickers, token keys and the set of revoked tokens.

The same data directory can also keep its cross-shard transaction ledger from the earlier log. Then the error also names each ledger row, with the B3408: lead:

B3408: 7 gtxn ledger row(s) hold a metadata log index above the last index 0 of the local metadata log: [gtxn a/4001 ack=2 live stamp=61; gtxn c/4001 next_pseq=3 live stamp=51; gtxn s/ consumed=51 live stamp=51; …]. …

Such a directory usually also keeps its audit trail. Then the error also names the audit trail, with the B3409: lead:

B3409: the audit trail _audit holds the highest key index 41 ABOVE the key base 0 plus the last index 0 of the local metadata log. …

What it means

On a --calvin node, each table, database and branch in the catalog carries the index of the metadata log entry that changed it last. A later DDL applies only when its index is higher. A node without --calvin stamps these entries with a local number. When that number is higher than the log, every later DROP, ALTER, CREATE INDEX, GRANT or REVOKE on the object does nothing. So the node refuses to start. The refusal changes nothing on disk.

An index carries two more such numbers: its registration and the build of its layout. A local value there makes each write of the index wait for ever, or it makes storage drop the index entry of the write. So the node also refuses for such an index.

A metadata row (a cluster setting, a user, a function and the others above) carries the index of the entry that changed it last in the row itself. An index of an earlier log is above every index of a new log, so every later change of the row does nothing: a SET CLUSTER answers "the cluster holds a later change of this setting", a DROP USER changes nothing. So the node also refuses for such a row.

The cross-shard transaction ledger carries the index of each cross-shard write. With an index of an earlier log, the node skips each new cross-shard write (the client gets OUTCOME UNKNOWN), gives a new write the result of an old one, or loses the ledger write of a new one. So the node also refuses for such a ledger.

Each row of the audit trail has a key that holds the index of its log entry. The node keeps a row that exists. Under a new log, a new event can get the key of an old row, and then the node does not write the event. So the node also refuses for such an audit trail.

The repair

Start the node once with KINEDB_RESTAMP_CATALOG_SLOTS=1:

KINEDB_RESTAMP_CATALOG_SLOTS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin

The node prints one B1403: re-stamped … line per slot and starts. The tables, their rows and their policies stay the same. Then unset the variable. The next start needs no variable.

This repair is for a node with no --join only: the directory becomes the first node of a new cluster. A --calvin --join boot on a directory that holds user tables and no identity file refuses before this check, and the variable does not change that. See A joiner does not start on a standalone data directory.

For each index of a B3179: error, the node also prints one B3179: reset index … line. The index keeps all its entries. A write-only index (a CREATE INDEX that did not finish) has no such repair: the node refuses also with the variable. Start the directory once without --calvin. That start removes the write-only index. Then create the index again.

For the rows of a B3341: error, the same start sets the index of each row to the last index of the log (a cluster setting to the last index plus 1), in the same commit as the slots. The value of each row stays. The node prints one numbered line for each row, B3341: re-stamped row i/n: …. The total line comes last.

Before that commit, the start reads the metadata log that the node keeps. Each entry of that log changed nothing on such a row: for example, a DROP USER of the user did not remove the user. The node prints one line with the range of the log first. Then it prints one numbered line for each such entry. The total line comes last. Entries that a compaction removed are not named, and the range line says so.

Each entry line starts with B3341: skipped entry i/n: index N: and names the op, the kind of row and the key of the row. A name is written in quotes, with each control character escaped. The node writes no file.

• ... was skipped; a later skipped entry (index M) on this key: a later entry changed the same row. Read the entries of the row in index order, and keep only the result that you want.
• ... rows can have been skipped (this boot does not decode the key): a SET CLUSTER, or an entry of a ticker, a function or a token key. The entry can also have applied on another row of that kind. Check the current value before you redo it.

When the node runs again, read the B3341: lines of that start and redo each change by hand, in index order, where it still applies. Examples: DROP USER, DROP ROLE, REVOKE ROLE … FROM, ALTER USER … PASSWORD, DROP CREDENTIAL, REVOKE AUDIT, REVOKE QUERY AS, ALTER USER … REVOKE SESSIONS, the logout of a token, SET CLUSTER, and the ticker, function and token key entries. Each entry has a numbered line of its own; the total line gives the count. A later version writes a file of runnable statements (B3435).

For the rows of a B3408: error, the same start sets the consumed index of the ledger (s/) to the last index of the log when it is above that index, and moves each other ledger row to the last index. The value of each row stays. The node prints one numbered line for each row, B3408: re-stamped gtxn row i/n: …. The total line comes last. Before that commit, the node prints one numbered line for each cross-shard write of the kept log that it skipped, B3408: skipped cross-shard write i/n: index N: gtxn=…. Run each such transaction again where it still applies.

The variable does not repair the ledger in two cases. The node refuses also with the variable, and no other variable changes that:

• The ledger holds a cross-shard transaction that is not complete (a t/ record). The error names each one: t/<G> gtxn=… participants=[shard … pseq …]. Restore the lost raft_wal directory, or use the manual fix of the error (kd-tool backup, then kd-tool restore into a new, empty node).
• The node is not a single node that starts with no --join (it has other voters, it starts passive, or its cluster-size high water is above 1). A repair on one member changes the ledger of that member only. Remove the node from the cluster, and join it again with an empty data directory.

A token key row or the stamp row of the revoked tokens that does not decode does not stop the start. The node prints one B3341: warning that names the row. The tokens then fail closed, as before.

For a B3409: error, the same start writes a key base for the audit trail, in the same commit as the slots. The key base is the highest key index of the trail. Each new event gets a key above every old row. Every old audit row stays, and no row is removed. The node prints one B3409: set the audit key base … line. A later start prints no such line.

If the node then refuses with a B2427: error, read the next section. The same directory can have both faults.

The node does not start with --calvin: rows have stamps above the Raft log

What you see

You start a data directory with --calvin, and earlier this directory was written without --calvin: by kinedb --data-dir DIR -e …, by the embedded library or by a server without --calvin. The node prints one boot error and exits with code 3. The error starts with B2427: and names every table that holds such rows:

B2427: 1 table(s) hold rows or index entries whose stamps are above the Raft log of their data group: [default.main.late]. A node without --calvin (the embedded engine, -e, or a standalone server) wrote these rows before the switch to --calvin, and it stamped them with a local counter or a clock, not with a log index. A write to such a row can be answered as success and not be written. A node that stopped inside a birth window (a merge, a rebuild, a shard copy) can also be flagged; the repair is safe for it. The node does not start; nothing was written. Fix: start the node once with KINEDB_REFLOOR_FOREIGN_STAMPS=1: the node starts, and each table gets new Raft groups above its highest stamp (one repair entry on the metadata log for each table). A write to such a table waits until its repair is applied. A write that was lost before the repair does not come back. Manual fix: start this directory without --calvin, run kd-tool backup against it, and kd-tool restore the dump into a new, empty --calvin node. Exit code 3 (needs an operator).

What it means

On a --calvin node, each row and each index entry carries a stamp: the index of the Raft log entry that wrote it. A write applies only when its index is higher than the stamp of the row. A node without --calvin stamps rows with a local counter or a clock. When such a stamp is higher than the log of the data group, the write loses to the old row. The node can answer success and write nothing. So the node does not start. The refusal changes nothing on disk.

The node scans each table once. It writes an audited marker in the node-local tree \x00_stamp_audit, so a later start does not scan the table again.

A node that stopped inside a birth window can also be flagged: after the commit of a shard merge, a shard rebuild or a shard copy, and before the birth of the new Raft group. Such a node has no durable floor of the new group yet, so the check compares the rows with index 0. The repair is safe for this node: it births new groups above every stamp.

The repair

Start the node once with KINEDB_REFLOOR_FOREIGN_STAMPS=1:

KINEDB_REFLOOR_FOREIGN_STAMPS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin

The same start also repairs the table when you set KINEDB_RESTAMP_CATALOG_SLOTS=1 (see the section above) on the same boot. The node starts and proposes one repair entry for each table on the metadata log. Every replica seals the old Raft groups of the table. Then each replica births new groups at a floor above the highest stamp of the table. The rows, the tombstones, the index trees and the file trees keep their data.

Note these points:

• A write to such a table waits for the repair, up to 10 seconds. After that, the write returns a retryable error and writes nothing. Send the write again.
• A write that was lost before the repair does not come back. The repair makes later writes safe.
• A write during the repair can start a second repair try of the same table. That try never seals a new group of the first repair. The leader of the new group refuses the seal, and the node log shows one [B2362_FRESH_SHARD_SEAL_REFUSED] line. The write then waits for the first repair (B3572).
• A table on a branch that is not the default branch is not repaired. The node refuses with every opt-in. Use the manual fix.
• Unset the variable after the start. The next start needs no variable.

The manual fix needs no variable. Start the directory without --calvin, run kd-tool backup against it, and run kd-tool restore with the dump into a new, empty --calvin node.

If the node refuses with a B2452: error, KINEDB_REFLOOR_FOREIGN_STAMPS=1 does not start it. Read the next section.

The node refuses to replay a data group over rows with foreign stamps

What you see

You start a data directory with --calvin. The directory ran with --calvin before, and then a node without --calvin wrote rows into it (kinedb --data-dir DIR -e …, the embedded library, or a server without --calvin). The node prints one boot error and exits with code 3. The error starts with B2452: and names every table that holds such rows. The node refuses also when you set KINEDB_REFLOOR_FOREIGN_STAMPS=1:

B2452: 1 table(s) hold rows or index entries whose stamps are above the retained Raft log of their data group, and that group replays its log at this start: [default.main.late]. A node without --calvin (the embedded engine, -e, or a standalone server) wrote these rows, and their stamps are not log indexes. The replay reads a stored stamp as the proof that a log entry is already applied, so it can skip a committed write that storage does not hold, and that write is lost. The repair runs only after the replay, so KINEDB_REFLOOR_FOREIGN_STAMPS=1 does not start the node, and the catalog re-stamp opt-in does not either. The node does not start; nothing was written. Fix: start this directory without --calvin, run kd-tool backup against it, and kd-tool restore the dump into a new, empty --calvin node. When another node holds every row of these tables, evict this node and add it again with --join. To accept the risk, start the node once with KINEDB_REPLAY_OVER_FOREIGN_STAMPS=1: the replay runs, then the node repairs each table as KINEDB_REFLOOR_FOREIGN_STAMPS=1 does. A write that the replay skips does not come back. Exit code 3 (needs an operator).

What it means

A data group that has a Raft log on this node replays the log at each start. The replay applies each entry again over the data on disk. It uses the stamp of a row as the proof that an entry is already applied: a stamp at or above the index of the entry means "skip". The rows of a node without --calvin have stamps that are not log indexes. So the replay can skip a committed write that the disk does not hold, and that write is lost.

The repair of the previous section runs only after the replay. So the node checks these groups before the replay, and it refuses. The node cannot tell which stamps are safe: a foreign stamp below the log looks the same as a stamp of the log. The refusal changes nothing on disk.

The repair

Use the manual fix. Start the directory without --calvin, run kd-tool backup against it, and run kd-tool restore with the dump into a new, empty --calvin node. When another node of the cluster holds every row of these tables, you can evict this node and add it again with --join instead.

If you accept the risk, start the node once with KINEDB_REPLAY_OVER_FOREIGN_STAMPS=1:

KINEDB_REPLAY_OVER_FOREIGN_STAMPS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin

The replay runs. Then the node repairs each table as KINEDB_REFLOOR_FOREIGN_STAMPS=1 does (see the previous section). A write that the replay skips does not come back. Unset the variable after the start.

A node stops with [REFLOOR_STAMP_ABOVE_FLOOR]

What you see

During the repair of a table with stamps above the Raft log, a node stops with a [B1082_FAIL_STOP] [REFLOOR_STAMP_ABOVE_FLOOR] line. The line names the table, the stamp and the floor. This section is the next step after KINEDB_REFLOOR_FOREIGN_STAMPS=1 (see the section above).

What it means

The node that proposed the repair set the floor above the highest stamp of its own copy and above its clock plus 1 day. A row or an index entry on this node has a stamp above that floor. A write of the repaired table could lose to that entry, so the node stops and writes nothing. A stamp comes from the clock of the node that wrote it. The floor covers a clock that ran up to 1 day ahead of the clock of the proposer. A clock that ran further ahead gives this stop. When each node holds the same rows, each node stops in the same way.

The repair

Do not delete the data directory. Make a copy of it. Then:

1. Start the copied directory without --calvin. Then no metadata log entry is applied again, and the node does not stop.
2. Run kd-tool backup against that node (see backup).
3. Start a new cluster with --calvin on empty data directories.
4. Run kd-tool restore with the dump into the new cluster.

Alternative, only when another node of the cluster holds every row of the table: remove this node with ALTER CLUSTER EVICT NODE '<id>'. Then start it on an empty data directory with --join. The node copies the table from the other nodes. The steps are the same as for [B2264_ADOPT_STAMP_ABOVE_FLOOR]: see a table has rows and no catalog entry.

The node does not start with --calvin: the Raft log of a member is lost

What you see

A node of a cluster keeps its data directory, but its Raft log is gone: somebody deleted <data-dir>/raft_wal, one directory under it, or only the hardstate file or the *.log files of one directory, KINEDB_RAFT_WAL_DIR points to a new place or is empty, a hardstate file under it is damaged, or the node cannot list a directory under it (for example, a permission error or a disk error). The node prints one boot error and exits with code 3. The error starts with B2048: and names every Raft group that the node is a member of and whose Raft state is lost:

B2048: the Raft state of 2 group(s) is lost, but /var/lib/kinedb/shard_configs names this node a member: [shard 0: no hardstate, no log and no snapshot floor in /var/lib/kinedb/raft_wal/0; shard 1 (metadata): no hardstate, no log and no snapshot floor in /var/lib/kinedb/raft_wal/1]. A member with term 0, no vote and an empty log must not vote: it can elect a leader that lacks committed entries, or vote two times in one term. The node does not start; nothing was written. Fix: restore the raft_wal directory (or set KINEDB_RAFT_WAL_DIR to its location) and start again. Or start the node once with KINEDB_REJOIN_PASSIVE_AFTER_LOST_RAFT_WAL=1: each group boots passive, never votes, and gets the log from its leader. Manual fix: move this data directory away, start the node on an empty one with --join (a new node id), and run ALTER CLUSTER EVICT NODE '0a911d8ae3c8d54f07acd16a033f52f4' for the old id. Exit code 3 (needs an operator).

A group can also lose only a part of its Raft state. Each group directory holds a log_acked file after the group acknowledged its first log entry. A group directory that an older kinedb wrote without that file gets it when the group acknowledges its next log entry. With that file, the error names two more cases:

• shard 7: no hardstate in /var/lib/kinedb/raft_wal/7, but its log_acked file shows that the node acknowledged log entries (the term and the vote are lost): the hardstate file is gone, and the log is there.
• shard 7: no log segment and no snapshot floor in /var/lib/kinedb/raft_wal/7, but its log_acked file shows that the node acknowledged log entries (the log is lost): the *.log files are gone, and the hardstate file is there.

A directory with a hardstate file and no *.log file and no log_acked file is a group that has no log entry yet. The node starts it as usual.

What it means

Each Raft group remembers its term, its vote and its log on disk. A member that lost them and votes again can help elect a leader that does not have the committed writes, and the other nodes then lose those writes. A member that lost only its term and vote can vote two times in one term. So the node refuses to start. The refusal changes nothing on disk.

The repair

If you still have the Raft log, put it back (or set KINEDB_RAFT_WAL_DIR to its place) and start the node again. For a partial loss, put back the missing hardstate file or the missing *.log files of that group directory. Do not delete the log_acked file to make the node start: the group then votes with a lost term or a lost log.

If the Raft log is gone, start the node once with KINEDB_REJOIN_PASSIVE_AFTER_LOST_RAFT_WAL=1:

KINEDB_REJOIN_PASSIVE_AFTER_LOST_RAFT_WAL=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node prints one [B2048_LOST_RAFT_WAL] line per group and one [B2048_PASSIVE_BOOT] line when it builds each group. Each such group is a passive follower: it does not vote and does not start an election. The leader sends it the log, and the group votes again when a committed configuration entry that names the node reaches it. The other members must hold a majority of each group while this node catches up. Then unset the variable.

The variable does not help when the node is the only voter of the metadata group (for example, a 1-node cluster): no other member holds the log, and a passive group of one voter never elects. The node then refuses with and without the variable. The error starts with B2048:, names shard 1 (metadata) and <data-dir>/shard_configs/1, and says that no other member holds the log. Restore the raft_wal directory or a backup of the data directory.

Without the variable: move the data directory away, start the node on an empty data directory with --join (it gets a new node id), and remove the old id with ALTER CLUSTER EVICT NODE '<old id>'.

The node does not start with --calvin: no record names the only metadata voter

What you see

A node with a store of an older kinedb version (often a 1-node cluster) prints one boot error and exits with code 3. The error starts with B3398::

B3398: /var/lib/kinedb/shard_configs/1 is missing, but the metadata Raft log in /var/lib/kinedb/raft_wal/1 has history, and this node is the only voter of the metadata group in that log. The store has no record of that, so the boot cannot find a later loss of the metadata Raft WAL: the node would start a new metadata log. The node does not start; nothing was written. Fix: start the node once with KINEDB_RECORD_LONE_METADATA_VOTER=1; it writes /var/lib/kinedb/shard_configs/1 with this node as the only voter, and starts. Manual fix: if this node is a member of a cluster with other voters, start it with --join to a live member. Exit code 3 (needs an operator).

What it means

A node that is the only voter of the metadata group now keeps the file <data-dir>/shard_configs/1. The boot check of a lost Raft log (the Raft log of a member is lost) finds a lost metadata log only through this file. An older version did not write the file for a lone voter. Without the file, a lost metadata log starts a new log at index 0, and the store and the log no longer agree. The refusal changes nothing on disk.

The repair

Start the node once with KINEDB_RECORD_LONE_METADATA_VOTER=1, for example in the environment of the service for that one start:

KINEDB_RECORD_LONE_METADATA_VOTER=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin

The node prints one [LONE_METADATA_RECORD_OPT_IN] line and one [LONE_METADATA_RECORD] line, writes the file and starts. Then unset the variable.

If this node is a member of a cluster with other voters, do not set the variable. Start the node with --join to a live member.

The node does not start with --calvin: the Raft log of a member has a hole

What you see

The node prints one boot error and exits with code 3. The error starts with B2117: and names every Raft group that the node is a member of and whose Raft log has a hole, with the first missing index of each:

B2117: the Raft WAL of 2 group(s) has a hole: [shard 7: index 5 is missing, the log holds entries up to 8, in /var/lib/kinedb/raft_wal/7; shard 9: index 101 is missing, the log holds entries up to 300, in /var/lib/kinedb/raft_wal/9]. A log with a hole names entries that it does not hold, so a leader can count this node in a quorum for a missing entry. The node does not start; nothing was written. Fix: start the node once with KINEDB_RAFT_WAL_CUT_AT_HOLE=1: the node cuts each such log at its first missing index, keeps the entries before it, and gets the rest from its leader. The node never fills a hole. Manual fix: move this data directory away, start the node on an empty one with --join (a new node id), and run ALTER CLUSTER EVICT NODE '0a911d8ae3c8d54f07acd16a033f52f4' for the old id. Exit code 3 (needs an operator).

What it means

A version before B2117 removed the end of a Raft log oldest file first. A power loss between two removals kept a log with a hole, for example the entries 1 to 4 and 7 to 8. Such a log names entries that it does not hold. A leader can then count this node for a missing entry, and the cluster can lose that entry. So the node refuses to start. The refusal changes nothing on disk.

The repair

Start the node once with KINEDB_RAFT_WAL_CUT_AT_HOLE=1:

KINEDB_RAFT_WAL_CUT_AT_HOLE=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node cuts each named log at its first missing index: it keeps the entries before the hole and removes every later entry. It prints one [B2117_WAL_CUT_AT_HOLE] line per group. The node never fills the hole: the leader sends the removed entries again. The other members must hold a majority of each group while this node catches up. Then unset the variable.

Without the variable: move the data directory away, start the node on an empty data directory with --join (it gets a new node id), and remove the old id with ALTER CLUSTER EVICT NODE '<old id>'.

The node does not start with --calvin: the Raft log of a member is damaged in its last segment

What you see

The node prints one boot error and exits with code 3. The error starts with B2971: and names every Raft group that the node is a member of and whose last Raft log file is damaged. For each group it names the file, the offset of the record that does not decode, and the valid record that follows it:

B2971: the Raft WAL of 1 group(s) is damaged in its last segment: [shard 7: the record at offset 4096 of /var/lib/kinedb/raft_wal/7/00000000000000000001.log does not decode, but a valid record (index 52) follows at offset 4160; the last good index is 50]. A torn write leaves only a prefix of the last append, so a valid record after a record that does not decode is damage, not a torn tail. A cut at the bad record loses the valid record and every later one, and this node can have acknowledged them. The node does not start; nothing was written. Fix: start the node once with KINEDB_RAFT_WAL_CUT_AT_DAMAGE=1: the node cuts each such log at its bad offset, keeps the entries before it, and gets the rest from its leader. Manual fix: move this data directory away, start the node on an empty one with --join (a new node id), and run ALTER CLUSTER EVICT NODE '0a911d8ae3c8d54f07acd16a033f52f4' for the old id. Exit code 3 (needs an operator).

What it means

A power loss during a write keeps only the start of the last write in a Raft log file. A record that does not decode, with nothing valid after it, is such a torn write, and the node removes it at the start. A valid record AFTER a record that does not decode is not a torn write: media damage, or a write of an older version after a failed write, made it. A version before B2971 removed the bad record and every record after it with no message. This node can have acknowledged those entries to its leader, so the cluster can lose them. So the node refuses to start. The refusal changes nothing on disk.

The repair

Start the node once with KINEDB_RAFT_WAL_CUT_AT_DAMAGE=1:

KINEDB_RAFT_WAL_CUT_AT_DAMAGE=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node cuts each named log file at the offset of its bad record: it keeps the entries before it and removes the bad record and every later one. It prints one [B2971_RAFT_WAL_CUT_AT_DAMAGE] line per group. The leader sends the removed entries again. The other members must hold a majority of each group while this node catches up. Then unset the variable.

Without the variable: move the data directory away, start the node on an empty data directory with --join (it gets a new node id), and remove the old id with ALTER CLUSTER EVICT NODE '<old id>'.

The node does not start with --calvin: the Raft timings break the lease bound

What you see

The node prints one boot error and exits with code 3. The error starts with B2888: and names the two values, the read lease window, the window plus one heartbeat, and the election timeout:

B2888: KINEDB_RAFT_ELECTION_US=1000000 and KINEDB_RAFT_HEARTBEAT_US=500000 break the lease bound: the read lease window is 1250000 us, and the window plus one heartbeat is 1750000 us, which is not under the election timeout of 1000000 us. A read lease could then outlive the leadership that granted it, and a follower could serve a stale read. The node does not start; nothing was written. Fix: set KINEDB_RAFT_ELECTION_US to 1750001 or more (the smallest value for this heartbeat), or lower KINEDB_RAFT_HEARTBEAT_US. Exit code 3 (needs an operator).

A value that is not a whole number of microseconds (for example 500ms), and a heartbeat of 0, stop the start the same way. That error names the variable and its value.

What it means

A follower serves a local read only while it holds a read lease from its leader. The read lease window is two heartbeats plus four safety margins of one eighth of a heartbeat: 1.25 s for the default heartbeat of 500 ms. The window plus one heartbeat must stay under the election timeout. Otherwise the group can elect a new leader while a follower still holds a lease of the old leader, and that follower can serve a stale read. So the node refuses to start. The check runs before the node opens the data directory, so the refusal changes nothing on disk.

The repair

There is no variable that allows the start. Set a correct pair: raise KINEDB_RAFT_ELECTION_US to the value that the error names or more, or lower KINEDB_RAFT_HEARTBEAT_US. For a heartbeat of 8 ms or more, the election timeout must be more than 3.5 heartbeats. For example, the defaults:

KINEDB_RAFT_ELECTION_US=3000000 KINEDB_RAFT_HEARTBEAT_US=500000 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node does not start with --calvin: the Raft log of shard 0 holds an old write with no table bind

What you see

The node prints one boot error and exits with code 3. The error starts with B2114: and names each write of the Raft log of shard 0 that a version before B2114 wrote, with its log index, its table and its key, and each table:

B2114: the Raft log of shard 0 holds 1 write(s) with no table bind, from a version before B2114: [index 1 table "acct" key "a"]. Tables: ["acct"]. At each start the node applies this log again, and the replay of such a write finds its table by NAME: it can write a row of a dropped table into a re-created table of the same name. The node does not start; nothing was written. The rows of the named tables MAY already be wrong. Check: compare each named table on each replica (the same rows on every replica). Opt-in: start the node once with KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1. The node then SKIPS these entries and logs each skip. That start writes a durable record of the skipped entries. When the write of the record succeeds, the next start needs no variable. When the record is larger than its bound, the start writes nothing and logs [B2764_ACK_RECORD_TOO_LARGE], and the next start needs the variable again. A binary from B2114 up to B2764 ignores the record, so it needs the variable again. A binary from B2318 up to B2427 also does not serve a metadata point. A skip CAN LOSE A WRITE: an entry that was not yet applied, or that was applied but was not yet durable, is lost (the storage WAL is fsynced every 5 seconds, and a SIGKILL, a power cut or a SIGTERM stop of an older version does no final sync). Exit code 3 (needs an operator).

What it means

Since B2114 each write names the table generation (the TableId) that its statement was translated against. The apply refuses a write whose table is not that table any more. A version before B2114 wrote no table generation. At each start the node applies the retained Raft log of shard 0 again, and such a write then finds its table by its NAME. After a DROP TABLE and a CREATE TABLE of the same name, the replay writes a row of the dropped table into the new table, on this node only. So the node refuses to start. The refusal changes nothing on disk. A table whose layout is still on shard 0 (a version before the per-table shards) is not refused.

The repair

1. On each replica, compare each named table with the same table on the other replicas. An older version could already have written a row into the wrong table.
2. Start the node once with KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1:

KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node prints one [B2114_SHARD0_UNBOUND_WRITES] line at the start. The apply of shard 0 skips each named write and prints one [B2114_SKIP_UNBOUND_SHARD0_WRITE] line with its index, table and key. Every other write applies as before. A skip can lose a write: a write that was not yet applied, or that was applied but was not yet on the disk, is lost. The storage WAL is synced every 5 seconds. A SIGKILL, a power cut or a SIGTERM stop of an older version does no final sync. To make this less likely, stop the writes to the old version, wait 10 seconds, and stop it before you start the new version.

Before the start with the variable continues, it writes one durable record of the skipped writes on this node (the key b2764_acked_unbound_shard0_writes of the node-local tree \0_stamp_audit). The record holds 52 bytes and the table bytes for each skipped write. It does not hold the row key. A start with the variable adds the new writes to a good record and removes nothing. Only a record that does not decode is replaced: the new record holds only the writes of the current log (see the next section). When the write of the record succeeds, you can unset the variable: each later start reads the record, skips the same writes and prints one [B2764_ACKED_UNBOUND_WRITES] line. A write that the record does not name still stops the start with the B2114: error. If the write of the record fails, the node does not start (a B2764: error, exit code 3).

A record of more than 16 MiB (about 300 000 writes with a table name of 4 bytes) is not written. The node then starts and skips the writes as the variable says, and it writes nothing. It prints one [B2764_ACK_RECORD_TOO_LARGE] line with the count, the size and the bound. The record on disk, if any, stays as it is. The next start needs the variable again.

A version at or after B2427 and before B2764 does not read the record, so it refuses again with B2114:. A version from B2318 up to B2427 does not know the tree \0_stamp_audit. With the tree, that version does not serve a metadata point to a joining node. Do not go back to such a version after the record exists.

The node does not start with --calvin: the record of the acknowledged shard-0 writes does not decode

What you see

The node prints one boot error and exits with code 3. The error starts with B2764:, names the record and the decode error, and names each write of the Raft log of shard 0 that a version before B2114 wrote:

B2764: the record of the acknowledged writes with no table bind of shard 0 (the key "b2764_acked_unbound_shard0_writes" of the tree "\0_stamp_audit") cannot be read: the version byte is 9, and this binary reads only version 2. The Raft log of shard 0 holds 1 write(s) with no table bind, from a version before B2114: [index 1 table "acct" key "a"]. Tables: ["acct"]. The node does not start; nothing was written. The rows of the named tables MAY already be wrong. Check: compare each named table on each replica (the same rows on every replica). Then start the node once with KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1. That start SKIPS each listed entry and writes a new record in place of the bad one. When the write of the record succeeds, the next start needs no variable. When the record is larger than its bound, the start writes nothing and logs [B2764_ACK_RECORD_TOO_LARGE], and the next start needs the variable again. A binary from B2114 up to B2764 ignores the record, so it needs the variable again. A binary from B2318 up to B2427 also does not serve a metadata point. A skip CAN LOSE A WRITE: an entry that was not yet applied, or that was applied but was not yet durable, is lost. Exit code 3 (needs an operator).

What it means

An earlier start with KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1 wrote a record of the writes that it skipped (see the section above). This start cannot decode that record: a newer version wrote it (an unknown version byte), or the value is damaged. The node does not know which writes the record names, so it does not skip a write by the record. It refuses to start and changes nothing on disk.

The repair

1. On each replica, compare each named table with the same table on the other replicas.
2. Start the node once with KINEDB_SKIP_UNBOUND_SHARD0_WRITES=1. That start skips each listed write, writes a new record of these writes in place of the bad one, and prints one [B2764_ACK_RECORD_REPLACED] line with the decode error. The next start needs no variable. The skip has the same data-loss risk as in the section above. When the new record is more than 16 MiB, that start writes nothing and prints one [B2764_ACK_RECORD_TOO_LARGE] line. The bad record then stays, and the next start without the variable shows this error again.

The new record holds only the writes of the current log. A write that the bad record named and that the log does not hold any more is not in the new record. The variable also replaces a record that a newer version wrote (an unknown version byte).

If the error says that the storage read or write of the record failed, the variable cannot help. Check the disk and the store (free space, permissions, errors in the system log), then start the node again. If the fault comes again on a healthy disk, keep the data directory as it is and report the error in an issue.

The node does not start with --calvin: a shard merge has no retired-source row

What you see

The node prints one boot error and exits with code 3. The error starts with B2301: and names each shard merge that a version before B2301 installed, with its survivor shard and its metadata log index:

B2301: 1 shard merge(s) that a version before B2301 installed have no retired-source row in the global transaction ledger: [survivor shard 2147483655 at metadata log index 7]. Since B2301 the ledger refuses a cross-shard transaction that names a shard that a merge retired. No replicated row names the sources of these merges, so this version cannot add the rows. A node that routes with a layout from before such a merge can still give a slice of a cross-shard transaction to a retired source, and the transaction then loses that slice. The node does not start; nothing was written. Check: run SHOW SHARDS on every node. A node that does not list a named survivor routes with an old layout: send no cross-shard transaction to it. Opt-in: start the node once with KINEDB_START_WITH_UNRETIRED_MERGES=1. The node then starts, writes the named merges into the file b2301_unretired_merges_ack in the data directory, and does not refuse a cross-shard transaction on a source of these merges, as before B2301. Exit code 3 (needs an operator).

What it means

A shard merge (MERGE SHARDS) replaces two source shards with one survivor shard. Since B2301 the merge also writes one row for each source into the global transaction ledger, and the ledger refuses a cross-shard transaction that names a source. The client gets a retryable error and sends the transaction again against the new layout. A version before B2301 wrote no such row. No replicated row names the sources of such a merge, so the new version cannot add the rows. The refusal changes nothing on disk.

The repair

1. Run SHOW SHARDS on every node. A node that does not list a named survivor shard routes with a layout from before the merge. Send no cross-shard transaction to it.
2. Start the node once with KINEDB_START_WITH_UNRETIRED_MERGES=1:

KINEDB_START_WITH_UNRETIRED_MERGES=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node prints one [B2301_UNRETIRED_MERGE] line for each merge and writes the survivors into the file b2301_unretired_merges_ack in the data directory. A later start does not refuse a merge that the file names. Then unset the variable.

The node does not start with --calvin: a shard has no voter record on this node

What you see

The node prints one boot error and exits with code 3. The error starts with B2376: and names each shard that has no voter record and no Raft log on this node, with its table:

B2376: 1 shard(s) have no voter record and no Raft log on this node: [shard 2147483655 of table "docs"]. An older version wrote no voter record on a node outside the voter set of a new shard. The same state also follows a stop of this node after the commit of a new shard and before the start of its Raft group. The boot cannot tell such a shard from a shard whose voter record is lost. A boot of such a shard makes this node its only voter, and that voter serves data that is not complete. The node does not start; nothing was written. Check: run SHOW RAFTGROUP <shard id> on the other nodes. The peers column of a node that holds a replica lists the voters of the shard. Opt-in: start the node once with KINEDB_START_WITH_UNRECORDED_NON_VOTER_SHARDS=1. The node then starts, builds no Raft group of the named shards at boot, and writes them into the file b2376_unrecorded_non_voter_shards_ack in the data directory. The shard acquisition then uses the cluster view to find the voters. Exit code 3 (needs an operator).

What it means

On a cluster with more nodes than the replication factor, each shard has voters on some nodes only. A node keeps a voter record of each shard in its data directory. An older version wrote no such record on a node outside the voters of a new shard. The same state also follows a stop of the node after the commit of a new shard and before the start of its Raft group. Since B2384 the commit of a CREATE TABLE, a CREATE COLLECTION or a TRUNCATE also holds the voters of the new shard: after such a stop the node starts the shard with those voters and does not refuse. Since B2399 the commit of a shard split also holds the voters of its two new shards. Since B2394 the commit of a branch materialize (the first write on a branch table) also holds the voters of each shard of the branch table. Since B2739 this is true also when the cluster had one node at the commit (for example the audit collection that the first node creates). Since B4091 a node that got a shard through a metadata point install (a node that joined after the shard was made) records that fact. At a start, it builds no Raft group of such a shard and does not refuse. A shard that an older version made still gets the refusal. At a start, the node does not yet know the members of the cluster. With no record and no Raft log, the node cannot tell whether it is a voter. If it started the shard, it would be the only voter of the shard and would serve data that is not complete. The refusal changes nothing on disk.

The repair

1. Run SHOW RAFTGROUP <shard id> on the other nodes. The peers column of a node that holds a replica of the shard lists the voters of the shard.
2. Start the node once with KINEDB_START_WITH_UNRECORDED_NON_VOTER_SHARDS=1:

KINEDB_START_WITH_UNRECORDED_NON_VOTER_SHARDS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

The node prints one [B2376_UNRECORDED_NON_VOTER] line for each shard and writes the shard ids into the file b2376_unrecorded_non_voter_shards_ack in the data directory. It builds no Raft group of these shards at the start. The shard acquisition then uses the cluster view to find the voters. A later start does not refuse a shard that the file names. Then unset the variable.

The node does not start with --calvin: a ticker exists only on this node

What you see

The node prints one boot error and exits with code 3. The error starts with B2332: and names each ticker that only this node holds, with its creator and its CREATE TICKER statement:

B2332: 1 ticker(s) exist only on this node: [ticker "decay" (created by "bob"): CREATE TICKER decay EVERY 60000ms CALL decay_fn OVERLAP SKIP]. A version before B2332 wrote each of these rows with a local write, and no metadata log entry holds it. The other nodes do not have these tickers, and the metadata state of this node does not match the other nodes. The node does not start; nothing was written. Fix: start the node once with KINEDB_DROP_LOCAL_TICKERS=1; it removes each named ticker on this node and logs its CREATE TICKER statement. Then run each CREATE TICKER statement again, as its creator: the statement then goes through the metadata log to every node. Exit code 3 (needs an operator).

What it means

Since B2332 a CREATE TICKER or a DROP TICKER on a --calvin node goes through the metadata log, and every node writes the same ticker row. A version before B2332 wrote the row only on the node that ran the statement. The other nodes do not have that ticker. Only the metadata leader runs a ticker, so such a ticker ran only while that node was the leader. A standalone server (no --calvin) keeps its tickers on its own node, and it does not refuse. The refusal changes nothing on disk.

The repair

1. Copy each CREATE TICKER statement and its creator from the error.

2. Start the node once with KINEDB_DROP_LOCAL_TICKERS=1:

   KINEDB_DROP_LOCAL_TICKERS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

   The node removes each named ticker on this node, prints one [B2332_DROP_LOCAL_TICKER] line with the CREATE TICKER statement of each ticker, and starts. Then unset the variable. In a cluster, do this on each node that refuses.

3. Log in as the creator of each ticker, and run its CREATE TICKER statement again. The ticker is then on every node. If two nodes held a ticker of the same name, run the statement of the ticker that you want to keep.

A ticker row that a node of this version wrote is never removed.

The node does not start with --calvin: a cluster setting is local to this node

What you see

The node prints one boot error and exits with code 3. The error starts with B3279: and names each cluster setting that only this node holds, with its SET CLUSTER statement:

B3279: 1 cluster setting(s) of this node are local: ["cluster.split.max_shards": SET CLUSTER cluster.split.max_shards = 7]. A standalone server wrote each of these rows, and no metadata log entry holds it. The data directory belongs to a cluster of 2 nodes, so the other nodes do not have these values. The node does not start; nothing was written. Fix: start the node once with KINEDB_DROP_LOCAL_CLUSTER_SETTINGS=1; it writes the SET CLUSTER statement of each named row into the file b3279_dropped_cluster_settings of the data directory and removes the row on this node. Then run each statement on the cluster: it then goes through the metadata log to every node. Exit code 3 (needs an operator).

Two other causes give the same error with another sentence in the middle:

• A start with --join on a data directory of a single node: The node starts with --join, and a joiner takes the settings of the cluster, so the other nodes do not have these values.
• A last_peer_count file or a shard_configs/1 file that does not read: The cluster size high water of the data directory does not read: … So this node cannot prove that the directory belongs to one node only. Another fix: repair or remove the named file. After the repair, a single-node data directory keeps its local values. The error names the path of the file.

What it means

A standalone server (no --calvin) applies a SET CLUSTER on its own node: the value is local. When a single-node data directory later starts with --calvin and no --join, the node keeps each local value, and a later SET CLUSTER goes through the metadata log as usual. The node refuses to start in three cases:

• The data directory was a node of a cluster of more nodes (its cluster size high water is above 1), and it ran as a standalone server between.
• The node starts with --join. A joiner takes the settings of the cluster that it joins.
• A file of the high water does not read. The node cannot prove that it is a single node.

In each case the other nodes do not have the local values. The check runs before the node writes its identity file, so the refusal changes nothing on disk.

Two starts without --join. A data directory of a cluster of more nodes that starts with --calvin and no --join stops with exit code 3 and the B3279: error. After the drop start below, the next check can stop it with exit code 1 and the I8 error:

FATAL: "/var/lib/kinedb" has prior MULTI-NODE cluster identity (last known peer count = 2) but --join was not provided.

That error names no cluster setting. Start the node again with --join.

The repair

1. Copy each SET CLUSTER statement from the error.

2. Start the node once with KINEDB_DROP_LOCAL_CLUSTER_SETTINGS=1:

   KINEDB_DROP_LOCAL_CLUSTER_SETTINGS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

   The node first appends to the file b3279_dropped_cluster_settings of the data directory: one header line # drop of local cluster settings at unix second N, then the SET CLUSTER statement of each setting on a line of its own. It fsyncs the file, then the directory. It prints one [B3279_DROP_LOCAL_CLUSTER_SETTING] line for each setting. Then it removes the settings on this node in one durable commit, prints one [B3279_DROP_LOCAL_CLUSTER_SETTINGS_DONE] line, and starts. Each line names the file. The node never truncates or deletes the file. Then unset the variable.

3. Run each SET CLUSTER statement that you want to keep on the cluster. The value is then on every node. A line that starts with # is a header, not a statement. A string value with a newline or a carriage return has no one-line SET CLUSTER form: its line starts with NOT RUNNABLE: and gives the exact value as a JSON string. Set that value by hand.

If the high water does not read, you can also repair or remove the named file (for example /var/lib/kinedb/last_peer_count) instead of the drop. After the repair, a data directory of a single node starts and keeps its local values.

A setting that a --calvin node wrote is never removed.

The node does not start without --calvin: the data directory is of a cluster

What you see

A server without --calvin, a run with -e or -f (also with --calvin), or the shell prints one boot error and exits with code 3. The error starts with B3290::

B3290: this start runs no Raft group (a server without --calvin, or -e or -f), and the data directory is of a cluster of more nodes. The data directory belongs to a cluster of 3 nodes (the cluster size high water is 3). A write of this start is in no log entry, so the other replicas never get it. The node does not start; nothing was written. Fix: start the node with --calvin as a member of its cluster. To use the directory without a cluster, start it once with KINEDB_DETACH_FROM_CLUSTER=1: the start writes the file b3290_detached_from_cluster into the data directory, and the directory can then never start with --calvin again. Exit code 3 (needs an operator).

A last_peer_count file or a shard_configs/1 file that does not read gives the same error with another sentence in the middle: The cluster size high water of the data directory does not read: … So this start cannot prove that the directory belongs to one node only. The error names the path of the file.

What it means

The data directory was a node of a cluster of more than one node (its cluster size high water is above 1). A start without a Raft group writes data that no log entry holds. The other replicas never get that data, and a later --calvin start of the directory can then differ from them. So the start refuses before the store opens. The refusal changes nothing on disk. A new data directory and a data directory of a single node are not refused.

The repair

• To keep the node in its cluster, start it with --calvin (and --join, as before).

• To use the directory without a cluster, start it once with KINEDB_DETACH_FROM_CLUSTER=1:

  KINEDB_DETACH_FROM_CLUSTER=1 kinedb --data-dir /var/lib/kinedb -e "SHOW TABLES"

  The start writes the file b3290_detached_from_cluster into the data directory, makes it durable, prints one [B3290_DETACHED_FROM_CLUSTER] line and opens the store. Each later start without --calvin opens with no variable. The directory never starts with --calvin again (see the next section). Then unset the variable.

• If a file of the high water does not read, repair the named file, or detach the directory as above.

The node does not start with --calvin: the data directory is detached from its cluster

What you see

The node prints one boot error and exits with code 3. The error starts with B3290::

B3290: this node starts with --calvin, and its data directory holds the file /var/lib/kinedb/b3290_detached_from_cluster: a start without --calvin detached it from its cluster. The store can hold writes that no replica has, so it cannot be a replica again. The node does not start; nothing was written. Fix: start the directory without --calvin, run kd-tool backup against it, and kd-tool restore the dump into a new, empty --calvin node. Or move the directory away, start the node on an empty one with --join, and evict the old node id. There is no override. Exit code 3 (needs an operator).

What it means

An earlier start with KINEDB_DETACH_FROM_CLUSTER=1 detached the directory from its cluster (see the section above). After that, starts without --calvin could write data that no replica has. The check runs before the store opens and before the node writes its identity file, so the refusal changes nothing on disk. No environment variable overrides it.

The repair

1. When you need the data, start the directory without --calvin. Run kd-tool backup against it, and kd-tool restore the dump into a new, empty --calvin node.
2. To bring the node back into its cluster, move the data directory away. Start the node with --calvin --join and an empty data directory. Then evict the old node id.

The node does not start with --calvin: a structure is older than the catalog

What you see

The node prints one boot error and exits with code 3. The error starts with B2987: and names each index tree or table of this node that is older than the catalog:

B2987: 2 structure(s) of this node are older than the catalog: [index tree "\u{1}idx_by_name" (build 30, catalog build 60, 2 row(s)); index tree "\u{1}idx_by_age" (build 42, no catalog index, table catalog changed at 70, 2 row(s))]. A version before B2972 installed a metadata point and kept these layouts and their rows. Index trees: the catalog names another build, or no index. Tables: the catalog names another table id. The node does not start; nothing was written. Fix when every item says 'no catalog index': start the node once with KINEDB_PURGE_SUPERSEDED_POINT_LAYOUTS=1. It removes each named index tree and its rows in one local commit, then the node starts. Fix for an item with 'catalog build' or a table (the variable does not repair it; with it set the node still refuses and writes nothing): remove this node from the cluster, delete its data directory, and join it again as a new node. Exit code 3 (needs an operator).

What it means

A node that fell behind the metadata log installs a metadata point from another node. A version from B2318 to B2972 installed the catalog of the point, but it kept the old index trees and tables of this node with their rows. So this node can hold:

• an index tree of a dropped index (no catalog index): the catalog names no such index. table catalog changed at gives the last change of the catalog entry of its table;
• an index tree of an older build of an index (catalog build): the index was dropped and created again. The entries of the new build are not on this node;
• a table with an older table id (layout id, catalog id): the table was dropped and created again. The rows on this node belong to the dropped table, and this node does not have the rows of the new table.

The check runs only on a --calvin node that installed a metadata point. The refusal changes nothing on disk.

The repair

When every item says no catalog index:

1. Start the node once with KINEDB_PURGE_SUPERSEDED_POINT_LAYOUTS=1:

   KINEDB_PURGE_SUPERSEDED_POINT_LAYOUTS=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

   The node removes each named index tree and its rows, prints one [B2987_PURGED_SUPERSEDED_LAYOUT] line for each tree, checks again and starts. Then unset the variable.

When an item says catalog build, or names a table, the variable does not repair the node. With the variable set, the node still refuses and writes nothing, also for the no catalog index items. The index or the table holds data that this node does not have, and this node has no path to get it back in place. Remove the node from the cluster, delete its data directory, and join it again as a new node. The new node copies the data from the other nodes.

A log line [B2987_SUPERSEDED_LAYOUT_UNPROVEN] names an index tree that the catalog does not prove old or new. The node keeps it and starts.

A node logs [LAYOUT_ANSWER_UNEXPECTED]

What you see

A warning line of this form:

[LAYOUT_ANSWER_UNEXPECTED] site=reconcile peer=10.0.0.2:4820 answer=error response: writer lock poisoned

site is reconcile (the layout repair) or rollup (the cardinality roll-up). The node logs the line once for each site and peer. It logs the line again when the answer text changes, or when the peer fails again after a good answer.

What it means

The node asked the peer for its table layouts, and the peer did not send them: it sent an error or another answer kind. The node skips the peer until the next tick. No data changes, because the repair adopts only a layout that a peer sends. While the peer answers this way, the node does not take a newer layout from it, and the cardinality counts do not include the shards of the peer. A peer that does not answer at all gives the line [PEER_SYNC_FAIL] instead.

The repair

Read the log of the peer at the time of the line. The answer error response: writer lock poisoned means that a thread of the peer stopped with a panic while it held the writer lock. Find the panic in the log of the peer, and then restart the peer.

The node does not start: a collection with no CREATE in default

What you see

The node prints one boot error and exits with code 3. The error starts with B2040: and names every collection of default that no CREATE registered, and every other database:

B2040: database `default` holds 2 collection(s) with no CREATE: ['events', 'kv_cache'], and 1 other database(s) exist: ['app']. These collections are read from `default` now. Before B2040, a write to an unregistered collection of another database stored its rows here. The node does not start; nothing was written. Fix: start the node once with KINEDB_OPEN_WITH_UNREGISTERED_DEFAULT_COLLECTIONS=1: the node starts, changes nothing in the store, and writes the file b2040_unregistered_collections_ack in the data directory, so the next start does not refuse. Then, for each collection, read the rows in `default`, write the rows of each other database into `<db>.<collection>`, and delete them from `default`. Exit code 3 (needs an operator).

What it means

An INSERT or a PUT into a collection that no CREATE registered makes the collection (see SQL). A version before B2040 stored such a collection of a database other than default in the collection of the same name in default. Its rows were visible from both databases. This version stores it in its own database, so the old rows are now visible from default only. A collection does not record the database of a row, so kinedb cannot move the rows. The check is broad: a collection that you made in default with no CREATE also gives this error when another database exists. The refusal changes nothing on disk.

The repair

1. Start the node once with KINEDB_OPEN_WITH_UNREGISTERED_DEFAULT_COLLECTIONS=1:

   KINEDB_OPEN_WITH_UNREGISTERED_DEFAULT_COLLECTIONS=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   The node starts and prints one [B2040_UNREGISTERED_COLLECTION] line per collection. It changes nothing in the store. It writes the file b2040_unregistered_collections_ack in the data directory, so the next start does not refuse. Then unset the variable. In a cluster, start each node once with the variable.

2. Read the rows of each named collection in default. Find the rows that belong to another database.

3. Write those rows into <db>.<collection> on this version, and delete them from default. Do not move them on a version before B2040: it writes them into default again.

A data directory that this version made gets the file at its first start, so it never refuses. An older data directory gets the file only after a start with the variable.

If the error says that the check could not read the table layouts, the state is unknown. The variable does not help: repair the cause of the read error, then start the node again.

The node does not start: a table has rows and no catalog entry

What you see

The node prints one boot error and exits with code 3. The error starts with B2264: and names each table that has rows and no catalog entry, with its database and its layout id:

B2264: 2 table(s) have rows and no catalog entry: ['events' (database `default`, layout id 11), 'kv_cache' (database `app`, branch `dev`, layout id 12)]. A binary before B2251 (with --calvin) or before B2763 (without --calvin, -e, -f) created these tables with no CREATE, so they have no catalog entry. The node does not start; nothing was written. Fix: start the node once with KINEDB_ADOPT_IMPLICIT_TABLES=1: the node starts and registers each table in the catalog with its existing id and rows (with --calvin, one adoption on the metadata log). Or, with an older binary, move the rows of each table into a collection made with CREATE COLLECTION, and drop the old table. Exit code 3 (needs an operator).

When the B2040 check also refuses at the same start, the two texts are in ONE boot error.

What it means

Before B2251 (a --calvin server) and before B2763 (a server without --calvin, and -e or -f), the first write to a name with no table made the table at the commit of its first row. That table had a layout and rows, but NO catalog entry and no CREATE. Now the first write runs CREATE COLLECTION IF NOT EXISTS first, in every mode (see SQL). A table of an older version is still there with no catalog entry, and the node cannot serve it correctly. Rows that only the storage WAL holds count too. On a --calvin server the check runs after the Raft replay, so a catalog entry in the Raft log counts. The refusal changes nothing on disk. Every mode runs the check: a --calvin server, a server without --calvin, and -e and -f.

The repair without --calvin

Run the server, or the -e statement, once with KINEDB_ADOPT_IMPLICIT_TABLES=1:

KINEDB_ADOPT_IMPLICIT_TABLES=1 kinedb --port 4820 --data-dir /var/lib/kinedb

Before it serves a client, the node writes the catalog entry of each table with the existing table id, and the rows stay. It prints one [B2763_LOCAL_ADOPT] line per table. Unset the variable: a later start needs no variable. A write to such a table also writes its entry in the same way.

The repair with --calvin

1. Start the node once with KINEDB_ADOPT_IMPLICIT_TABLES=1:

   KINEDB_ADOPT_IMPLICIT_TABLES=1 kinedb --port 4820 --data-dir /var/lib/kinedb --calvin --join 10.0.0.1:4820

   The node starts and prints one [B2264_IMPLICIT_TABLE] line per table. When the node is ready, it sends one adoption per table to the metadata leader. The adoption seals the old Raft group of each shard of the table. It moves the rows to a new shard id and starts the new group at a stamp floor above every stored row. Then it writes the catalog entry with the existing table id. Each replica applies the same adoption. The node prints [B2264_ADOPT] proposed ... per table and [B2264_ADOPT] adopted <n> table(s) at the end.

2. Unset the variable. A later start needs no variable: the catalog entry is the record. In a cluster, start each node that refuses once with the variable.

While a table waits for its adoption, a write to it can answer a retryable error that names B2264. Send the write again. If the adoption does not finish in 5 minutes, the node prints [B2264_ADOPT] adopted <n> table(s); <k> table(s) not adopted: .... The next write to such a table sends the adoption again.

If you do not want to adopt a table: with a version before B2251, read the rows of the table, write them into a collection made with CREATE COLLECTION, and drop the old table.

When the variable does not help

• A table id is held by another table. The error also says that a table has a layout id that a catalog entry of another table holds. The node refuses also with the variable. The adoption would give two tables one id. With a version before B2251, read the rows of the named table, write them into a collection made with CREATE COLLECTION, and drop the old table. Then start this version.
• A stored row has a stamp above the floor. The node stops during the adoption with a [B1082_FAIL_STOP] [B2264_ADOPT_STAMP_ABOVE_FLOOR] line that names the table, the stamp and the floor. A row on this node is newer than the floor that the proposer read from its own copy, so the replicas do not hold the same rows. A write of the new group would lose to that row, so nothing was written. Do not delete the data directory. Make a copy of it. Compare the named table on the other replicas. When they hold every row, remove this node with ALTER CLUSTER EVICT NODE '<id>' and add it again on an empty data directory with --join. It then copies the table from the other nodes.

Other fail-stops of the adoption ([B2264_ADOPT_ID_MISMATCH], [B2264_ADOPT_RANGE_MISMATCH], [B2264_ADOPT_UNSEALED_GROUP], [B2303_ADOPT_DECISION]) name their repair in the error text.

A node repeats [B2303_ADOPT_SOURCE_STALLED]

A node that was behind when the cluster adopted a table can print this line again and again, at most once each 10 s for one shard:

[B2303_ADOPT_SOURCE_STALLED] table=events index=40 shard=408408298 fresh=2948262435 applied=3 barrier=5: ...

The node did not apply the old Raft group of the shard up to the adoption point (the barrier), and the other voters already removed that old group. So the node cannot build the new shard from its own copy. It asks the voters of the old group whether they serve the new shard of THIS adoption. When one does, the node takes an empty copy of the new shard, and it then copies the rows from that voter before it serves a read. Until then, a read of the table on this node gets a retryable refusal. A timeout never ends the wait. The node does not apply the metadata entries after the adoption while it waits.

The repair is the same as for a stalled merge source (below): run one build on every node, and make sure that a voter of the old group is up and reachable. Do not delete data on the node.

If the error says that the check could not read the table layouts, the state is unknown. The variable does not help: repair the cause of the read error, then start the node again.

The node does not start: a retired-shard slice has an old digest name

What you see

You start a data directory that an older kinedb wrote (a version before B1429). The node prints one boot error and exits with code 3. The error starts with B1435: and names every slice file under <data-dir>/retired_shards/ that has the old name:

B1435: 1 retired-shard slice file(s) under /var/lib/kinedb/retired_shards carry the v1 digest name of a writer before B1429, not their BLAKE3 digest: [7_0_1024_3f2a…]. The ShardForceCatchup coord and the RetiredSliceGc in the metadata log name each slice so, and the name stays. The node does not start; nothing was written. Fix: start the node once with KINEDB_ACCEPT_V1_SLICE_DIGESTS=1; it renames each file to <shard>_<lo>_<hi>_v1_<hex> (the same digest), and a read of that file verifies the v1 digest only. Manual fix: stop the node and rename each file the same way. Exit code 3 (needs an operator).

What it means

A slice file of a retired shard carries its digest in its name. A version before B1429 named the file with an older digest format (v1). The metadata log names the slice with the same v1 digest, so the digest in the name must stay. The node refuses to start until you allow the rename. The refusal changes nothing on disk.

The repair

Start the node once with KINEDB_ACCEPT_V1_SLICE_DIGESTS=1:

KINEDB_ACCEPT_V1_SLICE_DIGESTS=1 kinedb --port 4820 --data-dir /var/lib/kinedb

The node prints one B1435: renamed the retired-shard slice … line per file and starts. The bytes of each file stay the same. Then unset the variable. The next start needs no variable.

To repair by hand, stop the node and rename each named file from <shard>_<lo>_<hi>_<hex> to <shard>_<lo>_<hi>_v1_<hex>. Then start the node normally.

The node does not start: a table was created WITHOUT SHARDING

What you see

You start a data directory that an older kinedb wrote, and that version created a table with CREATE TABLE … WITHOUT SHARDING or CREATE COLLECTION … WITHOUT SHARDING. kinedb removed that clause: every table is sharded now. The node prints one boot error and exits with code 3. The error starts with B1614: and names every such table:

B1614: 1 table(s) in this store were created by an older binary with the removed WITHOUT SHARDING clause: table "accounts" (id 7). Every table is sharded since B1614. Fix: stop every node of the cluster, then start every node with the new binary and KINEDB_CONVERT_UNSHARDED_TABLES=1. The node then marks each named table as sharded in one commit; no data moves. The manual fix without the variable: with the old binary, run `kd-tool backup`; then run `kd-tool restore` with the new binary into a new, empty data directory.

What it means

The table has the same shards and the same data as any other table. Only one flag on disk says that it must never split. The new version has no such mode, so the node refuses to start until you allow the change. The refusal changes nothing on disk.

The repair

1. Stop every node of the cluster. Do not run old and new nodes together.

2. Start every node once with KINEDB_CONVERT_UNSHARDED_TABLES=1:

   KINEDB_CONVERT_UNSHARDED_TABLES=1 kinedb --port 4820 --data-dir /var/lib/kinedb

3. Each node prints one [B1614_CONVERT_UNSHARDED] table "accounts" id 7 commit … line per table and starts. No row moves. The table can split from now on, like every other table.

4. Unset the variable. The next start needs no variable.

The node does not start: a branch file tree has an old key shape

What you see

You start a data directory that a kinedb version between B599 and B1898 wrote, and that version forked a branch table with an index on its first branch write. The file tree of that branch table has the key shape of that version, which this version cannot read. The node prints one boot error and exits with code 3. The error starts with B1898: and names every such branch table:

B1898: 1 branch table(s) in this store hold a file tree in the shape of a build between B599 and B1898, which this build cannot read: table "default\u{1f}dev\u{1f}docs" (id 7): shard 9 has file rows under its own id 9, not under its prefix 3. Fix: stop every node of the cluster, then start EVERY node once with KINEDB_REKEY_BRANCH_FILE_TREES=1. Each node then re-keys the file rows of each named branch table onto the key prefix of its shard, in one commit; no row is lost, and every node computes the same new roots from the same rows. Or, on the previous build, save the files of the branch and DROP BRANCH <b>.

What it means

A branch table points to the files of main at the base of the branch. Each shard of the branch table reads its file rows under the key prefix of the main shard it came from. The older version wrote the file rows of the branch under the id of the branch shard instead. The refusal changes nothing on disk.

That older version also lost the files of main on such a branch at its first write. The repair does not bring them back: it keeps the files that the branch itself stored.

The repair

1. Stop every node of the cluster. Do not run old and new nodes together.

2. Start every node once with KINEDB_REKEY_BRANCH_FILE_TREES=1:

   KINEDB_REKEY_BRANCH_FILE_TREES=1 kinedb --port 4820 --data-dir /var/lib/kinedb

3. Each node prints one [B1898_REKEY_BRANCH_FILE_TREE] table "…" id 7 … commit … line per table and starts. Only the key prefix of each file row changes. Every node computes the same new roots from the same rows, so EVERY node must start once with the variable.

4. Unset the variable. The next start needs no variable.

Without the variable: on the previous build, save the files of the branch and run DROP BRANCH <b>.

The node does not start: an index slice holds entries of a retired shard

What you see

You start a data directory that a kinedb version before B1921 wrote. That version could write the late index backfill of a shard that had split into the index slice of a child shard. Every split of such a slice then fails with the error SortedStreamBuilder ... the input must ascend strictly. The node prints one boot error and exits with code 3. The error starts with B3646: and names every such slice:

B3646: 1 index slice(s) in this store hold entries under the prefix of a base shard that split or retired: index "t_code" (tree "\u{1}idx_t_code") on table "t", slice 2821019795 of base shard 2709766506: entries under the retired prefix(es) 617035239. A build before B1921 wrote them, and every split of such a slice refuses. Fix: stop every node of the cluster, then start EVERY node once with the kinedb server binary and KINEDB_REPAIR_RETIRED_PREFIX_INDEX_ENTRIES=1. The server binary rebuilds each named slice from the rows of its base shard, in one commit; no row changes, and every node with the same rows computes the same roots. An embedded open cannot run this repair. Or, for each named index, run DROP INDEX <name> and then CREATE INDEX <name> again (SHOW INDEXES prints the CREATE line).

What it means

An index is derived data: each entry starts with the 4-byte prefix of the base shard that owns its row. The named slice also holds entries under the prefix of a shard that no longer exists. No row is damaged. The refusal changes nothing on disk. An embedded engine (kinedb::Db) refuses the same store; only the server binary runs the repair, because the repair needs the index definitions of the catalog.

The repair

1. Stop every node of the cluster. Do not run old and new nodes together.

2. Start every node once with KINEDB_REPAIR_RETIRED_PREFIX_INDEX_ENTRIES=1:

   KINEDB_REPAIR_RETIRED_PREFIX_INDEX_ENTRIES=1 kinedb --port 4820 --data-dir /var/lib/kinedb

3. Each node loads its catalog, then rebuilds each named slice from the rows of its base shard in one commit, before it starts Raft or accepts a client. It prints one [B3646_REPAIR_RETIRED_PREFIX_INDEX_SLICE] tree "…" slice … commit … line per slice and starts. Only index entries change. Every node computes the same new roots from the same rows, so EVERY node must start once with the variable.

4. Unset the variable. The next start needs no variable.

Without the variable: start the node on the previous build and, for each named index, run DROP INDEX <name> and then CREATE INDEX <name> again. SHOW INDEXES prints the CREATE line.

The node does not start: the storage WAL holds rows of an index with no layout

What you see

You start a data directory that an OLDER kinedb version wrote. Before B2263 a commit wrote a batch to the storage WAL and then refused it, because the batch wrote to an index that had no layout (B934). Before B1973 a crash could lose the layout of an index while the WAL kept the rows of the index. The node prints one boot error and exits with code 3. The error starts with B2269:. It names every such batch by its WAL seq and its indexes, and it gives the counts:

B2269: the storage WAL holds 1 batch(es) with rows of an index that has no layout (B934): seq 7 (indexes [idx_code], 1 index row(s), 1 other row(s)). In total: 1 index row(s) to drop, 1 other row(s) to keep. Cause: an OLDER binary wrote this WAL: before B2263 a commit wrote a batch to the WAL and then refused it, and before B1973 a crash could lose the layout of an index. The open refuses, and the WAL keeps every record. Fix: start the node once with KINEDB_OPEN_DROPPING_UNREGISTERED_INDEX_ROWS=1. That open drops the index rows of each such batch, keeps every other row, and logs [B2269_RECOVERY_REFUSED_BATCH]. Then start the node without the variable. When the next start names an index as UNUSABLE (B979), run its DROP INDEX and CREATE INDEX fix.

What it means

An index is derived data. The rows of an index with no layout cannot go into a tree, so the open cannot replay them. Every other row of the batch (the table rows, and the rows of the indexes that have a layout) can go into its tree. The WAL replay of the start applies the other batches, as every start does.

The repair

1. Start the node once with KINEDB_OPEN_DROPPING_UNREGISTERED_INDEX_ROWS=1:

   KINEDB_OPEN_DROPPING_UNREGISTERED_INDEX_ROWS=1 kinedb --port 4820 --data-dir /var/lib/kinedb

2. The node prints one [B2269_RECOVERY_REFUSED_BATCH] seq=… indexes=[…] dropped_index_rows=… kept_rows=… line per batch and one total line. It drops only the index rows and keeps every other row.

3. Unset the variable. The next start needs no variable.

4. When the catalog still lists the index, the next start names it as UNUSABLE (B979) and exits with code 3. Repair it as an index is UNUSABLE tells: DROP INDEX <name>, then CREATE INDEX again.

The node does not start: the data directory is already open

What you see

You start a node, or you open an embedded database, on a data directory that another opener holds. The open fails with one error, and the node exits with code 1:

B2150: the database directory /var/lib/kinedb is already open (the lock /var/lib/kinedb/LOCK is held by process 4242). Close the other opener first; two openers on one directory corrupt it.

What it means

One data directory has one opener. Two openers write two WALs and two HEADs into the same files, and that corrupts the directory. So every open takes an exclusive lock on the file LOCK in the directory, and a second open is refused, also in the same process. The refusal reads and writes nothing in the directory. A direct open of the block store of the Rust crate (DiskBlockStore::open, S3BlockStore::open) takes the same lock (B2166).

A native opener (the server, the Python package, the Rust crate) also reads the LOCK file that the JavaScript package writes on Node (B2183). A live Node holder refuses the native open with the same error, and the error names the Node process. A Node holder that does not run any more is taken over. When the native opener cannot check the Node holder (another host, another pid namespace, a LOCK that does not parse or a LOCK.takeover-<pid> file), it refuses. Make sure that no process opens the directory, then delete the file that the error names.

The repair

1. Find the holder: the error names its pid. Run ps -p <pid> to see it.
2. Stop that process, or close its database. Do not start two nodes on one --data-dir.
3. Start again. The operating system frees the lock when the holder exits, also after a crash or a kill -9. Do not delete the LOCK file: a file with no live holder blocks nothing.

The JavaScript package (@kinedb/embedded)

The JavaScript hosts refuse a second opener with the same B2150 error (B2160). This is true for a second Node process, a second instance in one process, a second Web Worker and a second browser tab.

• Node. Node has no kernel file lock, so the node:fs host writes its pid, its host name, the start time of the process and its pid namespace into LOCK. close() and a normal process exit remove the file. After a crash the file stays. The next open checks the pid: a process that is gone, or a pid that a new process uses now (another start time), lets the open take the lock over.

• Node, the refusals that need you. The host refuses, and does not take over, when it cannot check the holder. This happens for a LOCK of another host or of another pid namespace (a container on a shared volume), and for a LOCK that holds only a pid (a native kinedb process wrote it). The host also refuses a LOCK that does not parse and a LOCK.takeover-<pid> file that stays after a crash. Each of these errors ends with one sentence that starts with To repair:. It names the check and the file to remove (B2185):

  The LOCK	The check	The removal
  Holds only a pid	ps -p <pid> on every host that mounts the directory	rm <dir>/LOCK
  Names another host	ps -p <pid> on that host	rm <dir>/LOCK
  Names another pid namespace	ps -p <pid> in that pid namespace, for example in the container that wrote it	rm <dir>/LOCK
  Does not parse	Make sure that no process on any host opens the directory	rm <dir>/LOCK
  A LOCK.takeover-<pid> file exists	Make sure that no process on this host opens the directory	rm <dir>/LOCK.takeover-<pid>

  For example:

  B2150: the database directory /data/shop is already open (the lock /data/shop/LOCK is held by process 4242 (a LOCK with the pid only, which names no host). To repair: run `ps -p 4242` on every host that mounts /data/shop; when no host shows that process, run `rm /data/shop/LOCK`). Close the other opener first; two openers on one directory corrupt it.

• Browser (OPFS). The host takes a Web Lock (navigator.locks) for the database. OPFS has no LOCK file, so the LOCK path in the error is only the name of the lock. The browser frees the lock when the tab or the worker closes. The error names "another tab or worker of this origin". Close the other tab or worker, then open again. A browser with no Web Locks API refuses every open.

The node does not start: a data directory of an older version has no membership marker

What you see

A cluster node stops with one error, and the node exits with code 3:

B2931: this data directory was written by a version before B2931, and 2 data shard(s) have a Raft log on this node: [shard 0 of table "primary"; shard 2147494193 of table "t"]. … Fix: when the live voters of every shard are the metadata voters of this node (…), stop every node of the cluster, then start EVERY node once with KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP=1. …

What it means

A version before B2931 added the nodes of a growing cluster to each data shard OUTSIDE the Raft log of the shard, and it did not always write a record of the new voters. So the voter record of a shard on this node can be missing or older than the live voters. Since B2931 every node boots a shard on its own voter record, and every change of the voters is an entry of the shard log. A node that booted on a missing or old record could form a quorum that does not overlap the quorum of the other nodes. Two leaders could then commit different entries, and a committed entry could be lost. The node refuses, and it changes nothing on disk. A data directory that this version created has a marker file and never gets this check.

The repair

The error says which repair applies. An error with high-water <N> > RF <RF> comes from a cluster that once had more nodes than the replication factor: use the repair at N > RF below. Every other error uses these steps:

1. Before you stop the cluster, run SHOW RAFTGROUP <shard id> for each shard that the error names. The peers column lists the live voters of the shard.
2. Stop every node of the cluster. A node of an older version still changes the voters outside the log, so do not mix the versions.
3. Start every node once WITHOUT the variable. Each node refuses and prints its metadata voter set. Compare the sets: every node must print the same set, and it must be the live voters of step 1.
4. When the sets match, start EVERY node once with KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP=1. Each node writes the metadata voters as the voter record of each of its data shards, writes the marker file and boots. Each shard leader then commits one ConfigChange of its live voters.
5. Start the nodes without the variable from then on.

When the sets do not match, do not set the variable. Ask for help with the output of SHOW RAFTGROUP from every node. The node also refuses with the variable when it cannot read its metadata voters, when its metadata voters do not name it, or when KINEDB_DISABLE_AUTO_REFORM=1 is set: the error names the cause and the fix.

The repair at N > RF

At N > RF no voter record on a node proves the live voters of a shard: a version before B2931 changed the voters outside the log, and a record can be old. Use one of the two ways below. A node whose metadata voter record does not name it refuses also at N > RF (B2998): the cluster removed it, so remove its data directory and add it again as a new node.

A rolling upgrade

Upgrade one node at a time, and keep a quorum of each shard up on the older version. Start the upgraded node with KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP=1. The node writes no voter record. It boots each data shard PASSIVE: the shard does not vote and does not start an election. The node asks the live leader of each shard for one ConfigChange of the live voters. When that ConfigChange applies on the node, it writes the voter record and ends the passive boot. A shard whose live voters do not name the node stays passive.

The voters file

Use this way when the cluster is stopped. Before the stop, run SHOW RAFTGROUP <shard id> on a node of the older version for each shard that the error names. Write a file with one line for each shard:

# <shard id>=<voter id>,<voter id>,...
2147494193=72ec059a65db558c6fcf4a97a5173e07,0c1d5e7a9b2f4c6d8e0f1a2b3c4d5e6f

Start the node once with KINEDB_ALLOW_RFN_MIRROR_MEMBERSHIP=1 and KINEDB_RFN_MIRROR_VOTERS_FILE=<path of the file>. A set that names the node becomes its voter record, and the shard commits one ConfigChange of its live voters. A set that does not name the node writes no record, and the shard boots passive. The node refuses with exit code 3 and changes nothing when the file does not name every shard of the error, when a line does not parse, or when the file does not read: the error names each cause.

The node does not start: the HEAD file holds no txn id

What you see

The node, or an embedded open, stops with one error, and the node exits with code 3:

NeedsOperator: B2640: the HEAD file /var/lib/kinedb/HEAD names commit 5903a67f…9ae4 and holds no txn id, and the WAL holds no commit marker. An older build wrote this file. The open takes the next txn id from the HEAD txn id, so without it the open can give out a txn id that a commit already stored. Fix: stop every older build that uses this directory. Then set KINEDB_ALLOW_HEAD_WITHOUT_TXN_ID=1 and start again: the open takes the txn id from the timestamp of the HEAD commit and writes the HEAD file again with it. A manual fix: write the HEAD file as two lines, the hash and then a txn id above every txn id that this directory gave out.

What it means

The HEAD file names the newest commit. Since B2640 it also holds the highest txn id that a commit stored. The open gives out the next txn ids above that value. A build before B2640 wrote the commit hash only. When the WAL holds no commit marker either (the last open of the older build made no commit), the open does not know the highest txn id. A start at txn id 1 could give out an id again that a stored record already holds, for example the created_txn_id of a branch. So the open refuses, and it changes nothing on disk. A node with S3 does the same check on the local HEAD file when it starts again on its data directory. The repair below is the same.

The repair

1. Stop every process of an older build that opens this data directory.

2. Start once with KINEDB_ALLOW_HEAD_WITHOUT_TXN_ID=1:

   KINEDB_ALLOW_HEAD_WITHOUT_TXN_ID=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   The open takes the txn id from the timestamp of the HEAD commit, logs one warning, and writes the HEAD file again with the txn id.

3. Unset the variable. The next start reads the txn id from the HEAD file.

The timestamp of the HEAD commit is the best value that the store holds. An id above it that an older build gave out to a commit that is not the HEAD commit can come back once. After this start, the txn id never moves down again.

The node does not start: a storage WAL file is damaged in the middle

What you see

The node, or an embedded open, stops with one error, and the node exits with code 3:

NeedsOperator: B2952: the WAL file /var/lib/kinedb/wal.d/00000000000000000007.wal does not decode at byte offset 1048576, and 3 valid commit marker(s) follow it (the first at byte offset 1048690). A torn tail never holds a whole commit marker after its first bad byte, so this is damage in the middle of the WAL: a replay that stops at byte offset 1048576 loses every commit after it. The open changed nothing on the disk. Fix: stop every process on this data directory. Then restore the WAL file from a backup, or accept the loss: set KINEDB_OPEN_CUTTING_DAMAGED_WAL=1 and start again. With it the open replays the WAL up to byte offset 1048576, drops every commit after it, and logs one warning for each damaged file.

The file is wal/wal.log (the HEAD pointer of the block store), a segment of wal.d/, or the legacy wal.log at the top of the data directory.

What it means

The open replays each storage WAL file up to the first entry that does not decode: a CRC error, a damaged length field, or an incomplete entry. A crash during the LAST append leaves such an entry at the end of the file (a torn tail), and the open discards it. A torn tail is a prefix of the last append, so it never holds a whole commit marker after its first bad byte. Here the open found whole, valid commit markers AFTER the bad entry. Those are commits that a replay which stops at the bad entry loses. A version before B2949 could write such a file (it appended the next commit after a torn append); a damaged disk can too. So the open refuses, and it changes nothing on disk.

The repair

1. Stop every process that opens this data directory.

2. Restore the named WAL file from a backup of THIS data directory, then start again.

3. With no backup, accept the loss of the commits after the bad offset. Start once with KINEDB_OPEN_CUTTING_DAMAGED_WAL=1:

   KINEDB_OPEN_CUTTING_DAMAGED_WAL=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   The open replays each damaged file up to the bad offset, drops every commit after it (the behaviour before B2952), and logs one line per damaged file:

   [kdtrace ts=… storage warn] [B2952_DAMAGED_WAL] the WAL file /var/lib/kinedb/wal.d/00000000000000000007.wal does not decode at byte offset 1048576, and 3 valid commit marker(s) follow it (the first at byte offset 1048690) — KINEDB_OPEN_CUTTING_DAMAGED_WAL=1: the open replays the WAL up to byte offset 1048576 and drops every commit after it

4. Unset the variable.

The same variable: a commit marker names a block that the store does not hold (B2975)

The open can also stop with this error, exit code 3:

NeedsOperator: B2975: the last commit marker in the WAL file /var/lib/kinedb/wal/wal.log names commit 9f3c…e1, and no segment of the store holds its block. The HEAD file names commit 41ab…07, and the store holds its block. A commit marker comes after the fsync of its blocks (B1076), so the segment files lost data: a lost write of the disk, or a segment file that a restore or a copy left out. The open changed no byte of HEAD and of the WAL file. Fix: stop every process on this data directory. Restore the segment files from a backup of THIS data directory, then start again. On a cluster node with no backup, replace the node (B1544): its replicas still hold the shards. To accept the loss, set KINEDB_OPEN_CUTTING_DAMAGED_WAL=1 and start again: the open starts with the last commit marker whose block the store holds (or the HEAD file), DROPS every later commit, and logs one warning for each dropped marker.

wal/wal.log holds the HEAD pointer of the block store: one commit marker per commit. The node writes the blocks of a commit and fsyncs them before it writes the marker. So a marker whose block no segment holds means that the segment files lost data. The open used to write that hash to HEAD and empty wal.log: the good HEAD file and the WAL were gone, and the next open failed with NotFound: HEAD. Now the open refuses before it changes a byte.

The repair:

1. Stop every process that opens this data directory.

2. Restore the segment files (segments/) from a backup of THIS data directory, then start again. A backup of another node does not fit.

3. On a cluster node with no backup, replace the node (B1544). Its replicas still hold the shards.

4. With no backup and no replica, accept the loss of the later commits. Start once with KINEDB_OPEN_CUTTING_DAMAGED_WAL=1. The open starts with the last commit marker whose block the store holds, or with the HEAD file when no marker block is held. It logs one line per dropped marker:

   [kdtrace ts=… storage warn] [B2975_WAL_MARKER_NOT_HELD] the WAL file /var/lib/kinedb/wal/wal.log holds a commit marker of commit 9f3c…e1, and no segment holds its block — KINEDB_OPEN_CUTTING_DAMAGED_WAL=1: the open drops this commit and starts with commit 41ab…07

   When neither a marker nor the HEAD file names a block that the store holds, the open refuses with the variable set too. Then only a restore helps.

5. Unset the variable.

The node does not start: the history floor file does not parse

What you see

The node, or an embedded open, stops with one error, and the node exits with code 3:

NeedsOperator: B2683: the history floor file /var/lib/kinedb/HISTORY_FLOOR does not parse. The file records where the true history of each table starts on this node, so the open refuses. Fix: stop the node. Restore the file from a backup of this data dir, or delete the file and start again: the open then writes a floor of every table at the parent of HEAD, so every history read (AS OF, SHOW DIFF) at a commit older than HEAD refuses on this node.

The text can also say does not open or does not read. An empty (0-byte) file is not this case: the open reads it as no file.

What it means

A node that receives the data of a table from another node (a join, a rebalance, a repair) installs it in one commit. Before that commit, the history of this node holds no true rows of the table. Since B2683 each install records the commit before it in the file HISTORY_FLOOR of the data directory, and a history read older than that commit refuses. A file that does not parse is never read as "no floor", because a read with no floor can answer wrong rows. So the open refuses. The refusal changes nothing on disk.

The repair

1. Stop the node.
2. Restore HISTORY_FLOOR from a backup of this data directory, or delete the file.
3. Start the node. When the file is not there, the open writes a floor of every table at the commit before HEAD, logs one warning line, and starts.

After a start with a deleted file, a history read on this node at a commit older than HEAD refuses with the error of the next section. Read that history on a node that held the table then.

A data directory of a build before B2683 has no HISTORY_FLOOR file. Its first start on this build does the same as step 3 with no refusal. A restore from S3 onto a new data directory does the same, because the S3 sync does not upload the file.

The node does not start: the data directory has no history gap file

What you see

The server stops with one boot error and exits with code 3:

Failed to check the history gap record: NeedsOperator: B3817: the data directory holds commits and no history gap file HISTORY_GAP (or a 0-byte one): a build before B3817 wrote it, or the record is lost. That build did not record the periods where this node did not host a shard of a table, so a history read (AS OF, SHOW DIFF, the row history) of any table at a commit older than HEAD 3f1c… can answer stale rows on this node. Fix: start the server once with KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1. The start sets a floor of every table at the parent of HEAD and writes an empty HISTORY_GAP; then every history read older than HEAD refuses on this node. No row and no commit changes. No other fix exists: the periods are not on disk. This refusal changed nothing on disk.

What it means

Since B3817, a node records each period where it did not hold the data of a shard (a removal from the voters of a shard, a split or a merge that left it with no copy, a repair) in the file HISTORY_GAP of the data directory. A build before B3817 did not record these periods. The commits of such a period can hold stale rows of the table, and a history read at one of them can answer those rows with no error. The data directory of an older build has commits and no HISTORY_GAP file, so the server does not start. The refusal changes nothing on disk. An embedded open (-e, -f, a library) does not do this check: it has no Raft group, so it never stopped holding a shard.

The repair

1. Start the server once with KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1:

   KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1 kinedb --port 4820 --data-dir /var/lib/kinedb

   The start sets a floor of every table at the commit before HEAD, writes an empty HISTORY_GAP, logs one warning line that starts with B3817:, and serves. On this node, every history read (AS OF, SHOW DIFF, CREATE BRANCH … AS OF) at a commit older than HEAD then refuses. Rows and commits do not change.

2. Remove the variable before the next start. A later start does not need it.

Do this on each node of the cluster when it starts on this build for the first time. No other fix exists: the periods are not on disk.

The node does not start: the node key cannot be loaded

What you see

The server stops with one boot error and exits with code 3:

the node key cannot be loaded: /var/lib/kinedb/node.key cannot be unsealed: … Restore /var/lib/kinedb/node.key and the lock key of this node (lock.key or lock.tpm) together from one backup; a new key is never made over it

What it means

The file node.key of the data directory holds the private half of the key pair of this node, sealed by its lock key (The key of a node). The seal does not open: a byte of the file changed, or the lock key is not the one that sealed it (another lock.key, another TPM, or a data directory that moved to another machine with a TPM lock). The node does not make a new key over the file: a new key would lose the identity of the node. The refusal changes nothing on disk.

The repair

1. Restore node.key and lock.key (or lock.tpm) of this node together from one backup, on the machine that sealed them.
2. Start the server again. The log line B3963: the node key is ready shows the old fingerprint.

The node does not start: the cluster key cannot be loaded

What you see

The server stops with one boot error and exits with code 3:

the cluster key cannot be loaded: … Restore /var/lib/kinedb/cluster.key and the lock key of this node (lock.key or lock.tpm) together from one backup; a new key is never made over it

What it means

The file cluster.key of the data directory holds this node's copy of the cluster key pair, sealed by its lock key (The cluster key). The copy does not open: a byte of the file changed, or the lock key is not the one that sealed it. The node does not make a new pair over the file: a new pair would differ from the pair of the other nodes. The refusal changes nothing on disk.

The repair

1. Restore cluster.key and lock.key (or lock.tpm) of this node together from one backup, on the machine that sealed them.
2. Start the server again. The log line B3964: the cluster key is ready shows the old version and fingerprint.

The joining node does not start: no KINEDB_JOIN_SECRET

What you see

A new server (no multi-node history) that starts with --join stops with one boot error and exits with code 3:

this node has no cluster key and no KINEDB_JOIN_SECRET: it cannot get the key from a running node. Set KINEDB_JOIN_SECRET to the join secret of the cluster

What it means

The data directory has no cluster.key, so the node must get the cluster key from a running node before it joins (The cluster key). The joining node and the running node prove the join secret KINEDB_JOIN_SECRET to each other. With no join secret the node can never get the key, so it refuses at once. The refusal changes nothing on disk.

The repair

1. Set KINEDB_JOIN_SECRET on this node to the join secret of the cluster: the value that the running nodes have.
2. Start the server again. The log line B3964: the cluster key is ready: … (from the join) shows that the node got the key.

The joining node exits: no running node gave the cluster key

What you see

A new server that starts with --join and has no cluster.key logs warnings join attempt <n> via <host:port> failed: no cluster key: …, and after KINEDB_JOINER_BOOT_TIMEOUT_MS (10 s by default) it exits with:

FATAL: no running node gave the cluster key: check KINEDB_JOIN_SECRET on this node and on the running nodes, the --join address, and that the running nodes are up.

What it means

The node asks each node once in each attempt: the --join node, then each node of its peer book. It joins only after it has the key. Each warning names the answer of each node:

• 403: the running node has another KINEDB_JOIN_SECRET.
• 503 … no join secret: the running node has no KINEDB_JOIN_SECRET.
• 503 … no cluster key: the running node has no cluster key.
• a connect error: the node is down or the address is wrong.

The repair

1. Give every node the same KINEDB_JOIN_SECRET, and restart the nodes that had another value or none.
2. Check the --join address, and that the node at that address runs.
3. Start the joining node again.

The node does not start: the history gap file does not parse

What you see

The node, or an embedded open, stops with one error, and the node exits with code 3:

NeedsOperator: B3817: the history gap file /var/lib/kinedb/HISTORY_GAP does not parse. The file records the tables whose shards this node stopped hosting, so the open refuses. Fix: stop the node. Restore the file from a backup of this data dir, or delete the file and start the server again with KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1: the start then writes a floor of every table at the parent of HEAD, so every history read (AS OF, SHOW DIFF) at a commit older than HEAD refuses on this node.

The text can also say does not open or does not read. An empty (0-byte) file is not this case: the open reads it as no file.

What it means

A node that stops holding the data of a table (a removal from the voters of a shard, a split or a merge that leaves it with no copy, a repair) records the commit of the stop in the file HISTORY_GAP of the data directory. A history read at or after that commit refuses until the node holds the data again, or until a DROP TABLE or a TRUNCATE removes the table. After that, a history read inside the old period still refuses, and the commits after the DROP or the TRUNCATE read. A file that does not parse is never read as "no gap", because a read with no gap can answer stale rows. So the open refuses. The refusal changes nothing on disk.

The repair

1. Stop the node.
2. Restore HISTORY_GAP from a backup of this data directory, or delete the file.
3. Start the node. When you deleted the file, start the server once with KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1: the start writes a floor of every table at the commit before HEAD, logs one warning line, and starts. Without the variable, the start refuses with the error of the previous section.

An older build does not read the file. After a downgrade to an older build and an upgrade again, delete HISTORY_GAP and start once with KINEDB_SET_HISTORY_FLOOR_AT_HEAD=1.

A history read refuses: this node stopped holding the data of the table

What you see

SELECT … AS OF <ref>, SHOW DIFF OF <table> FROM <ref> TO <ref> or CREATE BRANCH … AS OF <ref> fails with an error of the kind NotFound:

the AS OF target is at or after the commit where this node stopped holding the data of table t, so the history of this node holds no true rows of t at that target; read on a node that held the data then

What it means

The cluster removed this node from the voters of a shard of the table, or a split, a merge or a repair left the node with no copy of a shard. From that commit on, the commit chain of this node holds stale rows of the table. The read refuses, so that it never answers stale rows. When the node holds the data again, a read before that commit refuses with the error of the next section.

The repair

Run the read on a node that holds the table, or that held it at the ref.

A history read refuses: this node received the data of the table later

What you see

SELECT … AS OF <ref>, SHOW DIFF OF <table> FROM <ref> TO <ref> or CREATE BRANCH … AS OF <ref> fails with an error of the kind NotFound:

the AS OF target is older than the commit where this node received the data of table t, so the history of this node holds no true rows of t at that target; use a later target, or read on a node that held t then

What it means

A history read resolves its ref in the commit chain of the node that runs it. This node received the data of the table in a later commit (a join, a rebalance, a repair), or the ref is older than the first start of this build on a data directory of an older build (or of a restore from S3). At the ref, the chain of this node holds no true rows of the table. The read refuses, so that it never answers 0 rows or wrong rows.

The repair

Use a ref at or after the commit where the node received the data, or run the read on a node that held the table at the ref.

An embedded sync client does not open: a window holds the unnamed subscription

What you see

A program that links the kinedb-sync crate opens a SyncClient on its local database. The open fails with an error of the kind NeedsOperator. The error starts with B2204: and names each table:

B2204: the local database holds a window in the unnamed subscription of `orders`. A development build between B2202 and B2204 wrote that row; no release wrote it. Each window now has a row of its own. To repair it by hand: push the pending writes, then delete the local database and sync again. To move the row at this open: set KINEDB_SYNC_MOVE_WINDOW_ROW=1

What it means

A window (sync_window, B2202) keeps a range of a table on the device. A build of kinedb-sync from source between B2202 and B2204 stored the window in the unnamed subscription row of its table. Since B2204 each window has its own row, so a window, a sync() and forwarded queries can share one table. No published package or image has a window driver, so only a program built from that source can hold such a row. The refusal changes nothing on disk.

The repair

Open the client once with KINEDB_SYNC_MOVE_WINDOW_ROW=1 in the environment of the program:

KINEDB_SYNC_MOVE_WINDOW_ROW=1 ./my-app

The open moves each such row to the row of its window in one local commit, with the same text, token and state, and prints one [B2204_WINDOW_ROW_MOVE] line. Then unset the variable. The next open needs no variable.

To repair by hand: push the pending writes (push), then delete the local database and sync the tables again.

An embedded database does not open: the directory holds .pack-incomplete

What you see

A program opens a local database with the Rust crate, the Python package or the wasm engine. The open fails with an error of the kind NeedsOperator. The Python package and the wasm engine get the text only, with the kind as its first word:

NeedsOperator: the directory /var/lib/myapp/cache holds the marker .pack-incomplete: a pack that did not end. The open refuses it. Close every other opener of the directory, delete the directory, then open again and sync.

What it means

A pack is the fast first fill of an empty device. It writes the marker .pack-incomplete first, and removes it only after the first open of the complete database. A directory that still holds the marker is a pack that did not end: a broken stream, a closed program or a power cut. Its files can be a half database, so the open refuses it (B3451). The refusal reads the marker only, and changes nothing on disk. The JavaScript package deletes such a directory for you, and then its open fails once with a .pack-incomplete error.

The repair

1. Stop every program that opens the directory.
2. Delete the directory that the error names. It holds no data of its own: a pack never writes into a database that holds local data.
3. Open again, and sync. The sync fills the tables from the server.

An embedded sync logs: the server refused WINDOW

What you see

A sync() of @kinedb/embedded (JavaScript or Python) completes, but the fill is slow. The log has one warning for the first table of the session.

In Python, stderr has this warn line:

[kdtrace ts=… sync warn] [SYNC] table="orders": the server refused WINDOW (expected end of input, got WINDOW): the fill falls back to window 1; upgrade the server for a faster fill (B3033)

In JavaScript, console.warn has this line:

kinedb embedded: table "orders": the server refused WINDOW: the fill falls back to window 1; upgrade the server for a faster fill (B3033)

What it means

The embedded sync asks the server for a window of 4 fill pages (B3033): WATCH … SINCE … WINDOW 4. A server from before B3026 does not know the clause WINDOW and refuses the statement with a parse error. The parse error registers no watch. The session then sends the same WATCH with no clause, and the fill continues at window 1: one page for each round trip. The fallback holds for all the tables of the session, so the line comes one time. No row is lost and no row is applied two times.

At window 1, the fill time grows with the round trip time. In the B3033 measure, a fill of 1,954 pages took 144 s at window 1 and 36 s at window 4, with an RTT of 60 ms.

The repair

Update the server to a build that has B3026. The next session asks for the window again, and the line does not come back. You do not have to change the client or the local database.

A different error that names WINDOW (for example "WINDOW needs SINCE") comes from a new server. The sync does not fall back for it: it fails the sync(). Report it as a defect.

An embedded query rejects: "the client budget refuses a new subscription"

What you see

db.query(sql) of @kinedb/embedded, or SyncClient::query of the Rust client, fails with a text of this kind:

the client budget refuses a new subscription of `orders`: the table would hold 65 predicates, and the budget is 64 for each table. No subscription changed. Run the query with sql() for the local rows, or raise the budget

What it means

The query ran a second time within 300 seconds, so the client tried to make it a live subscription (B2205). The new subscription would pass one limit of the client budget: the predicates or the local rows, for each table or for the client. The client refused before it sent anything. No subscription changed, and the client did not make a wider subscription in its place.

The repair

• Run the query with sql(): it reads the local rows and needs no subscription.
• Or raise the budget: await db.setQueryBudget({ perTablePredicates, predicates, perTableRows, rows }) in JavaScript, SyncClient::set_budget(Budget { … }) in Rust. See How the client keeps its subscriptions small.
• Or use fewer, wider queries of your own: a held subscription that contains a query answers it with no new subscription.

The log says an index cannot serve a TEXT range

What you see

One WARN line per index, the first time a query asks that index for a range over a text value:

[B913] index idx_name on people has the pre-B913 LenPrefixed layout: its tree sorts TEXT by length, so it cannot serve a TEXT range and the range runs as a residual over the equality prefix or the full scan (the answer stays correct, the read is slower). Migrate it: DROP INDEX idx_name; CREATE INDEX idx_name ON people (name)

What it means

An older build created the index. Its entries are in an order that does not match the order of text values. The index still serves =, IN and number ranges. For a range over text (<, <=, >, >=, BETWEEN, LIKE 'prefix%'), the node reads the rows instead of the index. The answer is the same; the query is slower on a large table.

The repair

Run the two statements from the log line on a live node:

DROP INDEX idx_name;
CREATE INDEX idx_name ON people (name);

CREATE INDEX builds the index again in the layout of this build, and that layout serves text ranges. For a unique index, the log line says CREATE UNIQUE INDEX.

A login answers AUTH_THROTTLED

What you see

A login, or ALTER USER … REPLACE '<old>', fails with:

AUTH_THROTTLED: too many failed password checks; retry in <N> s

POST /login answers 429 Too Many Requests with the header Retry-After: <N>.

What it means

The node counted too many failed password checks: from the source address, for the user from that address, or for the user from all addresses. The node did not check the password. The answer does not tell if the password is correct (the limit of failed password checks).

What to do

• Wait <N> seconds, then log in once with the correct password.
• Look at SHOW AUDIT for refusals with the reason bad_credentials and throttled. They show which source address sends the wrong passwords.
• When all clients come through a reverse proxy and SHOW AUDIT shows only the address of the proxy, start the node with --trusted-proxy (TLS for the clients).
• A client that has a session token logs in with it. A token login checks no password, and the limit does not apply to it.

A logout answers AUTH_THROTTLED

POST /logout answers 429 Too Many Requests, with Retry-After: <N> and:

AUTH_THROTTLED: too many logouts of this user; retry in <N> s

The node counted more than 20 logouts of this user, plus 1 each 6 seconds. The logout did not happen, and the token stays valid. Wait <N> seconds and send the logout again. A client that logs in and out in a loop must keep one token and use it again (tokens).

POST /token/refresh uses the same limit and gives the same 429. The refresh did not happen, and the old token stays valid (refresh a token).

A login fails right after the first start

What you see

The node listens, but a login as root fails for some seconds after the first start of a new data directory. The answer is:

the user registry is not ready: the root user is not seeded yet; retry shortly

GET /health answers 503 Service Unavailable in the same window. Its status starts with starting: the root user is not seeded yet.

What it means

The first start creates the user root. On a node with a metadata Raft group (--calvin, or a cluster), the node opens its port first, because the peers must reach it to elect a leader. The root user commits through that leader some seconds later. The log line seeded superuser 'root' marks the end of the window.

The password is not wrong. The node gives this answer for every username until root exists, so the answer does not tell which users exist.

What to do

Wait until GET /health answers 200, then log in. A client can also retry the login: the server marks this answer as retryable (the retryable field of the error). After the window, a wrong password answers invalid username or password as before.

The Rust client gives this answer as an error of kind ErrorKind::Unavailable, and is_retryable() is true. A wrong password stays ErrorKind::Auth. The kinedb remote connection keeps its socket open after the refusal, so the caller can send the same login again on it.

A request answers "the node is not ready" right after a start

What you see

After a start or a restart, the node listens, but each request (a query, a login, a POST /sql) answers:

the node is not ready: starting: replaying the Raft log (...); retry

GET /health answers 503 Service Unavailable in the same window. Its status starts with starting: replaying the Raft log or starting: boot checks.

What it means

A node with Raft groups (--calvin, or a cluster) keeps its log on disk. At a start, each group applies that log again before the node serves a client, so a read never sees a state older than a committed write. The node also runs its boot checks after that replay. The log line [B2157_READY] marks the end of the window. The refusal has the kind not_ready and is retryable: nothing ran.

A node of a cluster that cannot reach a quorum stays in this window: it does not know which entries of its log are committed.

One data shard without a quorum does not keep the whole node in this window. When the other voters of that shard do not answer, the node becomes ready and serves its other shards. The log line [B2458_NO_QUORUM_LEFT_OUT] shard=<id> names each such shard. A read or a write of that shard gets a retryable error until its voters come back and it elects a leader. The metadata group has no such exception: without its quorum, the node stays in this window.

What to do

Wait until GET /health answers 200. The Rust client (query, authenticate) waits on this refusal by itself, for up to 20 s. The JavaScript and Python clients and the two embedded packages also wait on it by default, for up to 20 s (B2298). The kinedb CLI, kinedb-tool (restore, backup, import) and the Rust sync client send a statement again on this refusal and on a split write hold, for up to 20 s (B2299). retry: false (retry=False in Python) turns that wait off. An old client that does not know the kind still sees the retry flag.

Do not point a liveness probe at /health. It is a readiness signal: a node of a cluster without a quorum stays at 503 until the quorum comes back, and a liveness probe would restart it again and again.

A WebSocket connection is refused with 503

What you see

A client cannot open a WebSocket connection to a node that runs with --memory. The server answers the upgrade request with:

HTTP/1.1 503 Service Unavailable
Retry-After: 1

client connection limit reached; retry later

The Rust client gives an error of kind ErrorKind::Unavailable, and is_retryable() is true. The Python client raises a WebSocketError with retryable true. A browser shows only a failed connection: the JavaScript client rejects connect() with a KineDBError with retryable true.

What it means

The node is at its client connection limit. A node with --memory takes this limit from its budget, so the threads and buffers of the connections stay inside the budget (see The connection and peer buffers of a memory budget). The connections between the nodes of the cluster do not count, so the cluster itself keeps working. A node without --memory has no limit.

What to do

Retry the connection after the Retry-After delay, or connect to another node. To see the state, run SHOW METRICS on an open connection: client_admission.connections is the count, client_admission.connection_limit is the limit, and client_admission.refused_total counts the refused connections. Close the connections that your application does not use, for example an idle pool that is larger than it needs to be. If the node needs more connections, give it a larger --memory budget.

A query fails with "connection closed" while the client reads slowly

What you see

A large reply, a QueryStream or a WATCH fails in the middle. The JavaScript client rejects the call with a KineDBError with the message connection closed (WebSocket close code 1006). The Python client raises kinedb.client.errors.KineDBError with the message connection closed (close code 1006, empty reason). The server sends no close frame. The server log has one warn line with the tag [B1981_FRAME_DEADLINE]. This line comes from a test with a short budget (stall 300 ms, 16 MiB/s):

[B1981_FRAME_DEADLINE] client 127.0.0.1:54690: 4227072 of 8388828 bytes of a frame written when the frame deadline of 800.013113ms passed (stall 300ms + the frame at 16777216 bytes/s); the socket is shut inside the frame, with no close frame

The line names the client address (behind a proxy, the address of the proxy), the bytes written of the frame total and the frame deadline. The time used is equal to the deadline. An HTTP response names HTTP client instead of client.

What it means

The client did not read a reply frame in time, so the node ended the connection (B1981). Each frame must arrive in 60 s + frame length / 64 KiB/s. A client that stops reading loses its connection after one or two frame deadlines: in a measurement with 1 MiB frames, after 77 s and after 137 s. Common causes: a slow link, or an application that blocks the thread of the client while a reply arrives.

The client cannot tell this close from a network fault: both give the same error. The server cannot send a reason, because a close frame cannot go inside a partly written frame (RFC 6455 § 5.4). Only the server side tells the two causes apart.

What to do

Find the [B1981_FRAME_DEADLINE] line with the address of the client in the server log. Run SHOW METRICS on the node: client_outbound.write_timeouts_total counts the connections that the frame deadline ended. When no line names the address of the client, the cause is not the frame deadline: look for a network fault. Read the replies while the connection is open, and do not block the thread of the client. On a slow link, ask for smaller replies with a LIMIT or a QueryStream. Reconnect after the error, and resume a watch with WATCH … SINCE.

A write fails on a node that is joining the cluster

What you see

A SQL write to a node that is still joining the cluster fails, and the answer is:

passive joiner: no leader known yet and no --join target configured; retry shortly

The error has retryable set in both client libraries.

What it means

A joining node holds no data of its own, so it passes a SQL write on to the --join address it started with. This node has no --join address, so it cannot pass the write on. Nothing was written.

What to do

Send the write again. The answer has the retry flag set, and a KineDB client reads that flag, never the text (B501). A client with the option retry: true does this for you; the default client sends again only a hold refusal (B2298). The node stops giving this answer when the cluster adds it as a member. If the answer does not stop, send the write to a node that has joined the cluster.

A second form of the same answer is:

passive joiner: no leader known yet; a forwarded write is not forwarded again; retry shortly

Another node passed the write to this joining node. The joining node does not pass it on again, because its --join address can send the write back to it in a loop. The node that passed the write tries the next node first, so a client sees this answer only when no node can take the write yet. Nothing was written. Do the same as above.

The log says that a join attempt failed

What you see

A node that starts with --join writes lines such as:

join attempt 1 via 10.0.0.5:7400 failed: no SWIM peer answered the join ping within 1s; seeds tried: [10.0.0.5:7400]; next attempt in 250ms

What it means

The node sent its join message to the --join address, but no member of the cluster answered within 1 s. A message can be lost when the seed is busy, refuses the connection or restarts. The node sends the join message again: the wait starts at 250 ms and doubles, up to 30 s. One line shows each failed attempt.

What to do

Nothing, when a later attempt succeeds: the node then writes Joined cluster via <address>. A new node that does not join within 10 s stops with FATAL: --join target unreachable. Then make sure that the --join address is correct, that the seed runs, and that this node can open a TCP connection to the seed. Start the node again.

A read fails on a node that does not hold the data yet

What you see

A SQL read to a node fails for a short time, often right after the cluster starts or a node joins, and the answer is:

read shard(s) [2109745565] are not served by this node and it cannot forward the read (no-other-voter) — retry shortly

The word in the parentheses is no-other-voter, no-peer-info or no-node-id.

What it means

The node does not hold the data that the read asks for. It also does not know yet which node holds it, so it cannot pass the read on. The node does not answer from its own empty copy, because that answer can be empty or incomplete with no error.

What to do

Send the read again. The answer has the retry flag set, and a KineDB client reads that flag, never the text (B501). A client with the option retry: true does this for you; the default client sends again only a hold refusal (B2298). The answer stops when the node learns the members of the cluster. If it does not stop, send the read to another node.

An UPDATE, DELETE or UPSERT fails while a shard splits

What you see

An UPDATE or a DELETE with a WHERE that is not the primary-key point form, or an UPSERT, fails for a short time while a table splits, and the answer is:

SplittingRetry: this node is a born voter of the split child shard(s) [3611364880, 3611364881], and their Raft drivers did not register (the birth ended without them, or the request deadline passed); retry shortly

What it means

Such a statement finds its rows in the node's own copy, so the node must hold every shard that the statement reads. A split makes its new shards visible first and starts them on the node a short time later. The node waits until the start ends. The node gives this answer only when the start failed, or when the deadline of the request passed first. Nothing was written.

What to do

Send the statement again. The answer has the retry flag set, and a KineDB client reads that flag, never the text (B501). A client with the option retry: true does this for you; the default client sends again only a hold refusal (B2298). The node holds the new shards a moment later.

A write fails while a shard splits, merges or moves its leader

What you see

A write fails for a short time while a table splits, and the answer is:

calvin propose failed: shard is splitting (SplittingRetry budget exhausted); retry shortly

A merge gives shard is merging, and a leadership transfer gives shard is transferring leadership.

What it means

A split, a merge or a leadership transfer holds the writes of the shard for a short time: the write hold. The local gate measured holds of up to about 5 s (B495). Nothing was written. The answer carries the retry flag and a typed refusal, refusal: {kind: "write_hold", retry_ms}, where retry_ms is the wait that the server asks for.

What to do

Send the write again after retry_ms. The JavaScript and Python clients and the two embedded packages do this for you by default, for up to 20 s from the first send (holdBudgetMs or hold_budget_ms, B2298). The refusal kinds write_hold and not_ready get this retry. Every other retryable error still throws at once with err.retryable set, unless you set retry: true. retry: false (retry=False in Python) turns off the hold retry too. A Rust client with a RetryPolicy does this for up to RetryPolicy::hold_budget, also 20 s. Remote::execute of the Rust crate does not retry a hold refusal yet (B2299). A hold refusal does not count against the attempt count of the client. When a write still fails after 20 s, the hold did not clear. Look at the split, merge or transfer of that shard in the node log.

A new leader can replace a split entry before it commits. Then the split does not occur, and the node releases the hold when it applies the entry of the new leader. The node log shows [B2885_SPLIT_HOLD_REPLACED] and the [WRITE_HOLD] released line (B2885). A snapshot install that moves the node past the replaced entry releases the hold in the same way, with the same two lines (B2895).

A merge holds a shard longer than a split. MERGE SHARDS first seals each of the two shards: a seal entry in the Raft log of the shard. From the seal until the merge ends, the shard refuses every write with shard is merging and serves no read. The merge ends in one of two ways: the new shard takes over, and a retry lands on it; or the merge aborts, the seal goes, and a retry lands on the old shard again. The merge does not retry itself. When the node that ran MERGE SHARDS stops in the middle, the leader of each sealed shard aborts the merge after 120 s and removes its own seal. The node log then shows one [B141_ORPHAN_SEAL] line for the abort and one for the unseal. When the merge already has an abort or a refusal in the metadata log, the leader does not wait: it removes the seal in about one second, with one [B141_ORPHAN_SEAL] line (B3559).

A sealed shard can also lose the majority of its voters. The cluster then does not rebuild or remove that shard while the merge is pending. It proposes the abort of the merge one time, and the node log shows a [B2306_RECOVERY_DEFERRED_SEALED] line. When the abort applies, the next recovery attempt rebuilds or removes the shard. When the merge installed first, the shard is gone and the cluster does not rebuild it. A rebuild or a removal that arrives before the decision is refused on every node with a [B2306_SEALED_REFUSED] line.

The metadata leader also checks each rebuild request against its placement record of the shard (B3767). It refuses the request when it has no record for the shard, when the survivor is not a voter of the record, when the record has no attested progress, or when the attested commit bound is above the applied index of the survivor. The answer is ShardRebuildRejected, and its reason names the shard and the cause. A refusal changes no row. A correct survivor sends a rebuild only for a record that names it, so this refusal normally comes from a forged or a stale request. When the record of the metadata leader is older than the record of the survivor, the survivor tries again on its next pass.

A new shard of a split can lose the majority of its voters before its leader confirmed the progress for the first time (attested_progress is 0 in SHOW PLACEMENT). The cluster then does not rebuild that shard, and the survivor logs a [SHARD_REBUILD_REFUSED] line with the reason no shard leader attested the record (B3991). The operator decides.

The log says that a merge source stalled

What you see

After a MERGE SHARDS, the node log of one node repeats a line such as:

[B2307_MERGE_SOURCE_STALLED] survivor=3611364882 source=3611364880 applied=118 barrier=124: merge source 3611364880 has not reached barrier 124 (applied=118; reaped reports=1, a hint only). No peer is known to serve the survivor of this merge, so the MergeOp is NOT consumed and the drain retries it with the backoff. ...

The node writes this line at most once each 10 s for one source. The node runs, but it does not apply the metadata entries after the merge.

What it means

The node did not apply the log of a merged shard (the source) up to the merge point (the barrier). So it cannot build the new shard (the survivor) from its own copy. It can get the survivor only from another voter of the survivor that holds it. That voter must show, with its merge decision, that its survivor comes from THIS merge. Until a voter does that, the node waits. A timeout never ends the wait, because an install without that proof can lose data.

The node asks the voters of the survivor again, at most once each 2 s. When a voter serves the survivor of this merge, the log shows [B2307_SURVIVOR_HELD], and the wait ends. A line [B2307_SURVIVOR_PROBE_REFUSED] or [B2307_SNAPSHOT_POLL_OTHER_MERGE] shows a voter whose answer names another merge decision, or no merge decision.

The repair

1. Make sure that every node of the cluster runs the same build. A node with a build before this check sends no merge decision, so a new node refuses its snapshot. Stop every node and start the whole cluster again on one build.
2. Make sure that at least one voter of the survivor is up and that this node can reach it. Start a voter that is down.
3. The node then ends the wait by itself. Do not delete data on the node.

A write fails during a leader change

What you see

A SQL write fails while the cluster elects a new leader for a shard, and the answer is:

calvin write failed: a new leader replaced its log entry before it committed (superseded); the write was NOT applied; retry shortly (shard=… log_idx=… term=… committed_term=…)

What it means

The old leader wrote the write into its log, but it lost the leadership before a majority had the entry. The new leader put its own entry at that position. The write can never commit, and no node applied it.

What to do

Send the write again. The answer has the retry flag set, and a KineDB client reads that flag, never the text (B501). A client with the option retry: true does this for you; the default client sends again only a hold refusal (B2298). A retry cannot apply the write two times.

A different answer starts with calvin batch <n>: the write's outcome is unknown. The node gives it when it caught up from a snapshot while the write waited, so it cannot tell if the write applied. Read the row back before you send the write again.

The node gives the same kind of answer when the Raft driver of the shard stopped before the node applied the write: calvin batch <n>: the write's outcome is unknown — the Raft driver of shard <id> stopped (a drop, a truncate or an eviction) …. A DROP TABLE, a TRUNCATE TABLE or a move of the shard to other nodes stops the driver. The answer comes at once; before B2576 the client waited until its deadline. Read the row back before you send the write again.

The same failure on a DDL statement

A DDL statement (for example CREATE TABLE, DROP TABLE, ALTER TABLE or CREATE INDEX) can meet the same leader change. The answer is then:

metadata CREATE TABLE … failed: a new leader (term …) replaced its log entry (index …, term …) before it committed (superseded); the statement was NOT applied; retry shortly

No node applied the statement. Send it again. A different answer contains the outcome is unknown … it may have applied: read the catalog (for example SHOW TABLES) before you send the statement again.

The leader can also refuse the entry before it appends it, for example when it loses the leadership or hands it over. The answer is then retryable (B2784):

CREATE TABLE: the metadata log refused the proposal (this node lost the metadata leadership); the statement was NOT applied; retry shortly

The cause in the parentheses names the reason. No node applied the statement. Send it again. Before B2784 this answer was the terminal propose_create_table failed (lost leadership?), although nothing was applied.

A statement fails on a busy node: the node cannot reach the leader, or a DDL waited too long

What you see

A statement that a node forwards to the leader fails, and the answer is:

leader forward connect to 127.0.0.1:7001 failed: …; nothing was sent; retry shortly

Or a DDL statement (for example CREATE TABLE) fails with one of these answers:

[forward-deadline] CREATE TABLE: the outcome is unknown — the forward deadline ended while this node waited for the metadata apply; it may still apply
[forward-deadline] default.t: the DDL is committed, but this node did not apply it before the forward deadline
timeout waiting for local DDL apply of default.t after forward

What it means

The first answer: the node could not open a connection to the leader, so it sent nothing. No node ran the statement. Since B1984 the answer has retryable set; before, it was a plain error.

The other answers: the node waited for the metadata log to apply the DDL, and the wait ended first. This happens when the node is very busy. The DDL may still apply, so the outcome is unknown.

What to do

For the first answer, send the statement again. The answer has the retry flag set, and a KineDB client reads that flag, never the text (B501). A client with the option retry: true does this for you; the default client sends again only a hold refusal (B2298).

For a DDL with an unknown outcome, send the IF NOT EXISTS form of the statement (for example CREATE TABLE IF NOT EXISTS t …) on the same connection. The node answers Created only when the table exists on that node. Do not send the plain form again: it fails with already exists when the first statement applied.

A read hold is stuck

What you see

A read of one shard fails again and again for more than 30 seconds. The answer has the retry flag set, and its text contains the name of the hold:

[B2597_READ_HOLD] shard=4021 pseq=7 cursor=6 held_ms=31250: a gtxn part of this shard was acked before its apply on this node, so the node serves no read of the shard until the apply reaches the pseq; the read hold is stuck (older than 30000 ms), see web/docs/troubleshooting.md#a-read-hold-is-stuck

The log of the node that holds the reads has this ERROR line at 30 seconds, and again every 30 seconds while the hold stays:

[B2597_READ_HOLD_STUCK] shard=4021 pseq=7 cursor=6 held_ms=31250: the apply of this shard has not reached the part of an acked gtxn for more than 30000 ms; ...

Before 30 seconds, the log has a [B2574_READ_HOLD] WARN every 5 seconds, and the answer has the [B2597_READ_HOLD] text without the word "stuck".

A split child with a birth row has a longer bound: 30 seconds plus cluster.raft_retire_grace_ms. The child holds every read until the forced catchup of its parent announces the split, and the node proposes that catchup only after the grace. The text of the refusal and of the ERROR line names the bound that the node used. The age of the hold does not restart at the announce, so the longer bound stays until the hold ends.

What it means

A transaction over more than one shard (a gtxn) got a SUCCESS answer before the leader of this shard applied its part. The part is at the position pseq in the gtxn order of the shard. The leader then commits a read hold entry in the log of the shard. Each node that applies the entry (a read hold) serves no read of the shard until its own apply reaches pseq, and the leader gives no read lease meanwhile. A new leader, a restarted node and a split child of the shard hold in the same way. A merge of a split child gives the hold of the child to the merged shard. Without the hold, a read can miss a write that the client got SUCCESS for. When the leader cannot place the hold, the transaction asks the current leader of the shard. When that ask fails too, the client gets the answer OUTCOME UNKNOWN and not SUCCESS.

Normally the apply ends the hold in less than one second. A stuck hold means that the apply of the part did not happen. cursor is the last gtxn position that the shard applied; it stays below pseq. The node never ends the hold without the apply, because a stale read is worse than a refused read. So the reads of the shard stay refused, and each refusal stays retryable.

What to check

1. Look in the log of the shard leader for [B35_SLICE_REJECT] and [B35_SLICE_REJECTED] lines for the shard. A rejected slice does not move the gtxn cursor of the shard.
2. Run SHOW METRICS on the node. b35.gtxn.lag.shard_<id> is the count of gtxn parts that the shard did not apply yet. b2597.read_hold.shard_<id>.pseq and b2597.read_hold.shard_<id>.held_ms give the held position and the age of the hold. b2597.read_hold.stuck counts the stuck holds of the node.

What to do

No repair tool exists yet. File a bug and attach the log of the node that leads the shard. Reads of the other shards are not affected.

A shard withdrew its read lease after a poisoned lock

What you see

The log of a node has this ERROR line one time for a shard:

[B2696_READ_HOLD_POISONED] shard=4021 the node lock is poisoned: the read hold rows could not be set, the read lease is withdrawn and every lease read on this shard declines and forwards

What it means

An earlier panic on the node poisoned the lock of the Raft state of the shard. The node cannot set the read hold of the shard, so it can miss the part of an acked transaction. The node withdraws its read lease and serves no local read of the shard. Each read goes to another node or gets a retryable refusal. The node never gives itself the lease again.

What to do

Find the earlier panic in the log of the node. Restart the node. File a bug and attach the log, with the panic and the ERROR line.

A node does not start: the boot cannot take the writer lock

What you see

The node exits with code 3, and its log has one of these lines:

B1403: the boot cannot take the writer lock for the catalog stamp check: poisoned lock: another task failed inside. The node does not start. Exit code 3.
B3279: the boot cannot take the writer lock to stamp the local cluster settings: poisoned lock: another task failed inside. The node does not start. Exit code 3.

What it means

A panic earlier in the process poisoned the writer lock of the store. The boot needs that lock for the B1403 check of the catalog stamps and for the B3279 stamp of the local cluster settings. The boot does not skip a check: a skipped check can start a node on a store that the check must refuse. So the node does not start.

What to do

Find the earlier panic in the log of the node. Start the node again. If the line comes back, file a bug and attach the log, with the panic and the line.

A forwarded write fails: the outcome is unknown

What you see

A statement that a node forwards to the leader fails, and the answer is:

leader forward to 127.0.0.1:7001: the outcome is unknown — the request was sent, but the reply was lost (…); the write may have applied. Read it back before you send it again

The JavaScript and Python clients raise this answer as KineDBOutcomeUnknownError, a subclass of KineDBError. The error has retryable false, and its response.refusal.kind is outcome_unknown (TLV kind code 5, B2304).

What it means

The node wrote the request to the leader, and then the connection failed before the reply came back. The leader may have applied the write, or it may not. The node cannot tell which. Before B2304 the same failure answered leader forward to … failed: …, and a client could not tell it from a write that did not apply.

What to do

Do not send the write again at once: when the first write applied, a second write can apply it two times. No KineDB client sends it again, also not with the option retry: true.

1. Read the data back, for example SELECT the row by its key.
2. When the read shows the write, the write applied. Do nothing more.
3. When the read does not show the write, send the write again.

BEGIN, COMMIT or ROLLBACK is refused

What you see

A BEGIN, a COMMIT or a ROLLBACK sent as a statement answers:

BEGIN, COMMIT and ROLLBACK are not statements that a client sends alone: send the complete transaction in one call with transaction(). To read and then write in one transaction, CALL a function. Nothing was applied.

What it means

No transaction stays open on the server or in the embedded engine. A client sends ONE complete transaction in ONE call. The WebSocket, POST /sql, the embedded sql() and a statement of a transaction refuse the three statements. Nothing was written.

What to do

• Put every statement of the transaction in one transaction([[sql, params], …]) call of the client library. See Transactions.
• Use token for a safe retry and retry to send the whole transaction again after a temporary error.
• For a read, a decision and a write in one step, CALL a function.
• The kinedb command line sends a complete BEGIN … COMMIT block of a .sql file, of -e or of the interactive shell as one transaction. A block with no end is an error.

A statement with a qualified table name is refused

What you see

A statement that names its database or branch in a table name (app.orders, app@dev.orders) answers one of these:

database not found: app
branch not found: dev
statement targets two databases/branches: 'default@main' and 'app@main'; one statement targets one (database, branch)
transaction is pinned to database 'default'; 'app.orders' targets 'app@main'

What it means

• database not found and branch not found: the database or the branch does not exist on the node that received the statement. A database that the user cannot see in SHOW DATABASES also answers database not found. A branch belongs to one database, so app@dev needs a branch dev that was made in app.
• statement targets two databases/branches: one statement reads and writes one database on one branch. A bare table name is in the database and on the branch of the session, so a bare name next to a qualified name of another database names two targets.
• transaction is pinned: a transaction runs in one database: the db of the transaction() call, else the database of the session. Every statement of the transaction runs in that database, on main. Nothing of the transaction was written.

What to do

• Create the database or the branch, or correct the name. SHOW DATABASES lists the databases; USE <db> then SHOW BRANCHES lists the branches of a database.
• Name every table of the statement with the same database and branch, for example app.orders and app.customers.
• For a transaction in another database, send transaction() with the db option of that database.

A CALL fails: the function has no parameter types

What you see

A CALL, or the tick of a ticker, answers:

function <name> has no parameter types: replace it with CREATE OR REPLACE FUNCTION

In SHOW FUNCTIONS, the row of the function has typed = false. The Functions page of the web console marks the function with the same text.

What it means

An older release stored the function without types. Since B2172, every parameter and the result of a function have a type (the signature). The server keeps the old function, its body and its grants, but it refuses every CALL of it until an admin gives it types. A ticker that calls it does no work. CREATE TICKER refuses such a function, and kd-tool restore refuses a dump that holds one.

The repair

1. Read the body. A superuser or the run-as user of the function can read it:

   SHOW CREATE FUNCTION <name>;

2. Send CREATE OR REPLACE FUNCTION with the same body, a type for each parameter and a RETURNS clause. Write NULL after the type of a parameter that can be null or missing. Keep the RUN AS clause of the old function:

   CREATE OR REPLACE FUNCTION add_score(p TEXT) RETURNS INT LANGUAGE kd AS $$
     insert("scores", {player: p}); return 1;
   $$ RUN AS 'web';

3. Run SHOW FUNCTIONS. The row now has typed = true. The grants of the function and the schedule of its tickers stay.

A ticker function has no NOT NULL parameter, because a tick sends no arguments. A function that a conflict policy names has the fixed signature of a conflict function. The Edit form of the Functions page writes the types for you: it shows each parameter with no type, and you pick one.

The metadata apply of a node is halted after a metadata point install

A node that was far behind the metadata group catches up with a pulled metadata point. After the install, the node reloads its views in memory (the tables, the users, the revoked tokens, the token keys, the cluster settings and the functions) from the installed trees. When one of these reloads fails, the node applies no later metadata entry. This prevents a decision on an old view, which can make this replica differ from the others.

The sign

The node log holds this line, at the first failure and then at most once every 10 seconds:

[B2936_POINT_VIEWS_STALE] index=<point>: the views ["<name>", ...] did not read their trees. The metadata apply on this node is HALTED ...

The log also holds [B35_LEDGER_STALL] lines with the kind PointViews. A DDL, a GRANT or a user change waits or times out on this node. The other nodes work.

The repair

1. The node retries the reload with a backoff. When a retry succeeds, the log holds [B2936_POINT_VIEWS_FRESH] and the metadata apply continues. Do nothing else.
2. When the line repeats for minutes, read the log lines before it: they name the tree that did not read. Restart the node. The start loads each view again from its tree. A restart clears the halt, but the start does not halt when a load fails: read the start log for a failed load of the same tree.
3. When the start also fails to read the tree, the system trees of this node are damaged. Remove the node from the cluster and add it again with an empty data directory. It then pulls a new metadata point.

A node stays behind at a metadata point install: [B2972_POINT_SUPERSEDE_REFUSED]

A node that was far behind the metadata group installs a pulled metadata point. The node can hold a table or an index tree that the point replaced. Then the install removes the old tree. The result is the same as for a node that joins with an empty data directory. The install removes a tree only by a fixed rule. The rule uses the builds and the catalog slots.

The sign

The node keeps running. It logs a refusal line, and the install changes nothing:

[B2972_POINT_SUPERSEDE_REFUSED] the metadata point names "<name>" in another form than this node: <reason>. The install changed nothing. ...

The node logs [B2318_METADATA_POINT_REFUSED_INSTALL] with the same reason. The node then repeats the install once for each dwell of the metadata sweep. Each repeat gives the same refusal. The reason names the ids, the builds, the slot stamp and the local floor. Before an accepted removal, the log holds one [B2972_POINT_SUPERSEDED_LAYOUT] line for each tree.

What it means

The node holds a layout that is newer than the point, or a layout that the node did not apply from the metadata log. The install cannot show that the cluster replaced it. So the install deletes nothing. A newer point does not fix this. No environment variable overrides it. The node stays behind its peers until the operator wipes its data directory.

The rule, and what it does not prove

The install replaces a re-created table only when the point id is newer. The local id must also be at most the slot stamp. The slot stamp must be at most the local floor. A node-local table can have a counter id in that range. The install then deletes that table and its rows. A cluster node can hold a node-local table: a data directory that a standalone node wrote can start with --calvin (B2985, B3182). Do not start such a directory as a member of an existing cluster. A --calvin --join boot on such a directory refuses at the start ([B3182_JOIN_STORE_HAS_LOCAL_TABLES], below).

An index tree can have no build in its layout: a branch index, or an index from before B2108. The install then takes the registration of the index from the catalog slot of its base table. Another base table on the two sides replaces the tree. A registration on the point side only replaces the tree. A registration on the local side only is a refusal. A catalog slot that does not decode is a refusal. When neither side has a registration, the install keeps the tree and logs one [B2972_POINT_NONE_BUILD_UNPROVEN] line (B2991).

The repair

Stop the node. Wipe its data directory. Start it again so that it joins the cluster as a new node. It then pulls a new metadata point.

A joiner does not start on a standalone data directory: [B3182_JOIN_STORE_HAS_LOCAL_TABLES]

A node that starts with --calvin --join is a new member of a cluster. It gets every table from the cluster. Only the first --calvin boot writes the file identity in the data directory. A standalone server and the embedded engine (-e, -f) never write it. So a data directory with no identity file and with user tables is a directory that a node without --calvin wrote. A later metadata point install can delete such a table or refuse it. So the node refuses to start.

The sign

The process exits with code 3 before it writes anything. It logs ONE error:

[B3182_JOIN_STORE_HAS_LOCAL_TABLES] this node starts with --calvin --join, but its data directory has no identity file and its store holds <n> user table(s): [<database>.<branch>.<name> (id <id>, rows <rows>); ...]. A node without --calvin (a standalone server, or the embedded engine -e or -f) wrote this directory. ... There is no override. Exit code 3 (needs an operator).

The error names every table, its id and its row count. With more than 10 tables, one numbered line for each table comes first, and the summary is the last line. The node writes no identity file, so the next start shows the same error. No environment variable overrides it. KINEDB_RESTAMP_CATALOG_SLOTS=1 does not override it either.

A data directory with an identity file is a returning member, and this check does not apply. A --calvin boot with no --join on a standalone directory is the conversion of a single node (the B1403 check of the catalog stamps, and the adoption point of B3201), and this check does not apply either.

The repair

1. When you need the rows, start the directory without --calvin. Run kd-tool backup against it, and kd-tool restore the dump into the cluster.
2. Move the data directory away.
3. Start the node with --calvin --join and an empty data directory.

A member of another cluster does not join: [B4162_JOIN_OTHER_CLUSTER]

A data directory with an identity file and a cluster key copy (cluster.key) belongs to one cluster. The cluster key pair is the id of the cluster. A --calvin --join boot of that directory asks the --join node first, then the nodes of its peer book, on the route POST /join/member-check. The asked node answers its committed cluster public key and whether its metadata voters name the calvin id of the joiner. The check runs before the node writes anything and before its drivers start.

The sign

The process exits with code 3 within about 5 seconds. It logs ONE error:

[B4162_JOIN_OTHER_CLUSTER] this node starts with --calvin --join <host:port>, but its data directory belongs to another cluster. Its cluster key is version <v> fingerprint <own>; the cluster of <host:port> committed version <v> fingerprint <committed>, and its metadata voters do not name the calvin id <id> of this node. The node does not start; nothing was written. ... There is no override. Exit code 3 (needs an operator).

The node refuses ONLY when both conditions are true: its own key is not the committed key, and the committed voters do not name it. A known member with another key starts and goes into the quarantine of [B3974_CLUSTER_KEY_MISMATCH]. When no node gives a decision within 5 seconds (no node answers, a node of an older version, no committed key yet), the boot continues and logs B4162: the member check passes: no node answered .... A node with no cluster.key asks nothing. No environment variable overrides the refusal.

The repair

1. When this directory is a member of its own cluster, start it without --join or with the --join address of a node of its own cluster.
2. To join the other cluster: when you need the rows, run kd-tool backup against the old cluster and kd-tool restore the dump into the new cluster. Move the data directory aside. Start the node with --calvin --join and an empty data directory.

The first node does not start on a standalone data directory: [B3201_ADOPTION_POINT]

A --calvin boot with no --join on a data directory that a node without --calvin wrote ADOPTS that directory. The metadata log did not create the tables, the databases, the users and the other catalog objects of the directory. A joiner that replays the log from the start does not get them. So the first node compacts its metadata log above the adoption before a second node can join. Each joiner then installs a metadata point, which holds the full catalog.

The boot logs each step with the tag [B3201_ADOPTION_POINT]: marked, armed and compacted: floor=<n> last_adoption_index=<n> first_log_index=<n>. Until compacted, the node refuses each new voter. A crash before compacted leaves a marker in the data directory, and the next start compacts the log.

The same compaction runs at any boot of a node that is the ONLY voter of its metadata group, whose metadata log starts at index 1, and whose store holds a catalog object (B3237). A version before B3201 adopted a directory with no compaction, and its joiners missed the tables ([B2264_ADOPT] refused ... there is none). The boot logs B3237 armed, then compacted. If the replay of the log shows more than one voter, the boot logs B3237 skipped and does not compact. A seed that you created empty and filled through SQL also compacts once at its next start. This changes nothing for its data: each joiner installs a metadata point with the same catalog.

A cluster that a version before B3201 made from an adopted directory, and that has more than one voter, does not get this compaction. Each node of it refuses to start with [B3249_UNREPLICATED_ADOPTION] (the next section).

The sign

The process exits with code 3. It logs ONE error that starts with [B3201_ADOPTION_POINT] and names the step that failed: the read of the catalog, the write of the marker, the durable sync of storage, or the compaction. The compaction needs a node that is the leader and the ONLY voter of its metadata group. No environment variable overrides the error.

The repair

1. Repair the access to the data directory when the error names a read, a write or a sync. Then start the node again.
2. Start the directory as the only node: with --calvin, with no --join, and with no other node in the cluster. Start the other nodes with --join after the compacted line.
3. If the error stays, start the directory without --calvin. Run kd-tool backup against it, and kd-tool restore the dump into a new --calvin node.

A node of an old adopted cluster does not start: [B3249_UNREPLICATED_ADOPTION]

A version before B3201 adopted a standalone data directory on the first node and did not compact its metadata log. Other nodes then joined by a replay of that log. The log re-floors the adopted tables, but it never creates them. So each joiner refused the re-floor ([B2264_ADOPT] refused ... there is none) and does not hold these tables. A read of such a table on a joiner fails; the first node returns the rows.

The same version also adopted the objects that have no re-floor: an empty table, a database, a branch, a function, a ticker and a user. No entry of the log creates them, so a joiner that replayed the log does not hold them either (B3391).

The sign

At each start, after the replay and before the node serves clients, every node of the metadata group reads its metadata log when the log starts at index 1 and the group has more than one voter. If the log re-floors a table that no earlier entry creates, the process exits with code 3. It logs ONE error. Its summary line holds [B3249_UNREPLICATED_ADOPTION] B3249: and names each table as db.branch.name (re-floor at index <n>). With more than 10 tables, the error starts with one numbered line for each table ([B3249_UNREPLICATED_ADOPTION]: table 1/<n>: ...), and the summary line comes last. The first node and each joiner that replayed the log refuse in the same way, because they hold the same log. A passive joiner (see the limits below) refuses only while a leader stays up: if the other nodes exit with code 3 first, it waits without end. A joiner that installed a metadata point later has a log that does not start at index 1, so it does not refuse. If it holds the tables depends on the node that sent the point (NOT DETERMINED: no test proves it).

Under the same conditions, a node also refuses when its LOCAL store holds a catalog object that no committed entry of the log creates. The summary line then holds B3391: and names each object by its kind and its name: table <db>.<branch>.<name> (an empty table too), database <name>, branch <db>.<name>, function <name>, ticker <name> or user <name>. Only the first node refuses for these objects: a joiner that replayed the log holds only the objects that the log created. An exception: an object that an older binary wrote outside the log (a function before B301, a ticker before B2332) is on every node, so every such node refuses (see upgrade path U128). A table that the error names by its re-floor is not named a second time. A drop does not count: an object that exists needs a create in the log, before or after a drop.

The repair

1. Stop every node of the cluster.
2. Start the data directory of the first node without --calvin. Run kd-tool backup against it.
3. Run kd-tool restore with the dump into a new, empty --calvin cluster. The new first node compacts its metadata log (B3201), and each joiner installs a metadata point with every table.

The same repair applies to the B3391 objects: the backup of the first node holds them, and the restore creates each one through the log of the new cluster.

KINEDB_OPEN_WITH_UNREPLICATED_ADOPTION=1 starts the node anyway and logs each table and each object. The joiners still miss them, and the first node can retry a refused metadata point without end ([B2318_METADATA_POINT_RETRY], B3386). Use it only to read the data before the repair.

Known limits:

• When the old log segments with the first entries are gone (the first log index is above 1), the check does not see the shape, and no node refuses. The routine compaction of the metadata log can cause this on a cluster that ran for some time.
• In the case above, the B3391 check of the local objects is silent too. The joiners still miss the tables and the objects.
• The B3391 check reads tables, databases, branches, functions, tickers and users. It does not read roles, settings, audit rows or tokens: an adopted role is not seen (NOT DETERMINED how often).
• The B3391 check reads only a table with a catalog entry. A table with data and no catalog entry is the shape of [B2264_IMPLICIT_TABLE], and that check decides it.
• A node whose metadata Raft WAL was lost and that starts with the B2048 opt-in does not read its store in the B3391 check. Its log no longer holds the entries that made its objects. The check line then shows lost=true. So at that boot, no B3391 local object is named. For example, a first node of a standalone directory that a binary before B3201 adopted is not named when it also lost its metadata WAL. Before you use the opt-in on such a node, do the repair above: start its data directory without --calvin, run kd-tool backup, and restore the dump into a new --calvin cluster.
• A node that restarts with --join starts passive when it cannot reform from its saved voter list: the list is lost, it has fewer than 2 voters, it does not name this node (after ALTER CLUSTER EVICT NODE), or KINEDB_DISABLE_AUTO_REFORM is set to a value that is not empty. A node whose metadata Raft WAL was lost ([B2048_NO_REFORM]) also starts passive, but its metadata log is empty, so it does not wait. When its metadata log starts at index 1, it waits for a leader before the check. It logs [B3249_UNREPLICATED_ADOPTION] passive boot: wait for the catch-up before the check, then [B3249_UNREPLICATED_ADOPTION] passive boot: still waits for a leader every 10 s. With no leader it does not become ready.
• A passive node that no leader takes out of the passive mode waits without end: it stays in the boot checks, is not ready and logs no refusal. Examples: a node that ALTER CLUSTER EVICT NODE removed and that restarts with --join; a node whose metadata point install the cluster refuses again and again (B3386). When the node had more than one peer, a start without --join is refused (I8: "pass --join HOST:PORT to rejoin the existing cluster, OR delete the data directory to bootstrap a fresh cluster from this node"). For an evicted node, run ALTER CLUSTER READMIT NODE '<id>' on a member to let it back, or stop it, move its data directory away and start it on an empty data directory with --join (a new node id). Decision: the wait has no time bound. A bound re-opens the silent miss (priority 3, correctness); the hang costs the availability of one node that serves no data: an evicted node, or a node that should become a member but whose point install the cluster refuses (B3386). B3403 tracks a parked refusal for such a node.
• When the metadata Raft driver stops or retires during the wait, the wait ends, the check passes without a scan and the node goes on to ready. Production does not stop or retire the metadata driver today (INFERRED), so this path is not reached. The test of this end uses a fake driver only.
• The tests prove the decision and its text on each node of a real 3-node group. The exit with code 3, the place of the check before Ready and the end of the wait have no proof on each change: the tests of the old binary run only by hand until B3383.
• When the boot cannot start the thread of the boot checks, the checks run on the boot thread before the node accepts connections ([B2157_SPAWN_FAILED]). A passive node then can get no message from a leader and wait without end. This risk existed before B3249; the wait makes it longer.
• The scan holds the lock of the metadata Raft node while it reads the log: about 3 µs for each entry (34 ms for 10,000 entries, 311 ms for 100,000, debug build).

If a joiner logged [B2264_ADOPT] refused ... there is none, or misses an object that the first node has, do the repair above.

A node stops at a metadata point install: [B2985_POINT_LOCAL_TABLE_REFUSED]

A node that was far behind the metadata group installs a pulled metadata point. The node can hold a user table that the point does not name. There are two causes. The cluster dropped the table, and this node did not apply the DROP TABLE before the log moved past it. Or the table is node-local: a standalone node wrote it, and the data directory then started with --calvin. The install cannot tell the two causes apart. A keep brings back a dropped table on one node only, and a delete loses the rows of a node-local table. So the install refuses.

The sign

The node logs a refusal line, and the install changes nothing:

[B2985_POINT_LOCAL_TABLE_REFUSED] the metadata point <index> does not name <n> local user table(s) of this node: "<name>" (id <id>, slot stamp <stamp>, rows <rows>, derived trees [...]); ... The install changed nothing and does not retry this point. ...

The node logs [B2318_METADATA_POINT_REFUSED_INSTALL] with the same reason. The error names every such table, its id, its catalog slot stamp, its row count and its index and companion trees. Then the node logs [B3386_METADATA_POINT_NEEDS_OPERATOR] with the same reason and exits with code 3 (needs an operator). Before B3386, the node kept running behind its peers and repeated the install. No environment variable overrides it.

The repair

1. Decide whether a named table is node-local. Only a data directory that a standalone node wrote can hold one. Every other named table is a table that the cluster dropped.
2. When you need the rows of a node-local table, export them before the next step.
3. Stop the node. Wipe its data directory. Start it again so that it joins the cluster as a new node. It then pulls a new metadata point.

A node stops at a metadata point install: the record of the installed point does not decode

A node that was far behind the metadata group installs a pulled metadata point. Before the install, the node reads its record of the last installed point. The record is an 8-byte index in a node-local tree.

The sign

The node logs this line and exits with code 3 (needs an operator):

[B3386_METADATA_POINT_NEEDS_OPERATOR] [B2318_METADATA_POINT_OLDER_THAN_LOCAL] the record of the installed point has <n> bytes, not 8. ...

What it means

The record holds a value that is not 8 bytes. The local data of this node is damaged. A point of another node and a retry do not change it. A read error of the record is not this case: the node then logs [B2318_METADATA_POINT_RETRY] and tries again. It pulls the points again in the same reseed loop, and after the last attempt it tries again at the next sweep.

The repair

Stop the node. Wipe its data directory. Start it again so that it joins the cluster as a new node. It then pulls a new metadata point.

A node stays behind at a metadata point install: [B2992_POINT_SKELETON_BASE_MISSING] or [B2992_POINT_SKELETON_SLICES_REFUSED]

A node that was far behind the metadata group installs a pulled metadata point. When the point adds an index tree, the install slices the tree from the base table of THIS node, as a CREATE INDEX does. The base table of this node can have another split than the base table of the node that served the point.

The sign

The node keeps running. It logs a refusal line, and the install changes nothing:

[B2992_POINT_SKELETON_BASE_MISSING] the metadata point adds the tree "<tree>" over the base table id <id>, which is in no layout of this node or of the point. The point is not consistent. The install changed nothing and does not retry this point
[B2992_POINT_SKELETON_SLICES_REFUSED] the metadata point adds the tree "<tree>", and its slices over its base do not partition the key space: <reason>. The install changed nothing and does not retry this point

The refusal is fatal for that point: the node does not repeat the install of the same point.

What it means

The point is not consistent with itself, or with the layouts of this node. The install deletes nothing and adds nothing. This is a defect: report it with the log line.

The repair

Stop the node. Wipe its data directory. Start it again so that it joins the cluster as a new node. It then pulls a new metadata point.

A local sync database holds a version key that cannot be rewritten

What you see

The first open of a local sync database stops with this error. In JavaScript, the text follows kinedb embedded: the module refused a sync session on <url> (kdb_sync_open answered -1): . The text itself starts with NeedsOperator: B3028: :

NeedsOperator: B3028: the version shadow `kinedb_sync_row_version` holds 3 row key(s) that this build cannot rewrite into the new key shape (tables: `orders`). The open stops and drops no version. To repair it: delete the local database and sync again. A write that was not pushed is lost with the local database.

This refusal names no KINEDB_* variable. It belongs to the local database of an embedded sync, not to the node, so web/docs/run.md has no row for it.

What it means

Before B3028, the local sync database kept the server version of each cached row in the table kinedb_sync_row_version. A build with B3028 rewrites that table into kinedb_sync_row_version_v2 at the open. The rewrite decodes each old key. A key that does not decode is damaged, or a build other than kinedb wrote it. The open stops, changes nothing more, and drops no version. The error names the count and the tables. It names no key.

The repair

1. Push the writes that you can: a write that is not pushed is lost in step 2.
2. Delete the local database.
3. Open a new local database and sync again. The fill writes each row and its version.

A synced write in a transaction is refused

A synced write inside a transaction that you opened is refused while the old table exists:

B3028: the version shadow of this database is not in the new key shape yet (`kinedb_sync_row_version` is not rewritten into `kinedb_sync_row_version_v2`), and a transaction cannot rewrite it. Nothing was applied. Run the transaction with `SyncClient::transaction_local`, which rewrites the shadow before the transaction starts, or open the sync first.

The write is refused in the same way when the new table does not exist and no old table exists. The text then reads the version shadow table kinedb_sync_row_version_v2 does not exist in this database yet.

Nothing is applied and no write is lost. The repair: run the transaction with the Rust method SyncClient::transaction_local, or open the sync first. Both create or rewrite the shadow. A JavaScript app cannot open a transaction of its own for a synced write, so it does not meet this refusal. It opens the sync, and the open rewrites the shadow.

An older build fails with unknown column row_key

A build before B3028 does not read kinedb_sync_row_version_v2. Since B3070 a newer build keeps an empty placeholder table under the old name kinedb_sync_row_version. Its one column is b3028_moved_to_kinedb_sync_row_version_v2, and it has no row_key column. So the older build opens the database, and then its first fill or its first synced write fails with one of these errors:

Query: unknown column in WHERE: row_key
Query: unknown column in WHERE: table_name
Query: unknown column: row_key

The text names the column, not the table. A real build from before B3028 (the commit before a1664a7ce) gave these texts on a database of a build with B3070. The older build never reads version 0 silently, and the server loses no write.

The repair: use a build with B3028 or newer. Or push the writes that you can, delete the local database and sync again. A write that is not pushed is lost with the local database.

An open fails with B3252: this local database holds sync format

Since B3252 the local database records the format of its sync tables. The identity row sync_format of the table kinedb_sync_identity holds it. A build refuses a database that a newer build wrote with a higher format. The sync open, a synced write and a synced file put fail with this error (kind NeedsOperator):

B3252: this local database holds sync format 2, and this build of kinedb-sync reads sync format 1 or lower. A newer build wrote it. Nothing was changed. Open it with that newer build, or use a new local database.

The refusal comes before any change: no table is created, dropped or rewritten, and no write is applied. A value that is not a number gives the same refusal, with the text which is not a number.

The repair: open the database with the newer build. Or push the writes with the newer build, delete the local database and sync again with the older build. A write that is not pushed is lost with the local database. Do not edit the sync_format row by hand: the older build then reads tables of a shape that it does not know.

A synced write is refused: the op-log has no usable index

What you see

A write to a synced table (an embedded sync: the JavaScript, Python and wasm clients, and the Rust SyncClient) fails with this error. Nothing is applied:

B3064: the op-log of this database has no usable index `kinedb_sync_oplog_by_key`, so the synced write to `orders` is REFUSED and nothing was applied: a synced write reads the pending entries of its key through that index. ...

What it means

The local database keeps each write that is not pushed in the op-log kinedb_sync_oplog. A synced write reads the entries of its row key through the index kinedb_sync_oplog_by_key (B3064). The index is derived from the op-log. The sync open and each synced write outside a transaction build it when it is absent. Since B3084 the build runs for an op-log of any size: it writes the index in parts of at most 192 KiB, so its memory does not grow with the op-log. (A build of B3064 built no index above 100,000 op-log entries.) So the error comes only inside a transaction that you opened, after a DROP INDEX of the index or after a push that dropped it (see the section after the next). DDL cannot run in a transaction.

The repair

1. Write again outside a transaction. That synced write builds the index first.
2. In Rust, run a transaction with SyncClient::transaction_local, which builds the index before the transaction starts.

No write is lost.

A synced write is refused: the index key is too long

What you see

A write to a synced table fails with an engine error that contains this text. Nothing is applied:

is longer than MAX_KEY_LEN (65535 bytes)

What it means

The index kinedb_sync_oplog_by_key (B3064) holds one key for each op-log entry: the table name plus the base64 text of the primary key of the row. The engine refuses an index key that is longer than 65,535 bytes. So a synced write with a primary key of about 49 KB or more is refused. A build before B3064 had no index and accepted such a write. An older build after a downgrade also refuses it (INFERRED: the index stays in the schema, and the engine maintains every index of a table).

An entry with such a key that an older build wrote does not stop the other writes. Then the open builds no index, and a synced write reads the whole pending log of its table until a push sends that entry.

The repair

Use a shorter primary key for the synced table, for example a hash or a generated id. Keep the long value in a column that is not part of the primary key.

A push is refused: the op-log index does not match the op-log

What you see

A push sends no frame and fails with this error:

B3064: the op-log holds 12 rows of `orders` in the state that the read selects, and the read through the index `kinedb_sync_oplog_by_key` gave 11: the index does not match the op-log. The read is REFUSED, so a push of `orders` sends no frame. ...

What it means

The push reads the waiting writes of the table. It compares the writes that it read with the rows of the op-log table itself. A difference means that the index lost an entry, or that a row of the op-log has a NULL pk or table_name and the index was built before B3087 (such an index holds no entry for that row; an index built by B3087 or later holds one). A push that used the index alone would skip that write with no error. So the push refuses, sends nothing and loses nothing. It also drops the index, so that the next build reads the op-log again.

The repair

Push again. The push does not need the index: with the index absent, it reads the op-log table itself. The next synced write outside a transaction, or the next open of the sync, builds the index again.

When the push still refuses, a row of the op-log has a NULL pk or table_name. The rebuilt index holds that row, so the refusal then names the row itself, for example the op-log entry for `id` carries no primary-key cell. No sync client writes such a row; an SQL statement on kinedb_sync_oplog did. Report it with the error text. The repair is to push what you can, delete the local database, and sync again.

A write or a CREATE INDEX is refused: too many distinct elements for an array index

What you see

A write to a table with an index on an ARRAY column fails on a server with:

index own10m_readers on own10m: the row 42 has 1025 distinct elements in column readers, above the limit of 1024 (cluster.index.max_array_entries). Store fewer elements, or an administrator can raise the limit with SET CLUSTER cluster.index.max_array_entries = <n>.

An embedded database has no SET CLUSTER, so its text ends after Store fewer elements.

A CREATE INDEX on an ARRAY column fails with the same text and NOTHING was created. at the end.

What it means

An index on an ARRAY column holds one entry for each DISTINCT element of a cell. One row can have at most 1,024 such entries in one index, or the value of the cluster setting cluster.index.max_array_entries when an administrator set one. The count is of the distinct elements, not of cardinality(col): ARRAY['a', 'a'] has one entry, and -0.0 and 0.0 are one value.

• A write that ADDS an entry to a row above the limit is refused, and the row does not land. A write that does not add an entry passes: an UPDATE of another column, an UPDATE that only removes or reorders elements. An INSERT over the key with the same elements passes when the receiving node holds the row; on a cluster, a node that does not hold the shard of the row refuses it above the limit. A DELETE always passes.
• A CREATE INDEX over a table with a row above the limit is refused. The text names the row. No index is created, also on a cluster.

The repair

Store fewer distinct elements in the row that the text names. For a list of more than 1,024 values, put the values in a separate table, with one row for each value.

An administrator can raise the limit for the whole server or cluster:

SET CLUSTER cluster.index.max_array_entries = 2048;

The value is from 1 to 65536. A value out of that range is refused:

settings: `cluster.index.max_array_entries` must be between 1 and 65536, got 0

A higher limit has a cost. The bytes of one write grow with the elements, so the largest element that one UPDATE of a full row can hold gets smaller: about 304 bytes at 1,024 elements, about 140 bytes at 2,048, and about 10 bytes at 10,000 (a node with a 64 MiB --memory budget). From 14,463 elements up, no element fits such an UPDATE on a node with a 64 MiB --memory budget. A larger write is refused with exceeds this node's write-pipeline budget, and nothing is written. On a node with no --memory budget, the effect of such a large write is not known yet. A lower limit removes nothing: a row above it stays readable, and only a write that adds an entry to it is refused.

A replica logs [B169_ADOPT_REFUSED]: the minority holds a newer write

What you see

One replica of a shard has a different root from the other two. The node log of that replica holds this line:

[B169_ADOPT_REFUSED] shard=<id> table="<name>" captured_index=<n> reason=minority_holds_newer_write minority_newer_keys=<k> ... heal=<proposed|blocked|unplanned> ...

The lines [B169_DIVERGENT] and [B169_DIVERGENCE_CLASS] come before it. The node keeps running. The counter b169.adopt_refused_minority_newer_total in SHOW METRICS rises by one for each refusal.

The field heal= tells what happens next:

• heal=proposed carry_keys=<n>: the line is a warn. The shard heals by itself. The next lines are [B169_RECONCILE] ... outcome=HealPlanned(...) and, on the same node:

  [SHARD_HEAL_PROPOSED] shard=<id> table="<name>" parts=<p> keys=<n> via=<leader|forward> — ...

  Each replica of the shard then logs one line for each part:

  [SHARD_HEAL_APPLIED] shard=<id> index=<log index> table="<name>" carried=<n> written=<w> skipped=<s>

  On the minority written=0. On a replica of the majority written is the number of keys that it lacked. A later sweep then adopts the majority root, and the roots agree ([B169_RECONCILED]).

• heal=blocked blocked_keys=<n> blocked_samples=[(<key>,<reason>) ...]: the line is an error. The shard stays divergent. A test run that scans the node logs stops on this line.

• heal=unplanned: the line is an error. The plan of the heal failed (the error follows), or the classifier failed (reason=unclassified). The shard stays divergent.

What it means

The replica is the dissenting minority of its shard. The node tried to heal it with a copy of the majority root (B169). Before the copy, the node found that the minority holds a write with a newer timestamp than the same key on the majority. The copy would lose that write. Thus the node refuses the copy.

Then the node plans a heal (B1804): for each key, the version with the newer timestamp wins, and a tombstone is never undone. The leader of the shard writes the plan into the Raft log of the shard (B3392). Each replica writes each carried version, with its own timestamp, only where its own version is older or absent. A write that arrives after the plan has a newer timestamp, so the heal never overwrites it. After the heal the majority holds every write of the minority, and the copy of the majority root loses nothing.

A key blocks the heal, and the refusal stays, for these reasons (blocked_samples):

• DerivedTrees: the heal writes the rows only, so a tree that follows the rows must not change. An index of the table always blocks, also an empty one. The file tree and the idempotency tree of the shard block unless the minority and the majority hold the same root of that tree at the plan point. A different idempotency tree can hold a record of the minority only, and a retry of that client request would apply a second time. A root that can not be read, or a tree that one side does not have, blocks too.
• AbovePoint: the timestamp of the minority is above its own applied index. No committed write can have that timestamp.
• Unprovable: a timestamp does not decode.
• MaybePurgedOnB: an internal table that removes a deleted key holds the key on the minority only. The majority can have deleted it.

A replica also refuses a part of the heal, and writes nothing, when the shard split or merged before the part, or when an index build fenced the shard log before the part. The fence check reads the shard log only, so every replica decides the same. Then the line [SHARD_HEAL_APPLIED] ... written=0 refused: <reason> is a warn, and a later sweep plans again.

The class line tells the cause. With verdict=BenignTsSkew and real=0, the values are equal on all replicas and only the timestamps differ. Then no data is wrong. With a real count above 0, the values differ and the case is a real divergence (priority 1: corruption).

The nightly run 4266 of 2026-10-15 shows the first case (DIRECTLY OBSERVED, comments of B1804). A shard of the table bench had differing=144 ts_only=144 real=0 a_newer=144 after a shard split. The 144 keys had equal values on the three replicas. The node logged the refusal 10 times and nothing healed the shard. Why the timestamps differ: NOT DETERMINED.

What to do

With heal=proposed, do nothing. Check that the roots agree after the next sweeps (SHOW ROOTS on each node, or the line [B169_RECONCILED]). If the refusal comes back with heal=proposed again and again and no [SHARD_HEAL_APPLIED] line follows, file an issue with the lines.

With heal=blocked or heal=unplanned, the shard stays divergent. The operator does not repair it by hand.

1. Find the minority node. The [B169_DIVERGENT] line is in its log. It says "this node is the dissenting minority".
2. Read the [B169_DIVERGENCE_CLASS] line of the same shard.
3. Check that the cluster is healthy: every voter of the shard answers, and [B169_DIVERGENT] names no second dissenting node.

With real=0, no data is wrong. Do NOT move or delete a data directory. Keep the log lines and the data directories of all replicas. File an issue with the lines.

With real above 0, do the same. Name the issue priority 1 (corruption).

A mixed cluster can not heal: a node of a kinedb version before B3392 does not read the heal entry of the shard log (see Breaking changes).

The sweep that finds the divergence runs every 15th tick (about 30 seconds). The variable KINEDB_B169_DETECT_INTERVAL sets the number of ticks. The tests use it. Leave the default in a production node.

The line can also say reason=unclassified. The node could not compare the two roots. It installed nothing. Read the node log before and after the line, and file an issue with it.

A diverged sync is refused: more than one lowest common ancestor

What you see

The Rust storage call sync_remote_commit (or merge::find_common_ancestor) fails with a Sync error that contains this text (B3507). The two short hashes name the two ancestors:

the sync cannot merge: the two copies have more than one lowest common ancestor (<hash>, <hash>), a criss-cross history. No single merge base is correct. Nothing was written. To continue, sync from a copy that holds one common base, or merge main in first.

What it means

The two copies merged each other in a crossed order: each copy holds a merge commit of the same two commits, and then each copy wrote more. So the two HEADs have two lowest common ancestors, and neither one is behind the other. A 3-way merge from one of them can lose a change or bring back a deleted row. The sync refuses before any write: the HEAD and the rows do not change. A correct base for this history is open work (B3599).

The repair

Sync from a copy that holds one common base with this copy, or merge main into the two copies first. Then run the sync again.