History retention of the embedded database

An embedded database (the Rust crate kinedb) keeps all of its commit history by default. Each commit writes new blocks, and no old block is ever removed. So the data directory grows with each UPDATE and each DELETE, also when the rows stay the same.

The startup option history_retention sets how much history the database keeps. With the option, the database removes the blocks that only older commits use.

Open a database with the option

use kinedb::{Db, OpenOptions};

let mut db = Db::open_with_options(
    "/var/lib/myapp/data",
    OpenOptions {
        history_retention: Some("7d".to_string()),
        ..Default::default()
    },
)?;

The option applies to the whole database. Give it at each open. The database does not save it, and no SQL statement and no environment variable sets it.

The values

The grammar is the grammar of the server flag --gc-retention (Run kinedb).

Value	The history that the database keeps
not set (None)	All history. Nothing is removed. This is the default.
<N>commits, for example 100commits	HEAD and its N nearest older commits. 0commits keeps HEAD only.
<N>s, <N>m, <N>h, <N>d, for example 90s, 30m, 1h, 7d	Each commit of the last N seconds, minutes, hours or days.

The case and the spaces around the value do not matter: 100 commits and 1H are correct. A bad value stops the open with an InvalidInput error. The open then creates no directory:

history_retention '1x': unrecognised spec. Want <N>commits (e.g. 100commits) or <N><s|m|h|d> (e.g. 90s, 30m, 1h, 7d). Omit the option to keep all history.

What the database does with the option

• The database starts one background thread, kinedb-history-purge. The thread looks at the store every 15 seconds. It removes old blocks when the store grew by 1/8 since the last removal, or when no removal ran since the open. A failed removal waits longer before the next try. The server uses the same rules.
• The close of the database (the drop of the Db) stops the thread and waits for it.
• A read of an older commit (AS OF) inside the kept history gives the rows of that commit. A read of a commit outside the kept history fails with an error that contains beyond retention horizon.
• A synced database (kinedb-sync) keeps each write that it did not push yet, and the data of each file that it wrote offline, until the push.

The clock check of a time value

A time value (7d, 1h) reads the clock of the computer: the database removes the commits that are older than the clock minus the time. A wrong clock can remove history that you want to keep. So before each removal, the database compares the clock with the time of the newest commit (HEAD). It does not remove anything when:

• the clock is more than 24 hours after the time of HEAD;
• the clock is before the time of HEAD;
• HEAD has no time.

The database writes one log line when the check starts to refuse, and one when it passes again. A <N>commits value reads no clock, so it has no check.

Limits of the check:

• After the first commit at a wrong clock, HEAD has the wrong time, and the check passes. The check protects only the time before that commit.
• An application that writes nothing for more than 24 hours removes nothing until its next commit.

Where the option works

• Native Rust only: Db::open_with_options on Linux, macOS and Windows.
• The JavaScript and Python packages and the WebAssembly engine do not pass the option. A database that they open keeps all history.
• On a WebAssembly target (wasm32-unknown-unknown, wasm32-wasip1), Db::open_with_options refuses the option: the target has no thread to run the removal.
• Db::open_in_memory takes no options.

Note for code that uses Db::engine_mut: after an open with the option, QueryEngine::init_db returns false, because the purge thread holds a reference to the database.