Compaction

The compactor is responsible for taking groups of sorted runs (this doc uses the term sorted run to refer to both sorted runs and l0 ssts) and compacting them together to reduce space amplification (by removing old versions of rows that have been updated/deleted) and read amplification (by reducing the number of sorted runs that need to be searched on a read). It’s made up of a few different components:

[Compactor]: The main event loop that orchestrates the compaction process.
[CompactorEventHandler]: The event handler that handles events from the compactor.
[CompactionScheduler]: The scheduler that discovers compactions that should be performed.
[CompactionExecutor]: The executor that runs the compaction tasks.

The main event loop listens on the manifest poll ticker to react to manifest poll ticks, the executor worker channel to react to updates about running compactions, and the shutdown channel to discover when it should terminate. It doesn’t actually implement the logic for reacting to these events. This is implemented by [CompactorEventHandler].

The Scheduler is responsible for deciding what sorted runs should be compacted together. It implements the [CompactionScheduler] trait. The implementation is specified by providing an implementation of [CompactionSchedulerSupplier] so different scheduling policies can be plugged into slatedb. Currently, the only implemented policy is the size-tiered scheduler supplied by SizeTieredCompactionSchedulerSupplier.

The Executor does the actual work of compacting sorted runs by sort-merging them into a new sorted run. It implements the [CompactionExecutor] trait. Currently, the only implementation is the TokioCompactionExecutor, which runs compaction on a local tokio runtime.

Trivial moves

When every input SST has a non-overlapping effective key range, the compactor can reuse the existing SSTs in the destination sorted run. The coordinator completes this trivial move itself, without dispatching the job to an executor or reading and rewriting SST data. This reduces compaction I/O while still producing a sorted run of non-overlapping SSTs.

Trivial moves are disabled by default. Set CompactorOptions::enable_trivial_move to true to enable them.

To split data with different read/write profiles into independent LSM trees — each with its own compaction policy — and to retire whole key ranges cheaply, see Segmented Compaction.

Incremental compaction

A large compaction can run for a long time. Without incremental compaction, readers keep using the input SSTs until the compaction finishes, and garbage collection keeps those SSTs in storage. With incremental compaction, the compactor publishes the output that a running compaction already wrote. Readers then use that output for its keys. Storage can be freed earlier, and reads of those keys search fewer SSTs.

Incremental compaction is disabled by default. Set CompactorOptions::enable_experimental_incremental_compaction to true to enable it:

let options = slatedb::config::CompactorOptions {
    enable_experimental_incremental_compaction: true,
    ..Default::default()
};

The compactor publishes this output in incremental commits, at most once a minute. The commit of a finished compaction does not wait. Incremental commits apply to compactions that merge sorted runs. A compaction with L0 SSTs in its inputs publishes its output only when it finishes.

Each incremental commit is a manifest update. An SST can have several views, and a view limits the keys that readers can see in an SST. The commit replaces the input views for the keys that the written output covers with views of the output SSTs. The input keys that the output does not cover yet stay in the input views.

Before each manifest update, the compactor writes a checkpoint, a reference that keeps an older manifest available. Garbage collection deletes an input SST only after no view uses it and the checkpoint expires, so reads that started before the commit can still finish. CompactorOptions::checkpoint_lifetime sets how long the checkpoint lasts.

The counter slatedb.compactor.incremental_released_bytes estimates the bytes of the input SSTs that incremental commits release. It counts an SST once, after a successful commit removes its last view from all trees. A released SST is not deleted yet. It stays until garbage collection deletes it, so storage use goes down later than this counter goes up.

If a worker stops, the next worker resumes the compaction from its progress. Key ranges that are already complete do not run again.