Garbage Collection

SlateDB’s garbage collector runs as a background task in the client process, periodically checking for obsolete files in the database storage.

The garbage collector has a configurable minimum age and interval for each file type (WAL SSTs, WAL fence SSTs, compacted SSTs, manifests, and compactions). Garbage collection for a file type can be disabled by setting its options to None. The collector runs every interval seconds and will delete files older than min_age that are not referenced by any active manifest or checkpoint.

Each garbage collection directory type supports a dry_run option. When enabled, the collector logs files that would be deleted without actually deleting them. This is useful for testing or verifying garbage collection behavior before enabling actual deletion.

Boundary files

Before deleting old manifest or compactions metadata, SlateDB normally advances a durable boundary file. Metadata writers check that boundary after creating a new version, which prevents a stale writer from successfully publishing metadata that GC has already passed.

Boundary advancement can be disabled with GarbageCollectorOptions::boundary_files_enabled. This supports object stores without conditional overwrite (If-Match) while allowing GC to continue deleting eligible metadata. Metadata readers and writers still check any existing boundary; on a new database, no boundary file is created while all garbage collectors use this mode. Without a boundary, a SlateDB client or compactor can begin updating a manifest or compactions file, stop making progress (for example, because its process or host is suspended), then resume after min_age. GC may have deleted the ID it intended to create, so create-if-absent can reuse that ID and report a stale update as successful even though newer metadata has superseded it. Every garbage collector must use a compatible setting, and the manifest and compactions min_age values must exceed the maximum lifetime of a stale process.

Filtering Deletion Candidates

SlateDB supports custom filtering of garbage collection candidates through the GcFilter trait. This allows users to intercept files before deletion and approve or reject them based on custom logic.

The GcFilter::filter method receives a set of files that SlateDB has already determined are eligible for deletion (files older than min_age that are not referenced by any active manifest or checkpoint). The filter returns the subset of files that may be physically deleted. Any files not returned by the filter are retained.

A common use case for GcFilter is verifying that files have been replicated to another bucket before allowing deletion. For example, in a mirror-maker-like scenario for cross-region or cross-cloud replication, the filter can check that a file exists in the destination bucket before approving it for deletion from the source.

To configure a filter, use the with_gc_filter() method on GarbageCollectorBuilder:

let gc_filter = Arc::new(MyGcFilter::new());
let gc = GarbageCollectorBuilder::new(path, object_store)
    .with_gc_filter(gc_filter)
    .build()
    .await?;

The filter is optional. If no filter is configured, garbage collection proceeds normally based on age and reference checks alone.

Below is a diagram illustrating the high-level flow of garbage collection in SlateDB:

flowchart TD

    A["Start GC Cycle (interval timer)"] --> B[Remove Expired Checkpoints from Manifests]

    B --> C[Run WAL SST GC Task]
    C --> C1[List WAL SSTs older than last compacted ID]
    C1 --> C2[Filter by min_age and active references]
    C2 --> C2a[Apply GcFilter if configured]
    C2a --> C3[Delete eligible WAL SSTs]

    B --> D[Run Compacted SST GC Task]
    D --> D1[List all compacted and L0 SSTs]
    D1 --> D2[Gather active SST IDs from manifests]
    D2 --> D2a[Apply GcFilter if configured]
    D2a --> D3[Delete SSTs not referenced and older than min_age]

    B --> E[Run Manifest GC Task]
    E --> E1["List all manifests (exclude latest)"]
    E1 --> E2[Gather active manifest IDs from checkpoints]
    E2 --> E2a[Apply GcFilter if configured]
    E2a --> E3[Delete manifests not referenced and older than min_age]

    C3 & D3 & E3 --> G[Wait for Next Interval or Shutdown]

Default Garbage Collection Behavior

By default, garbage collection is enabled for all managed directories (manifest, WAL, WAL fence, compacted SSTs, and compactions) using standard interval and minimum age settings.

WAL fence garbage collection runs in dry-run mode by default. This means it logs files that would be deleted without actually deleting them. This conservative default prevents accidental data loss while still providing visibility into what would be cleaned up.

To enable actual deletion for WAL fence GC, set dry_run: false with a high min_age to safely clean up old fences. Alternatively, to silence the dry-run logging entirely, set wal_fence_options: None.