Run a Standalone Garbage Collector

Learn how to run garbage collection as a separate process from your database clients

By default, SlateDB starts a background garbage collector inside each Db instance. You can also run garbage collection as a separate service:

Keep your existing Db clients running.
Start one or more standalone garbage collectors against the same database path.

All processes must use the same ObjectStore and database path (the path is the prefix inside your object store).

Unlike the compactor coordinator, the garbage collector does not require fencing or leader election. Multiple garbage collectors may run in parallel, including the embedded collectors inside writer processes.

If you want a standalone process to own all garbage collection work, set Settings::garbage_collector_options to None in your Db clients.

Run from the CLI

For a single foreground pass over one file type, use run-garbage-collection:

slatedb --env-file .env --path <db-path> run-garbage-collection \
  --resource compacted \
  --min-age 5m

Valid resources are manifest, wal, wal-fence, compacted, and compactions.

To run garbage collection on a schedule in a standalone process, pass one or more schedules:

slatedb --env-file .env --path <db-path> schedule-garbage-collection \
  --manifest min_age=5m,period=1m \
  --wal min_age=5m,period=1m \
  --compacted min_age=5m,period=1m \
  --compactions min_age=5m,period=1m

The scheduled process runs until interrupted (Ctrl-C), then shuts down gracefully.

The command above does not collect WAL fence objects. Enable --wal-fence only when min_age is longer than any writer can run. Default garbage collection keeps WAL fence deletion in dry-run mode because deleting fences too early can allow stale WAL writes to succeed.

Embed the garbage collector

You can also embed the garbage collector in your own process using GarbageCollectorBuilder and call run:

use slatedb::object_store::local::LocalFileSystem;
use slatedb::GarbageCollectorBuilder;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let local_root = std::env::temp_dir().join("slatedb-standalone-gc-tutorial");
    std::fs::create_dir_all(&local_root)?;
    let object_store = Arc::new(LocalFileSystem::new_with_prefix(local_root)?);

    // Build a garbage collector without opening a Db. The database must already
    // exist (i.e. the manifest has been created).
    let gc = Arc::new(GarbageCollectorBuilder::new("db", object_store).build());

    let gc_task = {
        let gc = Arc::clone(&gc);
        tokio::spawn(async move { gc.run().await })
    };

    println!("Garbage collector running. Press Ctrl-C to stop.");
    tokio::signal::ctrl_c().await?;

    gc.stop().await?;
    gc_task.await??;

    Ok(())
}

With GarbageCollectorOptions::default(), garbage collection runs every 60 seconds and uses a 5 minute minimum age for managed directories. WAL fence deletion stays in dry-run mode by default.