This is the full developer documentation for SlateDB
# Block Transforms
> How SlateDB transforms SST blocks before it writes them to object storage
[`BlockTransformer`](https://docs.rs/slatedb/latest/slatedb/trait.BlockTransformer.html) lets SlateDB apply a reversible byte transform to SST blocks, usually for encryption. SlateDB applies it to data blocks and to filter, index, and stats blocks in both WAL SSTs and compacted SSTs.
SlateDB does not transform manifest files, compaction-state files, or the trailing [`SsTableInfo`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/db_state.rs) record and final offset/version footer in an SST. It reads that footer first so it can find the stored blocks and learn which compression codec to use.
## Order
[Section titled “Order”](#order)
On write, SlateDB encodes the block, optionally compresses it, passes the bytes to [`BlockTransformer::encode`](https://docs.rs/slatedb/latest/slatedb/trait.BlockTransformer.html#tymethod.encode), computes CRC32 over the transformed bytes, and writes the result to object storage.
On read, SlateDB fetches the stored bytes, verifies the CRC32, calls [`BlockTransformer::decode`](https://docs.rs/slatedb/latest/slatedb/trait.BlockTransformer.html#tymethod.decode), optionally decompresses the decoded bytes, and then decodes the block contents.
The checksum covers the transformed payload, not the original plaintext block. Corruption in storage is detected before SlateDB calls `decode`. A mismatched transformer usually fails later, during `decode`, decompression, or block decoding.
If you combine a transformer with [Compression](/docs/design/compression), the transformer sees compressed bytes, not raw row encodings.
## Contract
[Section titled “Contract”](#contract)
The trait has two async methods, [`encode`](https://docs.rs/slatedb/latest/slatedb/trait.BlockTransformer.html#tymethod.encode) and [`decode`](https://docs.rs/slatedb/latest/slatedb/trait.BlockTransformer.html#tymethod.decode). `decode` must invert `encode` for every block SlateDB writes. The methods are async so implementations can offload CPU-heavy work or call an external service.
SlateDB does not record transformer identity, key ID, or format version in `SsTableInfo`. If you need key rotation or format changes, make the stored block self-describing or keep compatible configuration outside SlateDB.
## Configuration
[Section titled “Configuration”](#configuration)
Every process that reads or writes SST blocks needs compatible transformer logic.
* Writers use [`DbBuilder::with_block_transformer`](https://docs.rs/slatedb/latest/slatedb/struct.DbBuilder.html#method.with_block_transformer).
* Standalone compactors use [`CompactorBuilder::with_block_transformer`](https://docs.rs/slatedb/latest/slatedb/struct.CompactorBuilder.html#method.with_block_transformer).
* Read-only handles use [`DbReaderBuilder::with_block_transformer`](https://docs.rs/slatedb/latest/slatedb/struct.DbReaderBuilder.html#method.with_block_transformer).
* SST inspection uses [`SstReader::new`](https://docs.rs/slatedb/latest/slatedb/struct.SstReader.html#method.new).
If a writer used a transformer and a reader does not, SlateDB can still open the footer metadata, but reads of the filter, index, stats, or data blocks fail. [Readers](/docs/design/readers) covers the reader-side configuration in more detail.
[`WalReader`](https://docs.rs/slatedb/latest/slatedb/struct.WalReader.html) does not expose a `BlockTransformer` setting today. WAL SSTs written with a custom transformer are not inspectable through that API.
# Caching
> How SlateDB caches SST data and object-store reads
SlateDB has two separate cache layers for SST reads. The block cache stores decoded SST blocks and metadata. The object-store cache stores raw bytes fetched from the object store. You can use either layer or both.
## Block Cache
[Section titled “Block Cache”](#block-cache)
The block cache sits in the SST reader. It stores decoded data blocks, index blocks, filter blocks, and stats blocks. A hit here avoids the remote `GET`, and it also avoids decoding the block again.
[`DbBuilder`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html) installs a [`SplitCache`](https://docs.rs/slatedb/latest/slatedb/db_cache/struct.SplitCache.html) by default. `SplitCache` keeps data blocks separate from SST metadata so large reads are less likely to evict indexes and Bloom filters.
You can replace the cache with [`DbBuilder::with_db_cache`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_db_cache), or disable it with [`DbBuilder::with_db_cache_disabled`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_db_cache_disabled). If you want to plug in your own implementation, use the [`DbCache`](https://docs.rs/slatedb/latest/slatedb/db_cache/trait.DbCache.html) trait.
[`FoyerHybridCache`](https://docs.rs/slatedb/latest/slatedb/db_cache/foyer_hybrid/struct.FoyerHybridCache.html) also belongs in this layer. It is a `DbCache` implementation, so it caches decoded SST entries, not raw object-store bytes. Its memory tier works like any other database cache. Its disk tier can still avoid remote I/O and decode work, but it adds a local disk read on a miss in memory. See [Configuring Foyer Cache](/docs/operations/foyer-cache) for tuning guidance.
## Object-Store Cache
[Section titled “Object-Store Cache”](#object-store-cache)
[`CachedObjectStore`](https://docs.rs/slatedb/latest/slatedb/cached_object_store/struct.CachedObjectStore.html) is a second cache layer for raw object-store bytes. It splits each object into fixed-size parts, stores those parts under a local root folder, and can serve later `GET` and `HEAD` requests from those local files when the needed parts are already present.
There are two ways to enable it. The first is configuration: set [`root_folder`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.root_folder) in [`ObjectStoreCacheOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html), and SlateDB builds the cache from those options and wraps the object store you passed to the builder in it. This is the only way to enable the cache from a settings file.
The second is to build the cache yourself with [`CachedObjectStore::builder`](https://docs.rs/slatedb/latest/slatedb/cached_object_store/struct.CachedObjectStore.html#method.builder) and pass it to SlateDB as the object store:
```rust
let cache = CachedObjectStore::builder("/var/slatedb-cache", object_store)
.with_cache_on_flush(true)
.with_cache_on_compaction(true)
.with_max_cache_size_bytes(Some(16 * 1024 * 1024 * 1024))
.build()
.await?;
let db = Db::builder(path, cache).build().await?;
```
Building it yourself lets you hold on to the cache instance, share it across several `Db` and `DbReader` instances in the same process, and warm it directly. Leave [`root_folder`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.root_folder) unset when you do this, otherwise SlateDB wraps your cache in a second one.
Either way the cache ends up as the innermost layer, closest to the object store you provided. SlateDB adds its own retry and metrics handling above it, so a read served from the cache is still counted as an object-store request. See [Metrics](/docs/operations/metrics) for what that means for the `slatedb.object_store.*` metrics.
This cache stores object-store bytes, not decoded SST blocks. It helps when the block cache is cold because it can avoid a remote read even though SlateDB still needs to read from local disk and decode the block afterward. On a miss, SlateDB aligns the requested range to the configured part size, fetches that larger range from the object store, and saves the returned parts locally. The default part size is 4 MiB.
The built-in object-store cache is disk-backed today. The root folder is a filesystem path, and the current implementation stores parts under that directory. If you want an in-memory cache, use the block cache layer instead.
## How Caches Get Filled
[Section titled “How Caches Get Filled”](#how-caches-get-filled)
Reads consult the block cache first. On a miss there, SlateDB fetches bytes through the object-store layer, which may itself hit the object-store cache before going remote.
[`ReadOptions::cache_blocks`](https://docs.rs/slatedb/latest/slatedb/config/struct.ReadOptions.html#structfield.cache_blocks) and [`ScanOptions::cache_blocks`](https://docs.rs/slatedb/latest/slatedb/config/struct.ScanOptions.html#structfield.cache_blocks) decide whether SlateDB inserts a decoded data block after fetching it on a miss. They do not disable cache lookups. Reads and scans cache SST metadata, such as indexes, filters, and stats, independently of `cache_blocks`.
The defaults reflect the usual access patterns. Point reads default to `cache_blocks = true` because they are more likely to revisit hot data. Scans default to `cache_blocks = false` so a long sequential read does not fill the cache with data blocks that probably will not be reused soon. Scans can still benefit from entries that are already hot. Internal tasks follow the same idea: WAL replay and compaction read SSTs without populating the foreground cache.
Writes fill the block cache as well. [`BlockCachePolicy`](https://docs.rs/slatedb/latest/slatedb/struct.BlockCachePolicy.html) selects which components of an SST are inserted as that SST is written, and you install it with [`DbBuilder::with_block_cache_policy`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_block_cache_policy). The policy applies to the block cache only. Admission into the object-store cache is controlled separately, as described below.
The policy holds one target list per write source. The default inserts data blocks, the index, and the filters after a memtable flush, and inserts only the index and the filters as compaction output is written, so compaction does not evict blocks that reads have made hot. Replace either list with [`BlockCachePolicy::with_flush_targets`](https://docs.rs/slatedb/latest/slatedb/struct.BlockCachePolicy.html#method.with_flush_targets) or [`BlockCachePolicy::with_compaction_output_targets`](https://docs.rs/slatedb/latest/slatedb/struct.BlockCachePolicy.html#method.with_compaction_output_targets). An empty list disables insertion for that source.
Each list holds [`CacheTarget`](https://docs.rs/slatedb/latest/slatedb/db_cache/enum.CacheTarget.html) values:
* `CacheTarget::Filters` inserts the SST filter blocks, if any exist.
* `CacheTarget::Index` inserts the SST index.
* `CacheTarget::Stats` inserts the SST stats block, if one exists.
* `CacheTarget::data(range)` inserts the data blocks whose key span overlaps `range`.
`CacheTarget::data` accepts any key range, so `CacheTarget::data::<&[u8], _>(..)` selects every data block, while `CacheTarget::data(b"user:".as_slice()..b"user;".as_slice())` selects only the blocks covering that key range. This policy caches all four components after a flush and nothing after compaction:
```rust
use slatedb::{BlockCachePolicy, CacheTarget, Db};
let policy = BlockCachePolicy::default()
.with_flush_targets(&[
CacheTarget::data::<&[u8], _>(..),
CacheTarget::Index,
CacheTarget::Filters,
CacheTarget::Stats,
])
.with_compaction_output_targets(&[]);
let db = Db::builder(path, object_store)
.with_block_cache_policy(policy)
.build()
.await?;
```
The disk cache stays disabled unless you enable it, either by setting `object_store_cache_options.root_folder` or by passing your own `CachedObjectStore` to the builder.
By default, writes go straight to the upstream object store and do not populate the object-store cache. Each SST write source has its own admission flag: [`cache_on_flush`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.cache_on_flush) stores SSTs written by memtable flushes locally, and [`cache_on_compaction`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.cache_on_compaction) does the same for compaction output. The builder exposes the same two flags as [`with_cache_on_flush`](https://docs.rs/slatedb/latest/slatedb/cached_object_store/struct.CachedObjectStoreBuilder.html#method.with_cache_on_flush) and [`with_cache_on_compaction`](https://docs.rs/slatedb/latest/slatedb/cached_object_store/struct.CachedObjectStoreBuilder.html#method.with_cache_on_compaction). Enabling them can help if readers are likely to touch freshly written SSTs soon afterward. WAL and manifest writes are never cached.
## Warming the Object-Store Cache
[Section titled “Warming the Object-Store Cache”](#warming-the-object-store-cache)
If you want the disk cache warm before serving traffic, and you configured it through [`ObjectStoreCacheOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html), set [`preload_disk_cache_on_startup`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.preload_disk_cache_on_startup) to [`PreloadLevel::L0Sst`](https://docs.rs/slatedb/latest/slatedb/config/enum.PreloadLevel.html#variant.L0Sst) or [`PreloadLevel::AllSst`](https://docs.rs/slatedb/latest/slatedb/config/enum.PreloadLevel.html#variant.AllSst). `Db` and `DbReader` load recent SSTs, or all SSTs, into the local cache on open until the cache size limit is reached.
That setting only applies to a cache SlateDB built for you. A cache you built yourself is warmed by you. Read the current SST set from [`DbMetadataOps::manifest`](https://docs.rs/slatedb/latest/slatedb/ops/trait.DbMetadataOps.html#tymethod.manifest), resolve each SST id to an object-store path with [`PathResolver::sst_path`](https://docs.rs/slatedb/latest/slatedb/struct.PathResolver.html#method.sst_path), and pass the paths to [`CachedObjectStore::load_files_to_cache`](https://docs.rs/slatedb/latest/slatedb/cached_object_store/struct.CachedObjectStore.html#method.load_files_to_cache). Construct the resolver with [`PathResolver::new`](https://docs.rs/slatedb/latest/slatedb/struct.PathResolver.html#method.new), which takes the manifest as well as the database path because a manifest can reference SSTs owned by another database after a clone. The `max_bytes` budget is applied in path order, so order paths by priority. Fetches are best-effort, and failures are logged and skipped.
## Sharing Between Instances
[Section titled “Sharing Between Instances”](#sharing-between-instances)
The two cache layers behave differently when you open multiple [`Db`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html) or [`DbReader`](https://docs.rs/slatedb/latest/slatedb/struct.DbReader.html) instances against the same cache.
For the object-store cache, sharing is straightforward. If multiple instances on the same machine use the same root folder, whether they set [`root_folder`](https://docs.rs/slatedb/latest/slatedb/config/struct.ObjectStoreCacheOptions.html#structfield.root_folder) or pass it to `CachedObjectStore::builder`, they reuse the same local cache directory. Instances in the same process can also share one `CachedObjectStore` instance directly. For the same database path, that lets one instance benefit from parts fetched by another. When you use [`Db::resolve_object_store`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.resolve_object_store) and provide different path arguments to builder methods like [`DbBuilder::new`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.new), SlateDB keeps the cached files under different path prefixes, so they do not clobber one another. Using the same part size gives the best reuse. Different part sizes can coexist, but they will not reuse the same cached part files.
If you provide an object store directly with `PrefixStore` or other custom wrapping instead of using `Db::resolve_object_store`, you must configure different root folder values for each instance to prevent cache collisions. SlateDB no longer automatically resolves root prefixes from metadata locations.
The block cache is different. Both [`DbBuilder::with_db_cache`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_db_cache) and [`DbReaderBuilder::with_db_cache`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbReaderBuilder.html#method.with_db_cache) let you pass in your own cache object, so you can choose to reuse the same process-local cache implementation across builders. For `Db`, that mainly gives you a shared memory or disk budget, not shared hits: SlateDB scopes each instance’s entries so one `Db` does not read another `Db`’s cached blocks by accident. If your main goal is cross-instance warming, the object-store cache is the better fit.
A cache you pass in stays owned by you: `Db::close()` and `DbReader::close()` never close it, so closing one instance does not affect the others sharing it. Once every instance using the cache is closed, call [`DbCache::close`](https://docs.rs/slatedb/latest/slatedb/db_cache/trait.DbCache.html#method.close) yourself so hybrid implementations can flush their memory tier to disk.
For `DbReader`, treat a caller-managed block cache as an advanced optimization. It can make sense to keep one reader-side cache per database in a single process, but if you want a cache that is naturally shared between writers and readers, or across many short-lived instances, the object-store cache is the simpler mechanism to share.
## Tradeoffs
[Section titled “Tradeoffs”](#tradeoffs)
The block cache saves decode work and can also save remote I/O. With a memory-only implementation, it avoids both. With [`FoyerHybridCache`](https://docs.rs/slatedb/latest/slatedb/db_cache/foyer_hybrid/struct.FoyerHybridCache.html), a hit in the disk tier still avoids remote I/O and re-decoding, but it adds a local disk read.
The object-store cache only saves the remote I/O. SlateDB still has to read from local disk and decode the block. Many deployments use both layers: a block cache for the hot decoded working set, and an object-store cache for colder bytes and range-aligned fetches. Block size matters here too, because it controls cache granularity and therefore affects hit rate and read amplification. [Tuning](/docs/operations/tuning) covers that knob.
If you enable [`FoyerHybridCache`](https://docs.rs/slatedb/latest/slatedb/db_cache/foyer_hybrid/struct.FoyerHybridCache.html) and also enable the object-store cache, SlateDB may store the same SST data twice on local disk. That can still be a reasonable trade if the object-store cache’s range prefetching saves enough remote reads, but the cost is more local storage traffic.
# Change Data Capture
> Stream SlateDB WAL files with a live SlateDbWalReader iterator
Change data capture (CDC) turns writes in SlateDB into a durable change stream that external systems can consume. SlateDB exposes native-WAL CDC through **slatedb::wal::SlateDbWalReader**.
The reader opens a live iterator from the first unconsumed WAL file ID. The iterator owns polling and waits internally when it reaches the current WAL tail; the caller owns durable cursor storage.
## When to use WAL-based CDC
[Section titled “When to use WAL-based CDC”](#when-to-use-wal-based-cdc)
Use WAL-based CDC when you need:
* A low-latency stream of inserts, updates, and deletes.
* A durable append-only feed that can be replayed.
* Integration with downstream systems such as Kafka, Pulsar, or data lakes.
WAL-based CDC is not a bootstrap or backfill API. Combine a database backfill with WAL streaming when an initial full copy is required.
## Streaming model
[Section titled “Streaming model”](#streaming-model)
Keep a durable cursor containing the last fully consumed WAL file ID. Use zero when starting from the beginning unless you have a more appropriate manifest-derived cursor.
To consume the stream:
1. Compute **cursor + 1**, checking for overflow, and create one iterator with an unbounded end range beginning at that WAL file ID.
2. Repeatedly call **next()**. When the iterator reaches the current tail, the call waits and polls internally for the next WAL file.
3. Emit every row in each **WalRows** batch.
4. After the whole batch succeeds, persist its **last\_consumed\_wal\_file\_id** as the new cursor.
5. After a restart, create a new unbounded iterator beginning at the persisted cursor plus one.
The live iterator does not report exhaustion at the current tail; **next()** remains pending until a new WAL file is visible or an error occurs. **last\_wal\_file\_id** remains available when an application needs a point-in-time tail snapshot, but it is not part of the streaming loop.
## Batches and cursor progress
[Section titled “Batches and cursor progress”](#batches-and-cursor-progress)
One **WalRows** batch represents one fully consumed WAL file and contains:
* **rows**: the file’s row entries, in sequence order
* **last\_consumed\_wal\_file\_id**: the durable cursor after that file
Empty fence WAL files produce a batch with no rows. The cursor still advances, so consumers must process the batch rather than flattening batches into rows and losing progress information.
Each row is a **RowEntry** with:
* **key**: key bytes
* **value**: a **ValueDeletable** value, merge operand, or tombstone
* **seq**: the globally increasing sequence number
* **create\_ts** and **expire\_ts**: optional timestamps
Deletes appear as tombstones. Merge operands remain raw merge values; downstream consumers must apply the merge logic themselves when materialized values are required.
## Visibility and durability
[Section titled “Visibility and durability”](#visibility-and-durability)
SlateDbWalReader only sees WAL SSTs flushed to object storage. WAL flushes occur when:
* The WAL buffer reaches its size threshold.
* The configured flush interval elapses.
* The caller explicitly requests a WAL or memtable flush.
A memtable flush first forces pending WAL data to durable storage.
The database must have its WAL enabled. WAL support is enabled by default.
## Basic live stream
[Section titled “Basic live stream”](#basic-live-stream)
This example creates one unbounded iterator, emits both existing and later writes through it, and advances a durable file cursor.
main.rs
```rust
use slatedb::config::{FlushOptions, FlushType};
use slatedb::object_store::{memory::InMemory, path::Path};
use slatedb::wal::{SlateDbWalReaderBuilder, WalReader as _, WalRows};
use slatedb::{Db, RowEntry, ValueDeletable};
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let object_store = Arc::new(InMemory::new());
let path = "/change-data-capture-example";
let db = Db::open(path, object_store.clone()).await?;
db.put(b"user:1", b"alice").await?;
db.put(b"user:2", b"bob").await?;
db.delete(b"user:2").await?;
flush_wal(&db).await?;
let wal_reader = SlateDbWalReaderBuilder::new()
.with_object_store(object_store)
.with_path(Path::from(path))
.build()?;
let mut cursor = 0_u64;
let start_wal_id = cursor
.checked_add(1)
.ok_or_else(|| anyhow::anyhow!("WAL cursor cannot advance"))?;
let mut iterator = wal_reader.iterator((start_wal_id..).into()).await?;
// Drain the writes that already exist. Empty fence WALs still advance the
// cursor, so stop based on emitted rows only after persisting every batch.
let mut emitted_rows = 0;
while emitted_rows < 3 {
let batch = iterator
.next()
.await?
.ok_or_else(|| anyhow::anyhow!("live WAL iterator ended unexpectedly"))?;
emitted_rows += emit_batch(&batch, &mut cursor);
}
// Keep the same iterator alive. Its next call observes this later WAL;
// callers do not need to discover a new tail or create another iterator.
db.put(b"user:3", b"carol").await?;
flush_wal(&db).await?;
while emitted_rows < 4 {
let batch = iterator
.next()
.await?
.ok_or_else(|| anyhow::anyhow!("live WAL iterator ended unexpectedly"))?;
emitted_rows += emit_batch(&batch, &mut cursor);
}
db.close().await?;
Ok(())
}
async fn flush_wal(db: &Db) -> Result<(), slatedb::Error> {
db.flush_with_options(FlushOptions {
flush_type: FlushType::Wal,
})
.await
}
fn emit_batch(batch: &WalRows, cursor: &mut u64) -> usize {
for row in &batch.rows {
emit_row(batch.last_consumed_wal_file_id, row);
}
// Persist only after every row in this WAL file has been emitted. This
// also advances across empty fence WALs.
*cursor = batch.last_consumed_wal_file_id;
println!("persist cursor={cursor}");
batch.rows.len()
}
fn emit_row(wal_id: u64, row: &RowEntry) {
let key = String::from_utf8_lossy(row.key.as_ref());
match &row.value {
ValueDeletable::Value(value) => {
let value = String::from_utf8_lossy(value.as_ref());
println!("wal_id={wal_id} seq={} upsert {key}={value}", row.seq);
}
ValueDeletable::Merge(value) => {
let value = String::from_utf8_lossy(value.as_ref());
println!("wal_id={wal_id} seq={} merge {key}+={value}", row.seq);
}
ValueDeletable::Tombstone => {
println!("wal_id={wal_id} seq={} delete {key}", row.seq);
}
}
}
```
Persisting the cursor only after a complete batch gives at-least-once delivery if a process fails while emitting that WAL file. Make the sink idempotent or transactionally couple sink writes with cursor persistence when duplicates are unacceptable.
## Ordering and resumability
[Section titled “Ordering and resumability”](#ordering-and-resumability)
WAL file IDs increase monotonically, and entries within each file are returned in sequence order. The minimal cursor is therefore the last fully consumed WAL file ID.
Do not advance the cursor per row. Doing so can lose progress for empty fence WALs and can incorrectly mark a partially emitted WAL file as complete.
If a requested file has already been garbage collected, iteration reports a truncation/data error. The consumer must not silently skip the gap.
## Backfills plus streaming
[Section titled “Backfills plus streaming”](#backfills-plus-streaming)
For a full backfill:
1. Begin live WAL streaming and buffer changes.
2. Create a detached database checkpoint.
3. Scan the checkpoint or a clone for existing data.
4. Apply buffered WAL changes in sequence order.
5. Delete the checkpoint.
6. Continue streaming from the persisted WAL cursor.
The sink should tolerate overlap between the backfill and WAL stream.
## WAL retention and GC
[Section titled “WAL retention and GC”](#wal-retention-and-gc)
WAL SSTs are garbage collected according to the configured retention policy. Set WAL GC minimum age long enough for the slowest consumer and monitor cursor lag. A consumer that requests a deleted WAL receives a truncation error instead of silently continuing.
## Dedicated WAL object stores
[Section titled “Dedicated WAL object stores”](#dedicated-wal-object-stores)
When a database uses a dedicated WAL object store, construct the reader with both stores: the main object store is used for manifest and GC-cutoff state, and the dedicated WAL object store is used for WAL SSTs. In Rust, use **SlateDbWalReader::new\_for\_db\_with\_wal\_object\_store**.
# Checkpoints
> How SlateDB pins a manifest version for reads and garbage collection
Checkpoints give SlateDB a cheap way to hold onto one stable view of the database while the live database keeps moving. They are useful for long-running reads, read-only clients that need a fixed view, [clones](/docs/design/clones), and administrative workflows that must keep old SSTs and manifests alive while compaction and garbage collection continue in the background.
A checkpoint is a durable reference to one manifest version. SlateDB stores checkpoints in the latest manifest. Each checkpoint records an ID, the pinned `manifest_id`, its creation time, an optional expiration time, and an optional name.
Creating a checkpoint does not copy WAL or SST files. It only adds metadata to the manifest, so it is cheap. The pinned manifest still matters, though, because it defines the L0 SSTs, sorted runs, and other database state that readers see.
Note
A checkpoint is not the same thing as a [`DbSnapshot`](https://docs.rs/slatedb/latest/slatedb/struct.DbSnapshot.html). A checkpoint is durable manifest metadata. It survives restarts and pins manifests and SSTs so garbage collection cannot remove them. A `DbSnapshot` is an in-process read view at a sequence number. It gives one running client a stable view, but it is not stored in the manifest and does not keep old manifests or SSTs alive by itself. See [Consistency](/docs/design/consistency) for more on snapshots.
## What It Protects
[Section titled “What It Protects”](#what-it-protects)
The garbage collector keeps any manifest and SST file that is still reachable from an active checkpoint. That is how SlateDB holds a stable read view open while compaction and garbage collection continue in the background.
When a checkpoint expires or is deleted, SlateDB removes that checkpoint entry from the manifest. After that, GC may reclaim manifests and SSTs that are no longer referenced by the latest database state or by any other checkpoint. [Garbage Collection](/docs/design/gc) covers the cleanup flow.
A new checkpoint can also be created from an existing checkpoint through [`CheckpointOptions::source`](https://docs.rs/slatedb/latest/slatedb/config/struct.CheckpointOptions.html#structfield.source). In that case, the new checkpoint reuses the source checkpoint’s `manifest_id`. This is useful when you want a different lifetime or name without pinning newer database state.
## How SlateDB Creates Checkpoints
[Section titled “How SlateDB Creates Checkpoints”](#how-slatedb-creates-checkpoints)
[`Db::create_checkpoint`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.create_checkpoint) creates a checkpoint from an open writer handle. [`CheckpointScope::All`](https://docs.rs/slatedb/latest/slatedb/config/enum.CheckpointScope.html#variant.All) flushes WAL data and memtables first, so the checkpoint includes writes issued before the call. [`CheckpointScope::Durable`](https://docs.rs/slatedb/latest/slatedb/config/enum.CheckpointScope.html#variant.Durable) skips that extra flushing work and only includes data that was already durable.
[`Admin::create_detached_checkpoint`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html#method.create_detached_checkpoint) edits the manifest directly. It does not flush a writer’s in-memory state first, so recent unflushed writes are not included.
[`CheckpointOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.CheckpointOptions.html) lets you set a lifetime, a source checkpoint, and an optional name. Names are labels for listing checkpoints. They are not unique.
## Readers and Background Tasks
[Section titled “Readers and Background Tasks”](#readers-and-background-tasks)
[`DbReader`](https://docs.rs/slatedb/latest/slatedb/struct.DbReader.html) uses checkpoints to separate reader lifetime from writer lifetime. You choose how the reader tracks database state by passing a [`DbReaderMode`](https://docs.rs/slatedb/latest/slatedb/enum.DbReaderMode.html):
* `ManagedCheckpoint` (default): The reader creates and maintains checkpoints while following the latest database state.
* `Checkpoint(id)`: The reader remains pinned to the database state referenced by the supplied checkpoint. It does not follow newer manifests or newer WAL data.
* `FollowLatest`: The reader follows the latest manifest without creating a checkpoint. This mode performs no object-store writes and provides no protection from garbage collection. Reads may fail if referenced objects are deleted. This mode is useful for read-only access to databases not being actively written to, for mirrored databases where manifest changes might not be allowed, or for readers willing to handle missing objects gracefully.
[`DbReaderOptions::checkpoint_lifetime`](https://docs.rs/slatedb/latest/slatedb/config/struct.DbReaderOptions.html#structfield.checkpoint_lifetime) controls how long a `ManagedCheckpoint` reader’s checkpoint lives before it must be refreshed.
SlateDB also uses short-lived internal checkpoints during some background transitions. For example, the compactor writes a temporary checkpoint before publishing a manifest that drops obsolete SST references. That keeps recently replaced SSTs readable until GC can safely remove them.
## Clones
[Section titled “Clones”](#clones)
Clones build on the same mechanism. A clone starts from a checkpoint in the source database, then writes new data into its own path while continuing to reference source SSTs. [RFC 0004](/rfcs/0004-checkpoints) describes the full checkpoint and clone design.
# Clones
> How SlateDB forks a database from a checkpoint without copying SSTs
A clone is a new SlateDB database created from an existing database at a different object-store path. You can think of it as a fork taken at one checkpoint in the source database: the clone starts from that source state, then continues with its own manifest, WAL, and future SST files.
Clone creation is shallow. SlateDB does not copy the SSTs that were already visible at the checkpoint. Instead, the clone manifest records which SST IDs still live in external databases and which checkpoints keep those files alive. The clone reads those SSTs from the source paths while writing new data into its own path.
## Source State
[Section titled “Source State”](#source-state)
[`Admin::create_clone`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html#method.create_clone) takes a source path and an optional checkpoint ID. If you pass a checkpoint ID, the clone starts from that exact manifest. If you do not pass one, SlateDB creates a short-lived checkpoint against the source database’s latest manifest and uses that as the base.
`Admin::create_clone` works from manifest state only. It does not coordinate with an open writer to flush its in-memory state first. If you need the clone to include recent writes from an active writer, create a checkpoint through [`Db::create_checkpoint`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.create_checkpoint) with [`CheckpointScope::All`](https://docs.rs/slatedb/latest/slatedb/config/enum.CheckpointScope.html#variant.All) and clone from that checkpoint.
## Shared Files
[Section titled “Shared Files”](#shared-files)
The clone reuses any L0 and sorted-run SSTs that were visible at the chosen checkpoint, including SSTs that the source database may already have inherited from an older clone. Reads can therefore span files from the clone’s own path and files from one or more external database paths.
WAL state is handled differently. SlateDB copies any WAL SSTs referenced by the chosen checkpoint into the clone’s `wal/` directory, then marks the clone initialized. Recovery for the cloned database uses those copied WAL files, and all future WAL files are written only under the clone path.
## GC Pins
[Section titled “GC Pins”](#gc-pins)
Clone creation also creates a durable checkpoint in each external database that the clone references. That checkpoint pins the same manifest as the clone’s source checkpoint, so [garbage collection](/docs/design/gc) in the external database cannot delete shared SSTs or older manifests that the clone still needs.
This is why a source checkpoint can be temporary when you omit `parent_checkpoint`. SlateDB first creates a short-lived source checkpoint, then creates a second checkpoint in each external database to keep the referenced state alive for the clone.
## Nested Clones
[Section titled “Nested Clones”](#nested-clones)
A clone can itself be cloned. In that case the new clone inherits the parent clone’s external database references and adds the immediate parent as another external database. The result is still shallow: the newest clone reads old SSTs from the original paths instead of copying them into each intermediate clone.
## Creation Flow
[Section titled “Creation Flow”](#creation-flow)
Clone creation uses two manifest states so retries are safe. SlateDB first writes the clone manifest with `initialized = false`, copies the required WAL SSTs, then updates the manifest to `initialized = true`. If the process stops after the first step, rerunning clone creation validates the same source path and checkpoint before continuing.
The design is described in detail in [RFC 0004](/rfcs/0004-checkpoints).
## Constraints
[Section titled “Constraints”](#constraints)
The clone path must be different from the source path. Clone creation also fails if the source database uses a separate WAL object store. Today, clones only support databases whose WAL and main data share the same object-store root.
# Compaction
The compactor is responsible for taking groups of sorted runs (this doc uses the term sorted run to refer to both sorted runs and l0 ssts) and compacting them together to reduce space amplification (by removing old versions of rows that have been updated/deleted) and read amplification (by reducing the number of sorted runs that need to be searched on a read). It’s made up of a few different components:
* \[`Compactor`]: The main event loop that orchestrates the compaction process.
* \[`CompactorEventHandler`]: The event handler that handles events from the compactor.
* \[`CompactionScheduler`]: The scheduler that discovers compactions that should be performed.
* \[`CompactionExecutor`]: The executor that runs the compaction tasks.
The main event loop listens on the manifest poll ticker to react to manifest poll ticks, the executor worker channel to react to updates about running compactions, and the shutdown channel to discover when it should terminate. It doesn’t actually implement the logic for reacting to these events. This is implemented by \[`CompactorEventHandler`].
The Scheduler is responsible for deciding what sorted runs should be compacted together. It implements the \[`CompactionScheduler`] trait. The implementation is specified by providing an implementation of \[`CompactionSchedulerSupplier`] so different scheduling policies can be plugged into slatedb. Currently, the only implemented policy is the size-tiered scheduler supplied by [`SizeTieredCompactionSchedulerSupplier`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/size_tiered_compaction.rs).
The Executor does the actual work of compacting sorted runs by sort-merging them into a new sorted run. It implements the \[`CompactionExecutor`] trait. Currently, the only implementation is the [`TokioCompactionExecutor`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/compactor_executor.rs), which runs compaction on a local tokio runtime.
## Trivial moves
[Section titled “Trivial moves”](#trivial-moves)
When every input SST has a non-overlapping effective key range, the compactor can reuse the existing SSTs in the destination sorted run. The coordinator completes this [trivial move](https://github.com/facebook/rocksdb/wiki/Compaction-Trivial-Move) itself, without dispatching the job to an executor or reading and rewriting SST data. This reduces compaction I/O while still producing a sorted run of non-overlapping SSTs.
Trivial moves are disabled by default. Set `CompactorOptions::enable_trivial_move` to `true` to enable them.
To split data with different read/write profiles into independent LSM trees — each with its own compaction policy — and to retire whole key ranges cheaply, see [Segmented Compaction](/docs/design/segmented-compaction).
# Compression
> How SlateDB compresses SST data and metadata
SlateDB compresses SST data in two different ways. One is built into the row format and is always on. The other is an optional codec you choose for WAL SSTs and regular SSTs.
* Prefix compression happens inside each SST data block. [`SstRowCodecV2`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/format/row_codec_v2.rs) encodes each key relative to the previous key, with restart points that store full keys so readers can still seek within the block. This reduces repeated key bytes even when block compression is disabled. [Data Modeling](/docs/operations/data-modeling#prefix-compression) covers the effect on key layout.
* Block compression is controlled by [`Settings::compression_codec`](https://docs.rs/slatedb/latest/slatedb/config/struct.Settings.html#structfield.compression_codec) and [`CompressionCodec`](https://docs.rs/slatedb/latest/slatedb/config/enum.CompressionCodec.html). This applies to all blocks (data, indexes, filters, and stats).
## Codecs
[Section titled “Codecs”](#codecs)
The available codecs are a build-time choice. The `slatedb` crate exposes Snappy, Zlib, LZ4, and Zstd behind Cargo features. The `compression` feature enables Snappy. The `zlib`, `lz4`, and `zstd` features enable those codecs directly.
The selected codec is stored in each SST’s metadata. Changing `compression_codec` only affects new WAL and SST files. Existing files keep the codec they were written with, so a database can temporarily contain a mix of uncompressed files and files written with different codecs while flush and compaction rewrite older data.
## Tradeoffs
[Section titled “Tradeoffs”](#tradeoffs)
Compression reduces object-store bytes, network transfer, and cache footprint. That usually helps storage cost and scan-heavy workloads. The cost is CPU on both write and read.
Block size changes the tradeoff. Larger blocks usually compress better. Smaller blocks reduce over-read on point lookups. The [Tuning](/docs/operations/tuning) page covers the operational knobs.
Prefix compression has its own tradeoff. It helps most when adjacent sorted keys share long prefixes, because that reduces repeated key bytes inside a block. The cost is extra decode work within each restart region: after seeking to a restart point, SlateDB might need to reconstruct several keys before it reaches the target entry.
# Consistency
> Learn about SlateDB's consistency model
Snapshots provide consistent read-only views over the entire state of the key-value store.
Once created, it supports all read operations—`get` or `scan`—giving you a stable, consistent view of the data as it existed at the instant the snapshot was taken.
## Creating a snapshot
[Section titled “Creating a snapshot”](#creating-a-snapshot)
Call `snapshot()` on any SlateDB handle.
The call is cheap and non-blocking; it merely pins the current state so it can’t be garbage-collected while the snapshot is alive.
main.rs
```rust
use slatedb::config::Settings;
use slatedb::{Db, Error};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Error> {
// Initialize database
let object_store = Arc::new(object_store::memory::InMemory::new());
let db = Db::builder("my_db", object_store)
.with_settings(Settings::default())
.build()
.await?;
// Write some data to the database
db.put(b"key1", b"original_value").await?;
db.put(b"key2", b"data_before_snapshot").await?;
// Create a snapshot
let snapshot = db.snapshot().await?;
println!("Snapshot created successfully");
// After creating snapshot, make changes to the database
db.put(b"key1", b"modified_value").await?;
// Read from snapshot - should see original data
let snapshot_value1 = snapshot.get(b"key1").await?;
println!(
"Snapshot key1: {:?}",
snapshot_value1.map(|v| String::from_utf8_lossy(&v).to_string())
);
// Read from database - should see updated data
let db_value1 = db.get(b"key1").await?;
println!(
"Database key1: {:?}",
db_value1.map(|v| String::from_utf8_lossy(&v).to_string())
);
Ok(())
}
```
## Reading from a snapshot
[Section titled “Reading from a snapshot”](#reading-from-a-snapshot)
Use the returned snapshot handle exactly like a regular `Db`, but every operation sees the same, immutable state.
main.rs
```rust
use slatedb::config::Settings;
use slatedb::{Db, Error};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Error> {
// Initialize database
let object_store = Arc::new(object_store::memory::InMemory::new());
let db = Db::builder("my_db", object_store)
.with_settings(Settings::default())
.build()
.await?;
// Write some data to the database
db.put(b"apple", b"red").await?;
db.put(b"banana", b"yellow").await?;
db.put(b"cherry", b"red").await?;
// Create a snapshot
let snapshot = db.snapshot().await?;
println!("Snapshot created successfully");
// After creating snapshot, make changes to the database
db.put(b"apple", b"green").await?; // Modify existing
db.put(b"date", b"brown").await?; // Add new key
db.delete(b"banana").await?; // Delete existing
// Scan from snapshot - should see original data
println!("\nScanning snapshot:");
let mut snapshot_iter = snapshot.scan(b"a".to_vec()..=b"z".to_vec()).await?;
while let Some(kv) = snapshot_iter.next().await? {
println!(
" {}: {}",
String::from_utf8_lossy(&kv.key),
String::from_utf8_lossy(&kv.value)
);
}
// Scan from database - should see updated data
println!("\nScanning database:");
let mut db_iter = db.scan(b"a".to_vec()..=b"z".to_vec()).await?;
while let Some(kv) = db_iter.next().await? {
println!(
" {}: {}",
String::from_utf8_lossy(&kv.key),
String::from_utf8_lossy(&kv.value)
);
}
Ok(())
}
```
# Files
A SlateDB object store bucket contains the following directories:
```plaintext
path/to/db/
├─ manifest/
│ ├─ 00000000000000000001.manifest
│ ├─ 00000000000000000002.manifest
│ └─ ...
├─ wal/
│ ├─ 00000000000000000001.sst
│ ├─ 00000000000000000002.sst
│ └─ ...
├─ compactions/
│ ├─ 00000000000000000001.compactions
│ ├─ 00000000000000000002.compactions
│ └─ ...
├─ compacted/
│ ├─ 01K3XYV1W2WR4FDVB7A9S319YS.sst
│ ├─ 01K3XYV9JFPSZ5BW3Y1DVMKDFS.sst
│ └─ ...
└─ gc/
├─ manifest.boundary
├─ compactions.boundary
└─ ...
```
The directory names are mostly self-explanatory. Let’s look at each file type:
## Manifest
[Section titled “Manifest”](#manifest)
The manifest directory contains an ordered list of manifest files in the format `.manifest`. `` is a zero-padded, 20 digit unsigned integer. Each manifest file is a complete snapshot of the database state at the time it was written. A manifest file can be updated by the following processes:
* **Writer**: When a new WAL SSTable is created, the manifest is updated to include the new SSTable.
* **Reader**: When a new checkpoint is created, deleted, or refreshed, the manifest is updated to include the new checkpoint.
* **Compactor**: When a new sorted run is created, the manifest is updated to include the new sorted run.
Each manifest is encoded as a [FlatBuffer](https://flatbuffers.dev). The schema is located in [schemas/manifest.fbs](https://github.com/slatedb/slatedb/blob/main/schemas/manifest.fbs).
See [RFC-0001](/rfcs/0001-manifest) for details on the manifest update protocol.
Note
Users often ask why SlateDB has a WAL. Since SlateDB batches WAL writes, its WAL looks a lot like level 0 in a standard LSM tree. We address this question in the [FAQ](/docs/get-started/faq#why-does-slatedb-have-a-write-ahead-log).
## Compactions
[Section titled “Compactions”](#compactions)
The `compactions` directory contains an ordered list of compaction-state snapshots in the format `.compactions`. `` is a zero-padded, 20 digit unsigned integer. The compactor creates the first file when it initializes, then writes newer versions as it persists submitted, running, and recently completed compactions.
See [RFC-0013](/rfcs/0013-compaction-state-persistence) for details on the compaction update protocol.
## SSTable
[Section titled “SSTable”](#sstable)
`.sst` files in the `wal` and `compacted` directory share the same file format. Files in the `wal` directory are named `.sst`. `` is a zero-padded, 20 digit unsigned integer. Files in the `compacted` directory are named `.sst`, where `` is a [ULID](https://github.com/ulid/spec).
The `compacted` directory contains both L0 (non-partitioned) SSTables and SRs (partitioned SSTables). As the compactor runs, it will drop compacted SSTables from the manifest. Such files will be left in the `compacted` directory until the [garbage collector](/docs/design/gc) runs.
## Garbage Collector Boundaries
[Section titled “Garbage Collector Boundaries”](#garbage-collector-boundaries)
The `gc` directory contains boundary files used for garbage collection coordination. Boundary files are named `*.boundary` (e.g., `manifest.boundary`, `compactions.boundary`). These files track the minimum sequence number of metadata objects (compactions and manifests) that are still younger than `min_age`, which enables safe deletion of expired metadata and compaction artifacts.
Each boundary file stores a single unsigned 64-bit integer representing an inclusive high-watermark. A boundary value `B` means that object IDs `<= B` are eligible for deletion. Before the garbage collector deletes old sequenced metadata files, it advances the namespace boundary. After a writer creates a sequenced metadata file, it checks the boundary before returning success. If the created ID is at or behind the boundary, the write is treated as failed.
Boundary files use conditional updates (ETag-based) to ensure monotonic advancement and prevent concurrent GC processes from interfering with each other. They provide a persistent marker that allows garbage collectors to safely delete objects below the boundary without risking deletion of still-referenced data.
Garbage collectors advance boundary files by default. This can be disabled through `GarbageCollectorOptions`; eligible metadata is still deleted, but the durable boundary is not advanced. Metadata writers always check any existing boundary. All garbage collectors for a database must use a compatible policy, and the metadata `min_age` settings must be long enough to outlive stale processes when boundary advancement is disabled.
# Garbage Collection
> Learn about SlateDB's garbage collection strategy
SlateDB’s garbage collector runs as a background task in the client process, periodically checking for obsolete files in the database storage.
The garbage collector has a configurable minimum age and interval for each file type (WAL SSTs, WAL fence SSTs, compacted SSTs, manifests, and compactions). Garbage collection for a file type can be disabled by setting its options to `None`. The collector runs every `interval` seconds and will delete files older than `min_age` that are not referenced by any active manifest or checkpoint.
Each garbage collection directory type supports a `dry_run` option. When enabled, the collector logs files that would be deleted without actually deleting them. This is useful for testing or verifying garbage collection behavior before enabling actual deletion.
## Boundary files
[Section titled “Boundary files”](#boundary-files)
Before deleting old manifest or compactions metadata, SlateDB normally advances a durable boundary file. Metadata writers check that boundary after creating a new version, which prevents a stale writer from successfully publishing metadata that GC has already passed.
Boundary advancement can be disabled with `GarbageCollectorOptions::boundary_files_enabled`. This supports object stores without conditional overwrite (`If-Match`) while allowing GC to continue deleting eligible metadata. Metadata readers and writers still check any existing boundary; on a new database, no boundary file is created while all garbage collectors use this mode. Without a boundary, a SlateDB client or compactor can begin updating a manifest or compactions file, stop making progress (for example, because its process or host is suspended), then resume after `min_age`. GC may have deleted the ID it intended to create, so create-if-absent can reuse that ID and report a stale update as successful even though newer metadata has superseded it. Every garbage collector must use a compatible setting, and the manifest and compactions `min_age` values must exceed the maximum lifetime of a stale process.
## Filtering Deletion Candidates
[Section titled “Filtering Deletion Candidates”](#filtering-deletion-candidates)
SlateDB supports custom filtering of garbage collection candidates through the `GcFilter` trait. This allows users to intercept files before deletion and approve or reject them based on custom logic.
The `GcFilter::filter` method receives a set of files that SlateDB has already determined are eligible for deletion (files older than `min_age` that are not referenced by any active manifest or checkpoint). The filter returns the subset of files that may be physically deleted. Any files not returned by the filter are retained.
A common use case for `GcFilter` is verifying that files have been replicated to another bucket before allowing deletion. For example, in a mirror-maker-like scenario for cross-region or cross-cloud replication, the filter can check that a file exists in the destination bucket before approving it for deletion from the source.
To configure a filter, use the `with_gc_filter()` method on `GarbageCollectorBuilder`:
```rust
let gc_filter = Arc::new(MyGcFilter::new());
let gc = GarbageCollectorBuilder::new(path, object_store)
.with_gc_filter(gc_filter)
.build()
.await?;
```
The filter is optional. If no filter is configured, garbage collection proceeds normally based on age and reference checks alone.
Below is a diagram illustrating the high-level flow of garbage collection in SlateDB:
```
flowchart TD
A["Start GC Cycle (interval timer)"] --> B[Remove Expired Checkpoints from Manifests]
B --> C[Run WAL SST GC Task]
C --> C1[List WAL SSTs older than last compacted ID]
C1 --> C2[Filter by min_age and active references]
C2 --> C2a[Apply GcFilter if configured]
C2a --> C3[Delete eligible WAL SSTs]
B --> D[Run Compacted SST GC Task]
D --> D1[List all compacted and L0 SSTs]
D1 --> D2[Gather active SST IDs from manifests]
D2 --> D2a[Apply GcFilter if configured]
D2a --> D3[Delete SSTs not referenced and older than min_age]
B --> E[Run Manifest GC Task]
E --> E1["List all manifests (exclude latest)"]
E1 --> E2[Gather active manifest IDs from checkpoints]
E2 --> E2a[Apply GcFilter if configured]
E2a --> E3[Delete manifests not referenced and older than min_age]
C3 & D3 & E3 --> G[Wait for Next Interval or Shutdown]
```
## Default Garbage Collection Behavior
[Section titled “Default Garbage Collection Behavior”](#default-garbage-collection-behavior)
By default, garbage collection is enabled for all managed directories (manifest, WAL, WAL fence, compacted SSTs, and compactions) using standard interval and minimum age settings.
WAL fence garbage collection runs in dry-run mode by default. This means it logs files that would be deleted without actually deleting them. This conservative default prevents accidental data loss while still providing visibility into what would be cleaned up.
To enable actual deletion for WAL fence GC, set `dry_run: false` with a high `min_age` to safely clean up old fences. Alternatively, to silence the dry-run logging entirely, set `wal_fence_options: None`.
# Merge Operators
> How SlateDB stores merge operands and resolves them on reads and rewrites
[`MergeOperator`](https://docs.rs/slatedb/latest/slatedb/trait.MergeOperator.html) lets writers record partial updates with [`Db::merge`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.merge), [`Db::merge_with_options`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.merge_with_options), and [`WriteBatch::merge`](https://docs.rs/slatedb/latest/slatedb/struct.WriteBatch.html#method.merge). Those APIs append merge operands to the row history instead of forcing every writer to read the current value first. SlateDB resolves that history later with the operator you install.
## Example
[Section titled “Example”](#example)
This example uses a merge operator that concatenates string fragments.
```rust
use bytes::Bytes;
use slatedb::{Db, Error, MergeOperator, MergeOperatorError};
use slatedb::object_store::{memory::InMemory, ObjectStore};
use std::sync::Arc;
struct StringConcatMergeOperator;
impl MergeOperator for StringConcatMergeOperator {
fn merge(
&self,
_key: &Bytes,
existing_value: Option,
operand: Bytes,
) -> Result {
let mut result = existing_value.unwrap_or_default().as_ref().to_vec();
result.extend_from_slice(&operand);
Ok(Bytes::from(result))
}
}
#[tokio::main]
async fn main() -> Result<(), Error> {
let object_store: Arc = Arc::new(InMemory::new());
let db = Db::builder("example", object_store)
.with_merge_operator(Arc::new(StringConcatMergeOperator))
.build()
.await?;
db.merge(b"greeting", b"hello, ").await?;
db.merge(b"greeting", b"world").await?;
let value = db.get(b"greeting").await?.unwrap();
assert_eq!(value.as_ref(), b"hello, world");
Ok(())
}
```
## Resolution
[Section titled “Resolution”](#resolution)
For one key, SlateDB reads the newest visible row first and walks backward until it reaches a plain value, a tombstone, or the end of history. It then applies the newer operands from oldest to newest on top of that base.
* `Value -> Merge -> Merge` reads as the base value plus both operands.
* `Tombstone -> Merge -> Merge` ignores everything older than the delete and applies only the newer operands.
* `Merge -> Merge` returns the merged operand bytes even though no base value exists yet.
When SlateDB flushes or compacts data, that last case may remain a single `Merge` row instead of turning into a plain value, because SlateDB still has merge operands but no base value for the key. If older SST levels contain an older value or tombstone for that key, a later read or compaction can still combine that older state with the newer merged operand.
## Contract
[Section titled “Contract”](#contract)
The `MergeOperator` trait has two methods:
* [`merge`](https://docs.rs/slatedb/latest/slatedb/trait.MergeOperator.html#tymethod.merge) combines one operand with an optional accumulated value.
* [`merge_batch`](https://docs.rs/slatedb/latest/slatedb/trait.MergeOperator.html#method.merge_batch) combines an optional accumulated value with a slice of operands ordered from oldest to newest.
The default `merge_batch` implementation calls `merge` pairwise. Override it if you can process a batch more efficiently.
The operator must be associative. SlateDB may regroup operands during reads, memtable flush, and compaction, so the result has to stay stable when those boundaries move. SlateDB installs one merge operator per database, but your implementation can dispatch by key prefix or value format if you need different merge rules.
## Where Merges Happen
[Section titled “Where Merges Happen”](#where-merges-happen)
### Reads
[Section titled “Reads”](#reads)
Normal reads wrap the iterator stack with [`MergeOperatorIterator`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/merge_operator.rs). Point lookups and scans can merge operands across the write batch, memtable, immutable memtables, L0 SSTs, and sorted runs.
The read path merges operands even when their `expire_ts` values differ. The returned row uses the minimum `expire_ts` across the merged operands and any base value. [Time](/docs/design/time) covers the TTL rules in more detail.
If no merge operator is configured and a read encounters a `Merge` row, SlateDB returns `MergeOperatorMissing`.
### Rewrites
[Section titled “Rewrites”](#rewrites)
Before a [`WriteBatch`](https://docs.rs/slatedb/latest/slatedb/struct.WriteBatch.html) is committed, SlateDB reduces consecutive merge operations for the same key when the writer has a merge operator configured. If the batch also contains a value or tombstone base for that key, the reduced row becomes a plain value. If it has only operands, the reduced row stays a `Merge`.
Caution
`Db::write(...)` and transaction commits reject batch-local merges for the same key when the rows being reduced have different effective `expire_ts` values. This includes multiple `merge_with_options(...)` calls with different TTLs, a `put_with_options(...)` followed by a merge with a different TTL, or a `delete(...)` followed by a merge that expires. Use one effective TTL for each key in a batch, or split the operations across separate batchces or commits.
Memtable flush and compaction use the same merge logic before they write new SSTs. Those rewrite paths are stricter than reads. They keep rows with different `expire_ts` values in separate groups so they can expire independently, and they stop at snapshot and durability retention boundaries so active snapshots or remote readers can still see the version chain they need.
### Raw APIs
[Section titled “Raw APIs”](#raw-apis)
[`WalReader`](https://docs.rs/slatedb/latest/slatedb/struct.WalReader.html) and WAL-based [Change Data Capture](/docs/design/change-data-capture) do not apply the merge operator. They expose raw `ValueDeletable::Merge` rows. Downstream consumers need to materialize those operands themselves if they need final values.
## Configuration
[Section titled “Configuration”](#configuration)
Every process that reads or rewrites stored merge rows needs compatible merge logic.
* Writers use [`DbBuilder::with_merge_operator`](https://docs.rs/slatedb/latest/slatedb/struct.DbBuilder.html#method.with_merge_operator).
* Standalone compactors use [`CompactorBuilder::with_merge_operator`](https://docs.rs/slatedb/latest/slatedb/struct.CompactorBuilder.html#method.with_merge_operator).
* Read-only handles use [`DbReaderBuilder::with_merge_operator`](https://docs.rs/slatedb/latest/slatedb/struct.DbReaderBuilder.html#method.with_merge_operator).
If one process writes merge operands and another opens the same database without a compatible operator, the raw rows can still exist in storage, but later reads, memtable flushes, or compaction can fail when they reach them.
# Overview
> A high-level overview of LSM trees and SlateDB’s design
SlateDB is a log-structured merge-tree (LSM tree). We must first understand how LSM trees work to understand SlateDB’s design. In this page, we’ll explore LSMs and see how SlateDB uses them.
## What is an LSM tree?
[Section titled “What is an LSM tree?”](#what-is-an-lsm-tree)
LSM trees are an in-memory and on-disk data structure used to store key-value (KV) data. KV storage engines such as [RocksDB](https://github.com/facebook/rocksdb) and [LevelDB](https://github.com/google/leveldb) are based on LSM trees. They are a type of key-value (KV) storage engine. LSMs provide operations common to most KV storage engines:
* `put(key, value)`: Insert a key-value pair.
* `get(key)`: Retrieve a key-value pair.
* `delete(key)`: Delete a key-value pair.
* `scan(range)`: Scan a range of keys.
* `flush()`: Flush the in-memory data to disk.
## LSM tree design
[Section titled “LSM tree design”](#lsm-tree-design)
LSM trees are made up of a *write-ahead log (WAL)*, a mutable in-memory map called a *MemTable*, *sorted strings tables (SSTables)*, and a *manifest*. We will discuss each of these components in detail.
### Write-ahead log (WAL)
[Section titled “Write-ahead log (WAL)”](#write-ahead-log-wal)
A put() call writes its data to a write-ahead log (WAL). A WAL is an append-only persistent log. WALs are used to recover data in the event of a crash. When the database restarts, it can replay the WAL to recover its state. WALs are not great for serving reads, though; they’re not optimized for random access. Every write is simply an appended to the end of the log. A get() operation must scan the WAL from the last write to the first write until it finds the key-value pair (or until it reaches the beginning of the WAL).
### MemTables
[Section titled “MemTables”](#memtables)
LSM trees avoid this linear scan by storing their data in an additional data structure called a MemTable. After a put() call writes its data to the WAL, it inserts its data into a mutable in-memory map called a MemTable. MemTables typically use a sorted map such as a [SkipList](https://en.wikipedia.org/wiki/Skip_list) map so that `scan()` operations work as well.
```
sequenceDiagram
participant C as Client
participant WAL as Write-Ahead Log (WAL)
participant MT as MemTable (mutable)
C->>WAL: (1) put()/delete()
WAL-->>C: append OK
C->>MT: (2) put()/delete()
MT-->>C: update OK
```
When a get() occurs, the database will search the MemTable for the key-value pair.
```
sequenceDiagram
participant C as Client
participant MT as MemTable (mutable)
C->>MT: get()
MT-->>C: key-value pair
```
### SSTables
[Section titled “SSTables”](#sstables)
But an in-memory MemTable will only allow us to have a database that fits in memory. We must persist our data on disk. We’ve already seen that the WAL is an inappropriate format for serving reads. LSM trees introduce a third data structure here: a sorted strings table (SSTable). SSTables are files that contain a sequence of key-value pairs sorted by key:
```plaintext
key1:value1
key2:value2
key3:value3
...
```
We’ve established that MemTables are sorted by key, so LSM trees need only write their MemTable to disk in the same order.
```
sequenceDiagram
box Memory
participant C as Client
participant MT as MemTable (mutable)
end
box Disk
participant SST as SSTable
end
C->>MT: put()
MT-->>C: update OK
MT->>SST: write
SST-->>C: write OK
```
There’s a problem with this design, though. The MemTable must be locked while we write to disk. `put()` calls will block until the write is complete. This can cause performance issues if the MemTable is large. To get around this, LSM trees have both a mutable MemTable and one or more immutable MemTables that are being written to disk. When the mutable MemTable reaches a certain size, it is *frozen* and written to disk in the background. A new mutable MemTable is then created (or pulled from a pool) to receive new writes while the froze MemTable is written in the background.
```
sequenceDiagram
box Memory
participant C as Client
participant MT as MemTable (mutable)
participant IMT as MemTable (immutable)
end
box Disk
participant SST as SSTable
end
C->>MT: put()
MT-->>C: update OK
MT->>IMT: freeze
IMT->>SST: write
```
When a get() occurs, the database will search all of the MemTables and SSTables for the key-value pair.
```
sequenceDiagram
box Memory
participant C as Client
participant MT as MemTable (mutable)
participant IMT as MemTable (immutable)
end
box Disk
participant SST as SSTable
end
C->>SST: get()
MT-->>C: key-value pair (if found)
IMT-->>C: key-value pair (if found)
SST-->>C: key-value pair (if found)
```
Imagine we have 1000s (or millions) of immutable MemTables and SSTables. A read would need to search them sequentially to find the most recent version of a key-value pair (a user might have updated a key multiple times). This is clearly not efficient. LSM trees solve this problem with *compaction*. Compaction is a process where the database merges multiple SSTables into a new sequence of range partitioned SSTables. Consider the following SSTables:
```plaintext
SSTable 1:
key1:value1a
key56:value56a
key99:value99
SSTable 2:
key43:value43
key56:value56b
key89:value89
SSTable 3:
key1:value1b
key44:value44
key77:value77
```
Each SST in the example above is unpartitioned—it can potentially contain any key-value pair. Compaction will merge these SSTables into a new sequence of range partitioned SSTables. For example, the SSTables above might be compacted into the following SSTables:
```plaintext
SSTable 1:
key1:value1b
key43:value43
key44:value44
key56:value56b
key77:value77
SSTable 2:
key89:value89
key99:value99
```
Now a read for `key56` will only need to search the first SSTable. Readers need only check the key range of each SSTable (stored in memory) to determine if it might contain the key. The unpartitioned tables, are called *level 0 (L0)* SSTables. The partitioned tables are called *sorted runs (SRs)*. L0 SSTables are part of the write path, as we saw above. Sorted runs are generated through a background compaction process.
Note
Sorted runs are a logical construct. The files written to disk are the sorted run’s SSTables. The manifest (which we’ll see in a moment) contains the mapping from a sorted run ID to its SSTables.
A `get()` will traverse the MemTables, L0 SSTables, and SRs to find the key-value pair.
```
sequenceDiagram
box Memory
participant C as Client
participant MT as MemTable (mutable)
participant IMT as MemTable (immutable)
end
box Disk
participant L0 as L0 SSTable
participant SR as SR (sorted run)
end
C->>SR: get()
MT-->>C: key-value pair (if found)
IMT-->>C: key-value pair (if found)
L0-->>C: key-value pair (if found)
SR-->>C: key-value pair (if found)
```
### Manifests
[Section titled “Manifests”](#manifests)
Another issue remains: upon restart, the database has no easy way to know what SSTables it should load. We might try to list all SSTs and read each file’s metadata, but this doesn’t work. RocksDB’s [MANIFEST page](https://github.com/facebook/rocksdb/wiki/MANIFEST) explains why:
> File system operations are not atomic, and are susceptible to inconsistencies in the event of system failure. Even with journaling turned on, file systems do not guarantee consistency on unclean restart. POSIX file system does not support atomic batching of operations either. Hence, it is not possible to rely on metadata embedded in RocksDB datastore files to reconstruct the last consistent state of the RocksDB on restart.
LSM tree implementations store their state in a manifest. The manifest is a file (or log of files) that stores the database’s state; this includes the location of all SSTables, the WAL recovery starting point, and many other details. The LSM tree reads the manifest on startup to reconstruct its in-memory state. As state changes, the LSM tree writes manifest changes atomically to ensure consistency.
We’ve now built a basic LSM tree. There are many, many details to be explored (deletion tombstones, consistency, WAL replay, and more). Still, this basic implementation will work. If you’re interested in experimenting more with LSMs, we highly recommend checking out [Mini-LSM](https://skyzh.github.io/mini-lsm/).
## Tradeoffs
[Section titled “Tradeoffs”](#tradeoffs)
LSM trees are elegant, simple data structures. However, they are not without tradeoffs. LSM trees work very well for high write-throughput workloads. It does so by moving as much work as possible out of the write path. Consequently, the design requires more background work (CPU), more files (storage space), and more variable read/latency (I/O) than other data structures. These tradeoffs are commonly framed in the following terms:
* Write amplification: The number of extra bytes written to storage per byte of live user data ingested.
* Read amplification: The number of extra bytes read from storage per byte of result data returned.
* Space amplification: The total on-disk byte size relative to the total live logical byte size.
LSM trees tend to have higher write throughput but worse read locality, more read and space amplification, more background compaction CPU, and higher tail latencies than B-trees. This subject is an active area of research and there are now many strategies to balance these tradeoffs.
## SlateDB’s LSM design
[Section titled “SlateDB’s LSM design”](#slatedbs-lsm-design)
SlateDB’s LSM design diverges very little from the basic LSM tree design we’ve seen above. SlateDB has a WAL, MemTables, SSTables, sorted runs, and a manifest. The only difference is that all of SlateDB’s data is written to object storage instead of a local disk or filesystem:
```
sequenceDiagram
box Memory
participant C as Client
participant MT as MemTable (mutable)
participant IMT as MemTable (immutable)
end
box Object Storage
participant SST as SSTable
participant SR as Sorted Run
end
C->>MT: put()
MT-->>C: update OK
MT-->>IMT: freeze
IMT-->>SST: write
SST-->>SR: compact
```
In the next section, we’ll look at SlateDB’s object store directory structure.
# Readers
> How DbReader serves read-only traffic from a checkpointed manifest
[`DbReader`](https://docs.rs/slatedb/latest/slatedb/struct.DbReader.html) is SlateDB’s read-only handle. It implements the same [`DbReadOps`](https://docs.rs/slatedb/latest/slatedb/trait.DbReadOps.html) operations as [`Db`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html), but it does not own a mutable memtable and it does not write WAL records. It reads from a [checkpointed](/docs/design/checkpoints) manifest, the SSTs referenced by that manifest. When WAL replay is enabled, it also updates reader-local immutable memtables from newer WAL SSTs.
## Opening
[Section titled “Opening”](#opening)
This example opens a database, flushes one key to object storage, then opens a `DbReader` and reads that key back.
```rust
use slatedb::{Db, DbReader, Error};
use slatedb::object_store::{memory::InMemory, ObjectStore};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Error> {
let object_store: Arc = Arc::new(InMemory::new());
let db = Db::open("example", Arc::clone(&object_store)).await?;
db.put(b"key", b"value").await?;
db.flush().await?;
let reader = DbReader::builder("example", object_store).build().await?;
assert_eq!(reader.get(b"key").await?, Some("value".into()));
Ok(())
}
```
Without [`DbReaderBuilder::with_checkpoint_id`](https://docs.rs/slatedb/latest/slatedb/struct.DbReaderBuilder.html#method.with_checkpoint_id), SlateDB creates a checkpoint in the latest manifest and sets its lifetime from [`DbReaderOptions::checkpoint_lifetime`](https://docs.rs/slatedb/latest/slatedb/config/struct.DbReaderOptions.html#structfield.checkpoint_lifetime). That checkpoint pins the initial L0 and sorted-run view. The reader then replays newer WAL SSTs into immutable memtables unless [`DbReaderOptions::skip_wal_replay`](https://docs.rs/slatedb/latest/slatedb/config/struct.DbReaderOptions.html#structfield.skip_wal_replay) is `true`.
With an explicit checkpoint ID, SlateDB opens exactly the manifest pinned by that checkpoint. It does not create a new checkpoint, refresh the user-managed checkpoint, delete it on close, or follow newer manifests or WAL SSTs.
When `skip_wal_replay` is `true`, the reader only sees data already referenced by the manifest it opened. If the reader owns the checkpoint, it can still pick up later memtable flushes and compaction results when the manifest changes, but it ignores WAL-only data between those manifest updates.
## State
[Section titled “State”](#state)
Internally, the reader tracks the pinned checkpoint and manifest, a deque of immutable memtables built from WAL replay, the last WAL file it replayed, and the highest durable sequence number it can expose. Reads go through the same internal [`Reader`](https://github.com/slatedb/slatedb/blob/main/slatedb/src/reader.rs) implementation that backs ordinary [`Db`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html) reads.
A `DbReader` has no mutable memtable of its own. Its precedence is replayed immutable memtables, then L0 SSTs, then sorted runs. When SlateDB replaces the reader’s checkpoint with a newer manifest, it drops WAL-replayed rows whose sequence numbers are now at or below the new manifest’s `last_l0_seq`, because L0 or lower levels already cover them.
## Refresh Loop
[Section titled “Refresh Loop”](#refresh-loop)
When the reader owns the checkpoint, it starts a background poller that wakes up every [`DbReaderOptions::manifest_poll_interval`](https://docs.rs/slatedb/latest/slatedb/config/struct.DbReaderOptions.html#structfield.manifest_poll_interval).
On each tick, SlateDB loads the latest manifest. If the visible SST layout changed, or if the latest L0 sequence number moved past the WAL state already replayed into the reader, it replaces the checkpoint and rebuilds state from the new manifest. Otherwise it keeps the current checkpoint and only looks for newer WAL SSTs.
SlateDB refreshes the checkpoint expiration after half of the configured lifetime has elapsed. [`DbReader::close`](https://docs.rs/slatedb/latest/slatedb/struct.DbReader.html#method.close) stops the poller and deletes the checkpoint the reader created for itself.
## Read Semantics
[Section titled “Read Semantics”](#read-semantics)
`get`, `get_key_value`, `scan`, and `scan_prefix` use the normal SST iterator stack. Options such as [`ReadOptions::cache_blocks`](https://docs.rs/slatedb/latest/slatedb/config/struct.ReadOptions.html#structfield.cache_blocks), [`ScanOptions::cache_blocks`](https://docs.rs/slatedb/latest/slatedb/config/struct.ScanOptions.html#structfield.cache_blocks), [`ScanOptions::read_ahead_bytes`](https://docs.rs/slatedb/latest/slatedb/config/struct.ScanOptions.html#structfield.read_ahead_bytes), and [`ScanOptions::max_fetch_tasks`](https://docs.rs/slatedb/latest/slatedb/config/struct.ScanOptions.html#structfield.max_fetch_tasks) still control caching and scan behavior.
Visibility is narrower than [`Db`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html). A reader does not share a writer’s live mutable memtable, so read options cannot expose unflushed writer state. `DbReader` only returns committed data that is already durable in object storage, either through the pinned manifest or through WAL SSTs it replayed itself.
Reader-side block and object-store caches use the same configuration surface as [`Db`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html). [Caching](/docs/design/caching) covers those layers.
# Reads
1. A get(), scan(), or similar read call is made on the client.
2. The read request is routed to the database engine, which determines the current read snapshot (either the latest state or a pinned snapshot if using the snapshot API).
3. The engine first checks the in-memory MemTable for the requested key or range. If the key is found and not expired, the value is returned immediately.
4. If the key is not found in the MemTable, the engine searches the immutable MemTable (if present, e.g., during a flush).
5. If the key is still not found, the engine concurrently creates iterators over all L0 SSTables (uncompacted, recently flushed files) and all compacted sorted runs (lower levels). These iterators are constructed in parallel to maximize throughput and minimize latency, with the degree of concurrency determined by the number of files and system configuration [\[source\]](https://github.com/slatedb/slatedb/pull/815).
6. Each SSTable is searched using its block index. A binary search locates the block that may contain the key. If multiple blocks start with the same key, the engine ensures it returns the first such block to avoid stale data and to support multiple versions of a key [\[source\]](https://github.com/slatedb/slatedb/issues/517).
7. As iterators are created, SlateDB asynchronously fetches and caches file blocks from object storage.
8. The engine merges the MemTable, immutable MemTable, L0 SSTable iterators, and sorted run iterators into a single merged iterator. The iterator respects sequence numbers and snapshot visibility, ensuring that only the correct version of each key is visible for the current read or snapshot.
9. The merged iterator returns the value for the requested key (or the next key in a scan). If the key is not found in any structure, None is returned.
Below is a diagram illustrating the high-level flow of a read in SlateDB:
```
flowchart TD
A[API Call: get/scan] --> B[Determine Snapshot Sequence Number]
B --> C[Check MemTable]
C --> D{Found?}
D -- Yes --> Z[Return Value]
D -- No --> E[Check Immutable MemTable]
E --> F{Found?}
F -- Yes --> Z
F -- No --> G["Create merge iterator (concurrent)"]
G --> J["Seek to Key"]
J --> K{Found?}
K -- Yes --> Z
K -- No --> L[Return None]
```
**Notes:**
* Iterator creation for L0 SSTables and compacted runs is parallelized for performance.
* Block cache is populated asynchronously during reads.
* Reads from snapshots always reflect the state at the time the snapshot was taken, regardless of later modifications.
* WAL SSTs are not read during normal operations; they are only accessed during recovery [\[source\]](https://github.com/slatedb/slatedb/issues/499).
# Segmented Compaction
> Split a dataset into independent LSM trees, each with its own compaction and retention policy
Segmented compaction splits a database into independent LSM trees keyed by a prefix. Segments let each tree independently configure compaction policy to match its access pattern needs, and let a whole segment be dropped cheaply once it ages out. All segments share a WAL and writes across segments can be done transactionally/atomically.
A [`PrefixExtractor`](https://docs.rs/slatedb/latest/slatedb/trait.PrefixExtractor.html) derives a segment prefix from each key, and each segment becomes its own logical LSM tree. Because segments own disjoint key ranges, they are compacted and retired independently, and reads/scans automatically prune to the segments overlapping the query.
Common fits:
* **Timeseries / append-ordered data**: one segment per time bucket or log segment; old segments freeze and age out as a unit.
* **Metadata/data separation**: isolate a churny, frequently-read metadata keyspace from bulky, write-once data so each gets its own compaction and caching policy.
* **Column-family-like isolation**: give small, frequently-overwritten structures an aggressive policy while slow-moving data uses a low-write-amplification one.
See [RFC 0024](/rfcs/0024-segment-oriented-compaction) for the full design.
## Enabling segments
[Section titled “Enabling segments”](#enabling-segments)
Configure an extractor at creation with [`DbBuilder::with_segment_extractor`](https://docs.rs/slatedb/latest/slatedb/struct.DbBuilder.html#method.with_segment_extractor). The application encodes segment boundaries in the key (e.g. an hour bucket, a log segment id); the extractor returns the prefix length that identifies the segment. The example below routes every key to the segment named by its first three bytes.
* Rust
```rust
use std::sync::Arc;
use slatedb::{Db, PrefixExtractor, PrefixTarget};
struct FixedThreeByteExtractor;
impl PrefixExtractor for FixedThreeByteExtractor {
fn name(&self) -> &str { "fixed_three_byte" }
fn prefix_len(&self, target: &PrefixTarget) -> Option {
let (PrefixTarget::Point(key) | PrefixTarget::Prefix(key)) = target;
(key.len() >= 3).then_some(3)
}
}
let db = Db::builder(path, object_store)
.with_segment_extractor(Arc::new(FixedThreeByteExtractor))
.build()
.await?;
```
* Go
```go
type fixedThreeByteExtractor struct{}
func (fixedThreeByteExtractor) Name() string { return "fixed_three_byte" }
func (fixedThreeByteExtractor) PrefixLen(target slatedb.PrefixTarget) *uint64 {
var key []byte
switch t := target.(type) {
case slatedb.PrefixTargetPoint:
key = t.Key
case slatedb.PrefixTargetPrefix:
key = t.Prefix
}
if len(key) < 3 {
return nil
}
n := uint64(3)
return &n
}
builder := slatedb.NewDbBuilder("example-db", store)
if err := builder.WithSegmentExtractor(fixedThreeByteExtractor{}); err != nil {
panic(err)
}
db, err := builder.Build()
```
* Java
```java
import io.slatedb.uniffi.PrefixExtractor;
import io.slatedb.uniffi.PrefixTarget;
class FixedThreeByteExtractor implements PrefixExtractor {
@Override
public String name() {
return "fixed_three_byte";
}
@Override
public Long prefixLen(PrefixTarget target) {
byte[] key =
switch (target) {
case PrefixTarget.Point point -> point.key();
case PrefixTarget.Prefix prefix -> prefix.prefix();
};
return key.length < 3 ? null : 3L;
}
}
DbBuilder builder = new DbBuilder("demo-db", store);
builder.withSegmentExtractor(new FixedThreeByteExtractor());
Db db = builder.build().get();
```
* Node.js
```js
import { DbBuilder } from "@slatedb/uniffi";
class FixedThreeByteExtractor {
name() {
return "fixed_three_byte";
}
prefix_len(target) {
const key = target.tag === "Point" ? target.key : target.prefix;
return key.length < 3 ? undefined : 3;
}
}
const builder = new DbBuilder("demo-db", store);
builder.with_segment_extractor(new FixedThreeByteExtractor());
const db = await builder.build();
```
* Python
```python
from slatedb.uniffi import DbBuilder, PrefixExtractor, PrefixTarget
class FixedThreeByteExtractor(PrefixExtractor):
def name(self) -> str:
return "fixed_three_byte"
def prefix_len(self, target: PrefixTarget) -> int | None:
key = target.key if target.is_point() else target.prefix
return None if len(key) < 3 else 3
builder = DbBuilder("demo-db", store)
builder.with_segment_extractor(FixedThreeByteExtractor())
db = await builder.build()
```
Caution
Whether segmentation is enabled is fixed for the life of the database. The extractor’s `name()` is persisted in the manifest; opening with a different name, or adding or removing an extractor, fails. This is primarily as a soft check against accidental misuse.
List the segments that currently exist (in the manifest or in memtables) with [`DbStatus::list_segments()`](https://docs.rs/slatedb/latest/slatedb/struct.DbStatus.html#method.list_segments).
## Evolving the key schema
[Section titled “Evolving the key schema”](#evolving-the-key-schema)
An extractor’s name is fixed, but its implementation may evolve when the change is backward-compatible. For example, a schema byte can reserve disjoint key ranges for successive segmentation policies:
```text
# all records below use a schema marker as the first part of the key prefix
# initially use weekly partitions
[01][week]... -> [01][week] # v1: weekly segments
# later migrate to daily partitions
[02][day]... -> [02][day] # v2: daily segments
```
The new implementation must keep extracting the original weekly prefixes for all v1 keys while adding daily prefixes only for v2 keys. In general:
* Keep the same stable name and preserve the behavior of previous PrefixExtractor implementations.
* Ensure segment prefixes from all versions form an antichain: no prefix may be a prefix of another (a schema discriminator at the start of the key is a simple way to reserve disjoint ranges).
SlateDB checks the name and, on open, asks the configured extractor to recognize every persisted segment prefix. It also rejects new writes that would introduce a nested prefix. These are guardrails rather than complete compatibility verification: SlateDB does not persist the implementation’s version or revalidate every existing key. Maintaining backward-compatible routing under a stable name is therefore the application’s responsibility.
## How it compacts
[Section titled “How it compacts”](#how-it-compacts)
Each segment is compacted on its own schedule. The default size-tiered [`CompactionScheduler`](/docs/design/compaction) applies per segment, and compactions in different segments are parallel-safe (disjoint keys, no cross-segment ordering). Embed a semantic hint in the prefix (e.g. a time bucket) and a custom scheduler can vary policy per segment.
Note
Writing to N segments at once produces up to N× the L0 objects (smaller each); the WAL is shared, so it is unaffected. Most data models keep one active segment, so this rarely bites in practice.
## Deleting Segments Wholesale
[Section titled “Deleting Segments Wholesale”](#deleting-segments-wholesale)
The default scheduler never drops data; segment retention is the application’s job. To retire a whole segment, schedule a `CompactionSpec::drain_segment` compaction. Unlike a merge, a drain reads and rewrites nothing. Instead it detaches the segment’s L0s and sorted runs from the manifest, leaving a drain marker (`l0=[]`, `compacted=[]`) that the writer prunes, after which the garbage collector reclaims the files.
A drain is a `CompactionSpec::drain_segment` submitted through [`Admin.submit_compaction`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html), built from the segment’s L0 SSTs and sorted runs read from `read_compactor_state_view`. Every binding mirrors this API:
* Rust
```rust
use bytes::Bytes;
use slatedb::admin::Admin;
use slatedb::compactor::{CompactionSpec, SourceId};
let admin = Admin::builder(path, object_store).build();
let view = admin.read_compactor_state_view().await?;
let segment = Bytes::from_static(b"hour=10/");
if let Some(seg) = view.manifest().segment(&segment) {
// Drain every L0 SST and sorted run currently visible in the segment.
let sources: Vec = seg
.l0()
.iter()
.map(|v| SourceId::SstView(v.id))
.chain(seg.compacted().iter().map(|sr| SourceId::SortedRun(sr.id)))
.collect();
admin.submit_compaction(CompactionSpec::drain_segment(segment, sources)).await?;
}
```
* Go
```go
admin, err := slatedb.NewAdminBuilder("example-db", store).Build()
if err != nil {
panic(err)
}
view, err := admin.ReadCompactorStateView()
if err != nil {
panic(err)
}
var sources []slatedb.SourceId
for _, seg := range view.Manifest.Segments {
if !bytes.Equal(seg.Prefix, []byte("hour=10/")) {
continue
}
for _, v := range seg.L0 {
sources = append(sources, slatedb.SourceIdSstView{Field0: v.Id})
}
for _, r := range seg.Compacted {
sources = append(sources, slatedb.SourceIdSortedRun{Field0: r.Id})
}
}
_, err = admin.SubmitCompaction(slatedb.CompactionSpecDrainSegment{
Segment: []byte("hour=10/"),
Sources: sources,
})
```
* Java
```java
import io.slatedb.uniffi.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
Admin admin = new AdminBuilder("demo-db", store).build();
var view = admin.readCompactorStateView().get();
byte[] prefix = "hour=10/".getBytes();
List sources = new ArrayList<>();
for (Segment seg : view.manifest().segments()) {
if (!Arrays.equals(seg.prefix(), prefix)) continue;
seg.l0().forEach(v -> sources.add(new SourceId.SstView(v.id())));
seg.compacted().forEach(r -> sources.add(new SourceId.SortedRun(r.id())));
}
admin.submitCompaction(new CompactionSpec.DrainSegment(prefix, sources)).get();
```
* Node.js
```js
import { AdminBuilder, CompactionSpec, SourceId } from "@slatedb/uniffi";
const admin = new AdminBuilder("demo-db", store).build();
const view = await admin.read_compactor_state_view();
const prefix = Buffer.from("hour=10/");
const seg = view.manifest.segments.find((s) => prefix.equals(Buffer.from(s.prefix)));
const sources = [
...seg.l0.map((v) => SourceId.SstView(v.id)),
...seg.compacted.map((r) => SourceId.SortedRun(r.id)),
];
await admin.submit_compaction(CompactionSpec.DrainSegment(seg.prefix, sources));
```
* Python
```python
from slatedb.uniffi import AdminBuilder, CompactionSpec, SourceId
admin = AdminBuilder("demo-db", store).build()
view = await admin.read_compactor_state_view()
seg = next(s for s in view.manifest.segments if s.prefix == b"hour=10/")
sources = [SourceId.SST_VIEW(v.id) for v in seg.l0]
sources += [SourceId.SORTED_RUN(r.id) for r in seg.compacted]
await admin.submit_compaction(CompactionSpec.DRAIN_SEGMENT(segment=seg.prefix, sources=sources))
```
To retire segments automatically (e.g. by a TTL), implement [`CompactionScheduler`](https://docs.rs/slatedb/latest/slatedb/compactor/trait.CompactionScheduler.html) and return `drain_segment` specs from `propose`, wired in with `CompactorBuilder::with_scheduler_supplier`.
# Sorted String Tables (SSTs)
> How SlateDB stores WAL and compacted data in SST files
SlateDB stores durable rows in SST files under `wal/` and `compacted/`. Both use the same block and footer format, but they serve different jobs. WAL SSTs record writes in sequence number order for recovery and CDC. Compacted SSTs store L0 and sorted-run data in key order and serve normal reads. The same rows can appear in a different order depending on the SST type.
WAL SST (sequence order)
```text
seq=41 key=banana value=yellow
seq=42 key=apple value=green
seq=43 key=carrot value=orange
```
Compacted SST (key order):
```text
key=apple seq=42 value=green
key=banana seq=41 value=yellow
key=carrot seq=43 value=orange
```
## Layout
[Section titled “Layout”](#layout)
An SST stores data blocks first. A compacted SST may then append a Bloom filter block. The Bloom filter block stores a probabilistic membership filter that lets point reads skip the SST on a definite miss. Next comes the index block. The index block stores one entry per data block with the block offset and a search key or sequence bound so readers can jump to candidate blocks. A compacted SST may also append a stats block. The stats block stores file-level and per-block counters and byte totals for puts, deletes, merge operands, keys, and values. Every SST ends with `SsTableInfo`, followed by the 8-byte metadata offset and the 2-byte format version.
```
%%{init: {"packet": {"bitsPerRow": 14, "bitWidth": 50, "showBits": false}}}%%
packet-beta
title SST file layout
+14: "data blocks..."
+14: "filter block (optional)"
+14: "index block"
+14: "stats block (optional)"
+14: "SsTableInfo"
+14: "metadata offset: u64 LE"
+14: "format version: u16 LE"
```
Each stored block ends with a CRC32 checksum. That applies to data blocks and to footer blocks such as the filter, index, stats, and `SsTableInfo` metadata.
## Data Blocks
[Section titled “Data Blocks”](#data-blocks)
SlateDB groups rows into blocks up to the target size configured with [`DbBuilder::with_sst_block_size`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_sst_block_size). When the next row would overflow the current block, the writer closes that block and starts a new one. Compacted SSTs write rows in key order. WAL SSTs write rows in sequence number order.
Within a data block, each stored row is a `RowEntry`: the logical fields are the key bytes, a `ValueDeletable` payload (`Value`, `Merge`, or `Tombstone`), the sequence number, and the optional `expire_ts` and `create_ts` timestamps.
In the current SST row codec (`SstRowCodecV2`), keys are prefix-compressed against the previous key in the block. `shared_bytes` records how many leading bytes match the previous key, `unshared_bytes` records the remaining suffix length, and `key_suffix` stores that suffix. At restart points, which current writers insert every 16 entries, `shared_bytes` is `0`, so the entry stores the full key and readers can seek into the block without decoding from the beginning.
The rest of the row stores the value length and payload, followed by the sequence number, a flags byte, and any optional timestamps. Tombstones set the tombstone flag, encode `value_len = 0`, and omit the value bytes. Merge operands use the same payload field as normal values but set the merge flag instead.
```
%%{init: {"packet": {"bitsPerRow": 14, "bitWidth": 50, "showBits": false}}}%%
packet-beta
title RowEntry encoding in a data block
+14: "shared_bytes: varint"
+14: "unshared_bytes: varint"
+14: "value_len: varint"
+14: "key_suffix: bytes"
+14: "value: bytes (omitted for tombstones)"
+14: "seq: u64"
+14: "flags: u8"
+14: "expire_ts: i64 (optional)"
+14: "create_ts: i64 (optional)"
```
## Index Blocks
[Section titled “Index Blocks”](#index-blocks)
The index stores one entry per data block. Each entry says where that block starts in the file and what boundary SlateDB should use to decide whether to search that block.
In compacted SSTs, the boundary is a key: for the first block it is empty, and for later blocks it is the smallest key that should map to that block. That boundary is often a shortened prefix between the previous block’s last key and this block’s first key, so SlateDB does not need to store the full first key for every block.
In WAL SSTs, the index uses sequence numbers instead of keys and stores the first sequence number in each block.
Readers binary-search the index to find the first block that may contain a target key or the block range that overlaps a scan. In a sorted run, SlateDB first uses SST view boundaries and then the per-SST block index inside the selected file.
## Filter Blocks
[Section titled “Filter Blocks”](#filter-blocks)
Compacted SSTs can include a Bloom filter. SlateDB writes one when the file has at least [`Settings::min_filter_keys`](https://docs.rs/slatedb/latest/slatedb/config/struct.Settings.html#structfield.min_filter_keys) rows and sizes it with [`Settings::filter_bits_per_key`](https://docs.rs/slatedb/latest/slatedb/config/struct.Settings.html#structfield.filter_bits_per_key). Point lookups can skip the whole SST on a negative result. WAL SSTs do not include a filter because SlateDB reads them sequentially during replay.
## Stats Block
[Section titled “Stats Block”](#stats-block)
The current compacted SST builder also writes a stats block. It records counts of puts, deletes, and merge operands, plus raw key and value bytes and per-block stats. This file is useful for read optimization. See [RFC-0020](/rfcs/0020-range-metadata) for more information.
## Footer
[Section titled “Footer”](#footer)
`SsTableInfo` stores the high-level metadata needed to reopen and interpret the SST. That includes the file’s key or sequence bounds, where SlateDB can find auxiliary blocks such as the index, Bloom filter, and stats block, and format details such as compression and whether the file is a WAL SST or a compacted SST.
If you want the exact field-by-field layout, see the [`SsTableInfo` FlatBuffers schema](https://github.com/slatedb/slatedb/blob/main/schemas/sst.fbs).
# Time
> How SlateDB records wall-clock timestamps and expiration metadata
SlateDB separates ordering from time. Sequence numbers decide visibility. Wall-clock timestamps are metadata recorded in milliseconds since the Unix epoch and used for TTL, checkpoint lifetime, and background scheduling.
## Ordering
[Section titled “Ordering”](#ordering)
Every committed write batch gets a sequence number and a creation timestamp. The sequence number is the authoritative ordering key. The timestamp is descriptive metadata.
This matters when multiple writes land in the same millisecond. Two versions of a key can share the same timestamp and still be ordered correctly because SlateDB compares sequence numbers first.
SlateDB stamps the committed batch once. Every row produced by that batch inherits the same `create_ts`, while per-row TTL can still produce different `expire_ts` values. The batch timestamp is exposed through [`WriteHandle::create_ts()`](https://docs.rs/slatedb/latest/slatedb/db/struct.WriteHandle.html#method.create_ts).
## Clock
[Section titled “Clock”](#clock)
SlateDB gets wall-clock time from [`SystemClock`](https://docs.rs/slatedb-common/latest/slatedb_common/clock/trait.SystemClock.html). The default implementation uses the process wall clock. [`DbBuilder::with_system_clock()`](https://docs.rs/slatedb/latest/slatedb/db/struct.DbBuilder.html#method.with_system_clock) lets tests and specialized environments replace it.
Internally, SlateDB wraps that clock in a monotonic clock. If the underlying clock moves backwards, SlateDB waits briefly for it to catch up. If it is still behind, the operation fails instead of writing timestamps that move backwards.
When an immutable memtable is flushed to L0, SlateDB persists the newest flushed clock tick in the manifest as `last_l0_clock_tick`. On restart it seeds the monotonic clock from that value so new timestamps stay ahead of data that is already durable in L0 or lower levels.
The same clock is also used for checkpoint expiry and background tasks such as compaction and garbage collection.
## Row Metadata
[Section titled “Row Metadata”](#row-metadata)
Metadata-aware reads expose the stored timestamps. [`Db::get_key_value()`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.get_key_value) and [`DbIterator::next()`](https://docs.rs/slatedb/latest/slatedb/struct.DbIterator.html#method.next) return [`KeyValue`](https://docs.rs/slatedb/latest/slatedb/struct.KeyValue.html), which includes `seq`, `create_ts`, and `expire_ts`.
[`Db::get()`](https://docs.rs/slatedb/latest/slatedb/struct.Db.html#method.get) returns only value bytes. If you need timestamp metadata, use `get_key_value()` or iterate `KeyValue` results from a scan.
SlateDB also preserves the same metadata in the WAL path, which is why [Change Data Capture](/docs/design/change-data-capture) can expose `create_ts` and `expire_ts` for downstream consumers.
## Expiration
[Section titled “Expiration”](#expiration)
TTL is stored as an absolute expiration timestamp in milliseconds since the Unix epoch, not as a relative duration. On commit, SlateDB computes `expire_ts = create_ts + ttl_millis`.
[`Settings::default_ttl_millis`](https://docs.rs/slatedb/latest/slatedb/config/struct.Settings.html#structfield.default_ttl_millis) sets a default TTL in milliseconds for puts and merges. [`PutOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.PutOptions.html) and [`MergeOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.MergeOptions.html) can override that per operation:
* [`Ttl::NoExpiry`](https://docs.rs/slatedb/latest/slatedb/config/enum.Ttl.html#variant.NoExpiry) — store the value without expiration
* [`Ttl::ExpireAfterMillis(u64)`](https://docs.rs/slatedb/latest/slatedb/config/enum.Ttl.html#variant.ExpireAfterMillis) — expire after a relative duration in milliseconds
* [`Ttl::ExpireAtMillis(i64)`](https://docs.rs/slatedb/latest/slatedb/config/enum.Ttl.html#variant.ExpireAtMillis) — expire at a fixed Unix timestamp in milliseconds
Deletes write tombstones and do not carry TTL.
Caution
Batch-local merges requires one effective `expire_ts` per key. `Db::write(...)` and transaction commit return `ErrorKind::Invalid` if a single batch tries to merge the same key across different expiration timestamps. Effective timestamps are computed after applying `Settings::default_ttl_millis`, so `Ttl::Default` can still differ from `Ttl::NoExpiry` or from another explicit TTL.
Note
When SlateDB materializes a value from merge operands, the resulting row reports the earliest `expire_ts` across the merged operands and any base value. A later merge with a longer TTL does not extend the lifetime of older merged data.
The current read path does not consult wall-clock time to hide expired rows. Instead, SlateDB returns the recorded `expire_ts` to metadata-aware callers and uses `expire_ts` during compaction. Applications that need strict read-time TTL enforcement should compare `expire_ts` with their current time.
Compaction is what turns expiration into deletion. Ordinary values may be rewritten to tombstones so older versions in lower levels do not become visible again. Expired merge operands are dropped instead of converted to tombstones because a tombstone would also erase older merge history. Physical removal still depends on later compaction and [garbage collection](/docs/design/gc).
## Mapping Between Timestamps and Sequence Numbers
[Section titled “Mapping Between Timestamps and Sequence Numbers”](#mapping-between-timestamps-and-sequence-numbers)
For approximate conversion between sequence numbers and wall-clock time, the manifest stores a bounded sequence tracker. [`Admin::get_timestamp_for_sequence()`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html#method.get_timestamp_for_sequence) and [`Admin::get_sequence_for_timestamp()`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html#method.get_sequence_for_timestamp) query that tracker, and `slatedb-cli` exposes the same conversion. The mapping is lossy by design: recent history has finer granularity, and older history is downsampled to keep manifest state bounded. See [RFC-0012](/rfcs/0012-sequence-tracker/) for its design.
# Writes
Writes are moved to a background task as quickly as possible to prevent blocking the client. `put()`, `write()`, and `delete()` return a `WriteHandle` after the write reaches the in-memory WAL and MemTable. Call `handle.await_durable().await` to wait for that write to reach object storage, or call `flush()` explicitly. The synchronous flow is as follows:
1. A `put()`, `write()`, or `delete()` call is made on the client.
2. The key/value pair is written to the mutable, in-memory WAL table.
3. The key/value pair is written to the mutable, in-memory MemTable.
The following asynchronous flows occur:
* The WAL flusher periodically checks if the WAL table is full. If it is, it freezes the mutable WAL table and triggers an asynchronous write to object storage. Durability waiters are notified when the durable sequence number advances.
* The MemTable flusher periodically checks if the MemTable is full. If it is, it freezes the mutable MemTable and triggers an asynchronous write to object storage. With the WAL disabled, this advances the durable sequence number and notifies durability waiters.
Below is a diagram illustrating the high-level flow of a write in SlateDB:
```
flowchart TD
A[API Call: put/delete/write] --> B[Create WriteBatch]
B --> C["Send to Write Task (channel)"]
C --> D[Assign Sequence Number]
D --> E{WAL Enabled?}
E -- Yes --> F[Append to WAL Buffer]
E -- No --> G[Insert into Memtable]
F --> G[Insert into Memtable]
G --> H{Memtable Full?}
H -- Yes --> I[Freeze Memtable]
I --> J[Flush Immutable Memtable to SSTable]
F --> K{WAL Buffer Full or Flush Interval?}
K -- Yes --> L[Flush WAL to SSTable]
J & L --> M[Write SSTable to Object Storage]
M --> N[Send Durability Notification]
```
## User-supplied sequence numbers
[Section titled “User-supplied sequence numbers”](#user-supplied-sequence-numbers)
Every committed write batch is stamped with a `u64` sequence number. By default SlateDB’s internal *oracle* — a monotonic counter shared by all writers — assigns one as the batch is dequeued from the write channel. [`WriteOptions::seqnum`](https://docs.rs/slatedb/latest/slatedb/config/struct.WriteOptions.html#structfield.seqnum) lets the caller override that counter and stamp the batch with a specific value instead.
### Semantics
[Section titled “Semantics”](#semantics)
The default value is `0`, which means *“let the oracle assign the seqnum”*. Sequence numbers issued by the oracle start at `1`, so `0` is unambiguously a sentinel. Any non-zero value is treated as a request to commit at that exact sequence number, subject to one rule:
* The supplied seqnum must be **strictly greater** than the current max sequence number known to the database. Otherwise the write fails with a [`slatedb::Error`](https://docs.rs/slatedb/latest/slatedb/enum.Error.html) of kind `Invalid`, with the message `invalid sequence number, must be greater than the current max. provided=..., current=...`.
When the check passes, the oracle is advanced to the supplied value, so subsequent auto-assigned writes resume above it. User-supplied and oracle-assigned writes can be mixed freely as long as the seqnums monotonically increase across the write stream.
```rust
use slatedb::config::{PutOptions, WriteOptions};
// Stamp this write with seqnum = 42 (e.g., the offset returned by your
// external WAL after appending the record).
let opts = WriteOptions { seqnum: 42, ..Default::default() };
db.put_with_options(b"key", b"value", &PutOptions::default(), &opts).await?;
```
### Ordering responsibility
[Section titled “Ordering responsibility”](#ordering-responsibility)
Sequence numbers are assigned on the single-writer event loop that drains the write channel — but the *caller* picks the value before the batch is enqueued. That means concurrent writers on your side can race:
```text
thread A: pick seqnum 1, send batch to channel
thread B: pick seqnum 2, send batch to channel
```
If the OS schedules thread B’s send before thread A’s, the seqnum `2` batch arrives first, advances the oracle, and the seqnum `1` batch is rejected as no-longer-greater-than-current.
To avoid this, funnel all writes that use `seqnum` through a single ordering agent on your side — typically the same component that assigned the seqnums (e.g., the producer that appends to your external log). One writer, one channel, no races. If you genuinely need multiple writer threads, serialize them behind a mutex or a single-producer queue before calling `put`/`write`.
### Caveats
[Section titled “Caveats”](#caveats)
* **Non-contiguous seqnums.** Auto-assigned seqnums are not guaranteed to be strictly contiguous either (the memtable flusher tolerates gaps), so user-supplied jumps are consistent with existing behavior. Anything that relies on dense seqnums — don’t.
* **Recovery.** On restart, SlateDB recovers the max seqnum from the manifest and any unflushed WAL. If you continue assigning seqnums from your external log, make sure the next value is above whatever SlateDB recovered, or your first post-restart write will be rejected.
* **Transactions.** Conflict detection still works on user-supplied seqnums — the read/write set is tracked against the snapshot’s seqnum and the commit seqnum, regardless of who picked them. Just keep the monotonic-increase invariant.
* **Durability.** The oracle is advanced as soon as the batch is processed by the write loop, before durability is confirmed. The seqno is still consumed if a later durability wait fails. Use the returned `WriteHandle::await_durable()` when confirmation is required.
# FAQ
> Frequently asked questions about SlateDB
## Is SlateDB for OLTP or OLAP?
[Section titled “Is SlateDB for OLTP or OLAP?”](#is-slatedb-for-oltp-or-olap)
SlateDB is designed for key/value (KV) online transaction processing (OLTP) workloads. It is optimized for lowish-latency, high-throughput writes. It is not optimized for analytical queries that scan large amounts of columnar data. For online analytical processing (OLAP) workloads, we recommend checking out [Tonbo](https://github.com/tonbo-io/tonbo).
## Can’t I use S3 as a key-value store?
[Section titled “Can’t I use S3 as a key-value store?”](#cant-i-use-s3-as-a-key-value-store)
You can definitely use S3 as a key-value store. An object path would represent a key and the object its value. But then you pay one PUT per write, which gets expensive.
To make writes cheaper, you will probably want to batch writes (write multiple key-value pairs in a single `PUT` call). Batched key-value pairs need to be encoded/decoded, and a sorted strings table (SST) is a natural fit. Once you have SSTs, a log-structured merge-tree (LSM) is a natural fit.
## Why does SlateDB have a write-ahead log?
[Section titled “Why does SlateDB have a write-ahead log?”](#why-does-slatedb-have-a-write-ahead-log)
Some developers have asked why SlateDB needs a write-ahead log (WAL) if we’re batching writes. Couldn’t we just write the batches directly to level 0 (L0) as an SST?
We opted to have the WAL separate from the L0 SST so that we could frequently write SSTs without increasing the number of L0 SSTs we have. Since reads are served from L0 SSTs, having too many of them would result in a very large amount of metadata that needs to be managed in memory. By contrast, we don’t serve reads from the durable WAL, we only use it for recovery.
A separate WAL lets SlateDB frequently flush writes to object storage (to reduce durable write latency) without having to worry about the number of L0 SSTs we have.
## How is this different from RocksDB-cloud?
[Section titled “How is this different from RocksDB-cloud?”](#how-is-this-different-from-rocksdb-cloud)
RocksDB-cloud does not write its write-ahead log (WAL) to object storage. Instead, it supports either local disk, Kafka, or Kinesis for the WAL. Users must decide whether they want to increase operation complexity by adding a distributed system like Kafka or Kinesis to their stack, whether they want to use local disk for the WAL, or whether they want to use EBS or EFS for the WAL.
SlateDB, by contrast, writes everything (including the WAL) to object storage. This offers a simpler architecture, better durability, and potentially better availability at the cost of increased write latency (compared to local).
RocksDB-cloud also does not cache writes, so recently written data must be read from object storage even if reads occur immediately after writes. SlateDB, by contrast, caches recent writes in memory and also supports optional local-disk caching of object-store data.
Finally, it’s unclear how open RocksDB-cloud is to outside contributors. The code is available, but there is very little documentation or community.
## How is this different from RocksDB on EBS?
[Section titled “How is this different from RocksDB on EBS?”](#how-is-this-different-from-rocksdb-on-ebs)
Amazon Web Services (AWS) elastic block storage (EBS) runs in a single availability zone (AZ). To get SlateDB’s durability and availability (when run on S3 standard buckets), you would need to replicate the across three AZs. This complicates the main write path (you would need synchronous replication) and add to cost.
S3 is inherently much more flexible and elastic. You don’t need to overprovision, you don’t need to manage volume sizes, and you don’t have to worry about transient space amplification from compaction.
S3 also allows for more cost/perf/availability tradeoffs. For example, users can sacrifice some availability by running one node in front of S3 and replacing it on a failure. [The Cloud Storage Triad: Latency, Cost, Durability](https://materializedview.io/p/cloud-storage-triad-latency-cost-durability) talks more about these tradeoffs.
## How is this different from RocksDB on EFS?
[Section titled “How is this different from RocksDB on EFS?”](#how-is-this-different-from-rocksdb-on-efs)
Amazon Web Services (AWS) elastic file system (EFS) is very expensive ($0.30/GB-month for storage, $0.03/GB for reads, and $0.06/GB for writes). It’s also unclear how well RocksDB works with network file system (NFS) mounted filesystems, and whether [close-to-open consistency](https://docs.aws.amazon.com/efs/latest/ug/features.html#consistency) breaks any internal assumptions in RocksDB.
## How is this different from DynamoDB?
[Section titled “How is this different from DynamoDB?”](#how-is-this-different-from-dynamodb)
DynamoDB has a different cost structure and API than SlateDB. In general, SlateDB will be cheaper. DynamoDB charges $0.1/GiB for storage. If you use S3 standard with SlateDB, storage starts at $0.023/GiB (nearly 5 times cheaper).
S3 standard charges $0.005 per-1000 writes (PUT, DELETE, etc.) and $0.0004 per-1000 reads. DynamoDB charges in read and write request units (RRU and WRU, respectively). Writes cost $1.25 per-million write units and $0.25 per-million read units. Depending on consistency and data size, a single request can cost multiple units (see [here](https://aws.amazon.com/dynamodb/pricing/on-demand/) for details). SlateDB batches writes by default, so it’s usually going to have a less expensive API bill. If you batch DyanmoDB writes, you might be able to get similar fees.
DynamoDB offers 99.999% SLA while an S3 standard bucket offers 99.99%, so DynamoDB is more available.
DynamoDB also requires partitioning. SlateDB doesn’t have partitioning. Instead, you must build a partitioning scheme on top of SlateDB if you need it. Though, since SlateDB fences stale writers, partition management should be fairly straightforward.
SlateDB also offers some unique features like the ability to create snapshot clones of a database at a specific point in time.
## What happens if the process goes down before SlateDB flushes data to object storage?
[Section titled “What happens if the process goes down before SlateDB flushes data to object storage?”](#what-happens-if-the-process-goes-down-before-slatedb-flushes-data-to-object-storage)
Any in-flight data that hasn’t yet been flushed to object storage will be lost.
SlateDB’s write APIs return after updating the in-memory WAL and MemTable. Call `handle.await_durable().await` on the returned [`WriteHandle`](https://docs.rs/slatedb/latest/slatedb/struct.WriteHandle.html) to wait for one write to reach object storage, or call `db.flush().await` to flush all pending writes.
## Does SlateDB support column families?
[Section titled “Does SlateDB support column families?”](#does-slatedb-support-column-families)
SlateDB does not support [column families](https://github.com/facebook/rocksdb/wiki/column-families), but it does offer [segments](https://github.com/slatedb/slatedb/blob/main/rfcs/0024-segment-oriented-compaction.md), which partition a single keyspace into independent, prefix-derived key ranges that each get their own compaction and retention lifecycle. If you instead need fully isolated keyspaces with independent configuration, opening multiple SlateDB databases is cheap and remains the recommended approach.
## Are there any limits to key and value sizes?
[Section titled “Are there any limits to key and value sizes?”](#are-there-any-limits-to-key-and-value-sizes)
Keys are limited to a maximum of 65 KiB (65,535 bytes). Values are limited to a maximum of 4 GiB (4,294,967,295 bytes). Larger values require more memory and will take longer to write, so we recommend testing performance at your expected value size to ensure it meets your requirements.
# Introduction
> Learn about SlateDB, an embedded database built on object storage
SlateDB is an embedded storage engine built as a [log-structured merge-tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree). Unlike traditional LSM-tree storage engines, SlateDB writes all data to object storage.
## Vision
[Section titled “Vision”](#vision)
Object storage is an amazing technology. It provides highly-durable, highly-scalable, highly-available storage at a great cost. And recent advancements have made it even more attractive:
* Google Cloud Storage supports multi-region and dual-region buckets for high availability.
* All object stores support compare-and-swap (CAS) operations.
* Amazon Web Service’s S3 Express One Zone has single-digit millisecond latency.
We believe that the future of object storage are multi-region, low latency buckets that support atomic CAS operations. Inspired by [The Cloud Storage Triad: Latency, Cost, Durability](https://materializedview.io/p/cloud-storage-triad-latency-cost-durability), we set out to build a storage engine built for the cloud. SlateDB is that storage engine.
## Features
[Section titled “Features”](#features)
* **Zero-Disk architecture**: SlateDB is easy to operate. It can run as an in-process storage engine with no local state, no control plane, and no replication protocol.
* **Single-writer**: SlateDB is designed for a single writer. Partitioning can easily be built on top of SlateDB since fencing is supported.
* **Multiple-readers**: Multiple readers on different nodes can all read the same SlateDB database.
* **Read caching**: SlateDB supports in-memory and (optional) on-disk read caching to reduce latency and API cost.
* **Snapshot isolation**: SlateDB supports snapshot isolation, which allows readers and writers to see a consistent view of the database.
* **Transactions**: Transactional writes are supported.
* **Object store persistence**: SlateDB writes all data to object storage, which means SlateDB has the same durability, scalability, and availability as your object store.
* **Writer fencing**: SlateDB enforces writer fencing. Zombie writer processes are detected and prevented from writing to the database.
* **Pluggable compaction**: SlateDB supports pluggable compaction, so you can use the compaction strategy that fits your needs.
## Use Cases
[Section titled “Use Cases”](#use-cases)
SlateDB is a great fit for use cases that are tolerant to 50-100ms write latency, are tolerant to data loss during failure, or are willing to pay for frequent API PUT calls. Such use cases include:
* Stream processing
* Serverless functions
* Durable execution
* Workflow orchestration
* Durable caches
* Data lakes
# Quick Start
> Get started with SlateDB in minutes
SlateDB is implemented in Rust and ships official bindings for Go, Java, Node.js, and Python. Pick your language below to install the library and run a minimal example backed by an in-memory object store.
## Installation
[Section titled “Installation”](#installation)
* Rust
```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
```
* Go
```bash
go get slatedb.io/slatedb-go
```
The Go binding uses cgo and links against the `slatedb_uniffi` shared library. You need Go 1.25+, `CGO_ENABLED=1`, a C toolchain, and the `slatedb_uniffi` library on your loader path.
* Java
Gradle:
```gradle
dependencies {
implementation("io.slatedb:slatedb-uniffi:")
}
```
Maven:
```xml
io.slatedb
slatedb-uniffi
<version>
```
Requires Java 22 or newer.
* Node.js
```bash
npm install @slatedb/uniffi
```
Requires Node.js 20 or newer.
* Python
```bash
pip install slatedb
```
Requires Python 3.10 or newer.
## Usage
[Section titled “Usage”](#usage)
SlateDB reads and writes through an object store. Every binding exposes `ObjectStore.resolve(...)` which accepts any URL supported by Rust’s [`object_store`](https://docs.rs/object_store/latest/object_store/fn.parse_url_opts.html) crate, including `memory:///`, `file:///...`, `s3://`, `gs://`, and `az://`. Pass the database path separately to `Db::open`, `DbBuilder`, or `NewDbBuilder`, as the examples below do.
* Rust
SlateDB uses [`tokio`](https://crates.io/crates/tokio) as its async runtime and [`object_store`](https://docs.rs/object_store/latest/object_store/) to interact with object storage.
main.rs
```rust
use slatedb::{object_store::memory::InMemory, Db, Error};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Error> {
// Setup
let object_store = Arc::new(InMemory::new());
let kv_store = Db::open("/tmp/slatedb_full_example", object_store).await?;
// Put
let key = b"test_key";
let value = b"test_value";
kv_store.put(key, value).await?;
// Get
assert_eq!(kv_store.get(key).await?, Some("test_value".into()));
// Delete
kv_store.delete(key).await?;
assert!(kv_store.get(key).await?.is_none());
kv_store.put(b"test_key1", b"test_value1").await?;
kv_store.put(b"test_key2", b"test_value2").await?;
kv_store.put(b"test_key3", b"test_value3").await?;
kv_store.put(b"test_key4", b"test_value4").await?;
// Scan over unbound range
let mut iter = kv_store.scan(..).await?;
let mut count = 1;
while let Ok(Some(item)) = iter.next().await {
assert_eq!(item.key, format!("test_key{count}").into_bytes());
assert_eq!(item.value, format!("test_value{count}").into_bytes());
count += 1;
}
// Scan over bound range
let mut iter = kv_store.scan("test_key1"..="test_key2").await?;
let kv1 = iter.next().await?.unwrap();
assert_eq!(kv1.key, b"test_key1".as_slice());
assert_eq!(kv1.value, b"test_value1".as_slice());
let kv2 = iter.next().await?.unwrap();
assert_eq!(kv2.key, b"test_key2".as_slice());
assert_eq!(kv2.value, b"test_value2".as_slice());
// Seek ahead to next key
let mut iter = kv_store.scan(..).await?;
let next_key = b"test_key4";
iter.seek(next_key).await?;
let kv4 = iter.next().await?.unwrap();
assert_eq!(kv4.key, b"test_key4".as_slice());
assert_eq!(kv4.value, b"test_value4".as_slice());
assert_eq!(iter.next().await?, None);
// Close
kv_store.close().await?;
Ok(())
}
```
* Go
```go
package main
import (
"bytes"
"fmt"
slatedb "slatedb.io/slatedb-go/uniffi"
)
func main() {
store, err := slatedb.ObjectStoreResolve("memory:///")
if err != nil {
panic(err)
}
defer store.Destroy()
builder := slatedb.NewDbBuilder("example-db", store)
defer builder.Destroy()
db, err := builder.Build()
if err != nil {
panic(err)
}
defer db.Destroy()
if _, err := db.Put([]byte("hello"), []byte("world")); err != nil {
panic(err)
}
value, err := db.Get([]byte("hello"))
if err != nil {
panic(err)
}
if value == nil || !bytes.Equal(*value, []byte("world")) {
panic("unexpected value")
}
fmt.Println(string(*value))
if err := db.Shutdown(); err != nil {
panic(err)
}
}
```
Handle types own Rust-side resources: call `Shutdown()` on databases and readers, and `Destroy()` on handles when you are done with them.
* Java
Most database operations return `CompletableFuture`. Native-backed handles implement `AutoCloseable` and should be closed.
```java
import io.slatedb.uniffi.Db;
import io.slatedb.uniffi.DbBuilder;
import io.slatedb.uniffi.ObjectStore;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
public final class Main {
private static T await(CompletableFuture future) throws Exception {
return future.get(30, TimeUnit.SECONDS);
}
public static void main(String[] args) throws Exception {
byte[] key = "hello".getBytes(StandardCharsets.UTF_8);
byte[] value = "world".getBytes(StandardCharsets.UTF_8);
try (ObjectStore store = ObjectStore.resolve("memory:///");
DbBuilder builder = new DbBuilder("demo-db", store)) {
Db db = await(builder.build());
try (db) {
await(db.put(key, value));
byte[] read = await(db.get(key));
if (read == null || !Arrays.equals(read, value)) {
throw new IllegalStateException("unexpected value");
}
System.out.println(new String(read, StandardCharsets.UTF_8));
await(db.shutdown());
}
}
}
}
```
* Node.js
Keys and values are binary; pass `Buffer` or `Uint8Array`. Most operations are async and should be awaited. Native-backed handles also expose `dispose()` for deterministic cleanup after `shutdown()` or when abandoning a builder.
```js
import assert from "node:assert/strict";
import { DbBuilder, ObjectStore } from "@slatedb/uniffi";
async function main() {
const store = ObjectStore.resolve("memory:///");
let db;
try {
const builder = new DbBuilder("demo-db", store);
try {
db = await builder.build();
} finally {
builder.dispose();
}
const key = Buffer.from("hello");
const value = Buffer.from("world");
await db.put(key, value);
const read = await db.get(key);
assert.deepEqual(read, value);
console.log(Buffer.from(read).toString("utf8"));
} finally {
if (db != null) {
await db.shutdown();
db.dispose();
}
store.dispose();
}
}
main().catch((error) => {
console.error(error);
process.exitCode = 1;
});
```
* Python
Most database operations are `async` and should be used with `asyncio`. `WriteBatch` is mutated synchronously and then consumed by `db.write(...)`. Call `db.shutdown()` to close native resources.
```python
import asyncio
from slatedb.uniffi import (
DbBuilder,
IsolationLevel,
KeyRange,
ObjectStore,
WriteBatch,
)
async def main() -> None:
store = ObjectStore.resolve("memory:///")
builder = DbBuilder("demo-db", store)
db = await builder.build()
try:
await db.put(b"user:1", b"Alice")
value = await db.get(b"user:1")
assert value == b"Alice"
batch = WriteBatch()
batch.put(b"user:2", b"Bob")
batch.put(b"user:3", b"Carol")
await db.write(batch)
scan = await db.scan_prefix(
b"user:",
KeyRange(
start=None,
start_inclusive=False,
end=None,
end_inclusive=False,
),
)
while (row := await scan.next()) is not None:
print(row.key, row.value)
tx = await db.begin(IsolationLevel.SERIALIZABLE_SNAPSHOT)
await tx.put(b"user:4", b"Dora")
await tx.commit()
finally:
await db.shutdown()
asyncio.run(main())
```
# Benchmarks
> Review SlateDB performance results and learn how to run your own benchmarks
## Release benchmarks
[Section titled “Release benchmarks”](#release-benchmarks)
SlateDB publishes release benchmark results at [benchmark.slatedb.io](https://benchmark.slatedb.io). The [slatedb/slatedb-benchmark](https://github.com/slatedb/slatedb-benchmark) repository contains the runner and workload definitions. It also stores the raw results published on the site.
The release suite starts from a shared database with 300 million records, about 120 GiB of logical data. The runner applies a fixed workload catalog to each SlateDB revision. The catalog combines relevant workloads from [YCSB Core](https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads) and RocksDB’s [`db_bench`](https://github.com/facebook/rocksdb/wiki/Benchmarking-tools). SlateDB-specific cases exercise idle behavior and transaction contention. Most workloads run 64 closed-loop clients after a five-minute warmup and record 15 minutes of activity.
## Other benchmarking tools
[Section titled “Other benchmarking tools”](#other-benchmarking-tools)
* [slatedb-bencher](https://github.com/slatedb/slatedb/tree/main/slatedb-bencher) runs configurable database, compaction, and transaction benchmarks against an object store. Its [README](https://github.com/slatedb/slatedb/blob/main/slatedb-bencher/README.md) documents the command-line options, and [benchmark-db.sh](https://github.com/slatedb/slatedb/blob/main/slatedb-bencher/benchmark-db.sh) provides an example workload matrix.
* SlateDB uses [Criterion](https://bheisler.github.io/criterion.rs/) for microbenchmarks of internal functions. The benchmark sources live in [slatedb/benches](https://github.com/slatedb/slatedb/tree/main/slatedb/benches).
## Nightly microbenchmarks
[Section titled “Nightly microbenchmarks”](#nightly-microbenchmarks)
The [nightly workflow](https://github.com/slatedb/slatedb/actions/workflows/nightly.yaml) runs the Criterion microbenchmarks on [WarpBuild](https://warpbuild.com)’s [warp-ubuntu-latest-arm64-8x](https://docs.warpbuild.com/cloud-runners) ARM runners. It also records profiles with [pprof-rs](https://github.com/tikv/pprof-rs) and uploads them to [pprof.me](https://pprof.me). The GitHub Actions job summary links to each profile.
## Benchmarking object stores
[Section titled “Benchmarking object stores”](#benchmarking-object-stores)
SlateDB benchmarks measure the database and object store together. Use [MinIO Warp](https://github.com/minio/warp) to measure raw S3-compatible object store performance without SlateDB in the request path. Warp runs concurrent GET, PUT, DELETE, and mixed-request benchmarks with configurable object sizes and concurrency. Run it from the same region and network used by your SlateDB clients, ideally on the same machine, so the network path and client capacity remain comparable.
# CLI
The SlateDB Admin CLI provides an interface for managing and maintaining SlateDB databases. It enables direct interaction with SlateDB manifests, checkpoints, and garbage collector.
## Installation
[Section titled “Installation”](#installation)
You can install the SlateDB CLI using Cargo. Run:
```plaintext
cargo install --locked slatedb-cli
```
Alternatively, build from source by cloning the repository and compiling the CLI:
```plaintext
git clone https://github.com/slatedb/slatedb.git
cd slatedb
cargo build --release -p slatedb-cli
```
The resulting binary will be available at `target/release/slatedb`.
## Setup
[Section titled “Setup”](#setup)
The CLI requires environment variables to configure access to your object store. You can set these variables in your shell environment or provide them via a `.env` file using the `--env-file` option. The required variables depend on your object store provider. See [admin.rs](https://github.com/slatedb/slatedb/blob/main/slatedb/src/admin.rs) for details.
Every command requires the `--path` option, which specifies the root directory in your object store.
## Usage
[Section titled “Usage”](#usage)
Run the CLI with `slatedb --help` to see a list of available commands and options.
## Run the standalone compactor
[Section titled “Run the standalone compactor”](#run-the-standalone-compactor)
Run compaction as a standalone process (instead of inside your `Db` clients):
```bash
slatedb --env-file .env --path run-compactor
```
The compactor runs until interrupted (Ctrl-C), then shuts down gracefully.
Note
The compactor expects the database to already exist (a manifest must be present). If you’re creating a new database, open it once before starting the compactor.
See [Run a Standalone Compactor](/docs/tutorials/standalone-compactor/) for how to disable the embedded compactor in your database clients.
## Run the standalone garbage collector
[Section titled “Run the standalone garbage collector”](#run-the-standalone-garbage-collector)
Run garbage collection as a standalone process (instead of inside your `Db` clients).
For a single foreground pass over one file type, use `run-garbage-collection`:
```bash
slatedb --env-file .env --path run-garbage-collection \
--resource compacted \
--min-age 5m
```
Valid resources are `manifest`, `wal`, `wal-fence`, `compacted`, and `compactions`.
To run garbage collection on a schedule in a standalone process, pass one or more schedules:
```bash
slatedb --env-file .env --path schedule-garbage-collection \
--manifest min_age=5m,period=1m \
--wal min_age=5m,period=1m \
--compacted min_age=5m,period=1m \
--compactions min_age=5m,period=1m
```
The scheduled process runs until interrupted (Ctrl-C), then shuts down gracefully.
Pass `--disable-boundary-files` to `run-garbage-collection` or `schedule-garbage-collection` to delete eligible manifest and compactions metadata without advancing boundary files. Use the same setting for every garbage collector operating on the database.
Note
The garbage collector expects the database to already exist (a manifest must be present). If you’re creating a new database, open it once before starting the garbage collector.
See [Run a Standalone Garbage Collector](/docs/tutorials/standalone-garbage-collector/) for how to disable the embedded garbage collector in your database clients.
# Compatibility
> What SlateDB currently guarantees across releases, storage formats, object stores, and tested targets
SlateDB follows Semantic Versioning and guarantees forward and backward compatibility for storage formats between adjacent releases. **We do not currently guarantee compile-time API compatibility.**
Shared deployments should move one release at a time. If several processes read and write the same database path, keep them within one release step of each other until the rollout is complete.
## Stored Data
[Section titled “Stored Data”](#stored-data)
A single database can contain a mix of older and newer SST files during normal operation. Readers and compactors must understand every format already present in the bucket. SlateDB version-checks its persistent formats and fails fast when it sees a version it does not understand.
Compression is part of compatibility too. A process can only read compressed SSTs for codecs that were compiled into that binary. If one deployment writes Zstd-compressed SSTs, every reader and compactor that touches that database needs the `zstd` feature enabled. [Compression](/docs/design/compression) covers the codec feature flags.
# Configuration
> How SlateDB loads Settings, which knobs live there, and what must be configured through builders
SlateDB configuration shows up in three main places:
1. [`Settings`](https://docs.rs/slatedb/latest/slatedb/config/struct.Settings.html) holds serializable runtime settings.
2. Builders such as [`DbBuilder`](https://docs.rs/slatedb/latest/slatedb/struct.DbBuilder.html) and [`DbReaderBuilder`](https://docs.rs/slatedb/latest/slatedb/struct.DbReaderBuilder.html) have `with_*` functions to handle complex types and dependency injection.
3. Operation-specific option types such as [`ReadOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.ReadOptions.html) and [`WriteOptions`](https://docs.rs/slatedb/latest/slatedb/config/struct.WriteOptions.html).
For the complete configuration surface and current defaults, read the [`slatedb::config`](https://docs.rs/slatedb/latest/slatedb/config/) rustdoc.
## Example
[Section titled “Example”](#example)
This example shows all three configuration layers in one place:
```rust
use slatedb::{
config::{DbReaderOptions, MetricLevel, ReadOptions, Settings},
db_cache::foyer::FoyerCache,
object_store::{memory::InMemory, ObjectStore},
Db, DbReader, Error,
};
use std::sync::Arc;
use std::time::Duration;
#[tokio::main]
async fn main() -> Result<(), Error> {
let object_store: Arc = Arc::new(InMemory::new());
// 1. Serializable runtime settings.
let mut settings = Settings::default();
settings.flush_interval = Some(Duration::from_millis(250));
settings.metric_level = MetricLevel::Debug;
// 2. Builder configuration for process-local components and wiring.
let db = Db::builder("example", Arc::clone(&object_store))
.with_settings(settings)
.with_db_cache(Arc::new(FoyerCache::new()))
.build()
.await?;
// 3. Per-call options for one operation.
let _value = db
.get_with_options(
b"key",
&ReadOptions::default().with_cache_blocks(false),
)
.await?;
// Reader-specific options configure the DbReader instance itself.
let mut reader_options = DbReaderOptions::default();
reader_options.skip_wal_replay = true;
let _reader = DbReader::builder("example", object_store)
.with_options(reader_options)
.build()
.await?;
Ok(())
}
```
`Settings::metric_level` controls which registered metrics are active. The default is `Info`; set it to `Debug` to enable debug-level metrics when using a metrics recorder. See [Metric levels](/docs/operations/metrics/#metric-levels) for more information.
## Garbage-collection boundary files
[Section titled “Garbage-collection boundary files”](#garbage-collection-boundary-files)
`GarbageCollectorOptions::boundary_files_enabled` controls whether manifest and compactions garbage collection advances durable boundary files before deleting eligible metadata. Boundary advancement is enabled by default. Set it to `false` for object stores that do not support conditional overwrite (`If-Match`), and use the same setting for every garbage collector operating on the database.
Metadata readers and writers always honor existing boundary files. Without boundary advancement, configure manifest and compactions garbage collection `min_age` values longer than any stale writer or compactor can remain alive.
## Reconfiguring Object Stores
[Section titled “Reconfiguring Object Stores”](#reconfiguring-object-stores)
Reconfiguring an object store only changes how SlateDB reaches storage. It does not migrate data between stores for you. The new `ObjectStore` must still have the existing database contents for the database path.
For the regular/main object store, pass the replacement store as the primary store to `Db::builder`, `DbReader::builder`, or `AdminBuilder::new`. Changing credentials, endpoints, or client implementations is fine as long as the new store still points at the same underlying bucket and prefix.
Caution
If you point the main store at a different location without first copying the manifest and SST data, SlateDB will treat that location as a different database root rather than migrating the existing database.
If you’re using a separate WAL object store, make the same kind of change through `.with_wal_object_store(...)`. If you need to move the WAL data, copy the existing WAL objects first and then reopen with `.with_wal_object_store(...)` pointing at the replacement store.
To migrate the data manually:
1. Stop writers, readers that replay WAL, compactor processes, garbage collection, and any admin jobs that can update the database state.
2. Copy the main database directories from the old main store and database path to the replacement main store and database path: `manifest/`, `compactions/`, and `compacted/`. If the database does not use a separate WAL store, copy `wal/` there too.
3. If the database uses a separate WAL object store, copy the old WAL store’s `wal/` directory to the replacement WAL store under the same database path. Do not move `manifest/`, `compactions/`, or `compacted/` into the WAL store; those remain in the main store.
4. Copy whole directory contents, not only the files referenced by the latest manifest. Older manifests, WAL SSTs, and compacted SSTs can still be needed until checkpoints and garbage collection release them.
5. Reopen SlateDB with the replacement store configuration.
# Data Modeling
> Best practices for designing keys in SlateDB
SlateDB uses bytewise lexicographic ordering for all keys. This is the same approach used by FoundationDB, DynamoDB, and RocksDB (by default). Custom comparators are intentionally not supported. This is a deliberate design decision that reduces complexity and avoids bugs observed in other implementations.
This means you are responsible for constructing keys that sort correctly under byte comparison. This document provides patterns for encoding common data types and designing effective key schemas.
## Key Constraints
[Section titled “Key Constraints”](#key-constraints)
Before diving into encoding patterns, be aware of SlateDB’s key and value limits:
| Constraint | Limit |
| ------------------ | ----------------------------- |
| Maximum key size | 65,535 bytes |
| Maximum value size | 4,294,967,295 bytes |
| Minimum key size | 1 byte (keys cannot be empty) |
## How Keys Are Used
[Section titled “How Keys Are Used”](#how-keys-are-used)
Understanding how SlateDB uses keys internally helps explain why key design matters for performance.
### Sorted Storage
[Section titled “Sorted Storage”](#sorted-storage)
All data in SlateDB is stored sorted by key. The in-memory MemTable uses a skip list that maintains key order, and on-disk SSTables store keys in sorted order within blocks. This sorted structure enables:
* **Efficient range scans**: Iterating over a key range (e.g., all keys starting with `user:123:`) requires no random access—just sequential reads through sorted data.
* **Binary search**: Point lookups use binary search to find the right SSTable and block, achieving O(log n) complexity instead of scanning all data.
### Prefix Compression
[Section titled “Prefix Compression”](#prefix-compression)
Within each SST block, SlateDB uses prefix compression. Each key is encoded relative to the previous key in the block, and restart points store a full key. For example, if a block contains:
```plaintext
user:123:profile
user:123:settings
user:123:preferences
```
The common prefix `user:123:` is stored once, and subsequent keys store only `settings`, `preferences`, etc. This means keys with shared prefixes compress better and use less storage. It also means that there is less of a storage penalty for repeated key prefixes.
### Block Caching
[Section titled “Block Caching”](#block-caching)
SlateDB caches SST blocks in memory. Each block contains multiple adjacent keys. When you read one key, the entire block is cached—subsequent reads for nearby keys (in sort order) are served from cache without hitting object storage.
**Implication**: Keys that are accessed together should be close together in sort order. This is why hierarchical key designs (like `tenant:user:resource`) work well—related data shares prefixes and lands in the same blocks.
### Forward-Only Iteration
[Section titled “Forward-Only Iteration”](#forward-only-iteration)
SlateDB currently supports only forward (ascending) iteration. If your access pattern requires descending order (e.g., “most recent first”), you must encode keys to sort in reverse order. See the [Descending Order](#descending-order) pattern below.
## Encoding Primitive Types
[Section titled “Encoding Primitive Types”](#encoding-primitive-types)
### Unsigned Integers
[Section titled “Unsigned Integers”](#unsigned-integers)
Use big-endian (network byte order) encoding. The most significant byte comes first, which ensures correct lexicographic ordering.
```rust
// Encoding a u64
let value: u64 = 42;
let encoded = value.to_be_bytes();
// Result: [0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x2A]
```
### Signed Integers
[Section titled “Signed Integers”](#signed-integers)
For signed integers, big-endian encoding alone doesn’t work because negative numbers have their sign bit set, making them appear “larger” than positive numbers in byte comparison.
The solution is to flip the sign bit (XOR the most significant byte with `0x80`):
```rust
fn encode_i64(value: i64) -> [u8; 8] {
let mut bytes = value.to_be_bytes();
bytes[0] ^= 0x80; // Flip sign bit
bytes
}
```
This transforms the two’s complement representation into an unsigned-comparable form where negative numbers sort before positive numbers.
### Floating Point Numbers
[Section titled “Floating Point Numbers”](#floating-point-numbers)
IEEE 754 floating point numbers require special handling:
* For **positive** floats: flip the sign bit only (XOR first byte with `0x80`)
* For **negative** floats: flip ALL bits (XOR each byte with `0xFF`)
```rust
fn encode_f64(value: f64) -> [u8; 8] {
let mut bytes = value.to_be_bytes();
if bytes[0] & 0x80 == 0 {
// Positive: flip sign bit
bytes[0] ^= 0x80;
} else {
// Negative: flip all bits
for byte in &mut bytes {
*byte ^= 0xFF;
}
}
bytes
}
```
This ensures the correct ordering: `-inf < -max < ... < -0 < +0 < ... < +max < +inf`
Note
NaN values require special handling. A common approach is to sort them after +infinity by detecting and encoding them with a special prefix.
### Strings
[Section titled “Strings”](#strings)
UTF-8 encoded strings are naturally ordered by Unicode code point, which is usually what you want.
For locale-aware sorting (e.g., case-insensitive, or treating `ä` as equivalent to `a`):
1. Apply a collation transformation using a library like ICU
2. Store the collation key as the sort key
3. Store the original string in the value if you need to retrieve it
## Composite Keys
[Section titled “Composite Keys”](#composite-keys)
Many applications need keys composed of multiple components (e.g., `tenant_id + user_id + timestamp`).
### Choosing a Delimiter
[Section titled “Choosing a Delimiter”](#choosing-a-delimiter)
The recommended delimiter is `0x00` (null byte):
* Sorts before all other byte values
* Natural choice for “end of component” semantics
**Critical requirement**: The delimiter must not appear within any component value. If your components are strings that never contain null bytes, you can use `0x00` directly.
### Escaping for Binary-Safe Keys
[Section titled “Escaping for Binary-Safe Keys”](#escaping-for-binary-safe-keys)
If components can contain arbitrary binary data (including null bytes), use an escape mechanism. Following FoundationDB’s tuple layer pattern:
* `0x00 0x00` = component terminator
* `0x00 0xFF` = literal null byte within a component
When encoding: replace each `0x00` in the data with `0x00 0xFF`, then append `0x00 0x00` to terminate.
### Variable-Length Components
[Section titled “Variable-Length Components”](#variable-length-components)
Two approaches:
1. **Null-terminated with escaping**: Use `0x00` terminator with the escape sequences above
* Preserves sort order within components
* Slightly more complex encoding/decoding
2. **Length-prefixed**: Prepend a fixed-size length before each component
* No escaping needed
* Doesn’t preserve sort order of the content itself (lengths sort first)
**Optimization**: Place variable-length components last when possible—no delimiter is needed for the final component.
### Fixed-Length Components
[Section titled “Fixed-Length Components”](#fixed-length-components)
No delimiter is needed between consecutive fixed-length components. Just concatenate them directly:
```rust
// topic_id (4 bytes) + offset (8 bytes) = 12 byte key
let mut key = Vec::with_capacity(12);
key.extend_from_slice(&topic_id.to_be_bytes());
key.extend_from_slice(&offset.to_be_bytes());
```
This saves space and simplifies encoding.
### Hierarchical Keys
[Section titled “Hierarchical Keys”](#hierarchical-keys)
Structure keys to enable efficient prefix scans:
```plaintext
usa#california#san_francisco#mission
usa#california#san_francisco#soma
usa#california#los_angeles#hollywood
usa#texas#austin#downtown
```
With this structure:
* Scan `usa#` to get all California and Texas data
* Scan `usa#california#` to get all California cities
* Scan `usa#california#san_francisco#` to get all SF neighborhoods
Each level of the hierarchy is independently scannable with a prefix query.
## Common Patterns
[Section titled “Common Patterns”](#common-patterns)
### Descending Order
[Section titled “Descending Order”](#descending-order)
For “most recent first” or “highest value first” access patterns:
```rust
// Option 1: Bit inversion
fn encode_descending(value: u64) -> [u8; 8] {
let mut bytes = value.to_be_bytes();
for byte in &mut bytes {
*byte ^= 0xFF;
}
bytes
}
// Option 2: Subtraction from max
fn encode_descending_alt(value: u64) -> [u8; 8] {
(u64::MAX - value).to_be_bytes()
}
```
### Timestamps
[Section titled “Timestamps”](#timestamps)
For chronological order, use big-endian encoding of Unix timestamps:
```rust
let timestamp_millis: u64 = 1704067200000;
let key = timestamp_millis.to_be_bytes();
```
For reverse chronological order (most recent first):
```rust
let key = (u64::MAX - timestamp_millis).to_be_bytes();
```
### UUIDs
[Section titled “UUIDs”](#uuids)
* **UUIDv7**: Time-ordered and naturally lexicographically sortable (recommended for new applications)
* **UUIDv4**: Random distribution, does not sort meaningfully by time
* **UUIDv1**: Can be made time-ordered with byte reordering, but UUIDv7 is simpler
### Geospatial Data
[Section titled “Geospatial Data”](#geospatial-data)
Z-order (Morton) curves and Hilbert curves can encode 2D or 3D coordinates as fixed-length integers that preserve spatial locality. These space-filling curve encodings are lexicographically sortable and work well as keys or key components.
## Key Design Tips
[Section titled “Key Design Tips”](#key-design-tips)
* **Keep keys short**: Shorter keys mean less memory usage, faster comparisons, and better cache efficiency
* **Design for access patterns**: Structure keys so your most common queries are efficient range scans
* **Co-locate related data**: Items frequently accessed together should share key prefixes
## Libraries and References
[Section titled “Libraries and References”](#libraries-and-references)
Rather than implementing encodings yourself, consider using a library:
* [storekey](https://github.com/surrealdb/storekey) - Rust crate providing a binary encoding format that ensures lexicographic sort order
For more detailed encoding algorithms and patterns:
* [FoundationDB Data Modeling](https://apple.github.io/foundationdb/data-modeling.html) - Comprehensive guide covering tuple encoding, integers, floats, and composite types
* [DynamoDB Sort Key Best Practices](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-sort-keys.html) - Hierarchical key patterns and version control
# Errors
> How SlateDB classifies public errors
All public SlateDB APIs return [`slatedb::Error`](https://docs.rs/slatedb/latest/slatedb/struct.Error.html). The part applications should treat as stable is [`Error::kind()`](https://docs.rs/slatedb/latest/slatedb/struct.Error.html#method.kind), which returns [`ErrorKind`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html). The formatted message and source chain are for logs and diagnosis; they can change between releases, so they are a poor place to hang recovery logic.
In practice, this means you should branch on the kind of failure, not on the exact text.
* [`ErrorKind::Transaction`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Transaction) means retry the whole transaction.
* [`ErrorKind::Unavailable`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Unavailable) means a dependency failed and the operation may succeed on retry.
* [`ErrorKind::Closed`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Closed) means the handle is done, and [`CloseReason`](https://docs.rs/slatedb/latest/slatedb/enum.CloseReason.html) explains why.
* [`ErrorKind::Invalid`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Invalid), [`ErrorKind::Data`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Data), and [`ErrorKind::Internal`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html#variant.Internal) usually mean you should stop and investigate instead of retrying blindly.
See the [`slatedb::ErrorKind`](https://docs.rs/slatedb/latest/slatedb/enum.ErrorKind.html) rustdoc for a complete explanation of each kind.
## Sources
[Section titled “Sources”](#sources)
[`Error`](https://docs.rs/slatedb/latest/slatedb/struct.Error.html) implements the standard source chain. This chain is useful for logs and for narrow diagnostics, such as checking whether a `Data` error came from an underlying `object_store::Error::NotFound`.
# Configuring Foyer Cache
> Configuration and tuning guide for SlateDB's FoyerHybridCache block cache
\[`FoyerHybridCache`]\[foyer-hybrid] wraps Foyer’s `HybridCache` to give SlateDB a two-tier block cache: a fast in-memory tier backed by a persistent disk tier. For background on how it fits into SlateDB’s cache layers, see the [Caching](/docs/design/caching) design doc.
## Upgrade note for existing disk caches
[Section titled “Upgrade note for existing disk caches”](#upgrade-note-for-existing-disk-caches)
When upgrading from SlateDB 0.12.x to SlateDB 0.13.0 or later, existing Foyer disk cache directories may need attention. SlateDB 0.8.x through 0.12.x used Foyer 0.18 and wrote the disk tier with Foyer’s older large-engine layout. SlateDB 0.13.0 uses Foyer 0.22’s block engine, and that engine cannot read all entries from the old on-disk layout. After upgrading, startup may log `foyer_storage` `coding error` messages while recovering the old cache directory.
These errors are cache misses from SlateDB’s point of view. SlateDB falls back to the object store, records the cache error in metrics, and reinserts accessed blocks into the new Foyer layout. A warm-up pass over the working set should therefore repopulate the cache, but expect extra object-store reads and recovery noise during that first pass. To avoid the startup noise, clear the Foyer disk cache directory as part of the upgrade and let SlateDB rebuild it.
## Disk write pipeline
[Section titled “Disk write pipeline”](#disk-write-pipeline)
Two mechanisms control whether an in-memory entry reaches disk. Both need configuration for the disk tier to be useful.
### Write policy and admission
[Section titled “Write policy and admission”](#write-policy-and-admission)
The write policy (`HybridCachePolicy`) decides *when* Foyer submits an entry to the disk flusher. `WriteOnEviction` (default) submits entries only when they are evicted from the memory tier. `WriteOnInsertion` submits them on insert.
The admission filter decides *whether* the flusher accepts a submitted entry. The default accepts everything.
These two checks run in series. Under the defaults, entries only reach the flusher when memory pressure forces an eviction. If the memory tier is large enough to hold the working set, nothing is ever evicted, and nothing reaches disk until the db is closed.
### Flusher pipeline tuning
[Section titled “Flusher pipeline tuning”](#flusher-pipeline-tuning)
Even after an entry is submitted, the default `BlockEngineConfig` settings can silently drop it. The defaults allocate a single flusher thread (`with_flushers(1)`) and a 16 MiB buffer pool (`with_buffer_pool_size`). The submit queue threshold defaults to twice the buffer pool size (32 MiB). When inserts outpace the flusher, entries are dropped in two ways: the submit queue discards entries once it exceeds `submit_queue_size_threshold`, and the IO buffer drops entries that don’t fit in the buffer pool.
If this happens, you will notice the `storage_queue_channel_overflow` / `storage_queue_buffer_overflow` counters increment, which are available in foyer’s internal metrics and accessible via `with_metrics_registry()` when you construct your cache.
Ensure that your flusher and buffer pools are configured large enough to avoid the backpressure mechanism (see section on Monitoring below for sizing guidance):
```rust
use foyer::{
BlockEngineConfig, DeviceBuilder, FsDeviceBuilder, HybridCacheBuilder, PsyncIoEngineConfig,
};
use slatedb::db_cache::CachedEntry;
use slatedb::db_cache::foyer_hybrid::FoyerHybridCache;
let cache = HybridCacheBuilder::new()
.with_name("slatedb_block_cache")
.memory(8 * 1024 * 1024 * 1024) // 8 GB memory tier
.with_weighter(|_, v: &CachedEntry| v.size())
.storage()
.with_io_engine_config(PsyncIoEngineConfig::new())
.with_engine_config(
BlockEngineConfig::new(
FsDeviceBuilder::new("/data/slatedb-cache")
.with_capacity(100 * 1024 * 1024 * 1024) // 100 GB disk tier
.build()
.unwrap(),
)
.with_block_size(64 * 1024)
.with_flushers(4)
.with_buffer_pool_size(256 * 1024 * 1024) // large buffer pool
.with_submit_queue_size_threshold(1024 * 1024 * 1024), // large queue size
)
.build()
.await
.unwrap();
let cache = FoyerHybridCache::new_with_cache(cache);
```
### Write policy choice
[Section titled “Write policy choice”](#write-policy-choice)
`WriteOnEviction` (default) only writes entries to disk under memory pressure or at `close()`. This is fine when the memory tier is smaller than the working set (so eviction happens naturally) or when you can guarantee a clean shutdown via `Db::close()` / `DbReader::close()`.
`WriteOnInsertion` writes entries to disk on insert. At `close()` time, in-memory entries already have disk copies, so the shutdown flush only waits for the current queue to purge.
Set the policy via `HybridCacheBuilder::with_policy()` before calling `.storage()`:
```rust
use foyer::{
BlockEngineConfig, DeviceBuilder, FsDeviceBuilder, HybridCacheBuilder,
HybridCachePolicy, PsyncIoEngineConfig,
};
let cache = HybridCacheBuilder::new()
// ...
.with_policy(HybridCachePolicy::WriteOnInsertion)
.storage()
// ...
.build()
.await
.unwrap();
```
## Shutdown
[Section titled “Shutdown”](#shutdown)
If the process exits without calling `close()` (`SIGKILL`, panic, or dropping the `Db` without closing), the in-memory tier is lost. Only entries that Foyer wrote to disk during normal operation survive.
A cache passed to `with_db_cache()` is owned by you, not by SlateDB: `Db::close()` and `DbReader::close()` will *not* close it. This makes it safe to share one cache across multiple databases — closing one database no longer shuts down the disk engine for the others. It also means the shutdown flush is your responsibility: call `DbCache::close()` on the cache yourself, after closing every `Db` and `DbReader` that uses it.
Caution
Always call `Db::close()` / `DbReader::close()` before shutdown, and then close the cache you passed in with `DbCache::close()`. Dropping without closing races with tokio runtime teardown and will lose in-memory cached entries.
## Monitoring
[Section titled “Monitoring”](#monitoring)
Foyer exposes internal metrics through `with_metrics_registry()` on the builder. Two counters are especially important to know whether you are dropping entries without tiering to disk:
| Counter | Meaning |
| -------------------------------- | ------------------------------------------------- |
| `storage_queue_channel_overflow` | Entries dropped because the submit queue was full |
| `storage_queue_buffer_overflow` | Entries dropped because the IO buffer was full |
If either counter is climbing, increase `with_flushers()`, `with_buffer_pool_size()`, or `with_submit_queue_size_threshold()` on your `BlockEngineConfig`.
# Logging
> Learn how to configure logging and tracing in SlateDB
## Tracing in SlateDB
[Section titled “Tracing in SlateDB”](#tracing-in-slatedb)
SlateDB uses the [`tracing`](https://github.com/tokio-rs/tracing) library for logging and diagnostics. This provides structured, contextual logging that helps debug and monitor your database operations.
Here’s a basic example showing how to consume tracing logs with `tracing_subscriber` and SlateDB:
main.rs
```rust
use slatedb::{bytes::Bytes, object_store::memory::InMemory, Db};
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Initialize tracing subscriber to see the logs
tracing_subscriber::fmt::init();
// Setup
let object_store = Arc::new(InMemory::new());
let db = Db::open("/tmp/slatedb_tracing_subscriber", object_store).await?;
// Put
let key = b"test_key";
let value = b"test_value";
db.put(key, value).await?;
// Get
assert_eq!(db.get(key).await?, Some(Bytes::from_static(value)));
// Delete
db.delete(key).await?;
assert!(db.get(key).await?.is_none());
// Close
db.close().await?;
Ok(())
}
```
# Metrics
> How to collect and export SlateDB metrics using the MetricsRecorder trait
SlateDB uses a recorder-based metrics system. You provide an implementation of the `MetricsRecorder` trait and SlateDB pushes metrics to it. If no recorder is configured, all metric operations are silently discarded with nearly no overhead.
## Configuring a recorder
[Section titled “Configuring a recorder”](#configuring-a-recorder)
Pass your recorder to `DbBuilder`:
```rust
use slatedb::{config::MetricLevel, Db, Settings};
use slatedb::object_store::memory::InMemory;
use slatedb_common::metrics::MetricsRecorder;
use std::sync::Arc;
let recorder: Arc = /* your implementation */;
let settings = Settings {
metric_level: MetricLevel::Debug,
..Settings::default()
};
let db = Db::builder("my_db", Arc::new(InMemory::new()))
.with_settings(settings)
.with_metrics_recorder(recorder)
.build()
.await
.unwrap();
```
### Metric levels
[Section titled “Metric levels”](#metric-levels)
`MetricLevel` controls which metrics are registered. Metrics below the configured level are replaced with no-op handles when they are registered.
`MetricLevel::Info` is the default and registers standard operational metrics. `MetricLevel::Debug` also registers debug-level metrics when SlateDB defines them. Configure the level with `Settings::metric_level`.
## The MetricsRecorder trait
[Section titled “The MetricsRecorder trait”](#the-metricsrecorder-trait)
`MetricsRecorder` has four registration methods, one per metric type:
```rust
pub trait MetricsRecorder: Send + Sync {
fn register_counter(&self, name: &str, description: &str, labels: &[(&str, &str)])
-> Arc;
fn register_gauge(&self, name: &str, description: &str, labels: &[(&str, &str)])
-> Arc;
fn register_up_down_counter(&self, name: &str, description: &str, labels: &[(&str, &str)])
-> Arc;
fn register_histogram(&self, name: &str, description: &str, labels: &[(&str, &str)], boundaries: &[f64])
-> Arc;
}
```
Each method returns a handle that SlateDB calls on the hot path:
| Trait | Method | Value type | Use case |
| ----------------- | ---------------- | ------------------------------- | ----------------------------- |
| `CounterFn` | `increment(u64)` | Monotonic | Request counts, bytes written |
| `GaugeFn` | `set(i64)` | Absolute | Memory usage, queue depth |
| `UpDownCounterFn` | `increment(i64)` | Additive (positive or negative) | In-flight compactions |
| `HistogramFn` | `record(f64)` | Distribution | Latency, I/O sizes |
`GaugeFn` and `UpDownCounterFn` are separate following OpenTelemetry semantics: gauges represent point-in-time snapshots (`set`), while up-down counters track additive changes (`increment` with positive or negative values).
Labels are fixed at registration time as `&[(&str, &str)]` key-value pairs. SlateDB uses labels to collapse related metrics into a single name (e.g. `slatedb.db.request_count` with an `op` label instead of separate counters per operation).
## Available metrics
[Section titled “Available metrics”](#available-metrics)
All metric names use dot-separated notation: `slatedb..`.
### Database (`slatedb.db.*`)
[Section titled “Database (slatedb.db.\*)”](#database-slatedbdb)
| Name | Type | Labels | Description |
| -------------------------------------------- | ------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| `slatedb.db.request_count` | counter | `op`: get, scan, flush | Number of DB requests |
| `slatedb.db.write_ops` | counter | | Write operations |
| `slatedb.db.write_batch_count` | counter | | Write batches |
| `slatedb.db.backpressure_count` | counter | | Backpressure events |
| `slatedb.db.l0_stall_count` | counter | `type`: num\_ssts, num\_ssts\_per\_key | L0 flush dispatch stalls by cause |
| `slatedb.db.immutable_memtable_flushes` | counter | | Immutable memtable flushes |
| `slatedb.db.total_mem_size_bytes` | gauge | | Total memory usage |
| `slatedb.db.l0_sst_count` | gauge | | L0 SST count summed across all segment trees |
| `slatedb.db.segment_max_l0_sst_count` | gauge | | Maximum L0 SST count across all segments (useful for detecting backpressure on specific segments) |
| `slatedb.db.sorted_run_count` | gauge | | Number of sorted runs across all trees (root tree plus each named segment tree per RFC-0024) |
| `slatedb.db.sst_view_count` | gauge | | Total number of SST views including L0 views and every sorted-run SST view across all trees |
| `slatedb.db.sst_count` | gauge | | Number of distinct physical SSTables (deduplicates sst\_view\_count by SsTableId, so sst\_count is at most sst\_view\_count) |
| `slatedb.db.external_db_count` | gauge | | Number of external databases referenced in the manifest |
| `slatedb.db.sst_filter_false_positive_count` | counter | | Bloom filter false positives |
| `slatedb.db.sst_filter_positive_count` | counter | | Bloom filter positives |
| `slatedb.db.sst_filter_negative_count` | counter | | Bloom filter negatives |
### Write-Ahead Log (WAL) (`slatedb.wal.*`)
[Section titled “Write-Ahead Log (WAL) (slatedb.wal.\*)”](#write-ahead-log-wal-slatedbwal)
| Name | Type | Labels | Description |
| ---------------------------------------- | ------- | ------ | ------------------------- |
| `slatedb.wal.wal_buffer_flushes` | counter | | WAL buffer flushes |
| `slatedb.wal.wal_buffer_flush_requests` | counter | | WAL buffer flush requests |
| `slatedb.wal.wal_buffer_estimated_bytes` | gauge | | Estimated WAL buffer size |
### Block cache (`slatedb.db_cache.*`)
[Section titled “Block cache (slatedb.db\_cache.\*)”](#block-cache-slatedbdb_cache)
| Name | Type | Labels | Description |
| ------------------------------- | ------- | -------------------------------------------------------------------- | -------------- |
| `slatedb.db_cache.access_count` | counter | `entry_kind`: filter, index, data\_block, stats; `result`: hit, miss | Cache accesses |
| `slatedb.db_cache.error_count` | counter | | Cache errors |
### Compactor (`slatedb.compactor.*`)
[Section titled “Compactor (slatedb.compactor.\*)”](#compactor-slatedbcompactor)
| Name | Type | Labels | Description |
| -------------------------------------------------- | ----------------- | ------ | ------------------------------------ |
| `slatedb.compactor.bytes_compacted` | counter | | Total bytes compacted |
| `slatedb.compactor.epoch` | gauge | | Current compactor epoch |
| `slatedb.compactor.last_compaction_timestamp_sec` | gauge | | Last compaction time (epoch seconds) |
| `slatedb.compactor.running_compactions` | up\_down\_counter | | Currently running compactions |
| `slatedb.compactor.total_bytes_being_compacted` | gauge | | Bytes in active compactions |
| `slatedb.compactor.total_throughput_bytes_per_sec` | gauge | | Compaction throughput |
### Garbage collector (`slatedb.gc.*`)
[Section titled “Garbage collector (slatedb.gc.\*)”](#garbage-collector-slatedbgc)
| Name | Type | Labels | Description |
| -------------------------- | ------- | ------------------------------------------------- | ----------------- |
| `slatedb.gc.deleted_count` | counter | `resource`: manifest, wal, compacted, compactions | Deleted resources |
| `slatedb.gc.count` | counter | | GC runs |
### Memtable flush (`slatedb.memtable_flush.*`)
[Section titled “Memtable flush (slatedb.memtable\_flush.\*)”](#memtable-flush-slatedbmemtable_flush)
| Name | Type | Labels | Description |
| ------------------------------------------------- | ------- | ------ | -------------------- |
| `slatedb.memtable_flush.memtable_freeze_count` | counter | | Frozen memtables |
| `slatedb.memtable_flush.checkpoint_request_count` | counter | | Checkpoint requests |
| `slatedb.memtable_flush.flush_request_count` | counter | | Flush requests |
| `slatedb.memtable_flush.l0_upload_count` | counter | | Completed L0 uploads |
| `slatedb.memtable_flush.l0_flush_count` | counter | | Completed L0 flushes |
| `slatedb.memtable_flush.manifest_refresh_count` | counter | | Manifest refreshes |
### Object store (`slatedb.object_store.*`)
[Section titled “Object store (slatedb.object\_store.\*)”](#object-store-slatedbobject_store)
All object store metrics carry four labels:
| Label | Values |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `component` | db, reader, gc, compactor |
| `store_type` | main, wal |
| `op` | get, put, delete |
| `api` | get, get\_range, get\_ranges, head, list, list\_with\_offset, list\_with\_delimiter, put, multipart\_init, multipart\_part, multipart\_complete, delete |
| Name | Type | Description |
| ----------------------------------------------- | --------- | ----------------------------------- |
| `slatedb.object_store.request_count` | counter | Total API calls (success and error) |
| `slatedb.object_store.error_count` | counter | Failed API calls |
| `slatedb.object_store.request_duration_seconds` | histogram | Per-request latency |
The instrumented store sits beneath the retrying layer, so each retry attempt is counted separately. It sits above the optional object-store cache, so these metrics count the calls SlateDB makes into the cache: a cache hit is counted as one request even though it never reaches the remote store, and the requests the cache issues against the remote store to fill a miss are not counted. Use the cache metrics below to tell hits from misses.
### Object store cache (`slatedb.object_store_cache.*`)
[Section titled “Object store cache (slatedb.object\_store\_cache.\*)”](#object-store-cache-slatedbobject_store_cache)
| Name | Type | Labels | Description |
| ---------------------------------------------- | ------- | ------ | ------------------- |
| `slatedb.object_store_cache.part_hit_count` | counter | | Cache part hits |
| `slatedb.object_store_cache.part_access_count` | counter | | Cache part accesses |
| `slatedb.object_store_cache.cache_keys` | gauge | | Cached keys |
| `slatedb.object_store_cache.cache_bytes` | gauge | | Cached bytes |
| `slatedb.object_store_cache.evicted_keys` | counter | | Evicted keys |
| `slatedb.object_store_cache.evicted_bytes` | counter | | Evicted bytes |
When the cache comes from `object_store_cache_options`, it records to the recorder configured on the database builder. When you build a `CachedObjectStore` yourself, it records to a no-op recorder unless you pass one with `with_metrics_recorder`, so these metrics stay empty until you do.
These are passed to `register_histogram` as the `boundaries` parameter.
## Using DefaultMetricsRecorder
[Section titled “Using DefaultMetricsRecorder”](#using-defaultmetricsrecorder)
`slatedb-common` ships a `DefaultMetricsRecorder` backed by atomics. It’s useful in tests and in some production scenarios that don’t have a dedicated metrics backend (e.g. periodic logging):
```rust
use slatedb_common::metrics::DefaultMetricsRecorder;
use std::sync::Arc;
let recorder = Arc::new(DefaultMetricsRecorder::new());
let db = Db::builder("test_db", object_store)
.with_metrics_recorder(recorder.clone())
.build()
.await
.unwrap();
// ... perform operations ...
let snapshot = recorder.snapshot();
for metric in snapshot.all() {
println!("{}: {:?} {:?}", metric.name, metric.labels, metric.value);
}
```
## Using the metrics-rs facade
[Section titled “Using the metrics-rs facade”](#using-the-metrics-rs-facade)
The [`metrics`](https://crates.io/crates/metrics) crate is a lightweight facade (similar to `log` for logging). Install any compatible exporter (e.g. `metrics-exporter-prometheus`) and wire it up with a thin adapter:
```rust
use metrics::{counter, describe_counter, describe_gauge, describe_histogram, gauge, histogram};
struct MetricsRsCounter(metrics::Counter);
impl CounterFn for MetricsRsCounter {
fn increment(&self, value: u64) {
self.0.increment(value);
}
}
// Gauge, UpDownCounter, and Histogram wrappers follow the same pattern.
pub struct MetricsRsRecorder;
impl MetricsRecorder for MetricsRsRecorder {
fn register_counter(
&self, name: &str, desc: &str, labels: &[(&str, &str)]
) -> Arc {
let labels: Vec<(String, String)> = labels.iter()
.map(|(k, v)| (k.to_string(), v.to_string())).collect();
describe_counter!(name.to_string(), desc.to_string());
Arc::new(MetricsRsCounter(counter!(name.to_string(), &labels)))
}
// ... other methods follow the same pattern
}
```
`MetricsRsRecorder` is stateless since the `metrics` facade manages all state globally.
## Implementing a Prometheus recorder
[Section titled “Implementing a Prometheus recorder”](#implementing-a-prometheus-recorder)
A Prometheus recorder maps each `register_*` call to a labeled time series within a `Family`:
```rust
use prometheus_client::metrics::counter::Counter as PromCounter;
use prometheus_client::metrics::family::Family;
use prometheus_client::registry::Registry;
type Labels = Vec<(String, String)>;
// Thin wrapper that bridges prometheus-client's Counter to SlateDB's CounterFn.
struct PromCounterHandle(PromCounter);
impl CounterFn for PromCounterHandle {
fn increment(&self, value: u64) {
self.0.inc_by(value);
}
}
struct PrometheusRecorder {
registry: Mutex,
counters: Mutex>>,
// ... gauges, histograms
}
impl MetricsRecorder for PrometheusRecorder {
fn register_counter(
&self, name: &str, desc: &str, labels: &[(&str, &str)]
) -> Arc {
let mut families = self.counters.lock().unwrap();
let family = families.entry(name.to_string()).or_insert_with(|| {
let f = Family::::default();
self.registry.lock().unwrap().register(name, desc, f.clone());
f
});
let labels: Labels = labels.iter()
.map(|(k, v)| (k.to_string(), v.to_string())).collect();
let counter = family.get_or_create(&labels).clone();
Arc::new(PromCounterHandle(counter))
}
// ... other methods follow the same pattern
}
```
## Implementing an OpenTelemetry recorder
[Section titled “Implementing an OpenTelemetry recorder”](#implementing-an-opentelemetry-recorder)
A recorder that maps directly to the OpenTelemetry SDK instruments:
```rust
use opentelemetry::metrics::MeterProvider;
use opentelemetry::KeyValue;
use opentelemetry_sdk::metrics::SdkMeterProvider;
struct OtelRecorder {
meter: opentelemetry::metrics::Meter,
}
impl OtelRecorder {
fn new(provider: &SdkMeterProvider) -> Self {
Self { meter: provider.meter("slatedb") }
}
}
impl MetricsRecorder for OtelRecorder {
fn register_counter(
&self, name: &str, desc: &str, labels: &[(&str, &str)]
) -> Arc {
let attrs: Vec = labels.iter()
.map(|(k, v)| KeyValue::new(k.to_string(), v.to_string()))
.collect();
let counter = self.meter.u64_counter(name.to_string())
.with_description(desc.to_string()).build();
Arc::new(OtelCounter { counter, attrs })
}
// ... other methods follow the same pattern
}
```
# Tuning
> Learn about SlateDB's performance characteristics and tuning
SlateDB’s performance characteristics are primarily determined by the object store being used. This page provides guidance on expected performance and how to tune SlateDB for your use case.
## Latency
[Section titled “Latency”](#latency)
SlateDB’s write latency is dominated by object store PUT operations. The following table shows expected latencies for different object stores:
| Object Store | Expected Write Latency | Notes |
| -------------------- | ---------------------- | -------------------------------- |
| S3 Standard | 50-100ms | Network latency dominates |
| S3 Express One Zone | 5-10ms | Single-digit millisecond latency |
| Google Cloud Storage | 50-100ms | Network latency dominates |
| Azure Blob Storage | 50-100ms | Network latency dominates |
| MinIO | 5-20ms | Depends on network and disk |
Read latency dominated by the database’s working set size and read pattern:
1. Working sets that fit in local (memorya and disk) caches will respond very quickly (< 1ms).
2. Read patterns that do sequential reads will respond very quickly (<1ms) since SST blocks are pre-fetched sequentially and cached locally.
3. Read patterns that access keys that are (lexicographically) close to each other will respond very quickly (< 1ms) since blocks are cached locally after the first read.
4. Reads that access data across the keyspace, and whose dataset is larger than a single machine’s memory or disk will, are more likely to see latency spikes similar to object storage latency levels (50-100ms for S3 standard). Such workloads can still work well, but require more tuning, partitioning, and so on.
## Throughput
[Section titled “Throughput”](#throughput)
SlateDB’s write throughput is limited by:
1. **Object store PUT rate limits**: Most object stores limit PUT operations to 3,500 requests per second per prefix.
2. **Network bandwidth**: The time it takes to upload SSTs to object storage.
Read throughput is limited by:
1. **Object store GET rate limits**: Most object stores limit GET operations to 5,500 requests per second per prefix.
2. **Network bandwidth**: The time it takes to download SSTs from object storage.
3. **Disk I/O**: The time it takes to read SSTs from disk when object store caching or [Foyer hybrid caching](https://foyer.rs/docs/tutorial/hybrid-cache) are enabled.
## Tuning
[Section titled “Tuning”](#tuning)
SlateDB provides several configuration options to tune performance. See [slatedb::config](https://docs.rs/slatedb/latest/slatedb/config/) for the current tuning surface.
### Write Performance
[Section titled “Write Performance”](#write-performance)
* **`flush_interval`**: How long SlateDB waits before flushing the mutable WAL to object storage. Lower values reduce durable write latency but increase object-store PUT frequency.
* **`l0_sst_size_bytes`**: The size of L0 SSTs. Larger SSTs provide better compression but take longer to upload.
* **`max_unflushed_bytes`**: The total amount of unflushed WAL and memtable data allowed in memory before SlateDB applies backpressure to writers.
* **`l0_max_ssts`**: The maximum number of L0 SSTs SlateDB allows before it stops flushing memtables and waits for compaction to catch up.
### Read Performance
[Section titled “Read Performance”](#read-performance)
* **`min_filter_keys`**: SlateDB only builds filters for SSTs with at least this many keys. Raising it avoids filter overhead on small SSTs.
* **`DbBuilder::with_filter_policies(...)`**: Configures the filter policies used for SST construction and evaluation. Defaults to a single bloom filter with 10 bits per key. Pass `BloomFilterPolicy::new(bits_per_key)` to tune bloom filter density (higher values reduce false positives but increase filter size), or supply a custom `FilterPolicy` (e.g., prefix bloom). See the `with_filter_policies` docs on `DbBuilder`, `CompactorBuilder`, and `DbReaderBuilder`.
* **`object_store_cache_options.root_folder`**: Enables the local disk cache for object-store data when set.
* **`object_store_cache_options.part_size_bytes`** and **`object_store_cache_options.preload_disk_cache_on_startup`**: Tune how the disk cache is chunked and optionally warmed on startup.
* **`DbBuilder::with_sst_block_size(...)`**: Sets SST block size. Smaller blocks can help point lookups; larger blocks usually favor scans and compression.
### Memory Usage
[Section titled “Memory Usage”](#memory-usage)
* **`max_unflushed_bytes`**: The primary writer-side memory bound for buffered but not yet flushed data.
# Connect SlateDB to Azure Blob Storage
> Learn how to connect SlateDB to Azure Blob Storage
This tutorial shows you how to use SlateDB on Azure Blob Storage (ABS). You would need an ABS account to complete the tutorial.
## Setup
[Section titled “Setup”](#setup)
[Install](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) the Azure CLI.
## Create Storage account
[Section titled “Create Storage account”](#create-storage-account)
The following steps creates a storage account and list the keys. This section can be skipped if you already have a storage account created.
```bash
# Set storage account names
StorageAccountName=
ContainerName=
ResourceGroupName=
# Login
az login
# Create Resource Group in the default subscription.
az group create --name $ResourceGroupName --location westus
# Create Azure Storage account.
az storage account create --name $StorageAccountName --resource-group $ResourceGroupName --location westus --sku Standard_LRS
# Create a storage container
az storage container create --name $ContainerName --account-name $StorageAccountName
# Get the keys.
az storage account keys list --resource-group $ResourceGroupName --account-name $StorageAccountName
```
## Create a project
[Section titled “Create a project”](#create-a-project)
Let’s start by creating a new Rust project:
```bash
cargo init slatedb-abs
cd slatedb-abs
```
## Add dependencies
[Section titled “Add dependencies”](#add-dependencies)
Now add SlateDB and the required dependencies to your `Cargo.toml`:
```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/azure
cargo add anyhow
```
Note
If you see “`object_store::path::Path` and `object_store::path::Path` have similar names, but are actually distinct types”, you might need to pin the `object_store` version to match `slatedb`’s `object_store` version. SlateDB also exports `slatedb::object_store` for convenience, if you’d rather use that.
## Write some code
[Section titled “Write some code”](#write-some-code)
This code demonstrates puts that wait for results to be durable, and then puts that do not wait.
main.rs
```rust
use object_store::azure::MicrosoftAzureBuilder;
use slatedb::{config::PutOptions, Db};
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// construct azure blob object store.
let blob_store = Arc::new(
MicrosoftAzureBuilder::new()
.with_account("")
.with_access_key("")
.with_container_name("")
.build()?,
);
println!("Opening the db");
let path = "/tmp/slatedb_azure_blob_storage";
let db = Db::open(path, blob_store.clone()).await?;
// Put a value and wait for the flush.
println!("Writing a value and waiting for flush");
db.put(b"k1", b"value1").await?;
println!("{:?}", db.get(b"k1").await?);
// Put 1000 keys, do not wait for it to be durable
println!("Writing 1000 keys without waiting for flush");
let write_options = slatedb::config::WriteOptions {
..Default::default()
};
for i in 0..1000 {
db.put_with_options(
format!("key{}", i).as_bytes(),
format!("value{}", i).as_bytes(),
&PutOptions::default(),
&write_options,
)
.await?;
}
// flush to make the writes durable.
println!("Flushing the writes and closing the db");
db.flush().await?;
db.close().await?;
// reopen the db and read the value.
println!("Reopening the db");
let db_reopened = Db::open(path, blob_store).await?;
println!("Reading the value from the reopened db");
// read 20 keys
for i in 0..20 {
println!(
"{:?}",
db_reopened.get(format!("key{}", i).as_bytes()).await
);
}
db_reopened.close().await?;
Ok(())
}
```
## Check the blob contents
[Section titled “Check the blob contents”](#check-the-blob-contents)
```bash
az storage blob list --container-name $ContainerName --account-name $StorageAccountName --prefix "tmp/slatedb_azure_blob_storage/" --delimiter "/" --output table
compactions/
wal/
manifest/
```
SlateDB uses four top-level directories over time:
* `manifest`: Contains the manifest files. Manifest files define the state of the DB, including the set of SSTs that are part of the DB.
* `wal`: Contains the write-ahead log files.
* `compacted`: Contains the compacted SST files (may not appear in short examples).
* `compactions`: Contains the compaction-state snapshots.
Let’s check the `wal` folder.
```bash
az storage blob list --container-name $ContainerName --account-name $StorageAccountName --prefix "tmp/slatedb_azure_blob_storage/wal/" --delimiter "/" --output table
Name Blob Type Blob Tier Length Content Type Last Modified Snapshot
----------------------------------------------------------- ----------- ----------- -------- ------------------------ ------------------------- ----------
tmp/slatedb_azure_blob_storage/wal/00000000000000000001.sst BlockBlob Hot 64 application/octet-stream 2024-09-07T01:15:49+00:00
tmp/slatedb_azure_blob_storage/wal/00000000000000000002.sst BlockBlob Hot 138 application/octet-stream 2024-09-07T01:15:49+00:00
tmp/slatedb_azure_blob_storage/wal/00000000000000000003.sst BlockBlob Hot 23388 application/octet-stream 2024-09-07T01:15:49+00:00
tmp/slatedb_azure_blob_storage/wal/00000000000000000004.sst BlockBlob Hot 64 application/octet-stream 2024-09-07T01:15:50+00:00
```
Each of these SST files is a write-ahead log (WAL) file, and each WAL file can contain many `RowEntry` values. They get flushed based on the `flush_interval` config or when `flush` is called explicitly.
```plaintext
```
# Checkpoint and Restore
> Learn how to checkpoint and restore SlateDB databases
## Creating a Checkpoint
[Section titled “Creating a Checkpoint”](#creating-a-checkpoint)
To create a checkpoint, use the `create_checkpoint` function provided by SlateDB.\
This operation captures a consistent and durable view of the database state, anchored to a specific manifest version.
main.rs
```rust
use slatedb::admin::AdminBuilder;
use slatedb::config::CheckpointOptions;
use slatedb::object_store::memory::InMemory;
use slatedb::Db;
use std::{sync::Arc, time::Duration};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let path = "/tmp/slatedb-checkpoint";
let object_store = Arc::new(InMemory::new());
let db = Db::open(path, object_store.clone()).await?;
db.close().await?;
let admin = AdminBuilder::new(path, object_store).build();
let result = admin
.create_detached_checkpoint(&CheckpointOptions {
lifetime: Some(Duration::from_secs(3600)), // expires in 1 hour
..Default::default()
})
.await?;
println!(
"Created checkpoint: ID = {}, Manifest = {}",
result.id, result.manifest_id
);
Ok(())
}
```
## Refreshing a Checkpoint
[Section titled “Refreshing a Checkpoint”](#refreshing-a-checkpoint)
To extend the lifetime of an existing checkpoint, use `Admin::refresh_checkpoint`. This updates the checkpoint’s expiration time, ensuring that the referenced SSTs remain protected from garbage collection for the specified duration.
main.rs
```rust
use slatedb::admin::AdminBuilder;
use slatedb::config::CheckpointOptions;
use slatedb::object_store::memory::InMemory;
use slatedb::Db;
use std::{sync::Arc, time::Duration};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let path = "/tmp/slatedb-checkpoint";
let object_store = Arc::new(InMemory::new());
let db = Db::open(path, object_store.clone()).await?;
db.close().await?;
let admin = AdminBuilder::new(path, object_store).build();
let result = admin
.create_detached_checkpoint(&CheckpointOptions {
lifetime: Some(Duration::from_secs(3600)), // expires in 1 hour
..Default::default()
})
.await?;
admin
.refresh_checkpoint(result.id, Some(Duration::from_secs(7200)))
.await?;
println!(
"Checkpoint refreshed with extended 2-hour lifetime. ID: {}",
result.id
);
Ok(())
}
```
## Deleting a Checkpoint
[Section titled “Deleting a Checkpoint”](#deleting-a-checkpoint)
To remove a checkpoint, use `Admin::delete_checkpoint`. This deletes the checkpoint metadata from the manifest, allowing the garbage collector to eventually reclaim any unreferenced SST files associated with that checkpoint.
main.rs
```rust
use slatedb::admin::AdminBuilder;
use slatedb::config::CheckpointOptions;
use slatedb::object_store::memory::InMemory;
use slatedb::Db;
use std::{sync::Arc, time::Duration};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let path = "/tmp/slatedb-checkpoint";
let object_store = Arc::new(InMemory::new());
let db = Db::open(path, object_store.clone()).await?;
db.close().await?;
let admin = AdminBuilder::new(path, object_store).build();
let result = admin
.create_detached_checkpoint(&CheckpointOptions {
lifetime: Some(Duration::from_secs(3600)), // expires in 1 hour
..Default::default()
})
.await?;
admin.delete_checkpoint(result.id).await?;
println!("Deleted checkpoint: ID = {}", result.id);
Ok(())
}
```
# Connect SlateDB to Google Cloud Storage
> Learn how to connect SlateDB to Google Cloud Storage
This tutorial shows you how to use SlateDB on Google Cloud Storage (GCS). You would need a GCS account to complete the tutorial.
## Setup
[Section titled “Setup”](#setup)
[Install](https://cloud.google.com/sdk/docs/install) the Google Cloud SDK (gcloud CLI).
## Create Storage bucket and Service Account
[Section titled “Create Storage bucket and Service Account”](#create-storage-bucket-and-service-account)
The following steps create a storage bucket and generate service account credentials. This section can be skipped if you already have a bucket and service account key.
```bash
# Set bucket name and project ID
BucketName=
ProjectID=
# Login to Google Cloud
gcloud auth login
# Set the default project
gcloud config set project $ProjectID
# Create a storage bucket (if needed)
gcloud storage buckets create gs://$BucketName --location=us-central1
# Create a service account for authentication
ServiceAccountName=slatedb-gcs-test
gcloud iam service-accounts create $ServiceAccountName \
--display-name="SlateDB GCS Test Service Account"
# Grant the service account access to the project (for storage operations)
gcloud projects add-iam-policy-binding $ProjectID \
--member="serviceAccount:$ServiceAccountName@$ProjectID.iam.gserviceaccount.com" \
--role="roles/storage.objectAdmin"
# Create and download a key for the service account
gcloud iam service-accounts keys create ~/slatedb-gcs-key.json \
--iam-account=$ServiceAccountName@$ProjectID.iam.gserviceaccount.com
```
Now you have your service account key file ready to use.
## Create a project
[Section titled “Create a project”](#create-a-project)
Let’s start by creating a new Rust project:
```bash
cargo init slatedb-gcs
cd slatedb-gcs
```
## Add dependencies
[Section titled “Add dependencies”](#add-dependencies)
Now add SlateDB and the required dependencies to your `Cargo.toml`:
```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/gcp
cargo add anyhow
```
Note
If you see “`object_store::path::Path` and `object_store::path::Path` have similar names, but are actually distinct types”, you might need to pin the `object_store` version to match `slatedb`’s `object_store` version. SlateDB also exports `slatedb::object_store` for convenience, if you’d rather use that.
## Write some code
[Section titled “Write some code”](#write-some-code)
This code demonstrates puts that wait for results to be durable, and then puts that do not wait.
main.rs
```rust
use object_store::gcp::GoogleCloudStorageBuilder;
use slatedb::{config::PutOptions, Db};
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// construct google cloud storage object store.
let gcs_store = Arc::new(
GoogleCloudStorageBuilder::new()
.with_service_account_path("")
.with_bucket_name("")
.build()?,
);
println!("Opening the db");
let path = "/tmp/slatedb_google_cloud_storage";
let db = Db::open(path, gcs_store.clone()).await?;
// Put a value and wait for the flush.
println!("Writing a value and waiting for flush");
db.put(b"k1", b"value1").await?;
println!("{:?}", db.get(b"k1").await?);
// Put 1000 keys, do not wait for it to be durable
println!("Writing 1000 keys without waiting for flush");
let write_options = slatedb::config::WriteOptions {
..Default::default()
};
for i in 0..1000 {
db.put_with_options(
format!("key{}", i).as_bytes(),
format!("value{}", i).as_bytes(),
&PutOptions::default(),
&write_options,
)
.await?;
}
// flush to make the writes durable.
println!("Flushing the writes and closing the db");
db.flush().await?;
db.close().await?;
// reopen the db and read the value.
println!("Reopening the db");
let db_reopened = Db::open(path, gcs_store).await?;
println!("Reading the value from the reopened db");
// read 20 keys
for i in 0..20 {
println!(
"{:?}",
db_reopened.get(format!("key{}", i).as_bytes()).await
);
}
db_reopened.close().await?;
Ok(())
}
```
## Check the bucket contents
[Section titled “Check the bucket contents”](#check-the-bucket-contents)
```bash
gcloud storage ls gs://$BucketName/tmp/slatedb_google_cloud_storage/
gs://$BucketName/tmp/slatedb_google_cloud_storage/compactions/
gs://$BucketName/tmp/slatedb_google_cloud_storage/manifest/
gs://$BucketName/tmp/slatedb_google_cloud_storage/wal/
```
SlateDB uses four top-level directories over time:
* `manifest`: Contains the manifest files. Manifest files define the state of the DB, including the set of SSTs that are part of the DB.
* `wal`: Contains the write-ahead log files.
* `compacted`: Contains the compacted SST files (may not appear in short examples).
* `compactions`: Contains the compaction-state snapshots.
Let’s check the `wal` folder:
```bash
gcloud storage ls -l gs://$BucketName/tmp/slatedb_google_cloud_storage/wal/
74 2025-10-05T14:26:52Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000001.sst
163 2025-10-05T14:26:56Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000002.sst
1047 2025-10-05T14:26:57Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000003.sst
36483 2025-10-05T14:26:58Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000004.sst
74 2025-10-05T14:27:02Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000005.sst
74 2025-10-05T15:42:19Z gs://sibasishtest/tmp/slatedb_google_cloud_storage/wal/00000000000000000006.sst
```
Each of these SST files is a write-ahead log (WAL) file, and each WAL file can contain many `RowEntry` values. They get flushed based on the `flush_interval` config or when `flush` is called explicitly.
# Range Scans
> Learn how to scan and seek over ranges of keys in SlateDB
SlateDB stores key-value pairs sorted by key in lexicographic byte order. A range scan creates an iterator over that sorted keyspace, so you can read a bounded slice, read every entry, or jump forward within an iterator.
## Scan all entries
[Section titled “Scan all entries”](#scan-all-entries)
Use `Db::scan(..)` to iterate over the full keyspace. Range scans follow bytewise key order, so `test_key10` sorts before `test_key2`. If numeric order matters, use fixed-width encodings such as `test_key02` and `test_key10`.
```rust
let mut entries = db.scan(..).await?;
while let Some(kv) = entries.next().await? {
println!("{:?} = {:?}", kv.key, kv.value);
}
```
## Choose range bounds
[Section titled “Choose range bounds”](#choose-range-bounds)
Pass a Rust range to `scan` when you know the keys you want. A bounded range like `"test_key2".."test_key4"` includes the start bound and excludes the end bound.
```rust
db.scan("test_key2".."test_key4").await?; // excludes test_key4
db.scan("test_key2"..="test_key4").await?; // includes test_key4
db.scan("test_key2"..).await?; // test_key2 and later
db.scan(.."test_key4").await?; // before test_key4
db.scan(..).await?; // all keys
```
## Seek within a scan
[Section titled “Seek within a scan”](#seek-within-a-scan)
Call `seek` on an iterator to fast-forward to the first key equal to or greater than the requested key. The seek target must stay inside the original scan range and must move forward from the iterator’s current position.
```rust
let mut iter = db.scan(..).await?;
iter.seek(b"test_key3").await?;
let kv = iter.next().await?.unwrap();
assert_eq!(kv.key.as_ref(), b"test_key3");
```
## Scan for prefixes
[Section titled “Scan for prefixes”](#scan-for-prefixes)
If your access pattern is “all keys that start with this prefix,” use `scan_prefix` instead of manually building a start and end range. The second argument restricts the scan to a range of key *suffixes* relative to the prefix; pass `..` to scan the prefix’s entire keyspace. You can pass ordinary Rust range syntax directly, including suffix ranges over byte slices or `Vec`.
```rust
let mut iter = db.scan_prefix(b"test_key", ..).await?; // all keys with the prefix
let mut iter = db.scan_prefix(b"test_key", b"2"..).await?; // test_key2 onward
```
## Complete example
[Section titled “Complete example”](#complete-example)
main.rs
```rust
use slatedb::object_store::memory::InMemory;
use slatedb::{Db, Error};
use std::sync::Arc;
#[tokio::main]
async fn main() -> Result<(), Error> {
let object_store = Arc::new(InMemory::new());
let db = Db::open("/tmp/slatedb_range_scans", object_store).await?;
db.put(b"test_key1", b"test_value1").await?;
db.put(b"test_key2", b"test_value2").await?;
db.put(b"test_key3", b"test_value3").await?;
db.put(b"test_key4", b"test_value4").await?;
db.put(b"test_key10", b"test_value10").await?;
// Scan over unbound range
let mut iter = db.scan(..).await?;
// Scans return bytewise key order
let expected = [
("test_key1", "test_value1"),
("test_key10", "test_value10"),
("test_key2", "test_value2"),
("test_key3", "test_value3"),
("test_key4", "test_value4"),
];
for (expected_key, expected_value) in expected {
let entry = iter.next().await?.unwrap();
assert_eq!(entry.key.as_ref(), expected_key.as_bytes());
assert_eq!(entry.value.as_ref(), expected_value.as_bytes());
}
assert_eq!(iter.next().await?, None);
// Scan over bound range
let mut iter = db.scan("test_key2"..="test_key3").await?;
let kv2 = iter.next().await?.unwrap();
assert_eq!(kv2.key, b"test_key2".as_slice());
assert_eq!(kv2.value, b"test_value2".as_slice());
let kv3 = iter.next().await?.unwrap();
assert_eq!(kv3.key, b"test_key3".as_slice());
assert_eq!(kv3.value, b"test_value3".as_slice());
assert_eq!(iter.next().await?, None);
// Seek ahead to next key
let mut iter = db.scan(..).await?;
let next_key = b"test_key4";
iter.seek(next_key).await?;
let kv4 = iter.next().await?.unwrap();
assert_eq!(kv4.key, b"test_key4".as_slice());
assert_eq!(kv4.value, b"test_value4".as_slice());
assert_eq!(iter.next().await?, None);
// Scan over prefix
let mut prefixed = db.scan_prefix(b"test_key1", ..).await?;
let expected_keys: [&[u8]; 2] = [b"test_key1", b"test_key10"];
for expected_key in expected_keys {
let kv = prefixed.next().await?.unwrap();
assert_eq!(kv.key.as_ref(), expected_key);
}
assert_eq!(prefixed.next().await?, None);
db.close().await?;
Ok(())
}
```
# Connect SlateDB to S3
> Learn how to connect SlateDB to Amazon S3 using LocalStack
This tutorial shows you how to connect SlateDB to S3. We’ll use LocalStack to simulate S3.
## Create a project
[Section titled “Create a project”](#create-a-project)
Let’s start by creating a new Rust project:
```bash
cargo init slatedb-playground
cd slatedb-playground
```
## Add dependencies
[Section titled “Add dependencies”](#add-dependencies)
Now add SlateDB and the required dependencies to your `Cargo.toml`:
```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/aws
cargo add anyhow
```
Note
If you see “`object_store::path::Path` and `object_store::path::Path` have similar names, but are actually distinct types”, you might need to pin the `object_store` version to match `slatedb`’s `object_store` version. SlateDB also exports `slatedb::object_store` for convenience, if you’d rather use that.
## Setup
[Section titled “Setup”](#setup)
You will need to have [LocalStack](https://localstack.cloud/) running. You can install it using [Homebrew](https://brew.sh/):
```bash
brew install localstack/tap/localstack-cli
```
```bash
localstack start -d
```
For a more detailed setup, see the [LocalStack documentation](https://docs.localstack.cloud/).
You’ll also need the AWS CLI:
```bash
brew install awscli
```
## Initialize AWS
[Section titled “Initialize AWS”](#initialize-aws)
SlateDB requires a bucket to work with S3.
### Create your S3 bucket:
[Section titled “Create your S3 bucket:”](#create-your-s3-bucket)
```bash
# Create S3 bucket
aws --endpoint-url=http://localhost:4566 s3api create-bucket --bucket slatedb --region us-east-1
```
## Write some code
[Section titled “Write some code”](#write-some-code)
Stick this into your `src/main.rs` file:
main.rs
```rust
use slatedb::Db;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let object_store = Arc::new(
object_store::aws::AmazonS3Builder::new()
// These will be different if you are using real AWS
.with_allow_http(true)
.with_endpoint("http://localhost:4566")
.with_access_key_id("test")
.with_secret_access_key("test")
.with_bucket_name("slatedb")
.with_region("us-east-1")
.build()?,
);
let db = Db::open("/tmp/slatedb_s3_compatible", object_store.clone()).await?;
// Call db.put with a key and a 64 meg value to trigger L0 SST flush
let value: Vec = vec![0; 64 * 1024 * 1024];
db.put(b"k1", value.as_slice()).await?;
db.close().await?;
Ok(())
}
```
Note
Our code example writes a 64 MiB value to object storage to trigger an L0 SST flush. This is just to show the `compacted` directory in your bucket.
## Run the code
[Section titled “Run the code”](#run-the-code)
Now you can run the code:
```bash
cargo run
```
This will write a 64 MiB value to SlateDB.
## Check the results
[Section titled “Check the results”](#check-the-results)
Now’ let’s check the database path in the bucket:
```bash
% aws --endpoint-url=http://localhost:4566 s3 ls s3://slatedb/tmp/slatedb_s3_compatible/
PRE compacted/
PRE compactions/
PRE manifest/
PRE wal/
```
There are four folders:
* `manifest`: Contains the manifest files. Manifest files define the state of the DB, including the set of SSTs that are part of the DB.
* `wal`: Contains the write-ahead log files.
* `compacted`: Contains the compacted SST files (may not appear in short examples).
* `compactions`: Contains the compaction-state snapshots.
Let’s check the `wal` folder:
```bash
% aws --endpoint-url=http://localhost:4566 s3 ls s3://slatedb/tmp/slatedb_s3_compatible/wal/
2024-09-04 18:05:57 64 00000000000000000001.sst
2024-09-04 18:05:58 67108996 00000000000000000002.sst
```
Each of these SST files is a write-ahead log (WAL) file, and each WAL file can contain many `RowEntry` values. They get flushed based on the `flush_interval` config. The last WAL file is 64 MiB because it contains the value we wrote.
Finally, let’s check the `compacted` folder:
```bash
% aws --endpoint-url=http://localhost:4566 s3 ls s3://slatedb/tmp/slatedb_s3_compatible/compacted/
2024-09-04 18:05:59 67108996 01J6ZVEZ394GCJT1PHZYY1NZGP.sst
```
Again, we see the 64 MiB SST file. This is the L0 SST file that was flushed with our value. Over time, the WAL entries will be removed, and the L0 SSTs will be compacted into higher levels.
# Run a Standalone Compactor
> Learn how to run compaction as a separate process from your database clients
By default, SlateDB starts a background compactor inside each `Db` instance. If you want to run compaction as a separate service (for example, on dedicated compute), you can:
1. Open the `Db` with compaction disabled.
2. Start a standalone compactor in a separate process.
Both processes must use the same `ObjectStore` and database path (the path is the prefix inside your object store).
## Create a Db with no compactor
[Section titled “Create a Db with no compactor”](#create-a-db-with-no-compactor)
To prevent `Db` from starting the embedded compactor, set `Settings::compactor_options` to `None` and use `Db::builder`:
main.rs
```rust
use slatedb::config::Settings;
use slatedb::object_store::local::LocalFileSystem;
use slatedb::Db;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// This example uses a local object store so it can be shared with a separate
// compactor process. In production, this could also be an object store like S3.
let local_root = std::env::temp_dir().join("slatedb-standalone-compactor-tutorial");
std::fs::create_dir_all(&local_root)?;
let object_store = Arc::new(LocalFileSystem::new_with_prefix(local_root)?);
// Disable the embedded compactor by clearing compactor options.
let settings = Settings {
compactor_options: None,
..Default::default()
};
let db = Db::builder("db", object_store)
.with_settings(settings)
.build()
.await?;
db.put(b"hello", b"world").await?;
db.flush().await?;
db.close().await?;
Ok(())
}
```
## Run a compactor without a Db
[Section titled “Run a compactor without a Db”](#run-a-compactor-without-a-db)
You can run the standalone compactor using the SlateDB CLI:
```bash
slatedb --env-file .env --path run-compactor
```
The compactor runs until interrupted (Ctrl-C), then shuts down gracefully.
Alternatively, you can embed the compactor in your own process using `CompactorBuilder` and call `run`:
main.rs
```rust
use slatedb::object_store::local::LocalFileSystem;
use slatedb::CompactorBuilder;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let local_root = std::env::temp_dir().join("slatedb-standalone-compactor-tutorial");
std::fs::create_dir_all(&local_root)?;
let object_store = Arc::new(LocalFileSystem::new_with_prefix(local_root)?);
// Build a compactor without opening a Db. The database must already exist (i.e.
// the manifest has been created).
let compactor = Arc::new(CompactorBuilder::new("db", object_store).build());
let compactor_task = {
let compactor = Arc::clone(&compactor);
tokio::spawn(async move { compactor.run().await })
};
println!("Compactor running. Press Ctrl-C to stop.");
tokio::signal::ctrl_c().await?;
compactor.stop().await?;
compactor_task.await??;
Ok(())
}
```
Note
The standalone compactor expects the database to already exist (a manifest must be present). If you’re creating a new database, open it once (like in the previous section) before starting the compactor.
# Run a Standalone Garbage Collector
> Learn how to run garbage collection as a separate process from your database clients
By default, SlateDB starts a background garbage collector inside each `Db` instance. You can also run garbage collection as a separate service:
1. Keep your existing `Db` clients running.
2. Start one or more standalone garbage collectors against the same database path.
All processes must use the same `ObjectStore` and database path (the path is the prefix inside your object store).
Unlike the compactor coordinator, the garbage collector does not require fencing or leader election. Multiple garbage collectors may run in parallel, including the embedded collectors inside writer processes.
If you want a standalone process to own all garbage collection work, set `Settings::garbage_collector_options` to `None` in your `Db` clients.
## Run from the CLI
[Section titled “Run from the CLI”](#run-from-the-cli)
For a single foreground pass over one file type, use `run-garbage-collection`:
```bash
slatedb --env-file .env --path run-garbage-collection \
--resource compacted \
--min-age 5m
```
Valid resources are `manifest`, `wal`, `wal-fence`, `compacted`, and `compactions`.
To run garbage collection on a schedule in a standalone process, pass one or more schedules:
```bash
slatedb --env-file .env --path schedule-garbage-collection \
--manifest min_age=5m,period=1m \
--wal min_age=5m,period=1m \
--compacted min_age=5m,period=1m \
--compactions min_age=5m,period=1m
```
The scheduled process runs until interrupted (Ctrl-C), then shuts down gracefully.
The command above does not collect WAL fence objects. Enable `--wal-fence` only when `min_age` is longer than any writer can run. Default garbage collection keeps WAL fence deletion in dry-run mode because deleting fences too early can allow stale WAL writes to succeed.
## Embed the garbage collector
[Section titled “Embed the garbage collector”](#embed-the-garbage-collector)
You can also embed the garbage collector in your own process using `GarbageCollectorBuilder` and call `run`:
main.rs
```rust
use slatedb::object_store::local::LocalFileSystem;
use slatedb::GarbageCollectorBuilder;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let local_root = std::env::temp_dir().join("slatedb-standalone-gc-tutorial");
std::fs::create_dir_all(&local_root)?;
let object_store = Arc::new(LocalFileSystem::new_with_prefix(local_root)?);
// Build a garbage collector without opening a Db. The database must already
// exist (i.e. the manifest has been created).
let gc = Arc::new(GarbageCollectorBuilder::new("db", object_store).build());
let gc_task = {
let gc = Arc::clone(&gc);
tokio::spawn(async move { gc.run().await })
};
println!("Garbage collector running. Press Ctrl-C to stop.");
tokio::signal::ctrl_c().await?;
gc.stop().await?;
gc_task.await??;
Ok(())
}
```
With `GarbageCollectorOptions::default()`, garbage collection runs every 60 seconds and uses a 5 minute minimum age for managed directories. WAL fence deletion stays in dry-run mode by default.
To delete manifest and compactions metadata without advancing boundary files, set `GarbageCollectorOptions::boundary_files_enabled` to `false` on every garbage collector for the database. Metadata readers and writers continue to honor any existing boundary. Without boundary advancement, a SlateDB client or compactor can begin updating a manifest or compactions file, stop making progress (for example, because its process or host is suspended), then resume after `min_age`. It can then recreate a deleted metadata ID and incorrectly report its stale update as successful. Set the manifest and compactions `min_age` values longer than the maximum lifetime of a stale process before using this mode.
Note
The standalone garbage collector expects the database to already exist (a manifest must be present). If you’re creating a new database, open it once with your client process before starting the garbage collector.
Garbage collection deletes only files that meet the configured `min_age` and are no longer referenced by the latest database state or active checkpoints.
# Manifest Design
Status: Accepted
Authors:
* [Vignesh Chandramohan](https://github.com/vigneshc)
* [Rohan Desai](https://github.com/rodesai)
* [Chris Riccomini](https://github.com/criccomini)
References:
* Add manifest design document. ([#24](https://github.com/slatedb/slatedb/pull/24))
* Add an alternative manifest design document ([#39](https://github.com/slatedb/slatedb/pull/39))
## Current Design
[Section titled “Current Design”](#current-design)
SlateDB currently holds its state in an in-memory structure called `DbState`. `DbState` looks like this:
```rust
pub(crate) struct DbState {
pub(crate) memtable: Arc,
pub(crate) imm_memtables: Vec>,
pub(crate) l0: Vec,
pub(crate) next_sst_id: usize,
}
```
These fields act as follows:
* `memtable`: The currently active mutable MemTable. `put()` calls insert key-value pairs into this field.
* `imm_memtables`: An ordered list of MemTables that are in the process of being written to object storage.
* `l0`: An ordered list of level-0 [sorted string tables](https://www.scylladb.com/glossary/sstable/) (SSTs). These SSTs are not range partitioned; they each contain a full range of key-value pairs.
* `next_sst_id`: The SST ID to use when SlateDB decides to freeze `memtable`, move it to `imm_memtables`, and flush it to object storage. This will become the MemTable’s ID on object storage.
SlateDB doesn’t have compaction implemented at the moment. We don’t have level-1+ SSTs in the database state.
### Writes
[Section titled “Writes”](#writes)
A `put()` is performed by inserting the key-value pair into `memtable`. Once `flush_interval` (a configuration value) has expired, a flusher thread (in `flush.rs`) locks `DbState` and performs the following operations:
1. Replace `memtable` with a new (empty) MemTable.
2. Insert the old `memtable` into index 0 of `imm_memtables`.
3. For every immutable MemTable in `imm_memtables`.
1. Encode the immutable MemTable as an SST.
2. Write the SST to object storage path `sst-{next_sst_id}`.
3. Remove the MemTable from `imm_memtables`.
4. Insert the SST’s metadata (`SsTableInfo`) into `l0`.
5. Increment `next_sst_id`.
6. Notify listeners that are `await`’ing a `put()` in this MemTable.
A `SsTableInfo` structure is returned when the SST is encoded in 3.1. It looks like this:
```rust
pub(crate) struct SsTableInfo {
pub(crate) first_key: Bytes,
// todo: we probably dont want to keep this here, and instead store this in block cache
// and load it from there
pub(crate) block_meta: Vec,
pub(crate) filter_offset: usize,
pub(crate) filter_len: usize,
pub(crate) block_meta_offset: usize,
}
```
The structure contains the `first_key` in the SST, block locations and key information, and the bloom filter location and size.
*NOTE: The block meta information in the `SsTableInfo` struct stores the location and first key for each block in the SST. SSTs are broken into blocks (usually 4096 bytes); each block has a sorted list of key-value pairs. This data can get somewhat large; we’ll probably move it out of the in-memory `SsTableInfo`.*
### Reads
[Section titled “Reads”](#reads)
Reads simply iterate over each MemTable and SST looking for the value for a key. This is done by:
1. Return `get(k)` from `memtable` if it exists.
2. Iterate over each `imm_memtables` (starting from index 0) and return `get(k)` if it exists.
3. Iterate over each `l0` SST (starting from index 0) and return `get(k)` if it exists.
## Problem
[Section titled “Problem”](#problem)
There are a couple of problems with the current design:
1. SlateDB loses the state of the database when the process stops.
2. SlateDB does not fence zombie writers.
If the process SlateDB is running in stops, all data in `DbState` is lost since it’s not persisted anywhere. SlateDB could read over every SST in object storage and reconstruct `l0`, but it does not.
Even if SlateDB were to recover its state by reading all SSTs in object storage, the process would be slow and could be wrong. Inaccurate `l0` state occurs if multiple writers write SSTs in parallel to the same SlateDB object store location. SlateDB assumes it’s the only writer, but it currently does not implement fencing to keep other writers from corrupting its state.
## Solution
[Section titled “Solution”](#solution)
### Goals
[Section titled “Goals”](#goals)
This design should:
* Work with a single writer, multiple readers, and a single compactor
* Allow the writer, compactor, and all readers to run on separate machines
* Allow parallel writes from a single writer
* Allow readers to snapshot state for consistent reads across time
* Work with object stores that support CAS
* Work with object stores that do not support CAS if a transactional store is available (DynamoDB, MySQL, and so on)
* Provide < 100ms-300ms read/write latencies
* Avoid write contention on the manifest
* Work for S3, Google Cloud Storage (GCS), Azure Blob Storage (ABS), MinIO, Radically Reprogrammable (R2) Storage, and Tigris
### Proposal
[Section titled “Proposal”](#proposal)
We propose persisting SlateDB’s `DbState` in a manifest file to solve problem (1). Such a design is quite common in [log-structured merge-trees](https://en.wikipedia.org/wiki/Log-structured_merge-tree) (LSMs). In particular, RocksDB [follows this pattern](https://github.com/facebook/rocksdb/wiki/MANIFEST). In SlateDB, we will persist `DbState` in a manifest file, which all clients will load on startup. Writer, compactor, and readers will all update the manifest in various situations. All updates to the manifest will be done using [compare-and-swap](https://en.wikipedia.org/wiki/Compare-and-swap) (CAS).
To prevent zombie writers, we propose using CAS to ensure each SST is written exactly one time. We introduce the concept of writer epochs to determine when the current writer is a zombie and halt the process. The full fencing protocol is described later in the *Writer Protocol* section.
### CAS on S3
[Section titled “CAS on S3”](#cas-on-s3)
Changes to the manifest file and SST writes both require CAS in this design. Most object stores provide CAS (a.k.a. preconditions, conditionals, `If-None-Match`, `If-Match`, and so on). But S3 does not, and we want to support S3. Fortunately, we can support CAS-like operations using several different patterns.
There are two scenarios to consider when we want to CAS an object:
1. The object we wish to CAS is immutable; it exists or it does not, but never changes once it exists (“create if not exists”).
2. The object we wish to CAS is mutable; it might exist and can change over time.
There are five different patterns to achieve CAS for these scenarios:
1. **Use CAS**: If the object store supports CAS, we can use it. This pattern works for both scenarios described above.
2. **Use a transactional store**: A transactional store such as DynamoDB or MySQL can be used to store the object. The object store is not used in this mode. This approach works for both scenarios described above, but does not work well for large objects.
3. **Use object versioning**: Object stores that support [object versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) can emulate CAS if the bucket has versioning enabled. Versioning allows multiple versions of an object to be stored. AWS stores versions [in insertion order](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html#API_ListObjectVersions_Example_3). We can leverage this property as well as AWS’s [read-after-write](https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/) consistency guarantees to emulate CAS. Many writers can write to the same object path. Writers can then list all versions of the object. The first version (the earliest) is considered the winner. All other writers have failed the CAS operation. This does require a [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html) after the write, but if CAS is needed, this is the best we can do. This pattern works for immutable files—scenario (1)—but not mutable files—scenario (2). This pattern does not work for S3 Express One Zone since it does not support object versioning.
4. **Use a proxy**: A proxy-based solution combines an object store and a transactional store. Two values are stored: an object and a pointer to object. Readers first read the pointer (e.g. object path), then read the object that the pointer points to. Writers read the current object (using the reader steps), update the object, and write it back to the object store at a new location (e.g. a new UUID). Writers then update the pointer to point to the new object. The pointer update is done conditionally if the pointer still points to the file the writer read. The pointer may reside in a file on an object store that supports CAS, or a transactional store like DynamoDB or MySQL. This pattern works for both immutable and mutable files. Unlike (2), it also works well for large objects.
5. **Use a two-phase write**: Like the proxy solution (4), a two-phase write consists of a write to both an object storage and a transactional store. Writers first write a new object to a temporary location. The writer then writes a record to a transactional store. The transactional record signals an intent to copy; it contains a source, destination, and a completion flag. “put-if-not-exists” semantics are used on the destination column. The writer then copies the new object from its temporary location to its final destination in object storage. Upon completion, the transactional store’s record is set to complete and the temporary file is deleted. If the writer fails before the transactional record is written, the write is abandoned. If the writer fails after the transactional record is written, a recovery process runs in the future to copy the object and set the completion flag. This pattern works for immutable objects. It also works well for large objects.
*NOTE: These patterns are discussed more [here](https://github.com/slatedb/slatedb/pull/39/files#r1588879655) and [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1597297640).*
Our proposed solution uses the following forms of CAS:
* CAS (1) for manifest and SST writes when the object store supports CAS
* Two-phase write (5) for manifest and SST writes when the object store does not support CAS
*NOTE: Two-phase write (5) was selected because the folder structure will be the same for CAS object stores. A proxy-based solution (4) would store objects at random locations; only the pointer record would contain the actual object name. (5) is in keeping with our philosophy with our belief that S3 will eventually get CAS. Once this happens, S3 clients can be replaced simply by swapping the transactional store write for a CAS write. Also, (4) requires a proxy read for every object read, while (5) requires an extra write. The extra write is preferred since it’s asynchronous (it can happen after an async acknowledgement) and obviates the proxy reads.*
#### Two-Phase CAS Protocol
[Section titled “Two-Phase CAS Protocol”](#two-phase-cas-protocol)
A description of the two-phase CAS protocol is described in this section. These are the operations to consider:
1. **Writes**: A process writes an object to object storage and fails if the object already exists.
2. **Deletes**: A process deletes an object from object storage.
3. **Recovery**: A process recovers a failed write or delete.
##### Writes
[Section titled “Writes”](#writes-1)
Suppose a writer wishes to write to the object location `manifest/00000000000000000000.manifest`. The writer follows these steps:
1. Write the new manifest to a temporary location, say `manifest/00000000000000000000.manifest.[UUID].[checksum]`.
2. Write a record to a transactional store, say DynamoDB, with the following fields:
* `source`: `manifest/00000000000000000000.manifest.[UUID].[checksum]`
* `destination`: `manifest/00000000000000000000.manifest`
* `committed`: `false`
3. Copy the object from `manifest/00000000000000000000.manifest.[UUID].[checksum]` to `manifest/00000000000000000000.manifest`.
4. Update the record in DynamoDB to set `committed` to `true`.
5. Delete the object from `manifest/00000000000000000000.manifest.[UUID].[checksum]`.
*NOTE: The source location is arbitrary. If randomness is used in the name—a UUID, ULID, or KSUID—collisions must be considered. In our case, collisions are catastrophic since they lead to corruption. We use a UUID and checksum to attempt to reduce collisions.*
Once the record is written to DynamoDB in step (2), the entire write is considered durable. No other writers may insert a record with the same `destination` field.
*NOTE: This design is inefficient on S3. A single SST write includes two S3 `PUT`s, two writes to DynamoDB (or equivalent), and one S3 `DELETE`. See [here](https://github.com/slatedb/slatedb/pull/43#issuecomment-2105368258) for napkin math on API cost. Latency should be minimally impacted since the write is considered durable after one S3 write and one DynamoDB write. Nevertheless, we are gambling that S3 will soon support pure-CAS like every other object store. In the long run, we expect to switch S3 clients to use CAS and drop support for two-phase CAS.*
##### Deletes
[Section titled “Deletes”](#deletes)
See the garbage collection section below for details on deletions.
##### Recovery
[Section titled “Recovery”](#recovery)
A writer or deletion process might fail at any point in the steps outlined above. Client processes may run recoveries at any point in the future to complete partial writes or deletes. The recovery process is as follows:
1. List all records in DynamoDB with `committed` set to `false`.
2. For each record:
1. Copy the `source` object to the `destination` object.
2. Set `committed` to `true`.
3. Delete the source object.
### File Structure
[Section titled “File Structure”](#file-structure)
Let’s look at SlateDB’s file structure on object storage:
```plaintext
some-bucket/
├─ manifest/
│ ├─ 00000000000000000000.manifest
│ ├─ 00000000000000000001.manifest
│ ├─ 00000000000000000002.manifest
├─ wal/
│ ├─ 00000000000000000000.sst
│ ├─ 00000000000000000001.sst
│ ├─ ...
├─ levels/
│ ├─ ...
```
*NOTE: Previous iterations of this design referred to the WAL as `l0` or `level_0`, and the first level in `levels` (formerly `compacted`) as `level_1+`. We’ve since shifted the terminology to match traditional LSMs. This shift is discussed in more detail [here](https://github.com/slatedb/slatedb/pull/39/files#r1589855599).*
#### `manifest/00000000000000000000.manifest`
[Section titled “manifest/00000000000000000000.manifest”](#manifest00000000000000000000manifest)
A file containing writer, compaction, and snapshot information—the state of the database at a point in time.
The manifest’s name is formatted as 20 digit zero-padded numbers to fit u64’s maximum integer and to support lexicographical sorting. The name represents the manifest’s ID. Manifest IDs are monotonically increasing and contiguous. The manifest with the highest ID is considered the current manifest. `00000000000000000002.manifest` is the current manifest in the file structure section above.
*NOTE: The current design does not address incremental updates to the manifest. This is something that prior designs had. See [here](https://github.com/slatedb/slatedb/pull/24/files#diff-58d53b55614c8db2dd180ac49237f37991d2b378c75e0245d780356e5d0c8135R67). We’ve opted to eschew incremental manifest updates for the time being, given their size (\~5MiB) and update frequency (on the order of minutes).*
##### Structure
[Section titled “Structure”](#structure)
A `.manifest` file has the following structure.
*NOTE: We describe the structure in a proto3 IDL, but the manifest could be any format we choose. I’m thinking [FlatBuffers](https://github.com/slatedb/slatedb/issues/41). See [#41](https://github.com/slatedb/slatedb/issues/41) for discussion.*
```proto
syntax = "proto3";
message Manifest {
// Manifest format version to allow schema evolution.
uint16 manifest_format_version = 1;
// The current writer's epoch.
uint64 writer_epoch = 2;
// The current compactor's epoch.
uint64 compactor_epoch = 3;
// The most recent SST in the WAL that's been compacted.
uint64 wal_id_last_compacted = 4;
// The most recent SST in the WAL at the time manifest was updated.
uint64 wal_id_last_seen = 5;
// A list of the SST table info that are valid to read in the `levels` folder.
repeated SstInfo leveled_ssts = 6;
// A list of read snapshots that are currently open.
repeated Snapshot snapshots = 7;
}
message SstInfo {
// Manifest format version to allow schema evolution.
uint16 sstinfo_format_version = 1;
// Globally unique ID for an SST in the `compacted` folder.
uint64 id = 1;
// The first key in the SST file.
string first_key = 2;
// Bloom filter offset in the SST file.
uint32 filter_offset = 3;
// Bloom filter length in the SST file.
uint32 filter_len = 4;
// Block metadata offset in the SST file.
uint32 block_meta_offset = 5;
// Block metadata offset in the SST file.
uint32 block_meta_len = 6;
}
message Snapshot {
// 128-bit UUID that must be unique across all open snapshots.
uint128 id = 1;
// The manifest ID that this snapshot is using as its `DbState`.
uint64 manifest_id = 2;
// The UTC unix timestamp seconds that a snapshot expires at. Clients may update this value.
// If `snapshot_expire_time_s` is older than now(), the snapshot is considered expired.
// If `snapshot_expire_time_s` is 0, the snapshot will never expire.
uint32 snapshot_expire_time_s = 3;
}
```
##### Size
[Section titled “Size”](#size)
Manifest size is important because manifest updates must be done transactionally. The longer the read-modify-write takes, the more likely there is to be a conflict. Consequently, we want our manifest to be as small as possible.
The size calculation (in bytes) for the manifest is:
```plaintext
2 // manifest_format_version
+ 8 // writer_epoch
+ 8 // compactor_epoch
+ 8 // wal_id_last_compacted
+ 8 // wal_id_last_seen
+ 4 + ~56 * leveled_ssts // array length + leveled_ssts (~32 byte key + 24 bytes = ~56 bytes)
+ 4 + 28 * snapshots // array length + snapshots (28 bytes each)
```
Conservatively, a manifest with 1000 snapshots and 100,000 compacted SSTs would be:
```plaintext
2 // manifest_format_version
+ 8 // writer_epoch
+ 8 // compactor_epoch
+ 8 // wal_id_last_compacted
+ 8 // wal_id_last_seen
+ 4 + ~56 * 100000 // array length + leveled_ssts
+ 4 + 28 * 1000 // array length + snapshots
= 5,628,042
```
This comes out to 5,628,042 bytes, or \~5.6 MiB. If a client gets 100MB/s to and from EC2 to S3, it would take 56ms to read and 56ms to write (plus serialization, network, and TCP overhead). All in, let’s say 250-500ms.
Whether this is reasonable or not depends on how frequently manifests are updated. As we’ll see below, updates should be infrequent enough that a 5.6 MiB manifest isn’t a problem.
#### `wal/00000000000000000000.sst`
[Section titled “wal/00000000000000000000.sst”](#wal00000000000000000000sst)
SlateDB’s WAL is a sequentially ordered contiguous list of SSTs. SST object names are formatted as 20 digit zero-padded numbers to fit u64’s maximum integer and to support lexicographical sorting. Each SST contains zero or more sorted key-value pairs. The WAL is used to store writes that have not yet been compacted.
Traditionally, LSMs store WAL data in a different format from the SSTs; this is because each `put()` call results in a single key-value write to the WAL. But SlateDB doesn’t write to the WAL on each `put()`. Instead, `put()`’s are batched together based on `flush_interval`, `flush_bytes` ([#30](https://github.com/slatedb/slatedb/issues/30)), or `skip_memtable_bytes` ([#31](https://github.com/slatedb/slatedb/issues/31)). Based on these configurations, multiple key-value pairs are stored in the WAL in a single write. Thus, SlateDB uses SSTs for both the WAL and compacted files.
*NOTE: We discussed using a different format for the WAL [here](https://github.com/slatedb/slatedb/pull/39/files#r1590087062), but decided against it.*
This design does not use [prefixes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html). AWS allows, “3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix.” This limits a single writer to 3,500 writes and 5,500 reads per-second. With a `flush_interval` set to 1ms, a client would write 1,000 writes per-second (not including compactions). With caching, the client should be far below the 5,500 reads per-second limit.
This design also does not expose writer epochs in WAL SST filenames (e.g. `[writer_epoch].[sequence_number].sst`). We discussed this in detail [here](https://github.com/slatedb/slatedb/pull/39/files#r1588892292) and [here](https://github.com/slatedb/slatedb/pull/24/files#r1585569744). Ultimately, we chose “un-namespaced” filenames (i.e. filenames without epoch prefix) because it’s simpler to reason about.
##### Attributes
[Section titled “Attributes”](#attributes)
Each SST will have the following [user-defined metadata fields](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata):
* `writer_epoch`: This is a `u64`. Writers will set this on each SST written into `wal`.
### Manifest Updates
[Section titled “Manifest Updates”](#manifest-updates)
A full read-modify-write Manifest update contains the following steps:
1. List `manifest` to find the manifest with the largest ID. Say it’s `00000000000000000002.manifest` in this example.
2. Read `manifest/00000000000000000002.manifest` to read the current manifest.
3. Update the manifest in memory.
4. Write the updated manifest to the next manifest slot (`manifest/00000000000000000003.manifest`).
(4) is a CAS operation. Clients will use either real CAS (pattern (1)) or two-phase write (pattern (5)) to update the manifest.
If the CAS write fails in (4), the client must retry the entire process. This is because the client now has an outdated view of the manifest. It must find the (new) latest manifest and re-apply its changes.
Three different processes update the manifest:
* **Readers**: Readers can update `snapshots` to create, remove, or update a snapshot. Readers can also update `wal_id_last_seen` to update the latest `wal` SST ID when a new snapshot is created.
* **Writers**: Writers must update `writer_epoch` once on startup.
* **Compactors**: Compactors must update `compactor_epoch` once on startup. Compactors must also update `wal_id_last_compacted`, `wal_id_last_seen`, and `leveled_ssts` after each compaction pass.
The union of these three processes means the manifest is updated whenever:
1. A reader creates a snapshot on startup
2. A reader updates a snapshot periodically
3. A reader removes a snapshot when finished
4. A writer increments `writer_epoch` and creates a snapshot on startup
5. A compactor increments `compactor_epoch` on startup
6. A compactor finishes a compaction periodically
If these occur too frequently, there might be conflict between the clients. Conflict can lead to starvation, which can slow down startup time and compaction time. Luckily, all events except (6) should occur at the minute granularity or larger. Compaction, (6), warrants some additional thought, which we discuss at the end of this document.
Let’s look at each of these in turn.
### Snapshots
[Section titled “Snapshots”](#snapshots)
This design introduces the concept of snapshots, which allow clients to:
1. Prevent compaction from deleting SSTs that the client is still using
2. Share a consistent view of the database state across multiple clients
A snapshot has three fields: `id`, `manifest_id`, and `snapshot_expire_time_s`.
* `id`: A UUID that must be unique across all open snapshots.
* `manifest_id`: The manifest ID that this snapshot is using as its `DbState`.
* `snapshot_expire_time_s`: The UTC Unix epoch seconds that the snapshot expires at.
Each client will use a different `id` for each snapshot it creates. Similarly, no two clients will share the same `id` for any snapshot. Clients that wish to share the same view of a database will each define different snapshots with the same `manifest_id`.
Clients set the `snapshot_expire_time_s` when the snapshot is created. Clients may update their snapshot’s `snapshot_expire_time_s` at their discretion by writing a new manifest with the updated value. A snapshot is considered expired if `snapshot_expire_time_s` is less than the current time. If `snapshot_expire_time_s` is 0, the snapshot will never expire.
*NOTE: Clients that set `snapshot_expire_time_s` to 0 must guarantee that they will eventually remove their snapshot from the manifest. If they do not, the snapshot’s SSTs will never be removed.*
A client creates a snapshot by creating a new manifest with a new snapshot added to the `snapshots` field.
A client may also update `wal_id_last_seen` in the new manifest to include the most recent SST in the WAL that the client has seen. This allows clients to include the most recent SSTs from the `wal` in a new snapshot. It also allows clients to determine which `wal` SSTs to ignore from the head of the WAL if the client is restarted. See [here](https://github.com/slatedb/slatedb/pull/39/files#r1588780707) and [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1601924366) for more details.
A new snapshot may only reference an active manifest (see [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1594444035) and [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1601981496) for more details). A manifest is considered active if either:
* It’s the current manifest
* It’s referenced by a snapshot in the current manifest and the snapshot has not expired
A compactor may not delete SSTs that are referenced by any active manifest.
The set of SSTs that are referenced by a manifest are:
* All `levels` files referenced in the manifest’s `leveled_ssts`
* All `wal` files with SST ID >= `wal_id_last_compacted`
*NOTE: The inclusive `>=` for `wal_id_last_compacted` is required so the compactor doesn’t delete the most recently compacted file. We need this file to get the `writer_epoch` during the recovery process detailed in the Read Clients section below.*
*NOTE: Clock skew can affect the timing between the compactor and the snapshot clients. We’re assuming we have well behaved clocks, a [Network Time Protocol](https://www.ntp.org/documentation/4.2.8-series/ntpd/) (NTP) daemon, or [PTP Hardware Clocks](https://aws.amazon.com/blogs/compute/its-about-time-microsecond-accurate-clocks-on-amazon-ec2-instances/) (PHCs).*
*NOTE: A [previous design proposal](https://github.com/slatedb/slatedb/pull/39/files#diff-d589c7beb3d163638e94dbc8e086b3efe093852f0cad96f04cb1283c3bd1eb74R105) used a `heartbeat_s` field that clients would update periodically. After some discussion (see [here](https://github.com/slatedb/slatedb/pull/39/files#r1588896947)), we landed on a design that supports both reference counts and snapshot timeouts. Reference counts are useful for long-lived snapshots that exist indefinitely. Heartbeats are useful for short-lived snapshots that exist for the lifespan of a single process.*
*NOTE: This design considers read-only snapshots. Read-write snapshots are discussed [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1596319141) and in \[[#49](https://github.com/slatedb/slatedb/issues/49)].*
### Writers
[Section titled “Writers”](#writers)
#### `writer_epoch`
[Section titled “writer\_epoch”](#writer_epoch)
This design introduces the concept of a `writer_epoch`. The `writer_epoch` is a monotonically increasing `u64` that is transactionally incremented by a writer on startup. The `writer_epoch` is used to prevent split-brain and fence zombie writers. A zombie writer is a writer with an epoch that is less than the `writer_epoch` in the current manifest.
`writer_epoch` exists in two places:
1. As a field in the manifest
2. As a user-defined metadata field in each `wal` SST
#### Writer Protocol
[Section titled “Writer Protocol”](#writer-protocol)
On startup, a writer client must increment `writer_epoch`.
1. List `manifest` to find the manifest with the largest ID.
2. Read the current manifest (e.g. `manifest/00000000000000000002.manifest`).
3. Increment the `writer_epoch` in the current manifest in memory.
4. Create a new snapshot in the current manifest in memory.
5. Write the manifest with the updated `writer_epoch` (e.g. `manifest/00000000000000000003.manifest`).
*NOTE: A snapshot is created in (4) to prevent the compactor from deleting `wal` SSTs while the writer is writing its fencing SST. The timeout for the snapshot will be twice as long as the writer’s snapshot refresh interval. See [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1594460226) and [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1601806626) for details.*
The writer client must then fence all older clients. This is done by writing an empty SST to the next SST ID in the WAL.
2. List the `wal` directory to find the next SST ID.
3. Write an empty SST with the new `writer_epoch` to the next SST ID using CAS or object versioning.
*NOTE: The writer may choose to release its snapshot created in (1) after writing the fencing SST in (3), or it may periodically refresh its snapshot.*
At this point, there are four potential outcomes:
1. The write is successful.
2. The write was unsuccessful. Another writer wrote an SST with the same ID and a lower (older) `writer_epoch`.
3. The write was unsuccessful. Another writer wrote an SST with the same ID and the same `writer_epoch`.
4. The write was unsuccessful. Another writer wrote an SST with the same ID and a higher (newer) `writer_epoch`.
If the write was successful (1), all previous (older) writers have now been fenced.
If an older writer beats the current writer to the SST slot (2), the newer writer increments its SST ID by 1 and tries again. This process repeats until the write is successful.
If another writer has the same `writer_epoch` (3), the client is in an illegal state. This should never happen since clients are transactionally updating `writer_epoch`. If this does happen, the client should panic.
If a newer writer beats the current writer to the SST slot (4), the current writer has been fenced. The current writer should halt.
*NOTE: This protocol implies the invariant that SST IDs in `wal` will be contiguous and monotonically increasing.*
#### Examples
[Section titled “Examples”](#examples)
Let’s consider some examples to illustrate the writer protocol.
Here’s an example where a new writer successfully fences an older writer (protocol scenario (1)):
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1
time 1, 00000000000000000001.sst, writer_epoch=1
time 2, 00000000000000000002.sst, writer_epoch=2
time 3, 00000000000000000002.sst, writer_epoch=1
time 4, 00000000000000000003.sst, writer_epoch=2
```
In the example above, writer 1 successfully writes SSTs 0 and 1. At time 2, writer 2 successfully fences writer 1, but writer 1 hasn’t yet seen the fence write. When writer 1 attempts to write SST 2, it loses the CAS write and halts because SST 2 has a higher `writer_epoch`. Writer 2 then continues with a successful write to SST 3.
Here’s an example where a new writer has to retry its fence write because an older writer took its SST ID location (protocol scenario (2)):
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1
time 1, 00000000000000000001.sst, writer_epoch=1
time 2, 00000000000000000002.sst, writer_epoch=1
time 3, 00000000000000000002.sst, writer_epoch=2
time 4, 00000000000000000003.sst, writer_epoch=2
```
In the example above, writer 1 successfully writes SSTs 0, 1, and 2. Writer 2 tries to fence older writers with a write to SST ID 2, but fails because writer 1 has already written an SST at that location. Writer 2 sees that the SST with ID 2 has a lower `writer_epoch` than its own and retries its fence write at the next SST ID location. This write to SST ID 3 at time 4 is successful. Writer 2 has successfully fenced writer 1. If writer 1 tries to write to SST ID 3, it will see a higher `writer_epoch` and halt.
Here’s an example where a new writer gets fenced before it can write its own fencing write (protocol scenario (4)):
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1
time 1, 00000000000000000001.sst, writer_epoch=1
time 2, 00000000000000000002.sst, writer_epoch=3
time 3, 00000000000000000002.sst, writer_epoch=2
time 4, 00000000000000000003.sst, writer_epoch=3
```
In the example above, writer 1 successfully writes SSTs 0 and 1. Writer 2 tries to fence older writers with a write to SST ID 2, but fails because writer 3 has already written an SST at that location. Writer 2 sees that the SST with ID 2 has a higher `writer_epoch` than its own, and halts.
#### Acknowledgements
[Section titled “Acknowledgements”](#acknowledgements)
SlateDB’s `put()` API is asynchronous. Clients that want to know when their `put()` has been durably persisted to object storage must call `await`. A writer client will not successfully acknowledge a `put()` call until the key-value pair has been successfully written to object storage in a `wal` SST.
#### Parallel Writes
[Section titled “Parallel Writes”](#parallel-writes)
The writer protocol we describe above assumes that writers are writing SSTs in sequence. That is, a writer will never write SST ID N until SST ID N-1 has been successfully written to object storage by the client.
But we know we will want to support parallel writes in the future. Parallel writes allow a single writer client to write multiple SSTs at the same time. This can reduce latency for the writer client. Our design should not preclude parallel writes.
The current writer protocol can be extended to support parallel writes by defining a `max_parallel_writes` configuration parameter. A new writer must then write `max_parallel_writes` sequential fencing SSTs in a row starting from the first open SST position in the `wal`. `max_parallel_writes` would also need to be stored in the manifest so readers, writers, and compactors can agree on it.
*NOTE: This strategy is discussed in more detail [here](https://github.com/slatedb/slatedb/pull/24/files#r1585894110).*
Consider this example with `max_parallel_writes` set to 2:
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1
time 1, 00000000000000000001.sst, writer_epoch=1
time 2, 00000000000000000002.sst, writer_epoch=2
time 3, 00000000000000000003.sst, writer_epoch=1
time 4, 00000000000000000002.sst, writer_epoch=1
time 5, 00000000000000000003.sst, writer_epoch=2
time 6, 00000000000000000004.sst, writer_epoch=2
time 7, 00000000000000000005.sst, writer_epoch=2
```
Writer 1 successfully writes SSTs 0, 1, and 3. Writer 1’s SST 2 write fails at time 4 because writer 2 has already taken that location at time 2. At this point, writer 1 immediately stops writing new SSTs and halts. Existing parallel writes run to completion (either failure or success).
Writer 2 does not know whether it’s successfully fenced writer 1 yet. It attempts to write SST 3 at time 5, but fails because writer 1 has already written an SST at that location at time 3. At this point, writer 2 begins again at position 4. Writer 2 successfully writes SST 4, then SST 5. Writer 2 has now written two consecutive SSTs to the `wal`. Since `max_parallel_writes` is set to 2, writer 2 has successfully fenced writer 1; it’s guaranteed that writer 1 will see at least one of these fencing writes in its `max_parallel_writes` window. Thus, no writes from writer 1 (or any older writer) will appear after SST 5.
The next question is what to do about writer 1’s SST 3 that was written at time 3.
The writer must decide whether to:
1. Block all ack’s until all previous SST writes have been successfully written. In this case, `put()`s with key-value pairs in SST 3 would be notified of a failure since SST 2 was not successfully written.
2. Successfully acknowledge SST 3 as soon as it’s written.
We propose allowing writers to successfully acknowledge writes as soon as they occur (2). This strategy has slightly looser semantics since it means a put in SST 3 would be durably persisted and readable in the future, while puts in SST 2 would not be durably persisted.
```plaintext
a = put(key1, value1)
b = put(key2, value3)
a.await() -> failed
b.await() -> succeeded
```
This should be fine since the client can always retry the failed `put()` call. Clients that want strong ordering across puts can use transactions.
*NOTE: Transactions are not something we’ve considered in detail. The current idea is to keep all transactional writes within a single SST. Thus, transactions should work with this protocol.*
All readers and the compactor must decide whether to:
1. Ignore SST 3 because an SST with a higher `writer_epoch` has been written to a previous slot.
2. Treat SST 3 as valid because writer 2 has not yet written `max_parallel_writes` SSTs in a row.
We propose allowing readers and the compactor to treat SST 3 as valid.
*NOTE: Astute readers might notice that `max_parallel_writes` is equivalent to a Kafka producer’s [`max.in.flight.requests.per.connection`](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#max-in-flight-requests-per-connection) setting.*
If more than 2 writers are writing in parallel, the protocol is the same. The only difference is that readers will see writes from multiple epochs before the youngest writer wins the fencing race.
To illustrate, consider an example where we have 3 writers. writer 1 is the old writer, while writer 2 and writer 3 are trying to fence:
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1 // success
time 1, 00000000000000000001.sst, writer_epoch=1 // success
time 2, 00000000000000000002.sst, writer_epoch=3 // success
time 3, 00000000000000000003.sst, writer_epoch=1 // success
time 4, 00000000000000000002.sst, writer_epoch=1 // failure (writer 3 won at time 2), writer 1 halts
time 5, 00000000000000000003.sst, writer_epoch=2 // failure (writer 1 won at time 3)
time 6, 00000000000000000003.sst, writer_epoch=3 // failure (writer 1 won at time 3)
time 7, 00000000000000000004.sst, writer_epoch=2 // success
time 8, 00000000000000000004.sst, writer_epoch=3 // failure (writer 2 won at time 7)
time 9, 00000000000000000005.sst, writer_epoch=3 // success
time 10, 00000000000000000005.sst, writer_epoch=2 // failure (writer 3 won at time 9), writer 2 halts
time 11, 00000000000000000006.sst, writer_epoch=3 // success
```
At time 11, writer 3 has successfully fenced writer 1 and writer 2. Readers will consider all SSTs valid from 0-6, though only SSTs 0, 1, and 3 will have key-value pairs since fencing SST writes are always empty.
*NOTE: Writers will only detect that they’ve been fenced when they go to write. Low-throughput writers can periodically scan `wal` for SSTs with a higher `writer_epoch` to proactively detect that they’ve been fenced. This would work with the current design, but isn’t in scope.*
### Readers
[Section titled “Readers”](#readers)
#### Reader Protocol
[Section titled “Reader Protocol”](#reader-protocol)
Readers must establish a snapshot on startup and load DbState:
1. Read the current manifest
2. List all `wal` to find the maximum contiguous SST
3. Update the current manifest in-memory
1. Set `wal_id_last_seen` to the ID of the maximum contiguous SST in (2)
2. Create a new snapshot that references the new manifest
4. Write the updated manifest to the next manifest slot, go to (1) if CAS fails
5. Load all `leveled_sst`s into `DbState.leveled_ssts`
6. Load all `wal` SSTs >= `wal_id_last_compacted` into `DbState.l0`
*NOTE: Readers read the current manifest (1) before listing the `wal` (2) so the reader can detect manifest changes while it’s listing the `wal` directory. This is important because the compactor and garbage collector might have deleted SSTs that the reader sees in (2). Without this protection, the reader might set `wal_id_last_seen` to an SST that no longer exists. See [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1603626303) for more details.*
*NOTE: We’ll probably want to rename `DbState.l0` to `DbState.wal`.*
*NOTE: `DbState.leveled_ssts` doesn’t currently exist because SlateDB does not have compaction. When we add compaction, we’ll need to introduce this variable.*
At this point, a reader has a consistent view of all SSTs in the database.
The reader will then periodically refresh its state by:
1. Polling the `wal` directory to detect new `wal` SSTs
2. Reloading the current manifest and creating a new snapshot to detect new `leveled_ssts`
*NOTE: Readers can also skip snapshot creation if they wish. If they do so, they must deal with SSTs that are no longer in object storage. They may do so by re-reading the manifest and updating their `DbState`, or by simply skipping the SST read. Lazy readers are outside the scope of this document.*
#### Parallel Writes
[Section titled “Parallel Writes”](#parallel-writes-1)
The reader protocol above assumes that all SSTs are contiguous and monotonically increasing. If parallel writes are allowed, a reader might see a `wal` with gaps between SSTs:
```plaintext
time 0, 00000000000000000000.sst, writer_epoch=1
time 1, 00000000000000000001.sst, writer_epoch=1
time 2, 00000000000000000003.sst, writer_epoch=1
```
In this example, the reader has two choices:
1. Include SST 3 in its DbState.
2. Wait for SST 2 to be written.
If the reader includes SST 3 in its `DbState`, it will need to periodically check for SST 2 by polling the `wal` or refreshing its `DbState` at some interval.
We propose waiting for SST 2 to be written. This style simplifies polling: the reader simply polls for the next SST (SST 2 in the example above). When SST 2 appears, it’s added to `l0` and the reader begins polling for SST 3.
This does mean that SST 3 will not be served by a reader until SST 2 arrives. For the writer client, this is not a problem, since the writer client is fully consistent between its own reads and writes. But secondary reader clients might not see SST 3 for a significant period of time if the writer dies. This is a tradeoff that we are comfortable with. If it proves problematic, we can implement (1), above, and poll for missing SSTs periodically.
*NOTE: There is some interdependence between when `wal_id_last_seen` and the semantics of a snapshot. If `wal_id_last_seen` were to include SST 3 in our example above, we’d have to decide whether to include SST 2 when it eventually arrives. The current design does not have this issue. This is discussed more [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1594783266).*
### Compactors
[Section titled “Compactors”](#compactors)
On startup, a compactor must increment the `compactor_epoch` in the manifest. This is done in a similar manner to the process described in the *Writer Protocol* section above.
After startup, compactors periodically do two things:
1. Merge SSTs from the `wal` into `leveled_ssts`
2. Merge SSTs from `leveled_ssts` into higher-level SSTs
To merge SSTs from the `wal` into `leveled_ssts` (1), the compactor must periodically:
1. List all SSTs in the `wal` folder > `wal_id_last_compacted`
2. Create a new merged SST in `leveled_ssts`’s level 0 that contains all `wal` SST key-value pairs
3. Update the manifest with the new `leveled_ssts`, `wal_id_last_compacted`, and `wal_id_last_seen`
*NOTE: This design implies that level 0 for `leveled_ssts` will not be range partitioned. Each SST in level 0 would be a full a-z range. This is how traditional LSMs work. If we were to go with range partitioning in level 0, we’d need to do a full level 0 merge every time we compact `wal`; this could be expensive. This design decision is discussed in more detail [here](https://github.com/slatedb/slatedb/pull/39/files#r1589855599).*
Merging leveled SSTs (2) is outside the scope of this design.
### Garbage Collectors
[Section titled “Garbage Collectors”](#garbage-collectors)
A garbage collector (GC) must delete both objects from object storage and records from the transactional store (when two-phase CAS is used). The garbage collector must delete inactive manifests and SSTs.
*NOTE: This design considers the compactor and garbage collector (inactive SST deletion) as two separate activities that run for one database in one process—the compactor process. In the future, we might want to run the compactor and garbage collector in separate processes on separate machines. We might also want the garbage collector to run across multiple databases. This should be doable, but is outside this design’s scope. The topic is discussed [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1596319141) and in \[[#49](https://github.com/slatedb/slatedb/issues/49)].*
#### Object Deletion
[Section titled “Object Deletion”](#object-deletion)
Four classes of objects must be deleted:
* Inactive manifests
* Inactive SSTs
* Orphaned temporary manifests (when two-phase CAS is used)
* Orphaned temporary SSTs (when two-phase CAS is used)
These objects reside in three locations:
* `manifest`
* `wal`
* `levels`
The garbage collector will periodically list all objects in the `manifest`, `wal`, and `levels` directories and delete the following:
1. Any `.manifest` that is not an active manifest.
2. Any other file in `manifest` that has an object storage creation timestamp older than a day.
3. Any `.sst` in `wal` that is < the minimum `wal_id_last_compacted` in the set of all active manifests.
4. Any other file in `wal` that has an object storage creation timestamp older than a day.
5. Any `.sst` in `levels` that is not referenced by any active manifest.
*NOTE: These rules follow the definitions for active manifests and active SSTs in the *Snapshots* section.*
#### Transactional Store Deletion
[Section titled “Transactional Store Deletion”](#transactional-store-deletion)
Two classes of records must be deleted from the transactional store:
* Inactive manifests
* Inactive SSTs
The garbage collector will periodically delete all transactional store records with the following criteria:
1. `destination` references a `.manifest` that is not the latest and is not referenced by any snapshot in the latest manifest.
2. `destination` references an `.sst` in `wal` that is < `wal_id_last_compacted` in the current manifest.
*NOTE: We use `<` not `<=` for (2) because the compactor must not delete the SST at `wal_id_last_compacted` so readers can recover the `writer_epoch` from the SST.*
## Rejected Solutions
[Section titled “Rejected Solutions”](#rejected-solutions)
### Update Manifest on Write
[Section titled “Update Manifest on Write”](#update-manifest-on-write)
A previous [manifest design](https://github.com/slatedb/slatedb/pull/24/) proposed transactional manifest updates on every `wal` SST write (discussed [here](https://github.com/slatedb/slatedb/pull/39#issuecomment-2094356240)). This was rejected due to the high cost of transactional writes when using object storage with CAS for the manifest.
### Epoch in Object Names
[Section titled “Epoch in Object Names”](#epoch-in-object-names)
A previous [manifest design](https://github.com/slatedb/slatedb/pull/24/) proposed namespacing `wal` filenames with a writer epoch prefix (discussed [here](https://github.com/slatedb/slatedb/pull/24/files#diff-58d53b55614c8db2dd180ac49237f37991d2b378c75e0245d780356e5d0c8135R114)). This was rejected because it was deemed complex. We might revisit this decision in the future if we have a need for it.
Another alternative to two-phase CAS was proposed [here](https://github.com/slatedb/slatedb/pull/43#discussion_r1597308492). This alternative reduced the operation to a single object storage PUT and single DynamoDB write. In doing so, [it required `writer_epoch` in object names](https://github.com/slatedb/slatedb/pull/43#discussion_r1597317062). This scheme prevented *real* CAS from working. We rejected this design because we decided to favor a design that worked well with real CAS.
### Object Versioning CAS
[Section titled “Object Versioning CAS”](#object-versioning-cas)
A [previous version](https://github.com/slatedb/slatedb/blob/6beba6949487a519b8bb6c69a3cf80f06778f540/docs/0001-manifest.md) of this design used object versioning for `wal` SST CAS. This design was rejected because S3 Express One Zone does not support object versioning.
### Use a `manifest/current` Proxy Pointer
[Section titled “Use a manifest/current Proxy Pointer”](#use-a-manifestcurrent-proxy-pointer)
A [previous version](https://github.com/slatedb/slatedb/blob/6beba6949487a519b8bb6c69a3cf80f06778f540/docs/0001-manifest.md) of this design contained an additional file: `manifest/current`. Once we decided to use incremental IDs for the manifest, we no longer needed this file.
### Two-Phase Mutable CAS
[Section titled “Two-Phase Mutable CAS”](#two-phase-mutable-cas)
We [briefly discussed](https://github.com/slatedb/slatedb/pull/43#discussion_r1597489292) supporting mutable CAS operations with the two-phase write CAS pattern. There was [some disagreement](https://github.com/slatedb/slatedb/pull/43#discussion_r1597749655) about whether this was possible. It’s conceivable that user-defined attributes might allow the two-phase CAS pattern to handle mutable CAS operations. We did not explore this since we decided to use incremental IDs for the manifest, which require only immutable CAS.
### DeltaStream Protocol
[Section titled “DeltaStream Protocol”](#deltastream-protocol)
We [considered using a protocol](https://github.com/slatedb/slatedb/pull/43#discussion_r1597543706) similar to DeltaStream’s. This protocol is similar to two-phase CAS, but uses a DynamoDB pointer to determine the current manifest rather than a LIST operation. This protocol wasn’t obviously compatible with SST writes, though. We opted to have a single CAS approach for both the manifest and SSTs, so we rejected this design.
### `object_store` Locking
[Section titled “object\_store Locking”](#object_store-locking)
Rust’s `object_store` crate has a [locking mechanism](https://docs.rs/object_store/latest/object_store/aws/enum.S3CopyIfNotExists.html). We briefly looked at this, but [rejected it](https://github.com/slatedb/slatedb/pull/43#discussion_r1597551287) because it uses time to live (TTL) timeouts for locks. We wanted a CAS operation that would not time out.
### Atomic Deletes
[Section titled “Atomic Deletes”](#atomic-deletes)
A [previous version](https://github.com/slatedb/slatedb/pull/43#discussion_r1600912132) of this design proposed using atomic deletes. This turned out to be unnecessary.
## Addendum
[Section titled “Addendum”](#addendum)
We found that Deltalake [has a locking library](https://github.com/delta-incubator/dynamodb-lock-rs). We thought it might be used for its LSM manifest updates, but it’s not. We discuss our findings more [here](https://github.com/slatedb/slatedb/pull/24/files#r1582427235).
While exploring CAS with object store versioning, we found that Terraform had explored this idea in their [S3 backend](https://github.com/hashicorp/terraform/issues/27070). Ultimately, they’re trying to create file locks, which is different from what we’re doing with our manifest.
# Compaction
Status: Accepted
Authors:
* [Rohan Desai](https://github.com/rodesai)
References:
*
*
*
*
*
*
*
*
*
## Current Implementation
[Section titled “Current Implementation”](#current-implementation)
SlateDB maintains a live memtable, a set of immutable memtables, and metadata about SSTs in the WAL. `put()` writes the key-value into the memtable. At some interval, the flushing thread closes the memtable and flushes the contents to a new WAL SST. Immediately before the flush, the memtable is converted to an immutable memtable.
`get()` operations search the current memtable, immutable memtables, and WAL SSTs in reverse-write order (most recent SSTs first). The search is terminated when the key specified in `get` is found.
## Problem
[Section titled “Problem”](#problem)
There are a few problems with the current implementation that we address with this design:
1. As the DB grows, `get()` becomes very expensive because it has to search an ever-growing list of SSTs.
2. Similarly, as the DB grows, start becomes very slow because SlateDB has to reload the state of all the SSTs
3. The storage space used grows without bound even if the key space is finite. This is less of a problem for SlateDB as compared to disk-based databases because space in s3 is practically unlimited, and is relatively cheap (e.g. .023$/GB vs .08$/GB for EBS (.16$/GB for HA)). Still it is not ideal - storage is cheaper but not free. More troubling is that SlateDB relies on caching metadata effectively for good read performance. This gets harder and harder to do as the number of SSTs grows.
## Goals
[Section titled “Goals”](#goals)
* Compact WAL SSTs into larger SSTs so that space from overwrites and deletes can be reclaimed, and reduce read amplification.
* Further compact compacted SSTs to further reduce read and space amplification.
* Define a simple initial compaction algorithm that balances the various types of amplification.
* Allow for flexibility in the specific compaction algorithm. This is likely going to be an area that we can uniquely innovate, and the design should support easily iterating on compaction scheduler.
* Split some compaction work between the writer and compactor processes to allow for better network utilization.
* Support manual major compaction that compacts the whole database.
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
The following are out-of-scope for this design. That said, the design should not preclude them. For some items, we will describe how the design can be adapted to achieve these goals in follow-on work.
* Optimize for specific workloads. For example, some databases optimize for in-order bulk inserts by avoiding compaction and simply writing out the database in its fully compacted form as inserts arrive.
* Similarly, specialized compaction policies optimized for specific workloads are out-of-scope. For example, some databases support specialized compaction for time series data with a finite lifetime.
* Resumable compaction. This proposal will not define how to resume a long-running compaction after a compactor restart. This is important to support, as compactions could take 10s of minutes, and the compactor should not have to restart them after a failure. I’ll include a section at the end on how we can extend the design to support this.
* GC of unused SSTs is excluded from this design
## Proposal
[Section titled “Proposal”](#proposal)
### Amplification
[Section titled “Amplification”](#amplification)
The design for compaction depends on the database and workload sensitivity to write, read, and space amplification. We will likely ultimately need to support a variety of compaction strategies to accommodate different types of workloads. However there are some common considerations given SlateDB’s object-store based architecture:
1. write amplification: write amplification refers to the added work done by compaction for every write. This is usually measured as some multiplier. For example, if a write is written to the database 5 times (once for the initial write and compacted 4 times) - write amplification is 5x. Write amplification is not much of a concern from a cost perspective. The major cloud providers don’t charge for data transfers to/from the “standard” tier of object storage (provided you set up networking correctly). There are charges to/from zonal tiers like s3express, but I expect that most of the data will reside in the regional tier. Write amplification is a concern from a bandwidth usage perspective. If write amplification is too high, writes will become bottlenecked on the compactor node’s network. This is usually a higher limit than local or attached disk, but not an order of magnitude higher (\~1.5-2x nw-out baseline:local disk write). So we will need to take care to avoid too much write amplification to support write-heavy workloads.
2. read amplification: read amplification refers to the added work done by the database for reads. This includes added cpu from searching through multiple possible locations for a read, and added i/o from reading multiple locations. We certainly don’t want to be doing multiple, if any, GET operations per SlateDB read, as GETs are expensive and slow. Read amplification can be mitigated by effective caching. Ideally, for optimal performance and cost, reads can be served entirely from cache, which spans memory and local disk. Still, we probably don’t want to be doing multiple disk reads per read or we may saturate the local/attached disks. Therefore it’s desirable that the filter and index blocks used for most reads fit in memory. All of this is to say that it is pretty important for SlateDB to reduce read amplification. We need the set of SSTs to search for a read to be small enough so that indexes/filters can fit in memory, and data being read fits in total cache most of the time.
3. space amplification: SlateDB is not as sensitive to space amplification as disk-based databases are because storage is practically unbound, and is cheap (about 1/5th-1/20th the cost of an on-disk db on s3 depending on replication factor, instance stores vs ebs, reserved vs on-demand, etc). One problem that we can totally sidestep is transient space amplification from compaction. Most dbs suffer from this problem and have to either over-allocate disk space or implement incremental compaction. This won’t be an issue for us. Our main concern with space amplification is that it grows the search space for reads as described above.
### High Level Overview
[Section titled “High Level Overview”](#high-level-overview)
In the rest of this document, we propose mechanisms for compacting together SSTs to purge duplicates and keep the total count of SSTs contained. Compaction will be executed both by the main writer and by a separate compactor process.
The main writer compacts the WAL to the first level of the database. This has a number of benefits: As we describe below, the main writer compacts the WAL directly from memory without reading the data back from S3 first. This saves cost for low-latency writers and more importantly, reduces load on the network. This offloads some of the compaction i/o from the compactor onto the writer. Assuming the main bottleneck for compaction will be network, this should help us achieve more throughput. It should be easier to coordinate between the compactor and writer for lower-frequency compactions to lower levels than WAL->L0 compactions.
The compactor is responsible for compacting L0 and the lower levels of the database. Compacted SSTs are maintained in a series of Sorted Runs (SRs). Each SR spans the full keyspace of the database. A SR is made up of an ordered series of SSTs, each of which contains a distinct subset of the total keyspace. We use Sorted Runs instead of large SSTs because it is a simple way to keep the size of the metadata blocks small enough so that they can be easily paged in and out of cache without polluting the cache. Larger SSTs have larger metadata blocks. It’s expensive to page them in and out of cache, and they are likely to displace other data that should be retained by the cache to achieve a high hit rate.
The SRs are themselves ordered by age. When executing point lookups, SlateDB looks up the value for a key in age-order, and terminates the search at the first SR that contains the key. Range scans read every SR and sort-merge the result.
### Manifest Changes
[Section titled “Manifest Changes”](#manifest-changes)
```plaintext
table SortedRun {
id: uint32;
ssts:[SstId];
}
table SstId {
high: uint64;
low: uint64;
}
table Compacted {
runs: [SortedRun];
}
table Manifest {
…
l0_last_compacted: CompactedSstId // The last compacted l0
l0: [SstId];
compacted: Compacted;
…
}
```
We propose to augment the manifest by adding the following fields:
`L0`: Contains a list of SST IDs in L0
`compacted`: Contains a single instance of `Compacted`. `Compacted` contains a list of `SortedRun` instances. A `SortedRun` instance defines a single sorted run. Each Sorted Run contains a list of SST IDs and has a unique ID. The list of SST IDs defines the SSTs that comprise the sorted run. A given SST belongs to at most 1 SR. The ID describes the SR’s position in the list of sorted runs in `compacted`. That is, an SR S with an S.id must occur after SR S’ with ID S’.id if S.id < S’.id (so the sorted run with ID 0 must be last in the list). The last SR in the list must have ID 0. The semantics of the ID will be important when we describe how to define compactions.
`l0_last_compacted`: Contains the ID of the last l0 SST that was compacted to `compacted`. The compactor uses this value to determine which l0 SSTs it needs to consider for compaction when refreshing its view of state with the manifest stored durably.
#### Naming Compacted SSTs (L0 and L1+)
[Section titled “Naming Compacted SSTs (L0 and L1+)”](#naming-compacted-ssts-l0-and-l1)
We will use ULIDs to name compacted SSTs. The ULID is stored in the manifest in the `SstId` table, with the `high` and `low` fields containing the high and low bits of the ULID, respectively.
In the Object Store, compacted SSTs are stored under the compacted directory. Each SST object is named using its ULID and the suffix `.sst`, e.g:
```plaintext
/compacted/01ARZ3NDEKTSV4RRFFQ69G5FAV.sst
/compacted/01BX5ZZKBKACTAV9WEVGEMMVRZ.sst
…
```
### DB Options
[Section titled “DB Options”](#db-options)
Compaction behavior is governed by the following DB options:
`l0_sst_size_bytes`: defines the target L0 SST size in bytes.
`l0_manifest_commit_interval_ms`: defines the minimum interval between memtable flush-related manifest updates in milliseconds.
`l0_compaction_threshold_ssts`: defines the threshold number of SSTs for compacting L0. (default 8)
`l0_max_ssts`: defines the maximum number of uncompacted L0 SSTs. (default 16)
`max_compactions`: defines the max number of concurrent compactions. (default 4)
`compaction_scheduler`: defines the compaction scheduler (currently only supports the value `tiered`)
`compaction_scheduler_tiered`: An options struct that defines scheduler options for tiered compaction.
`compaction_scheduler_tiered.level_compaction_threshold_runs`: defines the threshold number of sorted runs for compacting a lower level (specific to the tiered compaction scheduler). (default 8)
`compaction_scheduler_tiered.level_max_runs`: defines the maximum number of sorted runs at alower level (specific to the tiered compaction scheduler). (default 16)
### WAL->L0 Compaction
[Section titled “WAL->L0 Compaction”](#wal-l0-compaction)
The writer is responsible for compacting the WAL to the first level of the database. It does this directly from the memtable rather than reading back the WAL first. Instead of freezing the memtable when writing the WAL, it instead retains it until enough data has accumulated to fill an L0 SST. Only then does it freeze the memtable and write out an L0 SST. When it’s been longer than `l0_manifest_commit_interval_ms` since the last manifest update, the writer updates the manifest with the new SSTs. The L0 writes are fenced by virtue of committing L0 SSTs in a manifest update. The newly written L0 SSTs are only considered part of the database when the manifest has been updated.
Let’s look at the write and recovery protocols in more detail:
write:
1. `put` adds the key-value to the current memtable and updates the WAL using the protocol described in [the manifest design](https://github.com/slatedb/slatedb/blob/main/rfcs/0001-manifest.md).
2. If the current memtable is larger than `l0_sst_size`, freeze the memtable by converting it to an immutable memtable, and write the memtable to a new ULID-named SST S in `compacted`.
3. When S and all earlier SSTs S’, S’’, … for unflushed immutable tables are written:
1. update L0 in-memory by prepending S, S’, S’’, … to the list of SSTs in L0.
2. clear the immutable memtables for S, S’, S’’,…
3. if time since last manifest update > `l0_manifest_commit_interval_ms`:
1. if (number l0 SSTs > l0\_max\_uncompacted):
1. pause new writes
2. wait till number of l0 SSTs < l0\_max\_uncompacted
2. unpause writes if paused
3. update the manifest using CAS with the following modifications:
1. update `last_compacted` to the last WAL SST included in S
2. update `l0` by prepending S, S’, S’’, … to the list
4. If CAS from (3) fails:
1. If CAS fails and the manifest has a different writer epoch, exit
2. If CAS fails and the manifest has the same writer epoch, go back to 3.iii.c
5. If CAS from (3) succeeds, remove the immutable memtable.
write-recovery:
1. Fence older writers as described in the [manifest design](https://github.com/slatedb/slatedb/blob/main/rfcs/0001-manifest.md).
2. Create a new mutable memtable
3. For every WAL SST W after `last_compacted`, reapply the writes as described above in the write protocol.
TODO: should we have some way to bound the number of immutable memtables in memory? I left it out since we are free to purge them once their SSTs have been written (even before we update the manifest). But this doesn’t help us if we can’t write the SSTs to S3 for some reason - though in that case we likely can’t commit writes anyway.
### Compacting Lower Levels
[Section titled “Compacting Lower Levels”](#compacting-lower-levels)
The Compactor compacts L0 and the lower levels. It contains two logical processes: a Compaction Executor and a Compaction Scheduler. The Compaction Scheduler observes the current state of the database and schedules Compactions. The Compaction Executor bootstraps the Compactor, executes the Compactions scheduled by the Compaction Scheduler, and notifies the Compaction Scheduler about status. The Compaction Executor is fixed, while The Compaction Scheduler is modular to allow SlateDB to support different compaction styles. The specific scheduler is specified in the `compaction.scheduler` db option. Initially, we will implement a single scheduler that performs tiered compaction.
#### Interfaces
[Section titled “Interfaces”](#interfaces)
Lets start by defining the interface between the CompactionExecutor and CompactionScheduler. I’ll define them as Rust structs/traits. Then, we discuss how each component implements the interfaces.
##### Compactions
[Section titled “Compactions”](#compactions)
The Compaction Scheduler tells the Compaction Executor what compactions to execute. A Compaction is defined using the following parameters:
* Sources: A list of one or more sources of data to compact. A source can either be a single L0 SST, or a single SR. The Sources must be logically consecutive. This means that for any sources S1 and S2 where S2 appears immediately after S1 in the list:
* If S1 is an L0 SST, then S2 must either be the next L0 SST OR if S1 is the last L0 SST then S2 must be the first SR (SR with the highest ID)
* If S1 is an SR, then S2 must be the next SR.
* Destination: A destination SR. This can be a new SR, or it can be the SR from Sources with the lowest ID. If it’s a new SR, The SR must be logically consecutive to the last element of Sources (as described above).
Let’s look at some examples of valid/invalid compactions. I’ll use string IDs for SSTs here instead of ULIDs. Suppose our manifest looks like:
```plaintext
l0: [SST-4, SST-3, SST-2, SST-1]
compacted: [100, 50, 3, 1, 0]
```
Here are examples of valid/invalid compactions (I’m using the notation Sources->Destination)
`[SST-2, SST-1]->101`: This describes compacting the oldest 2 L0 SSTs to a new SR
`[SST-4, SST-3]->101`: This is invalid because it skips SST-2 and SST-1
`[SST-1, 100]->100`: This describes compacting the oldest L0 SST (SST-1) and SR 100 and saving the result as SR 100
`[100, 50]->2`: This is invalid because it writes the result to an SR that is not consecutive to 50 (3 is consecutive to 50)
`[SST-4, SST-3, SST-2, SST-1, 100, 50, 3, 1, 0]->0`: This describes a major compaction that compacts everything and saves it as SR 0
Observe that we can use this basic definition to describe compactions done by different compaction algorithms (this isn’t strictly true in the above proposal - e.g. it doesn’t currently support some-to-all compactions like compacting a single SST from one SR into another SR, but that’s a fairly straightforward extension to the definition of a source) - it’s up to the Compaction Scheduler to decide what compactions to execute. The scheduler can choose to implement leveled compaction by viewing each SR as a level and scheduling Compactions that always merge one SR into the next SR. Or it can implement tiered compaction by grouping SRs into levels and define compactions that merge all the SRs in a level into a new SR at the next level. The levels themselves are a logical construct maintained by the scheduler.
In Rust, this looks like:
```plaintext
union SourceId {
sorted_run: u32,
sst: Ulid,
}
struct Compaction {
id: u32 // a unique identifier for the compaction (it must be unique to the process’s lifetime)
sources: Vec
destination: u32
}
```
##### Compaction Executor
[Section titled “Compaction Executor”](#compaction-executor)
The Compaction Executor provides the following interface to the CompactionScheduler:
```plaintext
trait CompactionExecutor {
/*
* Notifies the compaction executor about a new compaction to execute. The result
* is Ok if the compaction was accepted by the executor. This does not mean that the
* compaction was completed. The executor validates the compaction and returns an
* error if the compaction is invalid.
*/
fn submit(&self, compaction: Compaction) -> Result<(), Error>
}
```
##### Compaction Scheduler
[Section titled “Compaction Scheduler”](#compaction-scheduler)
The Compaction Scheduler provides the following interface to the Compaction Executor:
```plaintext
enum CompactorUpdateKind { DBState, CompactionFinished }
struct DBStateUpdate {
kind: CompactorUpdateKind // always DBState
state: DBState, // we probably don’t want to use DBState here, but rather a subset that contains compactor-relevant state like l0 and compacted. But, you get the idea.
}
struct CompactionFinished {
kind: CompactorUpdateKind // always CompactionFinished
compaction_id: u32,
state: DBState,
}
union CompactorUpdate {
db_state: DBStateUpdate,
compaction_finished: CompactionFinished,
}
trait CompactionScheduler {
/*
* Notifies the scheduler that it should start evaluating the db for compaction. This method
* receives a channel over which the Executor sends the Scheduler updates about changes
* to the database. This includes updates about changes to the database (e.g. arrival of
* new L0 files), and completion of compactions.
*/
fn start(&self, executor: Box, chan: Receiver);
}
```
#### Compaction Executor
[Section titled “Compaction Executor”](#compaction-executor-1)
The Compaction Executor initializes as follows:
1. Update the `compactor_epoch` in the manifest using CAS
2. Initialize the Compaction Scheduler by creating an instance based on `compaction.scheduler`, creating, a channel, and calling `start`
3. Send the initial db state (as returned in the manifest) over the channel
Then, the compactor periodically polls the manifest. On every poll, the compactor:
1. Check if the `compactor_epoch` is different than the compactor’s epoch. If it is, exit.
2. If the db has new l0 SSTs, send a `DBStateUpdate` message over the scheduler channel.
##### Executing a Compaction
[Section titled “Executing a Compaction”](#executing-a-compaction)
The Compaction Executor implements `compact` by:
1. Validate the compaction by running through the following checks. If any fail, return error
1. Make sure the compaction is valid as defined above in the Compaction section
2. Make sure there is no other ongoing compaction that includes the SSTs or SRs referenced by the compaction.
2. At this point, the call to `compact` returns.
3. Schedule the compaction for execution in the background.
The Compaction Executor executes the compaction by reading the SSTs and SRs in `sources` and sort-merging them into a new SR.
The Compaction Executor needs to coalesce updates. If the same key appears in multiple sources, then it takes the value from the logically latest (i.e. most recent) source. The Compaction Executor handles destination SR 0 specially. If the destination SR is 0, and the value for a key is resolved to a tombstone, then the Compaction Executor will not include the key in the resulting SR.
The new SR is made up of ULID-named SSTs in the `compacted` directory (just like L0).
We should implement the sort-merge so that we can make good use of the available network. One good option here is to use `async` Object Store APIs to concurrently read the various sources, and then to write the resulting SSTs while we move on to the next key ranges. I think the details are something we can work out in the implementation, and it doesn’t have to be optimal in this iteration of work.
Note that compacting whole sorted runs can create a lot of temporary space amplification, especially for compactions that read the last level. This is not a major concern for SlateDB as ObjectStore capacity is practically infinite, and the usage is temporary so it should not contribute meaningfully to cost.
When the new SR has been fully written out, the Compaction Executor finishes the compaction by:
1. Read the existing manifest and verify that the `compactor_epoch` matches the compactor’s epoch. If it does not, exit.
2. Generate a new manifest that has the new SR in `compacted` and the SR and SSTs from `sources` removed.
3. Write the new manifest using CAS. If CAS fails, go back to 1
4. Send the Compaction Scheduler a CompactionFinished message with the compaction ID and the updated DB state.
#### Compaction Scheduler
[Section titled “Compaction Scheduler”](#compaction-scheduler-1)
The Compaction Scheduler is responsible for selecting the next Compaction. Our goal here is to implement something simple that works, and then iterate/optimize on it in future cycles. Initially we propose to implement basic tiered compaction, which tries to maintain sorted runs in size-based levels, and constrains the number of sorted runs in a given level by merging the runs together when there are too many of them, usually moving the resulting run to the next level.
We choose tiered compaction because it works well for workloads with a moderate to high volume of writes because it has lower write amplification than leveled compaction (usually by a factor of T where T is the fanout for each level). It still guarantees that the total number of runs is proportional to O(log(N)) where N is the size of the db, but allows multiple runs to accumulate at a level to reduce the amount of merging (at the cost of a multiplier T on the number of runs, where T is the number of runs that can accumulate at a level - so it’s really O(Tlog(N))). So the drawback is that there can be significant space and read amplification. However, I think there are reasonable ways to work around most of these drawbacks for now, and we can trade off some write amplification for better read/space amplification down the line by implementing “lazy-leveling” as described in [dostoevsky](https://scholar.harvard.edu/files/stratos/files/dostoevskykv.pdf):
* Tiered compaction maintains `level_compaction_threshold_ssts` sorted runs even at the lowest level, which means the entire keyspace may be copied `level_compaction_threshold_runs` times. This is somewhat concerning from a cost pov, however as explained at the beginning of the design Object Store costs are much lower than block device costs (by a factor of 5-20x), so this is less of a concern for SlateDB than traditional stores.
* The more concerning consequence of high space amplification is that it will likely directly lead to requiring a proportionally larger cache to achieve high hit rates for good read performance and cost. We can deal with this at first by allocating a larger cache. Eventually, we can do leveled compaction to the final level (meaning, maintain a single run at the final level) to dramatically reduce space amplification as described in the doestoevsky paper. Further, we can estimate space amplification by looking at the relative size of the non-last levels and final level, and execute compactions when amplification crosses some threshold.
* Tiered compaction also yields a db with more Sorted Runs, which adds to the cost of point lookups. Impact to point lookup cost should be dramatically reduced by SlateDB’s usage of bloom filters. However, more Sorted Runs does mean that we will need more SST reads on average (as the expected number of bloom filters that return a fp increases linearly with the number of Sorted Runs). There are a number of approaches to solving this problem:
* Universal compaction in RocksDB takes the approach of compacting when the total number of runs crosses some threshold.
* [Monkey](https://stratos.seas.harvard.edu/files/stratos/files/monkeykeyvaluestore.pdf) describes a simple optimization we can adapt that reallocates bloom filter bits from lower levels to higher levels to dramatically decrease the false-positive-rate (fpr) of higher-level filters at the cost of higher fpr at lower levels, but yielding a much lower average number of lookups (see [here](http://daslab.seas.harvard.edu/monkey/) for a good visualization).
* Short range-scans are fundamentally worse with tiered storage as they need to examine a roughly equal amount of data in every Sorted Run. If it’s problematic we can adjust compaction to more aggressively merge runs together with some goal for the total number of runs (similar to RocksDB’s universal compaction). Long range-scans are not as bad as they derive most of their cost from the final level, and with lazy-leveling there is a single final level.
All of this is to say, Tiered compaction feels like a great starting point for a general-purpose store, and we have reasonable short-term workarounds and long term solutions to most of the problems that we anticipate with tiered compaction.
SlateDB’s tiered Compaction Scheduler will work as follows:
1. Whenever a new CompactorUpdate arrives on the scheduler channel:
1. The Compaction Scheduler groups SRs into levels L1, L2,… A level with a larger index is considered “lower” (ugh - this is confusing). The size of the runs in LN is at most `l0_sst_size X l0_compaction_threshold X compaction.scheduler.tiered.level_compaction_threshold^N`
2. Iterate over the levels from lowest to highest. For each level, maybe schedule a compaction. The Compaction includes all SRs in the level as its source (we could probably include compactions for the higher level as well if it needs to be compacted, and so on - but we can add this later). The destination SR ID is the ID of the last SR in the level. Schedule a compaction for level N if:
1. The number of SRs in N > `compaction.scheduler.tiered.level_compaction_threshold_runs`
2. The number of SRs in N+1 < `compaction.scheduler.tiered.level_max_runs`
3. The number of uncompleted compactions < `max_compactions`
4. No ongoing compaction from level N
3. Maybe schedule a compaction for L0. The Compaction includes all SSTs in L0 as its source. The destination SR ID is the highest unused SR ID. Schedule a compaction for L0 if:
1. The number of SRs in L0 > `l0_compaction_threshold_ssts`
2. The number of SRs in L1 < `compaction.scheduler.tiered.level_max_runs`
3. The number of uncompleted compactions < `max_compactions`
4. No ongoing compaction from L0
### Back-Pressure
[Section titled “Back-Pressure”](#back-pressure)
The design described above applies back-pressure so that we don’t wind up writing faster than we can compact and get unbounded read/space amplification. SlateDB blocks new writes if the number of SSTs in L0 exceeds `l0_max_ssts`. This shouldn’t happen if compaction is keeping up and is able to merge SSTs from L0 to L1. Compactions to L1 are blocked if the number of runs in L1 is greater than `compaction.scheduler.tiered.level_max_runs`. Similarly, Compactions to L2, L3, … LN are blocked if the number of runs in those levels exceeds the threshold.
### Reads
[Section titled “Reads”](#reads)
*Point-Lookups*
The reader looks up keys in the following order, and terminates the search at the first item that contains a given key: memtable immutable memtable L0 SSTs, in order SRs, in order
*Range Scans*
To serve Range Scans, the reader needs to look through every memtable, immutable memtable, SST, and SR and sort-merge the key-values that fall within the range.
### Running the Compactor
[Section titled “Running the Compactor”](#running-the-compactor)
By default, the compactor will run alongside the main writer-reader in the same process. We will also support running the compactor in a separate process by initializing and running just the compactor.
### Looking Ahead
[Section titled “Looking Ahead”](#looking-ahead)
In this section I want to briefly sketch out how we can iterate on the design described above to add some additional features.
#### Resuming Compaction
[Section titled “Resuming Compaction”](#resuming-compaction)
Compactions targeting lower levels can take a long time. If the compactor restarts, we don’t want to lose compaction progress. We can record what compactions were ongoing in the manifest by defining a flatbuffer table schema that describes the Compaction struct defined above. Then, when the compactor restarts it knows what compactions were already scheduled.
It’s not enough to know what compactions were ongoing - a new compactor also needs to be able to reconstruct compaction progress. We can leave breadcrumbs to allow it to piece this information together:
To do this we can allow including uncompleted SRs in the list of SRs in the manifest. We can do this by including a `completed` flag in the SR definition. For a given SR ID there should only be one with `completed` set to `true`, which is what the reader would use to serve reads. Then, the new compactor can inspect the SSTs in the new SR, and see what key ranges have already completed compaction, and finish compacting the uncompacted key ranges.
#### Lazy-Leveling/Tiered+Leveled
[Section titled “Lazy-Leveling/Tiered+Leveled”](#lazy-levelingtieredleveled)
The tiered compaction scheduler proposed in this document will have very high space amplification, as it maintains multiple SRs at the lowest level. It’s likely that each SR at the lowest level contains most of the database, assuming a stable db size. As described above, to fix this (at the cost of added write amplification) we can always maintain a single run at the lowest level. In particular, our tiered compaction policy can always maintain SR 0 as its own level. When the earlier level is full, it can compact (at least) all SRs from that level into SR 0.
#### Time-Series
[Section titled “Time-Series”](#time-series)
Some databases (e.g. ScyllaDB) handle time-series data with a fixed lifetime specially. They maintain data in fixed time buckets that are themselves compacted according to some schedule (e.g. tiered or leveled), where each bucket corresponds to a distinct non-overlapping key range. Then, when a given bucket is no longer accessible (as defined by the fixed lifetime), the entire bucket is removed from the database.
We can implement such a Scheduler fairly easily. The main challenge is that we need to either:
1. relax the compaction constraints so that we allow compacting non-contiguous SRs. Then, the scheduler can map each SR to some bucket, and only compact SRs that are in the same bucket. Reads would still work because SR order is preserved for a time bucket.
2. Alternatively, we can extend the model in the manifest to allow a collection of `Compacted`. Then, the scheduler can maintain each bucket in its own `Compacted` instance that’s associated with some key range. Reads would need to be aware of this mapping. When deleting a bucket the whole `Compacted` instance is dropped from the mapping.
#### Distributed Compaction (e.g. scheduling larger compactions on temporary large instances)
[Section titled “Distributed Compaction (e.g. scheduling larger compactions on temporary large instances)”](#distributed-compaction-eg-scheduling-larger-compactions-on-temporary-large-instances)
One of the main advantages of running in cloud is that applications can dynamically provision resources for a short time to burst capacity, since compute is easily provisioned and billed only for the time its provisioned. This means that for really expensive compactions, we should be able to quickly spin up a short-lived but beefy compactor node to execute the compaction. There’s nothing in the design above that precludes us doing this. The compactor could in the future with some (probably pluggable depending on whether a compactor runs on k8s, or directly on a compute service like ec2) API for provisioning, provision a short-lived node, notify it about compaction work, and then commit the manifest when the compaction work is complete.
It’s worth calling out that we already get some of this benefit as most instances have burstable network (e.g. an instance with a 5Gbps baseline network can burst up to 25Gbps for a short duration).
## Appendix
[Section titled “Appendix”](#appendix)
### Network Bandwidth vs Disk Bandwidth on AWS
[Section titled “Network Bandwidth vs Disk Bandwidth on AWS”](#network-bandwidth-vs-disk-bandwidth-on-aws)
| Instance Type | Network Bandwidth MBps (full-duplex) (Baseline/Burst) | Instance Disk Bandwidth MBps (assumes 4K max io size - experimentally verified for xlarges) (Read/Write) | EBS Bandwidth MBps (full-duplex) (Baseline/Burst) |
| ------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| m5d.xlarge | 160/1280 | 230/113 | 143.75/593.75 |
| m5d.metal | 3200/3200 | 5468/2656 | 2375/2375 |
| i3en.xlarge | 537/3200 | 332/253 | 144.2/593.75 |
| i3en.metal | 12800/12800 | 7812/6250 | 2375/2375 |
| c5d.xlarge | 160/1280 | 156/70 | 143.75/593.75 |
| c5d.metal | 12800/12800 | 5468/2656 | 2375/2375 |
# Timestamps & Time-To-Live
Status: Accepted
Authors:
* [Almog Gavra](https://github.com/agavra)
References:
*
*
*
*
*
*
*
*
## Motivation
[Section titled “Motivation”](#motivation)
SlateDB does not have a native understanding of time in data operations (it does in regard to compaction/garbage collection, discussion on this later). This introduces additional complexity to operators that want to implement semantics that depend on the insertion/modification time of rows. The most common usages of timestamps are:
1. **Time-To-Live**: many applications, either for compliance or space-saving purposes require purging data a certain amount of time after its insertion. Use cases with time series data and privacy requirements such as GDPR come to mind here, though ensuring that deleted data is collected within a time bound is left for a future exercise.
2. **Resolving Out-of-Order Operations (out of scope)**: some applications may process incoming requests in a manner that results in out-of-order insertions to SlateDB (stream processing applications are frequently subject to this type of pattern). Using insertion timestamps to resolve which version of a key is more recent may be preferable to using the clock at the time of insertion.
3. **Historical Reads (out of scope)**: while SlateDB provides snapshot capabilities to ensure reads are consistent and repeatable, it requires a snapshot to have been taken at the desired time. Increasing the number of snapshots adds significant storage overhead. Instead, it is possible to trade read-latency for space amplification by maintaining multiple versions of keys at different timestamps to allow for similar “historical” read operations.
## Goals
[Section titled “Goals”](#goals)
* Define the API for specifying TTL on writes
* Define the changes to the on-disk format of stored rows
* Outline mechanism to support Time-To-Live (TTL) via Compaction Filters
## Out of Scope
[Section titled “Out of Scope”](#out-of-scope)
While the following are out of scope for the initial implementation, the design should not preclude:
* Handling out-of-order insertions is not in scope for this RFC, though we make sure the design accounts for the future possibility (see rejected alternatives section)
* Outlining compaction strategies that are optimized for TTL (such as Cassandra’s [TWCS](https://cassandra.apache.org/doc/stable/cassandra/operating/compaction/twcs.html)). This will require additional metadata per-SST and a compaction strategy to match, but is otherwise possible with what’s outlined in the scope of this design.
* Outlining compaction strategies that target efficient deletes (such as outlined in [Lethe](https://disc-projects.bu.edu/lethe/))
* Ensuring deletions are resolved within a time bound (a TTL does not ensure the data is deleted from the object store within the specified time)
* A public API for compaction filters (see ) - this RFC will implement TTL using a compaction filter but will not yet expose a public API for plugging in custom compaction filters. Instead, the implementation should ensure that such an extension is easy.
## Public API
[Section titled “Public API”](#public-api)
We will make the following changes to `DbOptions`:
* introduce a configuration for a default TTL, which defaults to `None`
SlateDB uses the system clock internally for TTL expiration. The clock returns the system time in milliseconds since the Unix epoch, and SlateDB enforces monotonicity by tracking the last tick and sleeping briefly if clock skew is detected.
```rust
pub struct DbOptions {
// ...
/// The default time-to-live (TTL), in milliseconds, for insertions (note that
/// re-inserting a key with any value will update the TTL to use default_ttl_millis)
///
/// Default: no TTL (insertions will remain until deleted)
default_ttl_millis: Option
}
```
A custom `SystemClock` implementation can be provided via the `Db::builder` API using `with_system_clock()`. This is primarily useful for testing or for environments with non-standard time sources:
```rust
use slatedb::Db;
use slatedb_common::clock::DefaultSystemClock;
use std::sync::Arc;
let db = Db::builder("my_db", object_store)
.with_system_clock(Arc::new(DefaultSystemClock::new()))
.build()
.await?;
```
`Db#put_with_options` can optionally specify a row-level TTL at insertion time by leveraging the `PutOptions` struct:
```rust
pub enum Ttl {
/// Use the default TTL configured in DbOptions
Default,
/// No expiration for this entry
NoExpiry,
/// Expire after the specified duration (in milliseconds)
ExpireAfterMillis(u64),
}
pub struct PutOptions {
/// The time-to-live (ttl) for this insertion. If this insert overwrites an existing
/// database entry, the TTL for the most recent entry will be canonical.
///
/// Default: the TTL configured in DbOptions when opening a SlateDB session
pub ttl: Ttl,
}
```
For this RFC we will not introduce the ability to disable filtering out TTL-expired records on retrieval. The default implementation will read the metadata for every retrieved row and filter out records where `ts + tll < now`. We can, as a future optimization, enable lazy deserialization of the metadata so that this doesn’t need to happen on each row retrieved if the user requests the ability to retrieve TTL’d rows that have not yet been compacted. This can be part of the API design for compaction filters (see )
## Implementation
[Section titled “Implementation”](#implementation)
### Data Format
[Section titled “Data Format”](#data-format)
Currently, SlateDB does not have a mechanism for backwards-compatibility of the SST data format.
We will first introduce the following fields to the flatbuffer definitions:
* `format_version`: field that specifies the codec version that encoded this SST
* `row_features`: an enumeration of metadata fields that are available with each encoded row. This makes it possible in the future to save space per row by disabling row attributes, but that is out of scope for this RFC
* `creation_timestamp`: the time at which this SST file was written. For this RFC this timestamp is used to schedule follow-up compactions (see [periodic compaction](#periodic-compaction)).
* `min_ts`/`max_ts`: metadata that indicates the earliest and latest timestamp values in the SST
```fbs
/// the metadata encoded with each row, note that the ordering of this enum is
/// sensitive and must be maintained
enum RowFeature: byte {
RowFlags,
Timestamp,
TimeToLive,
}
table SsTableInfo {
// ...
// The current encoding version of the data block
format_version: uint;
// The metadata attributes that are encoded with each row. These attributes will be encoded
// in order that they are declared in the RowAttributes enum
row_features: [RowFeature];
// The time at which this SST was created (milliseconds since Unix epoch)
creation_timestamp: long;
// The minimum timestamp of any row in the SST (milliseconds since Unix epoch)
min_ts: long;
// The maximum timestamp of any row in the SST (milliseconds since Unix epoch)
max_ts: long;
}
```
This value will be written when the SST is created. We will read this back using a codec that maps 1:1 with the format version. The version 0 has the following per-value format:
```plaintext
| u16 | var | u32 | var |
|---------|-----|-----------|-------|
| key_len | key | value_len | value |
|---------|-----|-----------|-------|
```
The `format_version` will be bumped to 1 and will be modified to have the following schema:
```plaintext
| u16 | var | var | u32 | var |
|---------|-----|------|-----------|-------|
| key_len | key | attr | value_len | value |
|---------|-----|------|-----------|-------|
```
The newly introduced `attr` field will be decoded using the `row_features` array specified for this SST. Which row attributes are included depend on the data that is being written. If any row in the SST has a `ttl`, the `TimeToLive` feature will be enabled.
As part of this proposal, we will no longer represent tombstones as `INT_MAX` value\_len and instead have an indicator `byte` in the attribute array which specifies the `RowFlags`. `RowFlags` is a bitmask of descriptors on the row. The first bit in `RowFlags` is set to `1` if the row is a tombstone. Other bits are reserved for future flags. Later we can add other types of row attributes as necessary (since flatbuffers does not support single-bit values, a byte is the smallest amount of data we can store here):
```plaintext
|-- attr --|
| u16 | var | byte | u32 | var |
|---------|-----|----------|-----------|-------|
| key_len | key | row_flag | value_len | value |
|---------|-----|----------|-----------|-------|
```
In cases where both `Timestamp` and `TimeToLive` are enabled, the row will look like:
```plaintext
|-------- attr --------------|
| u16 | var | byte | i64 | i64 | u32 | var |
|---------|-----|----------------------------|-----------|-------|
| key_len | key | row_flag | ts | expire_ts | value_len | value |
|---------|-----|----------|-----|-----------|-----------|-------|
```
Both `ts` and `expire_ts` are encoded as `i64` in milliseconds.
The metadata will be returned in the decoded `KeyValueDeletable` which will be modified:
```rust
pub struct KeyValueDeletable {
pub key: Bytes,
pub value: ValueDeletable,
pub attributes: RowAttributes,
}
pub struct RowAttributes {
pub ts: Option,
pub expire_ts: Option,
}
```
### Compaction
[Section titled “Compaction”](#compaction)
#### TTL Compaction Filter
[Section titled “TTL Compaction Filter”](#ttl-compaction-filter)
We will introduce the concept of a compaction filter, which serves as a mechanism for pluggable participation in the compaction process. Specifically, the filter (if configured) will run on each key during the compaction process and optionally remove or modify the value. This is independent of the periodic compaction interval specified in the following section.
For the scope of this RFC, we will be introducing a builtin TS/TTL compaction filter. This filter will add a tombstone to any value where `now > kv.attributes.expire_ts` (where `now` is the current system time in milliseconds). Note that tombstones will still contain a timestamp (see [insertion timestamps](#insertion-timestamps)). If a tombstone is added, the TTL (`expire_ts`) for the row will be cleared, indicating that a future run of this filter should not remove the tombstone.
Since this RFC does not cover the public API for compaction filters, this filter will simply run on any SST that specifies `row_features: [Timestamp && TimeToLive]`.
#### Periodic Compaction
[Section titled “Periodic Compaction”](#periodic-compaction)
If the SST is written with the `TimeToLive` row attribute enabled, the compaction scheduler will compare the SST’s `SsTableInfo#timestamp` with a newly introduced compaction config for periodic compaction time limits:
```rust
use std::time::Duration;
pub struct CompactorOptions {
// ...
// When there are SSTs in L0 (or Sorted Runs with SSTs in other levels) older than
// `periodic_compaction_interval`, SlateDB will include them in the next scheduled
// compaction -- this is a best effort interval and slateDB does not guarantee that
// compaction will happen within this interval (implementations of the compaction scheduler
// are free to implement jitters or other mechanisms to ensure that SSTs are not
// all compacted simultaneously)
periodic_compaction_interval: Duration
}
```
This will ensure that Sorted Runs or SSTs that are not otherwise scheduled by the compaction scheduler will be picked up and TTL / tombstones will be compacted. This is particularly relevant for the lower levels of the LSM tree.
## Rejected Alternatives & Follow-Ups
[Section titled “Rejected Alternatives & Follow-Ups”](#rejected-alternatives--follow-ups)
### Insertion Timestamps
[Section titled “Insertion Timestamps”](#insertion-timestamps)
A previous version of this proposal introduced the concept of row level timestamps — the ability to set a timestamp at each insertion. This introduced the ability to have out-of-order insertions, which in turn caused significant downstream complications.
In order to leave the door open for timestamps on insertion, we encode the clock time with each row in the SST as opposed to only maintaining the time that it should expire. This comes at the cost of an extra 4 bytes when cached in memory (compression will probably mean it is stored as fewer bytes).
Another design consideration in the RFC that leaves the door open for insertion timestamps is the association of a timestamp with tombstones. This is to handle the scenario where a record with a TTL is inserted with a lower timestamp than an existing record. Consider the following:
```plaintext
seq0: Insert K1 with TS=100 and TTL=100000
seq1: Insert K1 with TS=50 and TTL=100
```
If `seq0` and `seq1` result in writes in different levels of the LSM tree, when a compaction kicks in at TS=170 a tombstone is generated for `K1`. This tombstone should not result in a deletion of the original `seq0` insert as it logically happened “after” `seq1`.
## Updates
[Section titled “Updates”](#updates)
### September 25, 2024
[Section titled “September 25, 2024”](#september-25-2024)
* Removed out-of-order insertions
* Removed the functionality to retrieve metadata of a row. Without multiple copies of the row the insertion time is less interesting. we can consider adding this functionality back in upon request.
* Added the requirement for the clock to be monotonically increasing
* Added a timestamp field to `ManifestV1` to seed clock
* Added a creation\_timestamp field to `SsTableInfo` and `periodic_compaction_seconds` to the `CompactorOptions`
* Added a `RowType` metadata feature and used it to replace the current encoding of tombstones
### October 1, 2024
[Section titled “October 1, 2024”](#october-1-2024)
* Changed `RowType` to `RowFlags` and indicated that the first bit is used to identify tombstones
* Added `min_ts`/`max_ts` into `SsTableInfo`
* Updated the seeding of the clock min timestamp to look at the WAL SSTs as well as the manifest
* Modified the stored value in the SST from `ttl` to `expire_ts`
* Marked `periodic_compaction_interval` as “best effort” to allow for implementations to optimize performance characteristics such as adding a jitter or max number of SSTs to compact in one go
* Many misc. improvements to readability and clarifications
### October 6, 2024
[Section titled “October 6, 2024”](#october-6-2024)
* Minor changes and feedback. Most notable is renaming “meta features” to “row attributes”
### January 20, 2026
[Section titled “January 20, 2026”](#january-20-2026)
* **Removed the `LogicalClock` trait and user-defined clock configuration.** The original design allowed users to supply a custom clock implementation for TTL timestamps. This was removed because:
1. The custom clock functionality added complexity that introduced bugs
2. With the introduction of transactions (see RFC 0011), sequence numbers can solve many of the same use cases that user-defined clocks were intended to address (e.g., ordering operations)
* SlateDB now uses the system clock directly for TTL expiration timestamps. The `SystemClock` trait (from `slatedb-common`) returns `DateTime`, and SlateDB extracts milliseconds since epoch internally. A custom `SystemClock` can still be provided via `Db::builder().with_system_clock()` for testing or non-standard time sources.
* Updated `DbOptions` to remove the `clock` configuration option
* Updated `WriteOptions` to `PutOptions` with a `Ttl` enum that supports `Default`, `NoExpiry`, and `ExpireAfterMillis(u64)` variants
# Checkpoints and Clones
Status: Accepted
Authors:
* [Rohan Desai](https://github.com/rodesai)
* [Roman Khachatryan](https://github.com/rkhachatryan)
## Current Implementation
[Section titled “Current Implementation”](#current-implementation)
SlateDB runs a Writer, a Compactor - optionally in a separate process, and a Garbage Collector (GC) - also optionally in a separate process. The writer writes data to L0 SSTs. The Compactor then goes and compacts these L0 SSTs into Sorted Runs (SR) under the “compacted” directory, and then further compacts those SRs as they accumulate. Updates to the current set of L0s and Sorted Runs are synchronized between the Writer and Compactor through updates to the centralized manifest. When reading, the Writer gets a reference to a copy of the metadata for the current set of sorted runs and searches them for the requested data. Meanwhile, the GC scans the SSTs in Object Storage and deletes any SSTs that are not referenced by the current manifest and are older than some threshold.
## Problem
[Section titled “Problem”](#problem)
The current implementation poses a few problems and limitations:
1. There is nothing preventing the GC from deleting SSTs out from the Writer while it’s attempting to read their data. This isn’t a big deal today as we only support key lookups, which are fast. But this could become a more acute problem when we support range scans which can be long.
2. Currently only the Writer process can interact with the database. We’d like to support additional read-only clients, called Readers that can read data.
3. Some use cases can benefit from forking the database and establishing a new db that relies on the original DB’s SSTs but writes new data to a different set of SSTs from the original writer.
## Goals
[Section titled “Goals”](#goals)
In this RFC, we propose adding checkpoints and clones to address this limitation.
* Clients will be able to establish cheap db checkpoints that reference the current version of SlateDB’s manifest.
* Writers will use checkpoints to “pin” a version of the database for reads to avoid the GC from deleting the SSTs.
* Readers will use checkpoints similarly, and many Reader processes can coexist alongside a Writer.
* Finally, we will support creating so-called “clones” from a checkpoint to allow spawning new Writers that write to a “fork” of the original database.
### Out Of Scope
[Section titled “Out Of Scope”](#out-of-scope)
This RFC does not propose:
* Any mechanism to provide isolation guarantees from other concurrent reads/writes to the database.
## Proposal
[Section titled “Proposal”](#proposal)
At a high level, this proposal covers a mechanism for establishing database checkpoints, and reviews how the various processes will interact with this mechanism. Logically, a checkpoint refers to a consistent and durable view of the database at some point in time. It is consistent in the sense that data read from a checkpoint reflects all writes up to some point in time, and durable in the sense that this property holds across process restarts. Physically, a checkpoint refers to a version of SlateDB’s manifest.
### Manifest Schema
[Section titled “Manifest Schema”](#manifest-schema)
Checkpoints themselves are also stored in the manifest. We’ll store them using the following schema:
```plaintext
// Reference to an external database.
table ExternalDb {
// Path to root of the external database
path: string (required);
// Checkpoint ID on the external database used to create the initial state of this clone.
// For an entry added for the immediate parent, this is the user-supplied (or ephemeral)
// checkpoint passed to the clone operation. For entries carried over from the parent's
// own `external_dbs` (i.e. nested clones), this is set to the parent's
// `final_checkpoint_id` for that grandparent — a slatedb-generated, slatedb-owned id —
// rather than the original user-supplied checkpoint. This breaks the chain back to a
// user-supplied checkpoint that the user could delete at any time, which would otherwise
// invalidate every future clone in the chain.
source_checkpoint_id: Uuid (required);
// Checkpoint ID this database has placed on the external database that prevents referenced
// data files from being GC'd. Both final_checkpoint_id and source_checkpoint_id should resolve
// to the same manifest_id as long as they both still exist.
final_checkpoint_id: Uuid (required);
// Compacted SST IDs belonging to external DB that are currently being referenced.
sst_ids: [Ulid] (required);
}
// A view referencing a CompactedSsTable by its id. This indirection allows the same
// physical SST to appear multiple times in the manifest with different visible ranges
// (e.g. after projection and union), without duplicating SST metadata.
table CompactedSsTableView {
// Reference to a CompactedSsTable by its id (stored at the top level of ManifestV2).
id: Ulid (required);
visible_range: BytesRange;
}
table SortedRunV2 {
id: uint32;
ssts: [CompactedSsTableView] (required);
}
table ManifestV2 {
// List of external databases referenced by this manifest.
external_dbs: [ExternalDb];
// Flag to indicate whether initialization has finished. When creating the initial manifest for
// a root db (one that is not a clone), this flag will be set to true. When creating the initial
// manifest for a clone db, this flag will be set to false and then updated to true once clone
// initialization has completed.
initialized: boolean;
// Optional epoch time in seconds that this database was destroyed
destroyed_at_s: u64;
...
// All compacted SSTs referenced by l0 and compacted (sorted runs) views.
// This is the single source of truth for SST metadata (id, info, format_version).
ssts: [CompactedSsTable] (required);
// L0 and sorted run entries reference SSTs via CompactedSsTableView, which holds
// only the SST id and a visible_range. This allows the same SST to be referenced
// multiple times with different visible ranges.
l0: [CompactedSsTableView] (required);
compacted: [SortedRunV2] (required);
// A list of current checkpoints that may be active (note: this replaces the existing
// snapshots field)
checkpoint: [Checkpoints] (required);
}
table WriterCheckpoint {
epoch: u64;
}
union CheckpointMetadata { WriterCheckpoint }
table Uuid {
high: uint64;
low: uint64;
}
// Checkpoint reference to be included in manifest to record active checkpoints.
table Checkpoint {
// Id that must be unique across all open checkpoints.
id: Uuid (required);
// The manifest ID that this checkpoint is using as its `CoreDbState`.
manifest_id: ulong;
// The UTC unix timestamp seconds that a checkpoint expires at. Clients may update this value.
// If `checkpoint_expire_time_s` is older than now(), the checkpoint is considered expired.
// If `checkpoint_expire_time_s` is 0, the checkpoint will never expire.
checkpoint_expire_time_s: uint;
// The unix timestamp seconds that a checkpoint was created.
checkpoint_create_time_s: uint;
// Optional metadata associated with the checkpoint. For example, the writer can use this to
// clean up checkpoints from older writers.
metadata: CheckpointMetadata;
// Optional name associated with the checkpoint. The name can be used to list the checkpoints.
// Note that name may not be unique and the list operation can return multiple checkpoints with
// the same name.
name: string;
}
```
### Manifest V1 to V2 Migration
[Section titled “Manifest V1 to V2 Migration”](#manifest-v1-to-v2-migration)
The introduction of ManifestV2 requires a phased rollout to ensure forward compatibility across mixed-version deployments:
1. **Phase 1 (current):** Write V1, read V1+V2. All nodes can read the new V2 format, but manifests are still written in V1 format. This allows older nodes that only understand V1 to continue operating during a rolling upgrade. To support this, a `last_compacted_l0_sst_id` field is maintained alongside `last_compacted_l0_sst_view_id`, since V1 encodes `l0_last_compacted` as an SST ID rather than a view ID.
2. **Phase 2:** Write V2, read V1+V2. Once all nodes are upgraded and can read V2, the writer switches to emitting V2 manifests. V1 reading is retained for backward compatibility with existing manifests on disk.
3. **Phase 3:** Deprecate V1. Once no V1 manifests remain on disk (i.e., all have been superseded), V1 reading support can be removed.
### Public API
[Section titled “Public API”](#public-api)
We’ll make the following changes to the public API to support creating and using checkpoints:
```rust
/// Defines the scope targeted by a given checkpoint. If set to All, then the checkpoint will include
/// all writes that were issued at the time that create_checkpoint is called. SlateDB will flush WALs
/// (if enabled) and do a best-effort flush of memtables to L0 SSTs in order to optimize for
/// readers. The memtable flush will not await if blocked by backpressure. If set to Durable, then
/// the checkpoint includes only writes that were durable at the time of the call. This will be
/// faster, but may not include data from recent writes.
enum CheckpointScope {
All,
Durable
}
/// Specify options to provide when creating a checkpoint.
struct CheckpointOptions {
/// Optionally specifies the lifetime of the checkpoint to create. The expire time will be set to
/// the current wallclock time plus the specified lifetime. If lifetime is None, then the checkpoint
/// is created without an expiry time.
lifetime: Option,
/// Optionally specifies an existing checkpoint to use as the source for this checkpoint. This is
/// useful for users to establish checkpoints from existing checkpoints, but with a different lifecycle
/// and/or metadata.
source: Option
/// Optionally specifies a name for the checkpoint. Can be used to list the checkpoints.
pub name: Option,
}
#[derive(Debug)]
pub struct CheckpointCreateResult {
/// The id of the created checkpoint.
id: uuid::Uuid,
/// The manifest id referenced by the created checkpoint.
manifest_id: u64,
}
impl Db {
/// Creates a checkpoint of an opened db using the provided options. Returns the ID of the created
/// checkpoint and the id of the referenced manifest.
pub async fn create_checkpoint(
&self,
scope: CheckpointScope,
options: &CheckpointOptions,
) -> Result {
…
}
/// Refresh the lifetime of an existing checkpoint. Takes the id of an existing checkpoint
/// and a lifetime, and sets the lifetime of the checkpoint to the specified lifetime. If
/// there is no checkpoint with the specified id, then this fn fails with
/// SlateDBError::InvalidDbState
pub async fn refresh_checkpoint(
path: &Path,
object_store: Arc,
id: Uuid,
lifetime: Option,
) -> Result<(), SlateDBError> {}
/// Deletes the checkpoint with the specified id.
pub async fn delete_checkpoint(
path: &Path,
object_store: Arc,
id: Uuid,
) -> Result<(), SlateDBError> {}
/// Called to destroy the database at the given path. If `soft` is true, This method will
/// set the destroyed_at_s field in the manifest. The GC will clean up the db after some
/// time has passed, and all checkpoints have either been deleted or expired. As part of
/// cleaning up the db, the GC will also remove the database’s checkpoint from the parent
/// database. If `soft` is false, then all cleanup will be performed by the call to this
/// method. If `soft` is false, the destroy will return SlateDbError::InvalidDeletion if
/// there are any remaining non-expired checkpoints.
pub async fn destroy(path: Path, object_store: Arc, soft: bool) -> Result<(), SlateDbError> {
…
}
}
mod admin {
/// Creates a checkpoint of the db stored in the object store at the specified path using the provided options.
/// Note that the scope option does not impact the behaviour of this method. The checkpoint will reference
/// the current active manifest of the db.
pub async fn create_checkpoint>(
path: P,
object_store: Arc,
options: &CheckpointOptions,
) -> Result {}
/// Clone a Db from a checkpoint. If no db already exists at the specified path, then this will create
/// a new db under the path that is a clone of the db at parent_path. A clone is a shallow copy of the
/// parent database - it starts with a manifest that references the same SSTs, but doesn't actually copy
/// those SSTs, except for the WAL. New writes will be written to the newly created db and will not be
/// reflected in the parent database. The clone can optionally be created from an existing checkpoint. If
/// parent_checkpoint is None, then the manifest referenced by parent_checkpoint is used as the base for
/// the clone db's manifest. Otherwise, this method creates a new checkpoint for the current version of
/// the parent db.
pub async fn create_clone>(
clone_path: P,
parent_path: P,
object_store: Arc,
parent_checkpoint: Option,
) -> Result<(), Box> {
…
}
}
/// Configuration options for the database reader. These options are set on client startup.
#[derive(Clone)]
pub struct DbReaderOptions {
/// How frequently to poll for new manifest files. Refreshing the manifest file allows readers
/// to detect newly compacted data.
pub manifest_poll_interval: Duration,
/// For readers that refresh their checkpoint, this specifies the lifetime to use for the
/// created checkpoint. The checkpoint’s expire time will be set to the current time plus
/// this value. If not specified, then the checkpoint will be created with no expiry, and
/// must be manually removed. This lifetime must always be greater than
/// manifest_poll_interval x 2
pub checkpoint_lifetime: Option,
/// The max size of a single in-memory table used to buffer WAL entries
/// Defaults to 64MB
pub max_memtable_bytes: u64
}
pub struct DbReader {
…
}
impl DbReader {
/// Creates a database reader that can read the contents of a database (but cannot write any
/// data). The caller can provide an optional checkpoint. If the checkpoint is provided, the
/// reader will read using the specified checkpoint and will not periodically refresh the
/// checkpoint. Otherwise, the reader creates a new checkpoint pointing to the current manifest
/// and refreshes it periodically as specified in the options. It also removes the previous
/// checkpoint once any ongoing reads have completed.
pub async fn open(
path: Path,
object_store: Arc,
checkpoint: Option,
options: DbReaderOptions
) -> Result { ... }
/// Read a key an return the read value, if any.
pub async fn get(&self, key: &[u8]) -> Result, SlateDBError> { ... }
}
pub struct GarbageCollectorOptions {
/// Defines a grace period for deletion of a db. A db for which deletion is handled by the GC
/// will not be deleted from object storage until this duration has passed after it's deletion
/// timestamp.
pub db_delete_grace: Duration,
}
```
Finally, we’ll also allow users to create/refresh/delete/list checkpoints using the admin CLI:
```bash
$ slatedb create-checkpoint --help
Create a new checkpoint pointing to the database's current state
Usage: slatedb --path create-checkpoint [OPTIONS]
Options:
-l, --lifetime Optionally specify a lifetime for the created checkpoint. You can
specify the lifetime in a human-friendly format that uses years/days/min/s, e.g. "7days 30min 10s".
The checkpoint's expiry time will be set to the current wallclock time plus the specified
lifetime. If the lifetime is not specified, then the checkpoint is set with no expiry and
must be explicitly removed
-s, --source Optionally specify an existing checkpoint to use as the base for the newly
created checkpoint. If not provided then the checkpoint will be taken against the latest manifest.
-n, --name Optionally specify a name for the checkpoint. The name can be used to list
the checkpoints. Note that name may not be unique and the list operation can return multiple
checkpoints with the same name
$ slatedb refresh-checkpoint --help
Refresh an existing checkpoint's expiry time. This command will look for an existing checkpoint
and update its expiry time using the specified lifetime
Usage: slatedb --path refresh-checkpoint [OPTIONS] --id
Options:
-i, --id The UUID of the checkpoint (e.g. 01740ee5-6459-44af-9a45-85deb6e468e3)
-l, --lifetime Optionally specify a new lifetime for the checkpoint. You can specify
the lifetime in a human-friendly format that uses years/days/min/s, e.g. "7days 30min 10s". The
checkpoint's expiry time will be set to the current wallclock time plus the specified lifetime.
If the lifetime is not specified, then the checkpoint is updated with no expiry and must be
explicitly removed
$ slatedb delete-checkpoint --help
Delete an existing checkpoint
Usage: slatedb --path delete-checkpoint --id
Options:
-i, --id The UUID of the checkpoint (e.g. 01740ee5-6459-44af-9a45-85deb6e468e3)
$ slatedb list-checkpoints --help
List the current checkpoints of the db
Usage: slatedb --path list-checkpoints [OPTIONS]
Options:
-n, --name Optionally specify the name to filter the checkpoints. Note that name may not
be unique and the list operation can return multiple checkpoints with the same name. If not
specified, all checkpoints will be returned.
```
### Creating a Checkpoint
[Section titled “Creating a Checkpoint”](#creating-a-checkpoint)
Checkpoints can be created using the `create_checkpoint` API, the `create-checkpoint` CLI command, or by the writer/reader clients as part of their normal operations (see below sections).
Checkpoints can be taken from the current manifest, or from an existing checkpoint. Checkpoints cannot be created from arbitrary manifest versions.
Generally to take a checkpoint from the current manifest, the client runs a procedure that looks like the following:
1. Get the current manifest M at version V.
2. If M.`initialized` is false, or M.`destroyed_at_s` is set, exit with error.
3. Compute a new v4 UUID id for the checkpoint.
4. Create a new entry in `checkpoints` with:
* `id` set to the id computed in the previous step
* `manifest_id` set to V or V+1. We use V+1 to allow addition of the checkpoint to be included with other updates.
* other fields set as appropriate (e.g. expiry time based on relevant checkpoint creation params)
5. Write a new manifest M’ at version V+1. If CAS fails, go to step 1.
To take a checkpoint from an existing checkpoint with id S, the client runs the following procedure:
1. Get the current manifest M at version V.
2. If `destroyed_at_s` is set, exit with error.
3. Look for the checkpoint C with id S under `checkpoints`. If there is no such checkpoint, exit with error.
4. Check that C is not expired. If it is, then exit with error.
5. Compute a new v4 UUID id for the checkpoint.
6. Create a new entry in `checkpoints` with:
* `id` set to the id computed in the previous step
* `manifest_id` set to C.`manifest_id`
* other fields set as appropriate (e.g. expiry time based on relevant checkpoint creation params)
7. Write a new manifest M’ at version V+1. If CAS fails, go to step 1.
The `create-checkpoint` CLI and `create_checkpoint` API will run exactly these procedures when called without and with a source checkpoint ID, respectively.
Otherwise, the exact procedures used by the writer/reader clients vary slightly depending on the context. For example, the writer may include other updates along with the update that adds the checkpoint. These variants are detailed in the sections below.
### Writers
[Section titled “Writers”](#writers)
Writers will create checkpoints to “pin” the set of SRs currently being used to serve reads. As the writer consumes manifest updates, it will re-establish its checkpoint to point to the latest manifest version. Eventually, when we support range scans, the writer will retain older checkpoints to ensure that long-lived scans can function without interference from the GC.
#### Establishing Initial Checkpoint
[Section titled “Establishing Initial Checkpoint”](#establishing-initial-checkpoint)
The writer first establishes it’s checkpoint against the current manifest when starting up, as part of incrementing the epoch:
1. Read the latest manifest version V.
2. Compute the writer’s epoch E
3. Look through the checkpoints under `checkpoint` for any entries with `metadata` of type `WriterCheckpoint`, and remove any such entries.
4. Create a new checkpoint with:
* `id` set to a new v4 uuid
* `manifest_id` set to the next manifest version (V + 1)
* `checkpoint_expire_time_s` set to None
* `metadata` set to `WriterCheckpoint{ epoch: E }`.
5. Set `writer_epoch` to E
6. Write the manifest. If this fails with a CAS error, go back to 1.
#### Refreshing the Checkpoint
[Section titled “Refreshing the Checkpoint”](#refreshing-the-checkpoint)
The writer will periodically re-establish its checkpoint to pin the latest SSTs and SRs. It will run the following whenever it detects that the manifest has been updated. This happens either when the manifest is polled (at `manifest_poll_interval`), or when the writer detects an update when it goes to update the manifest with it’s local view of the core db state:
1. Read the latest manifest version V. (If the writer is applying an update from the local state then in the first iteration it can assume that it’s local version is the latest. If it’s wrong, the CAS will fail and it should reload the manifest from Object Storage))
2. Decide whether it needs to re-establish it’s checkpoint. The writer reestablishes the checkpoint whenever the set of L0 SSTs, or Sorted Runs has changed. This includes both changes made locally by the Writer, or by the Compactor.
1. If so, create a new checkpoint with
* `id` set to a new v4 UUID.
* `manifest_id` set to V+1.
* `checkpoint_expire_time` set to None.
* `metadata` set to `WriterCheckpoint{ epoch: E }`.
2. Delete the old checkpoint from `checkpoints`. When we support range scans, we’ll instead want to maintain references to the checkpoint from the range scans and delete the checkpoint when all references are dropped. (We should technically do this for point lookups too, but the likelihood of GC deleting an SST in this case is very low).
3. Apply any local changes
4. Write the new manifest. If this fails with a CAS error, go back to 1.
#### Clones
[Section titled “Clones”](#clones)
The writer will also support initializing a new database from an existing checkpoint. This allows users to “fork” an instance of slatedb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance.
To support this, we add the `Db::open_from_checkpoint` method, which accepts a `parent_path` and `Option`. When initializing the cloned database, the writer will do the following:
1. Read the current cloned manifest `Option`.
2. Read the current parent manifest `M_p` at `parent_path`.
* If `M_p.initialized` is false, exit with error.
4. If `Option` is present
* Validate `parent_path` and `Option` are contained in `M_c.external_dbs`.
* If not, exit with error.
* If `Option` is not set, accept any checkpoint.
* If `M_c.initialized` is
* **True**
* Validate all external databases have a final checkpoint. If not, exit with error.
* Go to step 11
* **False**: Go to step 10
5. Compute the clone’s `source_checkpoint_id`:
* If `Option` is present use it.
* Otherwise, create a new ephemeral checkpoint of parent’s latest manifest and use that. Ephemeral checkpoint will have TTL of 5 minutes, giving us an upper bound for retrying the clone operation.
7. Create a new clone manifest `M_c'`.
* Assign `final_checkpoint_id` to a new random v4 UUID.
* Resolve parent manifest `M_p'` at `source_checkpoint_id` and use it as the basis for the clone.
* Set `M_c.initialized` to **False**.
8. Write a new clone manifest `M_c'`. If CAS fails, go to step 1. `M_c'` fields are set to:
* external\_dbs:
* initial list copied from `M_p'` (this list is non-empty for nested clones)
* for each `entry` in the list:
* set `entry.source_checkpoint_id` to `entry.final_checkpoint_id` (the parent’s pinned final checkpoint on that grandparent). This breaks the chain back to the original user-supplied checkpoint on the grandparent: that checkpoint can be deleted by the user at any time, and if a later clone tried to re-pin it the operation would fail. The parent’s `final_checkpoint_id` is slatedb-generated and pinned by the parent, so it is safe to depend on for future clones.
* assign a fresh `entry.final_checkpoint_id` (a new random v4 UUID) for this clone to own.
* add an `entry` for `parent_path` with:
* `entry.path` set to `parent_path`
* `entry.source_checkpoint_id` set to `source_checkpoint_id`
* `entry.final_checkpoint_id` set to `final_checkpoint_id`
* writer\_epoch: set to the epoch from `M_p' + 1`.
* compactor\_epoch: copied from `M_p'`.
* wal\_id\_last\_compacted: copied from `M_p'`
* wal\_id\_last\_seen: copied from `M_p'`
* l0: copied from `M_p'`
* l0\_last\_compacted: copied from `M_p'`
* compacted: copied from `M_p'`
* checkpoints: empty
9. Assign `M_c = M_c'`
10. For each external database `ED` in `M_c`
* Ensure final checkpoint with `ED.final_checkpoint_id` exists. If not, create it with following options:
* `CheckpointOptions{ lifetime: None, source: Some(ED.source_checkpoint_id) }`
* If checkpoint creation fails, because the source checkpoint does not exist, exit with error. This can happen when ephemeral checkpoint expires.
11. Copy over all WAL SSTs between `M_c.last_compacted_wal_id` and `M_c.wal_id_last_seen` from `parent_path`.
12. Update `M_c` with `initialized` set to true. If CAS fails, go to step 1.
##### SST Path Resolution
[Section titled “SST Path Resolution”](#sst-path-resolution)
The DB may now need to look through multiple paths to find SST files, since it may need to open SST files written by an external db. To support this, we’ll materialize a map of external SSTs at startup by reading `external_dbs` from the manifest and pass it to the `PathResolver` for use.
##### Deleting a Database
[Section titled “Deleting a Database”](#deleting-a-database)
With the addition of clones, we need to be a bit more careful about how we delete a database. Previously we could simply remove it’s objects from the Object Store and call it a day. With clones, it’s possible that a SlateDB instance is holding a checkpoint that another database relies upon. So it’s not safe to just delete that data. We’ll instead require that users delete a database by calling the `Db::destroy` method. `Db::destroy` can specify that the delete is either a soft or hard delete by using the `soft` parameter. This way we can support both more conservative deletes that clean up data from the GC process after some grace period, and a simple path for hard-deleting a db.
A soft delete will simply mark the manifest as deleted, and rely on a separate GC process to delete the db. It requires that the GC be running in some other long-lived process from the main writer. The GC process will wait for both some grace period, and for all checkpoints to either expire or be deleted. At that point it will delete the db’s SSTs and exit. This is covered in more detail in the GC section. A soft delete will be processed by the writer as follows:
1. Claim write-ownership of the DB by bumping the epoch and fencing the WAL. The intent here is to fence any active writer. We assume that the writer will detect that it’s fenced and exit by the time the GC goes to delete the data.
2. Delete the writer’s checkpoint, if any.
3. Update the manifest by setting the `destroyed_at_s` field to the current time.
A hard delete will proceed as follows. It is up to the user to ensure that there is no active writer at the time of deletion:
1. Check that there are no active checkpoints. If there are, return `InvalidDeletion`
2. If `destroyed_at_s` is not set, then update the manifest by setting `destroyed_at_s` field to the current time. If CAS fails, go to step 1.
3. Delete all objects under `path` other than the current manifest.
4. Delete the current manifest.
### Readers
[Section titled “Readers”](#readers)
Readers are created by calling `DbReader::open`. `open` takes an optional checkpoint. If a checkpoint is provided, `DbReader` uses the checkpoint and does not try to refresh/re-establish it. If no checkpoint is provided, `DbReader` establishes its own checkpoint against the latest manifest and periodically polls the manifest at the interval specified in `manifest_poll_interval` to see if it needs to re-establish it. If the manifest has not changed in a way that requires the reader to re-establish the checkpoint, then the reader will refresh the checkpoint’s expiry time once half the lifetime has passed. The checkpoints created by `DbReader` use the lifetime specified in the `checkpoint_lifetime` option.
#### Establishing the Checkpoint
[Section titled “Establishing the Checkpoint”](#establishing-the-checkpoint)
To establish the checkpoint, `DbReader` will:
1. Read the latest manifest M at version V.
2. Check that M.`initialized` is true and M.`destroyed_at_s` is not set. Otherwise, return error.
3. Create a new checkpoint with id set to a new v4 uuid, `manifest_id` set to the next manifest version (V + 1), `checkpoint_expire_time_s` set to the current time plus `checkpoint_lifetime`, and `metadata` set to None.
4. Update the manifest’s `wal_id_last_seen` to the latest WAL id found in the wal directory
5. Write the manifest. If this fails with a CAS error, go back to 1.
Then, the reader will restore data from the WAL. The reader will just replay the WAL data into an in-memory table. The size of the table will be bounded by `max_memtable_bytes`. When the table fills up, it’ll be added to a list and a new table will be initialized. The total memory used should be limited by backpressure applied by the writer. This could still be a lot of memory (e.g. up to 256MB in the default case). In the future we could have the reader read directly from the WAL SSTs, but I’m leaving that out of this proposal.
##### Checkpoint Maintenance
[Section titled “Checkpoint Maintenance”](#checkpoint-maintenance)
If the reader is opened without a checkpoint, it will periodically poll the manifest and potentially re-establish the checkpoint that it creates if the db has changed in a way that requires re-establishing the checkpoint (the set of SSTs has changed). It does this by re-running the initialization process described above, with the following modifications:
* It will delete the checkpoint that it had previously created (when we support range scans, we’ll need to reference count the checkpoint).
* It will purge any in-memory tables with content from WAL SSTs lower than the new `wal_id_last_compacted`.
Additionally, if the time until `checkpoint_expire_time_s` is less than half of `checkpoint_lifetime` the Reader will update the `checkpoint_expire_time_s` to the current time plus `checkpoint_lifetime`.
#### Compactions
[Section titled “Compactions”](#compactions)
During compaction, if an external SST is dereferenced, we’ll remove its entry from the corresponding `external_dbs[].sst_ids` set and update the manifest.
#### Garbage Collector
[Section titled “Garbage Collector”](#garbage-collector)
The GC task will be modified to handle soft deletes and manage checkpoints/clones. The GC tasks’s algorithm will now look like the following:
1. Read the current manifest. If no manifest is found, exit.
2. If `destroyed_at_s` is set, and the current time is after `destroyed_at_s` + `db_delete_grace`, and there are no active checkpoints, then:
1. If the db is a clone, update the parent db’s manifest by deleting the checkpoint.
2. Delete all object’s under the db’s path other than the current manifest.
3. Delete the current manifest.
4. Exit.
3. Garbage collect expired checkpoints
1. Find any checkpoints that have expired and remove them
2. Write the new manifest version with checkpoints deleted. If CAS fails, go back to 1 to reload checkpoints that may be refreshed or added.
4. Delete garbage manifests. This includes any manifest that is older than `min_age` and is not referenced by a checkpoint.
5. Read all manifests referenced by a checkpoint. Call this set M
6. Clean up WAL SSTs.
1. For a given Manifest n Let referenced\_wal(n) be the set of all WAL SSTs between n.`wal_id_last_compacted` and n.`wal_id_last_seen`
2. Let W be the set of all referenced\_wal(n) for all n in M AND all WAL SSTs larger than `wal_id_last_compacted` of the current manifest.
3. Delete all WAL SSTs not in W.
7. Clean up SSTs.
1. Let S be the set of all SSTs from all manifests in M.
2. Delete all SSTs not in M with age older than `min_age`
8. Detach the clone if possible. If list of external databases is non-empty, then for each external database `DB`:
* If `DB.sst_ids` is empty at the latest version of the manifest and all existing checkpoints, then:
* Remove `DB.final_checkpoint_id` from the external db’s manifest.
* Update the manifest by removing `DB` from the list of external databases.
Observe that users can now configure a single GC process that can manage GC for multiple databases that use soft deletes. Whenever a new database is created, the user needs to spawn a new GC task for that database. When the GC task completes deleting a database, then the task exits. For now, it’s left up to the user to spawn GC tasks for databases that they have created.
### Manifest Projection and Union
[Section titled “Manifest Projection and Union”](#manifest-projection-and-union)
Clones create new databases that reference SSTs from a parent. In some cases, it is useful to create views over a subset of a manifest’s key range, or to combine multiple non-overlapping manifest views into a single manifest. In particular, this can be useful to implement database rescaling. We introduce two operations on manifests to support this: **projection** and **union**.
#### Projection
[Section titled “Projection”](#projection)
Projection creates a filtered view of a manifest restricted to a specific key range specified by the client. Given a source manifest and a target range, projection returns a new manifest containing only the SSTs whose key ranges overlap with the target range. SSTs that fall entirely outside the target range are excluded, and SSTs that partially overlap are annotated with a `visible_range` that constrains which keys are accessible.
Write requests that are fully or partially outside the `visible_range` will fail, as well as read requests completely outside the `visible_range`.
The new SSTs produced by Writer and Compactor do not have `visible_range` attached, although the actual range of keys in those SSTs lies inside `visible_range` (in the future, we might want to attach `visible_range` to the whole manifest).
The projection process works as follows:
1. Clone the source manifest.
2. For each L0 SST view, compute the intersection of the SST’s effective range with the projection range. L0 SSTs may overlap with each other, so each SST is evaluated independently against the projection range.
3. For each sorted run, compute the intersection of each SST’s range with the projection range. Since SSTs within a sorted run are non-overlapping and ordered, the range of each SST is bounded by the start key of the next SST in the run (i.e., the range is `[current_start, next_start)`). The last SST in a run extends to the end of its effective range.
4. For each SST view that has a non-empty intersection with the projection range, create a new view with the `visible_range` set to the intersection. SSTs with an empty intersection are excluded from the result.
5. Sorted runs whose SSTs were all excluded are removed entirely. Note that removing sorted runs may leave gaps in sorted run IDs (e.g., `[5, 3, 1]` if SR 4 and 2 were removed). This is acceptable because the compactor does not assume contiguous IDs — it matches sorted runs by ID, not by position. Preserving original IDs is important so that union can correctly match the same logical tier across projected manifests.
6. Remove excluded SST IDs from `external_dbs` entries so that the external database’s checkpoint can be released and the SSTs garbage collected once no other manifest references them. Entries whose `sst_ids` list becomes empty are removed entirely, allowing their checkpoint on the external database to be released.
The `visible_range` on an `SsTableView` constrains the keys that are visible when reading the SST. The `effective_range` (computed and not stored) is the intersection of the SST’s physical range (derived from its first key) and the `visible_range`. Scans and lookups against a projected SST only return keys within the effective range.
Projection is logically equivalent to deleting all entries outside the projected range. Once an SST has been projected, its `visible_range` can only be narrowed further, never extended — the entries outside the original projection are treated as if they no longer exist.
Note: currently, size estimation for the projected SSTs is not accurate, which might lead to suboptimal compactor decisions. This will be improved in the future iterations.
Note: currently, it is the client responsibility to maintain the ranges associated with SlateDB instances and supply them to Projection and Union.
#### Union
[Section titled “Union”](#union)
Union combines multiple non-overlapping manifests into a single manifest. This is the inverse of projection: given a set of manifests that each cover a distinct portion of the key space, union produces a manifest that covers the combined key space. The source databases don’t have to share the root database (e.g. they can be independent shards).
Union does not support merging WAL state at the moment. Input manifests must have their WAL fully compacted before performing a union. This can be achieved for example by calling `db.flush_with_options(... FlushOptions { flush_type: FlushType::MemTable });` to force a flush to L0 (all writes prior to that flush will be guaranteed to be in L0+ and in the manifest); or setting `wal_enabled: false` from the beginning.
The union process works as follows:
1. Sort the input manifests by the start bound of their key ranges.
2. Validate that the manifests are non-overlapping. Each manifest’s key range is computed from the effective ranges of all its L0 and compacted SSTs and optionally intersected with `visible_range`s for each manifest if they are provided by the user. If any two manifests have intersecting key ranges, the operation fails. If the `visible_ranges` are explicitly provided then validate that they are adjacent. Segmented manifests apply this check per segment rather than to the manifest as a whole; see [RFC 0024](/rfcs/0024-segment-oriented-compaction#interaction-with-projection-and-union).
3. Merge the contents of all input manifests:
* `external_dbs` entries from all input manifests are merged and deduplicated by `(path, source_checkpoint_id)`. `external_dbs` with the same `(path, source_checkpoint_id)` originated from the same external\_db entry via projection, so their `sst_ids` lists are merged (unioned). Entries with different keys for the same path represent distinct checkpoints and must be kept separate. Without this deduplication, repeated cycles of projection and union cause exponential growth of duplicated entries: projecting into N parts and unioning back multiplies the entry count by N each cycle. Note that for entries carried over from a clone’s parents, `source_checkpoint_id` is the parent’s `final_checkpoint_id` on that grandparent (a slatedb-owned, pinned id — see [Clones](#clones)), not the original user-supplied checkpoint; this is what makes the dedup key stable across the union, since all sibling projections share the same slatedb-owned id and the user cannot invalidate it by deleting their own checkpoint. New `final_checkpoint_id` values are generated for all entries — the old ones belong to the source databases’ clone relationships and are not valid for the new database.
* SSTs with the same ID but **originating** from different source databases are deduplicated by assigning a new ID (this might happen because ULID is not guaranteed to be globally unique) and updating the corresponding views accordingly. To preserve the order of ULIDs within the same database, subsequent SSTs might require new IDs as well. The old ID is kept in `CompactedSsTableV2` as `original_id` and in `external_db.sst_ids` to allow path resolution
* L0 SST views are copied from the source databases (preserving the order within each source database)
* Sorted runs are copied from the source manifests preserving their relative order within each source manifest. Their IDs are regenerated to maintain uniqueness. As an optimization, similarly sized SRs can be merged together to reduce LSM height and metadata size; only SRs from different manifests can be merged because they are guaranteed to be non-overlapping.
4. `last_l0_seq` is set to the maximum of `last_l0_seq` across all input manifests. This ensures that the new database’s writer assigns sequence numbers higher than any existing data. Without this, new writes could receive sequence numbers that collide with those in the carried-over SSTs, breaking snapshot isolation and MVCC ordering. Note that existing SSTs from independent sources may have overlapping sequence numbers, but this is safe because the non-overlapping key range validation (step 2) guarantees that no two entries for the same key can have conflicting sequence numbers.
5. The resulting manifest has `initialized` set to `false`. The caller must complete setup (e.g., creating final checkpoints in source databases) before setting `initialized` to `true`.
6. Each source database is added as an `external_dbs` entry in the resulting manifest, with a new `final_checkpoint_id` and its owned SST IDs recorded so that the new database can resolve SST paths and maintain checkpoints on the source databases to prevent their SSTs from being garbage collected. Owned SST IDs are those in the source manifest that are not already tracked by its own `external_dbs` (i.e., SSTs physically stored under the source’s path).
##### Merging SST
[Section titled “Merging SST”](#merging-sst)
SST views coming from different databases might refer to the same SSTs (but have different `visible_range`s). It might be beneficial to merge such views to achieve the following:
* reduce the chance of hitting `l0_max_ssts` on writes right after union
* improve accuracy of size estimates (used by Compactor)
* reduce the size of the metadata
However, there are a number of constraints that must be met when merging SST views:
1. Due to compaction, views over the same SST might not be adjacent, i.e. have gaps; preserving those gaps is necessary to provide correctness; therefore, only the adjacent views can be merged.
2. Sorted Run boundaries (and L0/SR boundary) can not be crossed
3. Within SR, SST views must be sorted by key and can not overlap
4. L0 ordering: newer SST views must precede views over older SSTs
The last constraint could be met by ordering SST views by their underlying SSTs’ logical creation time. This time is normally represented by ULID; however, there’s a slight chance that due to a wall clock skew, parents’ SST ULIDs are later than the children’s ones. Instead, logical creation time can be inferred from topological order of SST (with dependencies implied from the order of SST in L0 lists).
Apart from the above union process, cloning from multiple source databases is the same as clone (see [Clones](#clones)), including creation of final checkpoints in source databases (and their source databases, recursively).
There are a few minor differences:
1. **Writer and compactor epochs are 0.** The new database is a fresh instance with no prior writer or compactor history, so both epochs start at 0.
2. **Default manifest core fields.** Fields such as `next_wal_sst_id` and similar manifest core metadata are initialized to their default values rather than being copied from any source database. An exception is `last_l0_seq`, which is derived from the maximum across all sources during union (see step 5 above).
3. **WAL is not carried over from any source database.** All source manifests must have their WAL fully compacted before the operation. The new database starts with an empty WAL. That also allows to start with new `wal_id_last_seen` and `replay_after_wal_id` on union
4. SequenceTracker is re-initialized on union; Sequence Tracking only works for the records added afterwards
5. Due to a temporary spike in the number of L0 after **union**, **`l0_max_ssts` might be exceeded** blocking the writes (until compaction). This is partially mitigated by `max_unflushed_bytes` buffer that can absorb the spike. Alternatively, this can be improved by merging adjacent SST views; and/or checking `l0_max_ssts` against a maximum number of views per any key.
Note: **union** works correctly starting from Manifest V2 with the introduction of `SstViewRange`. Without it, SST in L0 and SortedRuns might be duplicated, which can lead to incorrect results and violates assumptions in Compactor and other places.
# Range Scans
Status: Accepted
Authors:
* [Jason Gustafson](https://github.com/hachikuji)
References:
*
*
*
*
*
## Motivation
[Section titled “Motivation”](#motivation)
The ability to scan the keys in a range is an essential feature for building applications. For example, this can be used to implement a collections of keys using a common prefix. Currently, SlateDB only supports fetching individual values for known keys.
## Goals
[Section titled “Goals”](#goals)
The goal of this work is to provide a flexible capability to query a range of keys in the database. It should be possible to specify unbounded ranges to allow querying all keys, or bounded ranges for finer precision and efficiency.
Range queries have the potential to be expensive from a resource perspective because they hold references to existing tables, which prevents garbage collection. Our goal here is to provide a simple initial approach to state management which optimizes for common cases, while leaving room for further optimization in the future.
## Public API
[Section titled “Public API”](#public-api)
We introduce a new `Db::scan` API which accept a `DbRange` defining the range of the scan. The result will be a `DbIterator` object which iterates the current records within the range. Users may either provide an explicit starting range in the call to `scan`, or they may `seek` using the returned iterator.
```rust
impl Db {
/// Scan a range of keys.
///
/// returns a `DbIterator`
pub async fn scan(
&self,
range: T
) -> Result where
T: RangeBounds
{
...
}
}
impl DbIterator {
/// Get the next record in the scan.
///
/// returns Ok(None) when the scan is complete
/// returns Err(InvalidatedIterator) if the iterator has been invalidated
/// due to an underlying error
pub async fn next(
&mut self,
) -> Result, SlateDbError> {
...
}
/// Seek to a key ahead of the last key returned from the iterator or
/// the lower range bound if no records have yet been returned.
///
/// returns Ok(()) if the position is successfully advanced
/// returns SlateDbError::InvalidArgument if `lower_bound` is `Unbounded`
/// returns SlateDbError::InvalidArgument if the key is comes before the
/// current iterator position
/// returns SlateDbError::InvalidArgument if `lower_bound` is beyond the
/// upper bound specified in the original `scan` parameters
/// returns Err(InvalidatedIterator) if the iterator has been invalidated
/// in order to reclaim resources
pub async fn seek(
&mut self,
next_key: Bytes,
) -> Result<(), SlateDbError> {
...
}
}
pub struct KeyValue {
pub key: Bytes,
pub value: Bytes,
}
```
Note that the `DbIterator` may be invalidated internally if any of its referenced resources must be reclaimed. In this case, a call to `DbIterator::next` or `DbIterator::scan` will return `SlateDb::InvalidatedError`. This is discussed in more detail below.
We will also provide `Db::scan_with_options`, which allows the user to specify the query isolation through a `ScanOptions` struct. Similar to `get_with_options`, the two isolation levels that will be supported are `Committed` and `Uncommitted`. The default when `scan` is called without explicit options will be `Committed`.
```rust
impl Db {
/// Scan a range of keys with the provided options.
///
/// returns a `DbIterator`
pub async fn scan_with_options(
&self,
range: T,
options: &ScanOptions,
) -> Result where
T: RangeBounds
{
...
}
}
pub struct ScanOptions {
/// The read commit level for read operations
pub read_level: ReadLevel,
/// The number of bytes to read ahead
pub read_ahead_bytes: usize,
/// Whether or not fetched blocks should be cached
pub cache_blocks: bool
}
```
## Implementation
[Section titled “Implementation”](#implementation)
A scan must iterate every table which may have keys within the query range. This includes tables in memory which which have not been flushed and L0/SR tables which may need to be loaded from object storage. Internally, we will rely on iterators which sort and merge these tables. The latest records will have precedence following the same rules that a normal `get` query follows.
We will rely on the latest checkpoint state in order to obtain references to memtables, SSTs, and sorted runs needed for the query. The `DbIterator` will expose the UUID of the checkpoint that it was generated from. The `DbIterator` will only maintain references to tables which cover the range specified in the `scan` parameters.
Note that if the `DbReader` was created with a reference to a specific checkpoint, then the `DbIterator` will only refer to that checkpoint.
The existing `MergeIterator` used for compaction provides much of the necessary foundation to unify multiple iterators. We will build an iterator which sorts and merges memtables (e.g. with `MemTableIterator`), L0 tables (`SstIterator`), and sorted runs (`SortedRunIterator`).
**Query Isolation**: When using the `Uncommitted` isolation, unflushed records recorded in WALs will be used as a source for the key range. Otherwise, only committed records in memtables, as well as L0 and SR tables will be used.
Range queries must access mutable table state, such as the current WAL. It is therefore possible for the query results to include some records which were written after the query began scanning its range of records. For example, a query may include only a subset of records that were inserted in the same `WriteBatch`. In other words, range queries will not have snapshot isolation initially.
When a `Db` or `DbReader` is closed, active iterators will be invalidated. This will cause calls to `DbIterator::next` or `DbIterator::seek` to fail with an `InvalidatedIterator` error which indicates that the database was shutdown.
[RFC-????](https://github.com/slatedb/slatedb/pull/260) proposes transactional semantics for SlateDB. This relies on a monotonic sequence number which is tagged on each key update. Range queries could leverage this sequence number to improve isolation by filtering any updates with sequence numbers larger than the latest at the time the query was executed. We leave this as a gap for future work to address.
### Checkpoint References
[Section titled “Checkpoint References”](#checkpoint-references)
We intend to rely on state checkpoints introduced in [RFC-0004](https://github.com/slatedb/slatedb/blob/main/rfcs/0004-checkpoints.md) in order to prevent the cleanup of durable state referenced by the query. When a range scan is created, it will create an internal reference to the current checkpoint. This reference will prevent cleanup of the checkpoint state until the scan is complete.
Checkpoints created by `DbReaders` have an expiry time, which is only refreshed as long as long as the checkpoint remains current. If a `DbIterator` instance remains active long enough for its referenced checkpoint to expire, then the `DbIterator` may eventually encounter a NotFound error from the object store. This error will be propagated to the next invocation of `DbIterator::next` or `DbIterator::seek`. After encountering this error, the iterator will be invalidated. Invalidated iterators will return `SlateDbError::InvalidatedIterator` on any subsequent call to `DbIterator::next` or `DbIterator::seek`.
In general, we make the initial simplifying assumption that scans will not be frequent enough or last long enough for checkpoint accumulation to be a problem. In the future, we could either limit the number of checkpoint references that are available for scanning or we could limit the time that a `DbIterator` will remain valid.
### Memory Usage
[Section titled “Memory Usage”](#memory-usage)
In addition to the checkpoint, a range scan may hold references to records in unflushed WALs or memtables. A potential optimization is to pre-merge these tables in order to filter the records within the scan range. This can be useful when targeting a small range of keys with the scan. This is left for future work.
Iterators will use prefetching to improve performance. Internally, this is already implemented by `SstIterator`. Users can influence prefetching to improve performance using `ScanOptions::read_ahead_size`, which indicates the number of bytes to prefetch. Internally, this is used to set `SstIterator::blocks_to_fetch`. The default `read_ahead_size` will be the size of a single block.
Users can also influence data block caching behavior using `ScanOptions::cache_blocks`. Internally, this will be used to set `SstIterator::cache_blocks`. The default behavior will be to use `cache_blocks=false`, so scans do not populate the cache with data blocks from large sequential reads unless explicitly requested. Point reads expose the same data-block-only control as `ReadOptions::cache_blocks`, defaulting to true. SST metadata such as indexes, filters, and stats is always cached when read, independently of scan and read options. Note that while an iterator is actively scanning from a given block, it will remain pinned in memory.
## Rejected Alternatives & Follow-Ups
[Section titled “Rejected Alternatives & Follow-Ups”](#rejected-alternatives--follow-ups)
**Improved isolation:** Transactional semantics proposed in [RFC-????](https://github.com/slatedb/slatedb/pull/260) will provide the foundation to support snapshot isolation or serializability of range scans. We anticipate future work to address the existing gaps.
**Space optimizations:**
* Rather than retaining references to in-memory tables for the full duration of the query, we can filter the set of records within the query’s range. As new records are returned, previous ones fall out of scope and may be cleaned up. This makes sense for bounded queries where the number of matching keys within the WAL or memtables may be low. For an unbounded query, we may prefer to hold a reference to the full table.
* While a range scan is in progress, some of its in-memory table references may get flushed to storage. It would be possible to substitute these references in order to allow the in-memory state to be cleaned up. This might make sense when the rate of updates is high enough that memory must be aggressively reclaimed.
**Resource Limits:** As mentioned above, it is possible to impose limits on active range scans in order to ensure that they do not exhaust memory or create unnecessary garbage accumulation.
**Persistent Queries** It may be useful in some use cases to persist query state in order to allow a process to resume after a migration or restart. This may require exposing the checkpoint that the query results were fetched from through the `DbIterator`.
## Updates
[Section titled “Updates”](#updates)
# Merge Operator
Status: Accepted
Authors:
* [David Moravek](https://github.com/dmvk)
References:
*
*
*
*
*
*
*
## Motivation
[Section titled “Motivation”](#motivation)
To utilize SlateDB as a fully-featured state backend for a stream processor, it must support three key types of state:
1. **Value**: Latest value or aggregating state.
2. **List / Bag / Buffering**: State that stores multiple items for later processing.
3. **Map**: Key-value state for fine-grained data.
Examples from the stream processing ecosystem:
* [Apache Flink](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/state/#using-keyed-state)
* [Apache Kafka Streams](https://kafka.apache.org/25/javadoc/org/apache/kafka/streams/state/KeyValueStore.html)
* [Apache Beam](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/state/State.html)
State-of-the-art implementations built on RocksDB often leverage the [Merge Operator](https://github.com/facebook/rocksdb/wiki/Merge-Operator) to efficiently manage types 1 and 2. A prime example is list state, where elements are appended continuously until triggered by an event like a window firing. While this method may not always be the absolute optimal solution, it serves as a solid foundation for future iterations and enhancements.
## Goals
[Section titled “Goals”](#goals)
The primary goal is to introduce a new user-facing record type (**Merge Operand**) and a **Merge Operator** interface to SlateDB, allowing applications to bypass the traditional read/modify/update cycle in performance-critical situations where computation can be expressed using an [associative](https://github.com/facebook/rocksdb/wiki/Merge-Operator#associativity-vs-non-associativity) operator.
**Supporting non-associative operators is not a goal of this RFC**. The assumption is that associative operators can cover the majority of use cases while keeping the user-facing API simple and significantly reducing the scope. [Badger](https://github.com/dgraph-io/badger/blob/v4.4.0/merge.go) has adopted the same limitation.
## Proposed Solution
[Section titled “Proposed Solution”](#proposed-solution)
We will introduce a new record type (**Merge Operand**) and a **Merge Operator** interface to SlateDB, allowing applications to bypass the traditional read/modify/update cycle in performance-critical situations where computation can be expressed using an associative operator.
This will allows users to express partial updates to an accumulator using an associative operator, which will be merged during reads/compactions to produce the final result.
This plays quite nice with existing record types, as merge operands can be interleaved with values and tombstones, and will be correctly merged during reads/compactions. Both tombstone and value act as a barrier that clears any previous merge history for the same key.
This approach is similar to how RocksDB and Badger handle merge operations. We’re not inventing anything new here, but rather combining ideas from these projects to fit our use case. The main innovation that is not present in either RocksDB or Badger is the handling of TTLs.
## Public API
[Section titled “Public API”](#public-api)
We will introduce two new methods for writing Merge Operands into the database. They follow the same structure as existing put methods.
```rust
impl Db {
pub async fn merge(
&self,
key: &[u8],
value: &[u8]
) -> Result<(), SlateDBError> {
...
}
pub async fn merge_with_options(
&self,
key: &[u8],
value: &[u8],
merge_opts: &MergeOptions,
write_opts: &WriteOptions
) -> Result<(), SlateDBError> {
...
}
}
impl WriteBatch {
pub fn merge(
&self,
key: &[u8],
value: &[u8]
) -> Result<(), SlateDBError> {
...
}
pub fn merge_with_options(
&self,
key: &[u8],
value: &[u8],
merge_opts: &MergeOptions
) -> Result<(), SlateDBError> {
...
}
}
#[derive(Clone, Default)]
pub struct MergeOptions {
/// Merges with different TTLs will only be merged together during read but will be stored separately
/// otherwise to allow them to expire independently. This ensures proper TTL handling while still
/// allowing merge operations to work correctly.
pub ttl: Ttl,
}
```
### Merge Operator Interface
[Section titled “Merge Operator Interface”](#merge-operator-interface)
We also need to introduce a new trait that will allow users to implement custom merging logic.
```rust
trait MergeOperator {
pub fn merge(
&self,
key: &Bytes,
existing_value: Option,
value: Bytes
) -> Result;
pub fn merge_batch(
&self,
key: &Bytes,
existing_value: Option,
operands: &[Bytes],
) -> Result {
// Default implementation: pairwise merging (backward compatible)
let mut result = existing_value;
for operand in operands {
result = Some(self.merge(key, result, operand.clone())?);
}
result.ok_or(MergeOperatorError::EmptyBatch)
}
}
```
**Error Handling**: The `MergeOperatorError::EmptyBatch` error is returned by the default `merge_batch` implementation when called with no `existing_value` and an empty `operands` slice. SlateDB’s merge iterator prevents this by returning early when both conditions are met (no base value or tombstone with no operands). However, custom implementations of `merge_batch` should handle this edge case if they don’t follow the default pairwise merging pattern.
Initially, the implementation is limited to a single optional merge operator per database. The user must ensure that both the compactor and writer use the same merge operator to guarantee correct results.
The merge operator can be implemented to route to different merge strategies based on key prefixes. Here’s an example:
```rust
struct MyMergeOperator;
impl MergeOperator for MyMergeOperator {
fn merge(
&self,
key: &Bytes,
existing_value: Option,
value: Bytes
) -> Result {
if key.starts_with(b"list:") {
// concat values
match existing_value {
Some(existing) => {
let mut result = existing.to_vec();
result.extend_from_slice(&value);
Ok(Bytes::from(result))
}
None => Ok(value)
}
} else if key.starts_with(b"counter:") {
// add values
let existing_num = existing_value
.map(|v| u64::from_le_bytes(v.as_ref().try_into().unwrap()))
.unwrap_or(0);
let new_num = u64::from_le_bytes(value.as_ref().try_into().unwrap());
Ok(Bytes::copy_from_slice(&(existing_num + new_num).to_le_bytes()))
} else {
Err(...)
}
}
// Optional: Override merge_batch for better performance
fn merge_batch(
&self,
key: &Bytes,
existing_value: Option,
operands: &[Bytes],
) -> Result {
if key.starts_with(b"counter:") {
// For counters, we can sum all operands at once (O(1) instead of O(N))
let sum = existing_value
.map(|v| u64::from_le_bytes(v.as_ref().try_into().unwrap()))
.unwrap_or(0)
+ operands.iter()
.map(|b| u64::from_le_bytes(b.as_ref().try_into().unwrap()))
.sum::();
Ok(Bytes::copy_from_slice(&sum.to_le_bytes()))
} else {
// For other prefixes, use default pairwise merging
let mut result = existing_value;
for operand in operands {
result = Some(self.merge(key, result, operand.clone())?);
}
result.ok_or(MergeOperatorError::EmptyBatch)
}
}
}
```
It’s **user’s responsibility to ensure** that the merge operator is correctly implemented and **the same instance of the operator is used across all components** (compactor, writer, garbage collector).
#### Performance Optimization: Overriding `merge_batch`
[Section titled “Performance Optimization: Overriding merge\_batch”](#performance-optimization-overriding-merge_batch)
While the default `merge_batch` implementation provides backward compatibility by calling `merge` pairwise, users may optionally override `merge_batch` to improve performance and reduce intermediate memory allocations. For example, a counter merge operator can sum all operands in a single pass instead of performing N-1 merge operations, avoiding the creation of N-1 intermediate `Bytes` objects. This optimization is particularly beneficial when processing large batches of operands, as it can reduce both function call overhead and memory allocations for certain use cases.
### Merge Operator Configuration
[Section titled “Merge Operator Configuration”](#merge-operator-configuration)
User can provide their own merge operator implementation via `DbOptions`. If not provided, the database will not support merge operations.
```rust
impl DbOptions {
#[serde(skip)]
/// The merge operator to use for the database. If not set, the database will not support merge operations.
pub merge_operator: Option>,
}
```
### Extending TTL support
[Section titled “Extending TTL support”](#extending-ttl-support)
The last public API change is extending the `Ttl` enum to support a new `ExpireAtMillis(timestamp_millis)` variant. This allows users to set a specific expiration time for a key, expressed as milliseconds since the Unix epoch, which overrides the default TTL behavior.
```rust
pub enum Ttl {
...
ExpireAtMillis(i64),
}
```
See [TTL Handling with Merge Operations](#ttl-handling-with-merge-operations) for more details.
## Implementation Details
[Section titled “Implementation Details”](#implementation-details)
### Internal Data Structures
[Section titled “Internal Data Structures”](#internal-data-structures)
We need to introduce an additional variant of `ValueDeletable` that denotes a **Merge Operand**. As a result of this change, we can no longer represent the deletable as `Option` or `Option<&[u8]>`, because a non-empty state can represent either a value or a merge operand. This requires reusing the ValueDeletable data structure throughout the stack—all the way from the memtable to the row codec.
```rust
#[derive(Debug, Clone, PartialEq)]
pub enum ValueDeletable {
...
Merge(Bytes)
...
}
```
### WAL / Memtable
[Section titled “WAL / Memtable”](#wal--memtable)
The WAL and memtable share a common underlying data structure (`WritableKVTable`). To support merge operations, we need to extend this structure to handle both merge operations and value overrides:
* For merge operations, we need to maintain multiple entries per key in order to preserve the sequence of merge operands
* For put and delete operations, we continue to override any previous values for that key
The `WritableKVTable` must preserve the order of operations to enable proper reconstruction of the merge sequence during reads, while still allowing puts and deletes to clear the merge history. Maintaining this hybrid approach is critical for handling merge-related error cases, such as:
1. If merging operands fails (e.g. due to incompatible data formats), we need the full history to potentially retry with different merge strategies or fall back to the last known good value
2. If merge operands have different TTLs, we need to track the individual expiry times and may need to fall back to earlier values when operands expire
In both cases, maintaining the operation history allows us to handle these edge cases gracefully rather than losing data.
### Row Codec
[Section titled “Row Codec”](#row-codec)
We need to slightly adjust the binary format of the SST file and introduce a new flag to denote a merge operand. Note that the `Tombstone` and `Merge` flags are mutually exclusive.
Aside from this, `Merge` rows can follow the same structure as `Value` rows.
This change is backwards compatible.
### Block Format
[Section titled “Block Format”](#block-format)
The block format remains unchanged. It’s useful to highlight that merge operands are also stored in the bloom filters, so they have the same filtering guarantees as regular values.
### Read Path
[Section titled “Read Path”](#read-path)
Merges are expected to be read using the standard `get` API.
For each key, SlateDB maintains a history of operations, which is subject to compaction.
Let’s denote each operation as `OPi`. A key (`K`) that has experienced `n` changes looks like this logically (physically, the changes could be spread across the memtable and multiple SST files):
```plaintext
K: OP1 OP2 OP3 ... OPn
earliest --------------------> latest
```
Currently, during a read operation, SlateDB always observes the latest known state of the database. In other words, it only needs to look up the latest operation. If it’s a `Value`, it simply returns it, and if it’s a `Tombstone`, it returns `None`. All previous operations can be ignored and will be removed by compaction at some point.
```plaintext
K: OP1 OP2 OP3 ... OPn
^
|
GET
```
With the addition of a **Merge Operand**, it may be necessary to look backwards at previous values, up to the point where we encounter either a `Value` or a `Tombstone`. Everything beyond that point can be ignored.
```plaintext
K: OP1 OP2 OP3 OP4 OPn
Value Merge Merge Merge
^
|
GET
```
In the above example, `get` should essentially return something like:
```plaintext
Merge(OP2, Merge(OP3, Merge(OP4, OPn)))
```
### Write Path & Compactions
[Section titled “Write Path & Compactions”](#write-path--compactions)
During compaction, we continuously reduce the history by merging entries according to the rules described above. This process performs partial merges along the way, combining merge operands with their base values when possible. We also merge entries in the memtable as they arrive to minimize unnecessary history storage.
Compaction operates on a set of source SST files and writes to a destination SST file. The process reads all source SSTs sequentially and produces a single destination SST containing the merged results.
The table below shows how different combinations of previous and new values are handled during compaction:
| Previous | New | Result |
| --------- | --------- | --------- |
| tombstone | value | value |
| tombstone | merge | value |
| tombstone | tombstone | tombstone |
| value | merge | value |
| value | value | value |
| value | tombstone | tombstone |
| merge | merge | merge |
| merge | value | value |
| merge | tombstone | tombstone |
#### Snapshot Isolation
[Section titled “Snapshot Isolation”](#snapshot-isolation)
*This part assumes that [Transactions RFC](https://github.com/slatedb/slatedb/pull/260) is already in place.*
Compaction must preserve snapshot isolation to avoid affecting externally observable state. RocksDB provides a [proven solution to this problem](https://github.com/facebook/rocksdb/wiki/Merge-Operator-Implementation#compaction) that we can adopt.
The key insight is that merging must stop whenever we encounter a snapshot barrier, as this represents a point where a transaction may be reading from. Below is an adapted example from RocksDB’s documentation showing how this works in practice, where the `+` symbol represents a merge operation:
```plaintext
K: 0 +1 +2 +3 +4 +5 2 +1 +2
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
We show it step by step, as we scan from the newest operation to the oldest operation
K: 0 +1 +2 +3 +4 +5 2 (+1 +2)
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
A Merge operation consumes a previous Merge operation and produces a new Merge operation
(+1 +2) => Merge(1,2) => +3
K: 0 +1 +2 +3 +4 +5 2 +3
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
K: 0 +1 +2 +3 +4 +5 (2 +3)
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
A Merge operation consumes a previous Put operation and produces a new Put operation
(2 +3) => Merge(2, 3) => 5
K: 0 +1 +2 +3 +4 +5 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
A newly produced Put operation hides any previous operations up to the snapshot barrier
(+5 5) => 5
K: 0 +1 +2 (+3 +4) 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
(+3 +4) => Merge(3,4) => +7
K: 0 +1 +2 +7 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
A Merge operation cannot consume a previous snapshot barrier
(+2 +7) can not be combined
K: 0 (+1 +2) +7 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
(+1 +2) => Merge(1,2) => +3
K: 0 +3 +7 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
K: (0 +3) +7 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
(0 +3) => Merge(0,3) => 3
K: 3 +7 5
^ ^ ^
| | |
snapshot1 snapshot2 snapshot3
```
### TTL Handling with Merge Operations
[Section titled “TTL Handling with Merge Operations”](#ttl-handling-with-merge-operations)
#### Supported TTL Behaviors
[Section titled “Supported TTL Behaviors”](#supported-ttl-behaviors)
SlateDB supports two TTL approaches:
1. **Operation-Level TTL**
* Each operation (put/merge) has its own independent TTL, specified via:
* `Ttl::ExpireAfterMillis(duration_millis)`: Expires after the specified duration in milliseconds (internally this is implemented as `create_ts + duration_millis`)
* `Ttl::ExpireAtMillis(timestamp_millis)`: Expires at the specified Unix timestamp in milliseconds
* Enables per-element expiration in collections
2. **TTL Renewal (NOT SUPPORTED NATIVELY)**
* Refreshing TTL for entire key would require READ + MODIFY + UPDATE
* Use standard `PUT` API if this behavior is needed
#### TTL Storage and Expiration
[Section titled “TTL Storage and Expiration”](#ttl-storage-and-expiration)
When merging values with different TTLs, the merge operation only combines values during reads while keeping entries separate in storage, allowing independent expiration per TTL.
Unlike regular values which become tombstones upon expiration, expired merge entries are simply removed, enabling per-element expiration in collections.
Users can implement custom TTL patterns by consistently using either `ExpireAtMillis` or `ExpireAfterMillis` across operations.
### Ordering Guarantees
[Section titled “Ordering Guarantees”](#ordering-guarantees)
If there are multiple merge operands available for the same key, they will be ordered by their corresponding sequence numbers.
### Error Handling
[Section titled “Error Handling”](#error-handling)
Merge operations can fail for several reasons:
* The user-defined merge operator returns an error during execution
* No merge operator is configured for the database
These failures can occur in three different code paths:
1. **Write Path**
* During merging of entries into the WAL or memtable
* On failure: Unmerged operands are preserved in the WAL/memtable and retry occurs on next startup
2. **Read Path**
* When merging entries during a read operation
* On failure: Error is propagated to the user
3. **Compaction Path**
* When the user-defined merge operator fails during compaction
* On failure: Unmerged operands are preserved in the SST file and retry occurs on next compaction
## Performance and Benchmarks
[Section titled “Performance and Benchmarks”](#performance-and-benchmarks)
We plan to validate the merge operator implementation with the following benchmarks focused on key use cases:
### List Buffering Workload
[Section titled “List Buffering Workload”](#list-buffering-workload)
* Append N elements to a list using merge operations and reading the final result vs read-modify-write
* Compare performance for different value sizes
* Expectation: Merge operations should show significant performance improvements over read-modify-write
### Aggregation Workload
[Section titled “Aggregation Workload”](#aggregation-workload)
* Increment counters using merge operations vs read-modify-write
* Compare performance for different value sizes
* Expectation: Merge operations should be faster than read-modify-write against cached records and should show significant improvements over read-modify-write for uncached records.
The benchmarks will be implemented and maintained as part of the existing benchmark suite to validate the merge operator performance characteristics. No benchmark results are available yet as the feature is still under development.
We expect the merge operator to show significant benefits for buffering use cases by eliminating read-modify-write cycles. The actual performance improvements will be quantified once the implementation is complete.
## Possible Follow-up Work
[Section titled “Possible Follow-up Work”](#possible-follow-up-work)
### Multiple Merge Operators
[Section titled “Multiple Merge Operators”](#multiple-merge-operators)
The currently proposed implementation allows for a single merge operator per database. While this may seem limiting, the current interfaces allow implementing support for multiple operators in userspace by routing to different implementations based on key prefixes or other criteria. This approach aligns with RocksDB’s design philosophy.
If compelling use cases emerge that demonstrate clear benefits of first-class support for multiple merge operators, we can consider adding this capability in a future iteration. However, the current single-operator design provides a good balance of simplicity and flexibility for most use cases.
### Persisting GET results
[Section titled “Persisting GET results”](#persisting-get-results)
One possible optimization is to introduce a new read option for persisting the result of a GET operation when it was merged from multiple operands. This would allow for faster subsequent GETs by avoiding the need to recompute the merge. The result could be stored as a regular value in the memtable or an SST file. Although this optimization sounds appealing, it’s not clear whether the benefits outweigh the added complexity and storage overhead. For certain use cases, such as buffering, it might have a negative performance impact.
### Read Options for faster termination
[Section titled “Read Options for faster termination”](#read-options-for-faster-termination)
It might be useful to have a read option that allows for early termination of merge operations. This could be beneficial for buffering use cases where the user wants to limit the number of operands that are merged together (effectively allowing partial iterations).
RocksDB provides an alternative approach by exposing `GetMergeOperands` which allows listing the unmerged operands directly.
# Error API
Status: Accepted
Authors:
* [Jason Gustafson](https://github.com/hachikuji)
## Motivation
[Section titled “Motivation”](#motivation)
Errors exposed through a public API are part of the public API. Users depend on clearly documented errors to drive failure handling logic and we cannot change them without considering how it affects compatibility. It is important to have common agreement among contributors about the guidelines to expose new public errors.
## Goals
[Section titled “Goals”](#goals)
The goal of this RFC is to establish a set of principles for exposing public errors and a framework for maintaining them.
## Principles
[Section titled “Principles”](#principles)
* **Errors should not expose implementation.** The intent is to reserve enough space for the implementation to change without affecting the public API.
* **Errors should be prescriptive.** Ideally it should be clear to the user what they need to do to resolve an issue. For example, we should indicate whether a failure may be transient and an operation can be retried. It may not always be possible to have clear guidance for unexpected errors, but ideally we can at least state whether the error is fatal.
* **Prefer coarse error types.** A useful criteria to decide whether a new error type is needed is whether the user would handle it differently than one of the existing types.
* **Use rich error messages.** Pack as much information as is needed to diagnose a problem into the error message, which will not be considered part of the public API.
## Public API
[Section titled “Public API”](#public-api)
All public APIs will document the complete set of errors that may be returned.
All public errors are exposed through the `slatedb::Error` struct. This struct includes an enum describing the kind of error that it represents, `slatedb::ErrorKind`. New error representations can be added to `slatedb::ErrorKind` only when the current representations don’t cover a general problem in the database.
Each exposed error type will expose documentation which contains a general description of the error and contains any prescriptive guidance (such as indicating whether a failed operation can be retried or is fatal).
Each error type will also expose a custom message field which is used to convey details about the specific instance of the error that was encountered. Unlike the error type, the message is not part of the public API and can be changed without notice. The intent is to pack as much detail into the message for someone to understand the root cause, and if possible, what they should do to resolve it.
New errors can be added by updating this RFC. Existing errors can be removed through semantic versioning. Typically, the need to remove an error suggests that some part of the internal implementation has been inadvertently leaked, so such cases should be rare if the exposed errors follow the principles above.
These are the currently supported `slatedb::ErrorKind`:
```rust
/// Represents the reason that a database instance has been closed.
#[derive(Debug, Clone, Copy)]
pub enum CloseReason {
/// The database has been shutdown cleanly.
Clean,
/// The current instance has been fenced and is no longer usable.
Fenced,
/// One or more background tasks panicked.
Panic,
}
/// Represents the kind of public errors that can be returned to the user.
///
/// These are less specific and more prescriptive. Application developers or operators must
/// decide how to proceed. Adding new [ErrorKind]s requires an RFC, and should happen very
/// infrequently.
#[non_exhaustive]
#[derive(Debug, Clone, Copy)]
pub enum ErrorKind {
/// A transaction conflict occurred. The transaction must be retried or dropped.
Transaction,
/// The database has been shutdown. The instance is no longer usable. The user must
/// create a new instance to continue using the database.
Closed(CloseReason),
/// A storage or network service is unavailable. The user must retry or drop the
/// operation.
Unavailable,
/// User attempted an invalid request. This might be:
///
/// - An invalid configuration on initialization
/// - An invalid argument to a method
/// - An invalid method call
/// - A user-supplied plugin such as the compaction schedule supplier or logical clock
/// failed.
///
/// The user must correct the code, configuration, or argument and retry the operation.
Invalid,
/// Persisted data is in an unexpected state. This could be caused by:
///
/// - Temporary or permanent machine or object storage corruption
/// - Incompatible file format versions between clients
/// - An eventual consistency issue in object storage
///
/// The user must fix the data, use a compatible client version, retry the operation,
/// or drop the operation.
Data,
/// An unexpected internal error occurred. Users should not expect to see this error.
/// Please [open a Github issue](https://github.com/slatedb/slatedb/issues/new?template=bug_report.md&title=Internal+error+returned)
/// if you receive this error.
Internal,
}
```
### Constructing public errors
[Section titled “Constructing public errors”](#constructing-public-errors)
The `slatedb::Error` struct includes functions to construct public errors. Each one of those functions is identified by the name variant that they represent. `slatedb::Error::system` creates `slatedb::ErrorKind::System` errors and so on.
If you want to propagate another error into a public error, you have to use the `with_source` function after constructing an error. For example if you want to propagate an `std::io::IOError`:
```rust
let error = slatedb::Error::unavailable("failed connection".to_string()).with_source(my_ioerror)
```
In general, you should not construct public errors directly. It’s recommended to use the internal `slatedb::SlateDBError` which is then transformed into a public error. This gives you more flexibility to construct your errors.
## Internal errors vs External errors
[Section titled “Internal errors vs External errors”](#internal-errors-vs-external-errors)
Internal errors can continue being described through `slatedb::SlateDBError`. This error type can be transformed into an `slatedb::Error` through the `From` trait.
### Internal errors message format
[Section titled “Internal errors message format”](#internal-errors-message-format)
`SlateDbError` uses [thiserror](https://docs.rs/thiserror) to define internal errors. This crate gives you flexibility in the formatting of error messages, as well as other errors that are bubbling up from the database.
We follow some conventions to generate message to ensure they are all consistent. This is the list of conventions:
* Try to be specific when you create an internal error. Instead of creating an error that can take multiple messages, create different errors that can be formatted following these rules.
* Write messages in lowercase: `read channel error` instead of `Read Channel Error`.
* If an error message includes variables, they’re added at the end of the message in KEY=`VALUE` format. `unexpected range key. key=\`foo\`, range=\`\[..]\``instead of`unexpected key \`foo\` in range \`\[..]“.
* Errors that propagate other errors don’t include the propagated errors in the message. They will be included automatically when the internal error is translated into a public error. See the example below:
```rust
// DO THIS
enum SlateDBError {
#[error("io error")]
IOError(#[from] std::io::Error)
}
// **DO NOT** DO THIS
enum SlateDBError {
#[error("io error: #{0}")]
IOError(#[from] std::io::Error)
}
```
# Synchronous Commit & Durability
Status: Accepted
Authors:
* [Li Yazhou](https://github.com/flaneur2020)
## Background
[Section titled “Background”](#background)
The discussion of commit semantics and durability began in the comments of . As the discussion evolved, it became clear that this is a complex topic. The semantics of commit and durability involve many intricate details, with subtle but important differences between various approaches.
We’re creating this RFC to facilitate a thorough discussion of these concepts in the code review board.
## Goals
[Section titled “Goals”](#goals)
This RFC aims to:
1. Define clear synchronous commit semantics and durability guarantees that SlateDB users can rely on with confidence.
2. Create a well-defined API allowing users to specify their commit semantics and durability requirements.
3. Account for tiered WAL in the design.
4. Plan and organize the necessary code changes to implement these features.
As previously discussed in meetings and comments, we intend to implement these changes in an additive way that preserves all existing capabilities.
## References
[Section titled “References”](#references)
* [Understanding synchronous\_commit in PostgreSQL](https://medium.com/@mihir20/understanding-synchronous-commit-in-postgresql-54cb5609a221)
* [RocksDB: WAL Performance](https://github.com/facebook/rocksdb/wiki/WAL-Performance)
## Other Systems
[Section titled “Other Systems”](#other-systems)
Let’s examine how other systems handle synchronous commits and durability guarantees before diving into the design. We’ll focus on these key aspects:
1. The API for users to specify their commit semantics & durability requirements
2. Use cases & trade-offs, and the default settings
3. Error handling
### PostgreSQL
[Section titled “PostgreSQL”](#postgresql)
PostgreSQL provides a flexible setting called `synchronous_commit` that controls transaction durability and commit behavior. It offers several levels:
* `off`: Commits complete immediately after the transaction finishes, without waiting for the WAL to be written to disk. This means data loss is possible if a crash occurs.
* `local`: Commits wait for the WAL to be written and flushed to the local disk before returning.
* `on` (default): Commits wait for the WAL to be written and flushed locally, plus wait for at least one standby server to apply the WAL if synchronous replication is configured.
* `remote_write`: Commits wait for the WAL to be written locally and replicated to standby servers, then wait for standbys to flush to their file systems.
* `remote_apply`: Commits wait for the WAL to be written and flushed locally, then wait for standbys to fully apply the WAL, this ensures the data consistent between primary and standby.
The article “Understanding synchronous\_commit in PostgreSQL” includes a helpful diagram showing how the `on` and `off` settings work (converted into a mermaid sequence diagram by @criccomini with ChatGPT):
```
sequenceDiagram
participant Client
participant ClientBackend
participant SharedBuffer
participant WALBuffer
participant WALWriter
participant Disk
Client ->> ClientBackend: Execute DML
ClientBackend ->> SharedBuffer: Write Data
ClientBackend ->> WALBuffer: Write WAL
alt synchronous_commit is off
ClientBackend -->> Client: Return Success (if off)
WALBuffer ->> WALWriter: Asynchronous WAL write
WALWriter ->> Disk: Asynchronous Write
else synchronous_commit is on
WALBuffer ->> WALWriter: Wait for WALWriter
WALWriter ->> Disk: Flush WAL
ClientBackend -->> Client: Return Success (if on)
end
```
The key distinction between `on` and `off` is that `on` ensures the WAL is written and flushed to local storage before proceeding. This highlights a central theme throughout this RFC: synchronous commit and durability fundamentally revolve around WAL handling.
Let’s summarize the use cases and trade-offs for different `synchronous_commit` levels:
1. For financial systems where data is highly sensitive and data loss is unacceptable, `on`, `remote_write`, or `remote_apply` should be used.
2. For mission-critical systems that cannot tolerate data inconsistency between primary and standby servers after a failover, `remote_apply` should be used to guarantee data consistency across servers.
3. For workloads like logging or stream processing where some data loss is acceptable and performance is paramount, `off` or `local` can be used to optimize throughput.
### RocksDB
[Section titled “RocksDB”](#rocksdb)
RocksDB describes synchronous commit in their documentation as follows:
> #### Non-Sync Mode
>
> [Section titled “Non-Sync Mode”](#non-sync-mode)
>
> When WriteOptions.sync = false (the default), WAL writes are not synchronized to disk. Unless the operating system thinks it must flush the data (e.g. too many dirty pages), users don’t need to wait for any I/O for write.
>
> Users who want to even reduce the CPU of latency introduced by writing to OS page cache, can choose Options.manual\_wal\_flush = true. With this option, WAL writes are not even flushed to the file system page cache, but kept in RocksDB. Users need to call DB::FlushWAL() to have buffered entries go to the file system.
>
> Users can call DB::SyncWAL() to force fsync WAL files. The function will not block writes being executed in other threads.
>
> In this mode, the WAL write is not crash safe.
>
> #### Sync Mode
>
> [Section titled “Sync Mode”](#sync-mode)
>
> When WriteOptions.sync = true, the WAL file is fsync’ed before returning to the user.
>
> #### Group Commit
>
> [Section titled “Group Commit”](#group-commit)
>
> As most other systems relying on logs, RocksDB supports group commit to improve WAL writing throughput, as well as write amplification. RocksDB’s group commit is implemented in a naive way: when different threads are writing to the same DB at the same time, all outstanding writes that qualify to be combined will be combined together and write to WAL once, with one fsync. In this way, more writes can be completed by the same number of I/Os.
>
> Writes with different write options might disqualify themselves to be combined. The maximum group size is 1MB. RocksDB won’t try to increase batch size by proactive delaying the writes.
Like PostgreSQL, RocksDB provides a `sync` option to control commit behavior and durability guarantees. When `sync = true`, a write is not considered committed until the data is `fsync()`ed to storage.
When `sync = false`, a write is considered committed immediately after the transaction completes, without waiting for the WAL to be written. While the WAL is still buffered in the kernel’s page cache, data loss can occur if a crash happens.
Unlike PostgreSQL’s `synchronous_commit` which offers multiple levels, RocksDB only provides a simple boolean option. This is because RocksDB is an embedded database and doesn’t have the primary/standby architecture that PostgreSQL has.
To optimize synchronous commit performance, RocksDB implements Group Commit, which is a common pattern in WAL-based systems. This mechanism batches multiple writes together into a single, larger WAL write and flush operation, which significantly improves I/O throughput compared to multiple small writes.
(SlateDB implemented a similar Group Commit mechanism through its Commit Pipeline, where multiple writes with `await_durable: true` will be batched into a single WAL write after `flush.interval` seconds or when the WAL buffer is full.)
It’s worth noting that RocksDB defaults to `sync = false`, meaning WAL writes are not crash-safe by default.
This default is likely a performance trade-off. In many distributed systems (RocksDB’s primary use case), some data loss on individual nodes is acceptable without compromising overall system durability. Examples include Raft clusters, distributed key-value stores, and stream processing state stores. For these use cases, enabling `sync: false` or `manual_wal_flush: true` can be beneficial for performance.
RocksDB allows mixing writes with different sync settings. For example, if transaction A commits with `sync = false` and transaction B starts afterwards, transaction A’s writes will be visible to readers in transaction B. When transaction B commits with `sync = true`, both transactions’ writes are persisted. This ordering guarantee means that when a `sync = true` write commits, all previous writes are guaranteed to be persisted as well.
Another important consideration is WAL write failures. RocksDB handles these by retrying writes until determining whether the failure is temporary or permanent. In cases of permanent failure (like a full or corrupted disk), RocksDB marks the database state as fatal, rolls back the transaction, and switches the database instance to read-only mode.
## Synchronous Commit in Summary
[Section titled “Synchronous Commit in Summary”](#synchronous-commit-in-summary)
Based on the PostgreSQL and RocksDB references above, we can summarize the key semantics of Synchronous Commit:
1. A write is only considered committed once the WAL has been persisted to storage. Until then, the data remains invisible to readers.
2. If there is a permanent failure while persisting the WAL during a Synchronous Commit, the transaction rolls back. The database instance enters a fatal state and switches to read-only mode.
3. It’s possible to have multiple levels of Synchronous Commit, or even disable it, allowing users to balance performance and durability requirements.
4. Synchronous and Unsynchronous Commits can be interleaved in different transactions. A transaction using Synchronous Commit can read writes from transactions that used Unsynchronous Commit. When a Synchronous Commit persists, it also persists any previous Unsynchronous Commit writes in the WAL.
## Current Design in SlateDB
[Section titled “Current Design in SlateDB”](#current-design-in-slatedb)
This section is based on @criccomini’s comment in .
SlateDB currently does not provide an explicit notion of Synchronous Commit. However, it does provide a `DurabilityLevel` enum to control the durability guarantees on both read and write operations.
The `DurabilityLevel` enum is defined as follows:
```rust
enum DurabilityLevel {
Memory,
Local, // not implemented yet
Remote,
}
```
The `WriteOptions` struct contains an `await_durability: DurabilityLevel` option to control the waiting behavior for durability. If `await_durability` is set to `DurabilityLevel::Remote`, the write will wait for the WAL to be written into S3 before returning.
It’s important to note that SlateDB’s commit semantics differ from other systems’ Synchronous Commit. Regardless of what `DurabilityLevel` is set in the write operation, this write is considered visible to readers with `DurabilityLevel::Memory` immediately after the write is appended to the WAL, not necessarily flushed to storage.
This difference exists because SlateDB’s WAL serves not only for crash recovery but also for data reads. The read path first accesses the WAL, then MemTable, then L0 SST, and finally SSTs at deeper levels.
In traditional Synchronous Commit semantics, data is considered committed only after the write is persisted to WAL storage. Users can specify the durability level as `DurabilityLevel::Remote` for read calls to ensure only committed/persisted data is read.
SlateDB differs from both PostgreSQL and RocksDB in multiple ways:
1. Unlike PostgreSQL, it’s not a distributed system with Primary/Standby nodes.
2. Unlike RocksDB, it stores data in S3 rather than local disk, resulting in slower write operations and additional costs for API requests.
These differences lead to several key considerations:
1. Group commit is essential. By batching multiple writes together, we can reduce both API costs and improve performance compared to multiple small writes. However, even with Group Commit, writes to S3 will still be slower than local disk writes.
2. Writes will inherently take longer due to S3 latency. Given this reality, it makes sense to allow readers who can accept eventual consistency to access unpersisted and uncommitted data while waiting for writes to be durably committed.
3. Writing WAL to S3 has a higher risk of permanent failures due to network instability compared to local disk. This makes it critical to implement robust auto-recovery mechanisms for handling I/O failures. [1](#user-content-fn-1)
4. We should provide a way to control the durability level of read operations, so users can choose to read only persisted data.
These unique characteristics of SlateDB must be carefully considered as we design our durability and commit semantics.
## Possible Improvements
[Section titled “Possible Improvements”](#possible-improvements)
Synchronous Commit is a critical feature for mission-critical systems. It guarantees full ACID compliance by ensuring writes remain invisible until they are committed to durable storage. It also allows for different levels of durability guarantees to balance various use cases and trade-offs.
However, when comparing SlateDB’s current model with PostgreSQL and RocksDB’s Synchronous Commit implementations, there are some challenges in replicating the same semantics.
For example, in a transaction intended to be synchronously committed, the write should not be considered committed until the data is flushed to storage. But in SlateDB’s current model, the data becomes visible to readers accepting unpersisted data (using `DurabilityLevel::Memory`) as soon as the write is appended to the WAL - before it’s actually persisted to storage. This means uncommitted data can potentially be read before it’s durably committed.
If users want to avoid reading uncommitted data, they can use `DurabilityLevel::Remote` to ensure they only read persisted data. However, this approach has drawbacks within transactions. If other writers make unpersisted writes (`DurabilityLevel::Memory`) to the same keys that the transaction is accessing, it will constantly encounter conflicts and rollbacks.
In short:
* We cannot guarantee Synchronous Commit semantics by setting writers to `DurabilityLevel::Remote` and readers to `DurabilityLevel::Memory`, since uncommitted data may be visible before it’s durably committed.
* Setting both writers and readers to `DurabilityLevel::Remote` also presents challenges, as transactions may frequently roll back due to conflicts with unpersisted writes made by other writers using `DurabilityLevel::Memory` on the same keys.
## Proposal
[Section titled “Proposal”](#proposal)
### Read Committed Data by Default
[Section titled “Read Committed Data by Default”](#read-committed-data-by-default)
This proposal aims to provide users with true Synchronous Commit semantics while preserving all capabilities of the current model.
It’s important to note that “Commit” and “Durability” are distinct concepts that aren’t necessarily coupled. Data can be considered “Committed” even if it hasn’t been flushed to persistent storage - it may exist only in memory. While such data isn’t persisted, it can still be treated as safely committed from a transactional perspective.
Later transactions can safely depend on these “Unpersisted” but “Committed” data without worrying about conflicts, since they represent the latest committed state of the data. This allows for consistent transaction semantics even when some committed data hasn’t yet been persisted to storage.
“Committed” data won’t contain data that is still in the process of being committed (which is possible in reads with `DurabilityLevel::Memory` in the current model).
By allowing users to read “Committed” data by default, we can address the challenges outlined earlier. Reading committed data, regardless of persistence status, enables proper transaction semantics while still allowing for performance optimizations through writes with lower durability requirements. This provides a cleaner separation between transaction consistency and durability guarantees.
Let’s assume we have a sequence of writes like this:
| seq | operation | marks |
| --- | ------------------------------------------------------------------------------------ | ---------------------------- |
| 100 | WRITE key001 = “value001” with (durability: Memory) | |
| 101 | WRITE key002 = “value002” with (durability: Memory) | |
| 102 | WRITE key003 = “value003” with (durability: Memory) | |
| 103 | WRITE key004 = “value004” with (durability: Remote) | <— last remote persisted seq |
| 104 | WRITE key005 = “value005” with (durability: Memory) | |
| 105 | WRITE key006 = “value006” with (durability: Memory) | <— last committed seq |
| 106 | WRITE key007 = “value007” with (durability: Remote), but still haven’t persisted yet | <— last committing seq |
While “Committed” is distinct from the “DurabilityLevel” concept, both the “Committed” and “Persisted” positions can be tracked using separate watermarks in the commit history of sequence numbers.
Reading “Committed” data should be considered the default behavior for read operations, as committed data is safe data.
In some cases, users may want to read only “Persisted” data, and they can use the `durability_filter` option to achieve this goal:
```rust
let opts = ReadOptions::new().with_durability_filter(DurabilityLevel::Local);
db.get(key, opts).await?;
```
Using this durability filter option, the read operation will only retrieve data that has been persisted to storage. This will make sure the data it read is at least durable at `DurabilityLevel::Local`, but it might not be the most up-to-date.
By default, no matter what `DurabilityLevel` is set in the `durability_filter` option, the read operation will always read the Committed data. But at some use cases like disk caching, users may not care about the commitness of the data, but only care about the data is durable at least at `DurabilityLevel::Local`.
If the writer write with `SyncLevel::Remote`, the data will not be considered as committed until it’s persisted to `Remote`. If the user want to read the data as soon as it’s persisted to `Local`, they can use an additional option `dirty: true` to make this possible:
```rust
let opts = ReadOptions::new()
.with_durability_filter(DurabilityLevel::Local)
.with_dirty(true);
db.get(key, opts).await?;
```
### Sync Commit
[Section titled “Sync Commit”](#sync-commit)
To implement Synchronous Commit semantics, the write side do not need to change a lot: the operation returns once the WAL is persisted according to the specified durability level.
In other systems, this option is often named as `sync`, to emphasize the idea of synchronous commit, like:
```rust
let opts = WriteOptions::new().with_sync(DurabilityLevel::Remote);
db.write(key, value, opts).await?;
```
However, for the writes that not expected to durablely persisted, this can be thought of as **disabling** synchronous commit entirely rather than synchronizing to memory. The code sample may be like:
```rust
enum SyncLevel {
Off,
Local,
Remote,
}
let opts = WriteOptions::new().with_sync(SyncLevel::Off);
db.write(key, value, opts).await?;
```
The benefits of having a `SyncLevel` enum are:
1. The name “Synchronous Commit” directly describes the feature we aim to support.
2. The term “sync” is commonly used across database systems and is well understood, familiar to users.
3. The names are shorter, easier to type & read.
4. It’s accurate to describe as sync commit as “off” when it comes with memory durability.
At the writer side, if `sync` is enabled, the write operation will wait until the data is persisted to storage according to the specified sync level.
If `sync` is set to `SyncLevel::Off`, the write operation will not await it self to be persisted remotely.
But there’s one worth noting: the commits are strictly ordered. The later write will wait for the previous write to be committed to commit itself. As a result, the write operation with `SyncLevel::Off` does NOT means the write will become visible immediately.
Give an example:
* thread A: seq 100: write keyXXX with `sync: Remote`
* thread B: seq 101: write keyYYY with `sync: Off`
The write operation B will NOT return immediately, but it will be queued in the WAL buffer, and await the previous write operation A to be committed, then it will be applied to MemTable and visible to readers, and finally return.
`commit()` always ensures “read your writes consistency”, as a result, it’s still a blocking operation even with `sync: Off`. If we want to have a no-wait write, we can consider add something like `commit_with_callback(cb: Callback)` interface, like [badger](https://pkg.go.dev/github.com/dgraph-io/badger/v4#Txn.CommitWith) did. This async commit interface is out of scope of this RFC, we can discuss it in a separate RFC or issue if needed.
### Difference between “await\_durable” and “sync”
[Section titled “Difference between “await\_durable” and “sync””](#difference-between-await_durable-and-sync)
`await_durable` and `sync` appear to have similar behaviors - on the write side, both wait until data reaches a persistent state before returning.
However, there are subtle semantic differences between them:
1. `sync` determines when data enters the committed state and returns after data is committed.
2. `await_durable` only waits for the data’s durability state.
For example, when a write has `Sync::Off` set, it is considered to enter the Committed state immediately, this write will become visible to readers with `LastCommitted` immediately.
However, at some point this data will still be eventually flushed to storage if the DB keeps running. If you want to subscribe or wait for when this `sync:Off` write becomes durably persisted, you can still use `await_durable` to achieve this goal.
Also, when using `sync` commit, it’s possible to queue the later Non-Sync writes to be queued in the WAL buffer, delaying them later to be applied to MemTable and visible to readers.
Sometimes, user might hope to always allow the writes to be visible to readers immediately, while still waiting for the data to be durably persisted. `await_durable` and `sync` can be used together to achieve this goal:
```rust
let opts = WriteOptions::new()
.with_sync(SyncLevel::Off)
.with_await_durable(DurabilityLevel::Remote) // on the writer side, i still await this until it's persisted in remote
db.put_with_options("key", "val", opts)
```
When WAL is disabled (no-WAL mode), the `sync` option has no effect since there is no WAL to sync. In this case, users should use `await_durable` to ensure their writes are persisted durably.
While `SyncLevel` and `DurabilityLevel` may appear similar, they serve distinct purposes. `SyncLevel` controls when writes become visible to readers by determining the commit semantics, while `DurabilityLevel` focuses on the durability of the write.
## Implementation
[Section titled “Implementation”](#implementation)
The key change is to separate “Append to WAL” and “Commit to make it visible” into two different steps.
In the current design, whenever a write operation is called, it’ll append the data to the WAL, the data is becoming visible to readers with `DurabilityLevel::Memory` immediately, as the readers will read the data from the WAL in front of MemTable. And when the WAL buffer reaches the `flush.interval` or `flush.size`, it’ll be flushed to storage, and apply to MemTable.
The sequence diagram is like this:
```
sequenceDiagram
participant WriteBatch
participant WAL
participant MemTable
participant Flusher
WriteBatch->>WAL: append write op to current WAL
Note over WAL: become visible to readers with DurabilityLevel::Memory
WAL->>WAL: maybe trigger a flush
Note over WAL: freeze the current WAL, and flush this WAL
WAL->>Flusher: Flush WAL
Flusher-->>WAL: ACK FlushWAL message
WAL->>MemTable: Merge WAL to MemTable
```
In this proposal, “Commit to make it visible” effectively means “applying the changes to the MemTable”, whenever the change is applied to MemTable, it’s considered as committed.
The changes to the read path are minimal. For reads with different durability levels, we can simply use different sequence numbers as watermarks to determine visibility.
On the write side, after the data is appended to the WAL, it’ll be applied to MemTable as soon as possible to make the change to be visible to readers.
The sequence diagram is like this:
```
sequenceDiagram
participant WriteBatch
participant WAL
participant MemTable
participant Flusher
WriteBatch->>WAL: append write op to current WAL
WAL->>WAL: Maybe trigger a flush
Note over WAL: when it reaches flush.interval or flush.size
WAL->>Flusher: Flush WAL
Flusher-->>WAL: ack FlushWAL message, increment last wal flushed position
WAL->>MemTable: apply write op to MemTable
Note over MemTable: become visible to readers with ReadWaterMark::LastCommitted
alt sync is Remote
WriteBatch->>WAL: await last applied position
end
MemTable-->>WriteBatch: wake up waiters of last applied position
```
### Handling No WAL Write
[Section titled “Handling No WAL Write”](#handling-no-wal-write)
It’s not possible to have Sync Commit when WAL is disabled. So let’s not allow users to enable `sync` when WAL is disabled.
When `sync` is not possible to set, all the writes will be considered as Committed immediately, and become visible to readers in the MemTable.
Users can still use `await_durable` to wait for the write to be durably persisted on the writer side.
### Handling WAL Write Failures
[Section titled “Handling WAL Write Failures”](#handling-wal-write-failures)
It’s more likely to have WAL write failures on object storage like S3 than local disk.
We do not have to fail the write operation immediately when the WAL write fails. Instead, we can retry the write operation with a backoff.
However, it’s possible to have some failure on flushing WAL to object storage which last for several minutes or even longer, and without a local disk which might help to buffer the WAL during the failure.
In this case, we can still buffer the WAL in memory as long as possible. When the failure is resolved, we can resume the flush operation.
However, the memory is limited, and we can’t buffer the WAL in memory forever. We already have a threshold setting `max_unflushed_bytes` for this. When this threshold is reached, we can’t buffer the WAL anymore, and we have to mark the db into an read-only state.
When the WAL fails to be flushed to storage, the write operation with `SyncLevel::Remote` should be failed with an `IOError` (or a concrete error type), and considered as not committed. This write operation should be invisible to all of the readers.
### Possible Code Changes
[Section titled “Possible Code Changes”](#possible-code-changes)
As above described, we need to tackle with several details to implement this proposal, like:
1. The synchronization of flushing the WAL to storage.
2. Notify the writers when the WAL is successfully flushed to storage.
3. Retry the flush operation when it fails with backoff, and finally mark the db into an read-only state if it’s permanent failure.
4. Flush the memory buffered WAL to storage in order.
5. Support tiered WAL storage like local disk and S3.
6. Handle the no-WAL mode.
Given the complexity of these changes, implementing everything in a single PR would be challenging. It’s difficult to estimate the implementation effort required or identify all the code that will need to change in this RFC. A better approach would be to break this down into several smaller PRs, starting with some small refactoring to make the subsequent changes easier to be implemented.
One possible code change would be introducing a `WALManager` struct to centralize management of the WAL buffer and flush operations, helping to encapsulate the complexity in a single place.
For inspiration, we can also consider to reference how Badger structures their code. In Badger’s implementation, they embed the WAL buffer inside the memtable struct, as shown here:
```go
// memTable structure stores a skiplist and a corresponding WAL. Writes to memTable are written
// both to the WAL and the skiplist. On a crash, the WAL is replayed to bring the skiplist back to
// its pre-crash form.
type memTable struct {
sl *skl.Skiplist
wal *logFile
maxVersion uint64
opt Options
buf *bytes.Buffer
}
```
It makes sense to put the manager of the WAL buffer inside the memtable struct, as WAL is closely related to the memtable, it actually make a good encapsulation: put the complexities behind some simple `put()` / `get()` interface.
However, it’s a different codebase, it would be better to keep code structure changes minimal with each iteration. It’ll make more sense to have several small PoC PRs. Introducing a `WALManager` might be a good first step to encapsulate the WAL buffer, flushing, and synchronous commit functionality. This would allow us to have more detailed discussions about code structure as we develop these PoCs.
## Updates
[Section titled “Updates”](#updates)
* 2025-02-20: added the comparison between `sync` and `await_durable`
* 2025-03-12: revise the api with `with_durability_filter` and `with_dirty`
## Footnotes
[Section titled “Footnotes”](#footnote-label)
1. The discussion in [RFC for Handling Data Corruption](https://github.com/slatedb/slatedb/pull/441) may also be related to this topic. [↩](#user-content-fnref-1)
# Separate Object Store for WAL
Status: Approved
Authors:
* [Sergei Sitnikov](https://github.com/taburet)
## Motivation
[Section titled “Motivation”](#motivation)
Sequential writes are a common pattern. Combined with `await_durable` set to `true`, which is the default, WAL performance is the main limiting factor for the sequential writes throughput. Even if `flush_interval` is set to an extremely low value, it can’t overcome the inherent latency of the underlying object store.
The obvious solution is switching to a lower latency object store; for instance, from S3 to S3 Express One Zone. But lower latency stores usually come with a cost, they might be significantly more expensive or might be limited in features. Using such stores for the main store containing the entire database might be undesirable.
## Goals
[Section titled “Goals”](#goals)
* Support an optional separate object store dedicated specifically for WAL.
* Provide a user-facing configuration mechanism.
* Avoid significant refactorings.
## Public API
[Section titled “Public API”](#public-api)
If user wants to use a separate WAL object store, the store instance should be provided while constructing a `Db`:
```rust
impl Db {
// ...
/// ...
///
/// ## Arguments
/// ...
/// - `wal_object_store`: an optional [object store](ObjectStore) instance
/// dedicated specifically for WAL.
///
/// If set to `None`, the passed `object_store` will be used for WAL storage.
///
/// NOTE: WAL durability and availability properties depend on the properties
/// of the underlying object store. Make sure the configured object
/// store is durable and available enough for your use case.
///
/// ...
pub async fn open_with_opts>(
path: P,
options: DbOptions,
main_object_store: Arc,
wal_object_store: Option>,
) -> Result {
// ...
}
// ...
}
```
Once RFC-10 “Settings design” is implemented, the WAL object store configuration should be moved to the builder proposed in RFC-10. With this more scalable approach the parameter set of `open_with_opts` will be nice and clean.
## Implementation
[Section titled “Implementation”](#implementation)
The configured WAL object store is propagated down to `TableStore` instance which internally will select the appropriate object store for each operation based solely on the `SsTableId` involved in the operation. The object path structure will remain unchanged for both stores.
There is already `transactional_wal_store` field in `TableStore`. The naming is misleading because apparently this store is used not only for WAL-related operations. The field should be renamed, the resulting field layout will look like this:
```rust
pub struct TableStore {
// ...
/// The main object store.
main_object_store: Arc,
/// The dedicated WAL object store, if any.
wal_object_store: Option>,
/// The transactional store wrapper of the main object store.
main_transactional_store: Arc,
/// The transactional store wrapper of the dedicated WAL object store, if any.
wal_transactional_store: Option>
}
```
The object store selection procedure will look like this:
```rust
fn object_store_for(&self, id: &SsTableId) -> Arc {
match id {
SsTableId::Wal(..) =>
if let Some(wal_object_store) = &self.wal_object_store {
wal_object_store.clone()
} else {
self.main_object_store.clone()
}
SsTableId::Compacted(..) => self.main_object_store.clone(),
}
}
```
The transactional object store selection logic would look about the same.
### Block and Object Cache
[Section titled “Block and Object Cache”](#block-and-object-cache)
WAL requires no caching support since it is read only once at startup. All operations on the dedicated WAL object store should be direct and bypass all the caches.
### Cloning Support
[Section titled “Cloning Support”](#cloning-support)
Public API methods like `admin::create_clone` could just accept an explicitly passed WAL object store instance, but this is undesirable because that introduces another point for a potential misconfiguration. Instead, a reference to the object store should be included in the associated `Manifest`, so the information about the WAL store is always encapsulated in the database itself.
`Manifest`-s are (de)serializable while `ObjectStore` instances are not. There is some support present in `object_store` crate for creating object stores from URIs, specifically `object_store::parse_url(_opts)`, but it’s very limited in features. For instance, it’s impossible to configure a store using a URI alone. At the same time, it’s impossible to re-adjust the configuration after the store creation. The configuration must be fully known before the construction and passed either to `parse_url_opts` or to some other custom construction procedure.
Additionally, there is no backward conversion from an object store instance to a URI. Getting store configuration information from an instance is even more problematic because the configuration details are private to the store instance. To avoid this problematic conversion, URIs must be used everywhere along with the configuration information encoded in the URIs themselves or as standalone objects. Alternatively, the store configuration might be obtained via some user-provided callbacks mechanism. Also, there should be a centralized URI-to-store resolver to construct and cache store instances.
All this raises many open questions that are not directly related to this RFC. Exact implementation details will be provided in a separate RFC/PR dedicated to the descriptive representation of store references and configurations.
# Settings design
Status: Approved
Authors:
* [David Calavera](https://github.com/calavera)
## Motivation
[Section titled “Motivation”](#motivation)
SlateDB is an embedded storage engine built on top of object storage. Since it’s designed to be used in many different environments, SlateDB exposes a number of configuration options to use fine-tune the engine for the environment it’s running in. It also exposes several components that users can swap with their own implementations to customize the engine, for example the `object_store` implementation, or the `clock` implementation.
These configuration options and components are somehow intertwined. Some components are included in the `DbOptions` struct and its related structs. Others are passed as parameters to the `Db::open` method.
This coupling makes it hard to understand where all the components are, what they do, and how they fit together with the configuration options. As an example of this coupling, you can see that the `DbOptions` struct has some fields marked with the macro `#[serde(skip)]` to avoid serializing them. This is because these fields are not part of the settings that a user can change, they are components that are internally used by the engine. One could make the argument that all components should be part of the `DbOptions` struct, but this would make the API more complex and harder to understand.
## Goals
[Section titled “Goals”](#goals)
This RFC aims to:
* Separate the concerns of configuration options and components.
* Make it easier to introduce new components without breaking existing public APIs.
## Proposal
[Section titled “Proposal”](#proposal)
### Separate the concerns of configuration options and components
[Section titled “Separate the concerns of configuration options and components”](#separate-the-concerns-of-configuration-options-and-components)
To make it easier to understand how SlateDB is configured, I propose to separate the concerns of configuration options, or settings, and components:
* Settings are options that a user can tweak to customize the engine for their use case
* Components are the parts of the database that are responsible for performing the work.
With that in mind, I propose to create a type alias for settings called `Settings` that will be used instead of the `DbOptions` struct. This struct will be cleaned from the components that are not settings. Currently, there are 3 components that are not settings:
* `block_cache`: The block cache to use.
* `clock`: The clock to use.
* `gc_runtime`: The runtime to use for garbage collection.
### Provide a builder API that allows users to customize the engine
[Section titled “Provide a builder API that allows users to customize the engine”](#provide-a-builder-api-that-allows-users-to-customize-the-engine)
Instead of having to pass all the components to the `Db::open` method, we will provide a builder API that allows users to customize the engine. This API will also include a method to provide the settings to the engine. The methods in this builder API will be named after the components they customize, for example `with_db_cache`, or `with_object_store`.
```rust
let object_store = Arc::new(InMemory::new());
let db = Db::builder("path/to/db", object_store)
.with_settings(Settings {
manifest_poll_interval: Duration::from_secs(1),
..Default::default()
})
.build();
```
This API makes forward changes easier to implement, as we can add new methods to the builder API without breaking backwards compatibility. For example, adding a new [WALL object store](https://github.com/slatedb/slatedb/pull/558) would not have to break the existing API since it would add a new method to the builder API, instead of changing the signature of the `open_with_opts` method.
## Backwards compatibility
[Section titled “Backwards compatibility”](#backwards-compatibility)
We can use the [`#[deprecated]` attribute](https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute) to signal to users that some parts of the current API are deprecated. This attribute allows you to signal that a function, struct, or field is deprecated and will be removed in a future release.
The `DbOptions` struct will be kept for backwards compatibility. It will be marked as deprecated and will be removed in a future release.
The fields that are not settings will be marked as deprecated and will be removed in a future release. While they are still in use, they can feed the builder API.
The `Db::open_with_opts` method will be marked as deprecated and will be removed in a future release. It will be replaced by the builder API. Until then, it will be implemented as a wrapper around the builder API.
The `Db::open` method can probably stay as is, since it’s probably the most used method at the moment. Internally, it will use the builder API to initialize the database.
# Transaction
Status: Accepted
Authors:
* [Li Yazhou](https://github.com/flaneur2020)
## Background
[Section titled “Background”](#background)
Transaction allows multiple operations to be executed in a single atomic operation, and control the isolation level of the operations. This is a planned feature in SlateDB, and is essential for many critical use cases, like:
* Using SlateDB as a metadata store or coordinator for a distributed system, such as the backend of [Kine](https://github.com/k3s-io/kine) or various metadata stores in OLAP systems.
* Using SlateDB as a underlying KV layer in an OLTP system, such as a [TiKV](https://tikv.org/) replacement or other SQL engines.
This RFC proposes the goals & design proposal of the transaction feature in SlateDB.
## Goals
[Section titled “Goals”](#goals)
* Define the Transaction API in SlateDB.
* Discuss the expected isolation level semantics in SlateDB.
* Organize the prerequisites needed to implement Transaction feature.
* Organize the roadmap for the implementation of transaction feature.
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Pessimistic Transaction support: although it is a supported feature in rocksdb, but it is not a goal for the initial implementation. Optimistic transaction is more widely used in practice, and considered as more scalable and efficient. We’ll not discuss the support of pessimistic transaction in this RFC.
* Distributed Transaction support: SlateDB is a single-writer database, and distributed transaction is not a goal for the initial implementation. We may consider support [Two Phase Commit](https://github.com/facebook/rocksdb/wiki/Two-Phase-Commit-Implementation) in the future, which may help users coordinate between multiple SlateDB instances, but it is a separate topic and not discussed in this RFC.
## Constraints
[Section titled “Constraints”](#constraints)
* The transaction API should be aligned with the existing APIs like Snapshot API and WriteBatch API.
* The transaction commit operation should be efficient, and should not involve IO overhead on checking conflicts.
* The transaction is better not to produce floating garbage in the storage, and should be able to be GCed after the transaction is committed or rolled back.
## References
[Section titled “References”](#references)
* [Transactions - RocksDB](https://github.com/facebook/rocksdb/wiki/Transactions)
* [Concurrent ACID Transactions in Badger](https://dgraph.io/blog/post/badger-txn/)
* [Transaction Internals: RocksDB](https://flaneur2020.github.io/posts/2021-08-14-rocksdb-txn/)
* [Transaction Internals: Badger](https://flaneur2020.github.io/posts/2021-08-01-badger-txn/)
* [Serializable Snapshot Isolation in PostgreSQL](https://arxiv.org/pdf/1208.4179)
## Requirements
[Section titled “Requirements”](#requirements)
Snapshot and WriteBatch are considered as the prerequisites for the Transaction feature. We could implement the Transaction feature upon them, with the following requirements:
1. Adding a **conflict check before writing a WriteBatch** to guarantee the transaction isolation.
2. Adding an **iterator that merges WriteBatch & Snapshot** to provide a consistent view that includes both uncommitted changes and the underlying snapshot.
In this section, we’ll discuss the requirements for implementing transactions, including the necessary capabilities of Snapshots and WriteBatches, as well as the semantics of different Isolation Levels.
### Snapshot
[Section titled “Snapshot”](#snapshot)
In LSM engines, Snapshot is mostly implemented by adding a sequence number to each key, and simply let each Snapshot bounds with a sequence number. All the read in the Snapshot filters out the values with bigger sequence number, including a new value or a tombstone on the same key.
At the moment of writing, adding sequence number to each key is still WIP in SlateDB. However, we’d **assume that each key has a sequence number** in this doc, because it’s considered as a common practice in LSM-tree based storage engines.
### Write Batch
[Section titled “Write Batch”](#write-batch)
This is a sample code of using `WriteBatch`:
```rust
use slatedb::{Db, WriteBatch};
let db = DB::open_with_opts(...);
let mut batch = WriteBatch::new();
batch.put(b"my key", b"my value");
batch.delete(b"key2");
batch.put(b"key3", b"value3");
db.write(batch); // Atomically commits the batch
```
You’ll notice that the `WriteBatch` API is very similar to the `Transaction` API we’ll propose below. The write operation from `WriteBatch` is also considered atomic. The only differences are:
1. the `WriteBatch` is committed by `db.write(batch)`, while the `Transaction` is committed by `txn.commit()`
2. beside the write operations, the `Transaction` should also support the read operations, and if one key is updated in the `Transaction`, the read operation should return the updated value during the transaction.
The `WriteBatch` is also considered as a prerequisite for the transaction feature, as it plays as a place to buffer the writes, and it makes MVCC possible with an iterator over it in front of the iterator over Snapshot.
Now WriteBatch has been implemented in SlateDB in [this PR](https://github.com/slatedb/slatedb/pull/264). We can further add the iterator over WriteBatch in front of the iterator over Snapshot while implementing the transaction feature.
### Isolation Levels
[Section titled “Isolation Levels”](#isolation-levels)
When implementing the transaction feature, we should also consider the isolation level semantics. The Snapshot Isolation (SI) and Serializable Snapshot Isolation (SSI) are the two most widely used isolation levels in the industry.
The Snapshot Isolation is widely used in many databases. It’s gives a lightweight Snapshot on each transactions started, and checks whether the writes against the snapshot has modified by others or not during the commit. It is very efficient and is considered as a good default isolation level for most of the use cases.
However, Snapshot Isolation still suffers Write Skew anomaly. For example, let’s assume we have two accounts, where account A has $600, and account B has $500. Also, the business rule is that Account A’s balance + Account B’s balance should always be >= $200.
Two transactions occur concurrently:
1. Transaction 1 first checks the total amount of balance in A and B, and then transfers $550 from account A to account C.
2. Transaction 2 first checks the total amount of balance in A and B, and then transfers $450 from account B to account D.
In this case, if the two transactions are executed concurrently, the total amount of balance in A and B might be less than $200, which violates the business rule. There’s no Write-Write conflict in the two transactions, but the Read-Write conflict is not detected by Snapshot Isolation.
To mitigate this anomaly, we could choose:
1. use Serializable Snapshot Isolation (SSI) level: increase the isolation level with stricter conflict checking between read and write operations, thus no longer allow the write skew anomaly to occur. For more details about conflict checking, please refer to the [Conflict Checking](#conflict-checking) section.
2. use `get_for_update()` to synchronize the critical read operations and write operations in the SI transaction.
In my understanding, the pros of the (2) approach is easy to implement and efficient, and free of floating garbage requires to be VACUUMed (when compares with innodb’s UNDO logs approach and the Postgres’s multi version rows approach). Also, in most of the time, users can make the decision by themselves about when to use `GetForUpdate()` to make the critical concurrency control correct.
However, it may produce massive writes on read intensive workloads when you hope to keep the read & write operations consistent, because `GetForUpdate()` simply transforms an read operation into a write operation, rewriting the key/value pair into WAL again. Also, some researches found that manual `GetForUpdate()` often introduces unexpected hard to diagnosis bugs, as described in the “2.2 Why Serializability” section of the [Serializable Snapshot Isolation in PostgreSQL](https://arxiv.org/pdf/1208.4179) paper:
> Given the existence of these techniques, one might question the need to provide serializable isolation in the database: shouldn’t users just program their applications to handle the lower isolation level? (We have often been asked this question.) Our view is that providing serializability in the database is an important simplification for application developers, because concurrency issues are notoriously difficult to deal with. Indeed, SI anomalies have been discovered in real-world applications \[14]. The analysis required to identify potential anomalies (or prove that none exist) is complex and is likely beyond the reach of many users. In contrast, serializable transactions offer simple semantics: users can treat their transactions as though they were running in isolation.
As above, I think it’s better to finally have SSI support in the transaction feature, we could consider to implement the Snapshot Isolation first, and then implement the SSI support after it.
We’ll cover the implementation details about the SI and SSI in the later section.
## Other Systems
[Section titled “Other Systems”](#other-systems)
Before diving into the details of the transaction implementation in SlateDB, let’s first review the transaction implementation of other LSM-tree based storage engines, to understand the possible approaches to implement the transaction feature in SlateDB.
### RocksDB
[Section titled “RocksDB”](#rocksdb)
RocksDB supports both optimistic and pessimistic transactions, but we’ll focus on the optimistic transaction here since it’s more widely used in practice.
The key idea of RocksDB’s optimistic transaction is simple: track the sequence number when transaction started, and check whether the modified keys are modified by others after the transaction started during commit.
The conflict checking in RocksDB is very straightforward:
1. Track the sequence number when transaction is started in the transaction state.
2. Track the modified keys in the transaction state.
3. When `commit()` is called, check whether the modified keys are modified by others after the transaction started.
RocksDB assumes each transaction is short-lived, and only checks the conflicts with the changes in recent. In LSM, the “recent changes” are buffered in MemTable, so RocksDB leverages MemTable to check the conflicts, making conflict checks a pure in-memory operation without any IO overhead.
However, since the size of MemTable is limited and might be flushed & rotated at any time, it cannot buffer ALL the changes since the earliest running transaction. To deal with this limitation, RocksDB’s solution is simply abort the transaction as ‘Expired’ if the MemTable’s earliest sequence number is bigger than the transaction’s sequence number.
Also, RocksDB’s Optimistic Transaction does not support Serializable Snapshot Isolation (SSI), because the read keys are not tracked in the transaction state.
The good part of this approach is that it’s very efficient and simple, and it’s free of floating garbage. The bad part is that it may abort the transaction when it’s not necessary, and tends to encourage users to set a bigger MemTable size to reduce the probability of the expired transaction.
### Badger
[Section titled “Badger”](#badger)
Badger takes a different approach from RocksDB, it provides Serializable isolation level by default. The key idea is to track both read and write operations in the transaction state, and check conflicts with the recent committed transactions tracked in a global Oracle during commit.
Badger maintains a global “Oracle” which tracks the recent committed transactions. When a transaction commits, it should push itself into the Oracle’s recent committed transactions list. The Oracle also maintains a sequence number generator, which is used to assign sequence numbers to transactions.
The conflict checking in Badger is as follows:
1. Track the sequence number when transaction started.
2. Track both read and write keys in the transaction state.
3. When `commit()` is called, check conflicts with the recent committed transactions in the Oracle which committed after the transaction started.
4. The conflict check verifies that: the transaction’s read and write keys do not conflict with other transactions’ write keys
This approach provides Serializable isolation level by detecting both Write-Write conflicts and Read-Write conflicts. It does not rely on MemTable size like RocksDB does, and provides more accurate conflict detection, without worrying about the MemTable size.
The downside is that it requires more memory to track the read keys and recent committed transactions. However, Badger mitigates this by:
1. Using u64 key fingerprints instead of full keys to track read/write keys.
2. Having an GC mechanism to clean up old committed transactions from the Oracle.
The GC is also straightforward - when all running transactions have sequence numbers greater than a committed transaction’s sequence number, that committed transaction can be removed from the Oracle.
## Proposal
[Section titled “Proposal”](#proposal)
### API
[Section titled “API”](#api)
The transaction API should be resembles with the non-transactioned API. All the api like `get()`, `put()`, `range()` should also be supported in the transaction API, and these APIs should be aligned with the Snapshot API + WriteBatch API. The only difference is that the transaction API should be able to `commit()` or `rollback()` the changes.
(note: at the time of this RFC, the Snapshot API and WriteBatch API are still not implemented, the transaction feature should be implemented after these APIs are implemented.)
The write operations like `put()` in the transaction should be buffered in the transaction state, and only be written to the underlying storage when `commit()` is called.
```rust
let object_store: Arc = Arc::new(InMemory::new());
let options = DbOptions::default();
let kv_store = Db::open_with_opts(
Path::from("/tmp/test_kv_store"),
options,
object_store,
)
.await
.unwrap();
// start transaction and write some data
let txn_opts = TransactionOptions::new().with_isolation_level(IsolationLevel::Serializable); // or IsolationLevel::Snapshot
let txn = kv_store.transaction_with_opts(txn_opts).await.unwrap();
let val1 = txn.get("key1").await;
// could batch the write operations
txn.put("key2", val1);
txn.put("key3", "val3");
// should see the new value of key2
assert_eq!(txn.get("key2").await, val1);
// commit, might fail on conflict, when it fails, the transaction should be rolled back
txn.commit().await?;
// one could also rollback the transaction by calling `rollback()` or dropping it when it's out of scope.
txn.rollback().await;
```
A list of the transaction API is looked like:
```rust
// it might not really be a trait, but a struct with the methods:
trait Transaction {
async fn commit(&self) -> Result<()>;
// the rollback operation should not be async, we need to call it in Drop impl
fn rollback(&self) -> Result<()>;
async fn get(&self, key: &[u8]) -> Result>;
async fn get_with_options(&self, key: &[u8], options: GetOptions) -> Result>;
async fn put(&self, key: &[u8], value: &[u8]) -> Result<()>;
async fn put_with_options(&self, key: &[u8], value: &[u8], options: PutOptions) -> Result<()>;
async fn delete(&self, key: &[u8]) -> Result<()>;
async fn scan(&self, range: Range) -> Result;
}
```
The list of the transaction API is not final, and it’s expected to be adjusted during the implementation. The rule of thumb is that the Transaction API should be as aligned as possible to the non-transactioned API in `DB` struct.
### Conflict Checking
[Section titled “Conflict Checking”](#conflict-checking)
The approach we choose on conflict checking is expected to be more similar to Badger’s global `Oracle` approach, since we hope to provide SSI in the transaction feature, and it’s less prone to unnecessarily abort the transaction with limited MemTable size.
Below is the simplified data structure of the global `Oracle` and the transaction state, with only the fields that are related to the conflict checking:
```rust
struct Oracle {
next_seq: AtomicU64,
recent_committed_txns: Deque>,
}
struct TransactionState {
started_seq: u64,
write_keys: HashSet,
read_keys: Vec>,
committed_seq: Option
}
```
`Oracle` is considered a global singleton in the DB, and it tracks the recent committed transactions in the `recent_committed_txns` deque. When a transaction commits, it should push itself into the `recent_committed_txns` deque, and the entries in `recent_committed_txns` can be GCed after they are no longer needed. Also, when a `TransactionState` is tracked in the `recent_committed_txns`, only the write keys need to be tracked, while `read_keys` should always be empty.
Besides the started sequence number of the transaction, we also need to track the sequence number of the transaction committed. During the execution of a transaction, some other transactions might commit, and the current transaction should check the conflicts with these committed transactions.
The conflict check simply checks whether the Read & Write keys of the current transaction are overlapped with the Write keys from related recent committed transactions.
Given an execution history of transactions like:
```plaintext
txn1: [seq:97] <--- ---- ---> [seq:99]
txn2: [seq:98] <--- ---- ---- ---- ---- ---- ---> [seq:105]
txn3: [seq:102] <--- ---> [seq:104]
txn4: [seq:101] <--- ---- ---- ---- ---> [seq:106]
```
Let’s say the current transaction is `txn4`, it should check conflicts with `txn2` and `txn3` during the commit, because `txn2` and `txn3` are committed after `txn4` started. During the conflict checking, `txn4`’s Read & Write keys should not conflict with the Write keys of `txn2` and `txn3`.
It does not need to check conflicts with `txn1`, because `txn1` is committed before `txn4` started. This could tell us when to GC the entry in the `recent_committed_txns` deque: when ALL the running transactions’ sequence number are bigger than the committed sequence number of the entry, then the entry could be GCed.
Please note that `read_keys` are represented as a set of `Range`. This allows us to efficiently track ranges of keys that were read during the transaction, and ensure Serializable Snapshot Isolation (SSI) on range conflicts. For example, if transaction A reads a range `["key01", "key10"]` and transaction B writes to `"key05"` which falls within that range, we can detect this conflict and abort one of the transactions to maintain serializability.
This requires we track the original write keys in the `TransactionState`, instead of the key fingerprints like Badger does. We could reuse the buffers of the original keys in WriteBatch to reference the write keys with slices of `Bytes`.
The non-transactioned writes operations should also be tracked in the `TransactionState` and `recent_committed_txns` for conflict checking. They could be regarded as a transaction that only contains a single write operation, and the start sequence number of this transaction is same as the committed sequence number.
This approach can be adjusted to support SI as well by only tracking the keys that are being written to. When user set the isolation level to SI, the `read_keys` could always be set as empty. The GC mechanism to release inactive transactions is still the same as the SSI.
#### GC & Expiration
[Section titled “GC & Expiration”](#gc--expiration)
As described above, the entries in `recent_committed_txns` could be released whenever ALL the running transactions’ sequence number are bigger than the committed sequence number of the entry.
We have to track the running transactions’s sequence number in the `Oracle`.
Also, it’s less make sense to allow a transaction to be running for 12 hours, the `recent_committed_txns` will also keep growing for such 12 hours, and the GC will not be able to release the entries. This is a potential problem that the memory usage of the `Oracle` will keep growing, and possible lead to OOM.
To mitigate this, we should allow users to set a default expiration time for all of the transactions & snapshots, and the transactions & snapshots will be considered as expired if it’s not finished within the expiration time.
```rust
struct Oracle {
running_txns: HashSet,
next_seq: u64,
discard_seq: u64,
recent_committed: Deque>,
}
struct TransactionState {
started_time: u64,
expired_time: u64,
started_seq: u64,
write_keys: HashSet,
read_keys: Vec>,
committed_seq: Option
}
```
We should let the `Oracle` to track the running transactions in `running_txns`, and when the transaction is finished, whether it’s committed or rolled back, the `Oracle` should remove the transaction from the `running_txns` set. If the transaction is expired, the `Oracle` should also remove the transaction from the `running_txns` set.
Please note that Snapshot could also be considered as a read only transaction, and it should also be tracked in `running_txns`. And Snapshot should also has a default expiration as well.
With `running_txns`, we could find out the earliest running transaction’s sequence number, and `Oracle` could use this sequence number to determine which entries in `recent_committed_txns` could be GCed. This could help us to ensure both `running_txns` and `recent_committed_txns` do not grow out of a time limit.
Also, we can maintain a `discard_seq` during `recent_committed_txns`’s cleanup. If the cleaned up txn’s create time is earlier than global expiration time, we could set `discard_seq` to the txn’s sequence number, and the compactor could respect this `discard_seq` to discard the values earlier than `discard_seq` safely when it found a value with multiple versions.
### How to Test
[Section titled “How to Test”](#how-to-test)
Testing transaction implementations requires verifying both correctness and isolation guarantees.
To test the transaction feature, we could write some test cases that cover the following aspects:
1. Basic transaction operations.
2. Isolation guarantees for both SI and SSI modes.
3. Edge cases and concurrency control.
We could consider to port the test cases from [RocksDB](https://github.com/facebook/rocksdb/blob/3495c94761629404f5476f732e2130134cfcc51b/utilities/transactions/optimistic_transaction_test.cc), [Badger](https://github.com/dgraph-io/badger/blob/36c461a435c53a8a81e7377c2b026b24d37eee0c/txn_test.go) to have a comprehensive coverage of the above aspects.
## Roadmap
[Section titled “Roadmap”](#roadmap)
1. Implement the WriteBatch API (related with ).
2. Add sequence to each key in the storage, and implement the Snapshot API.
3. Add the global Oracle to track the recent committed transactions for conflict checking, implement the SI transaction, then implement the SSI transaction.
## Updates
[Section titled “Updates”](#updates)
* 2024-10-13: Initial draft.
* 2024-10-20: Add a simple roadmap.
* 2024-11-17: Restructure the doc to align with the latest implementation.
# Tracking Timestamps for Sequence Numbers
Status: Accepted
Authors:
* [Almog Gavra](https://github.com/agavra)
## Motivation
[Section titled “Motivation”](#motivation)
Standardizing on sequence numbers for tracking progress in SlateDB makes it straightforward to reason about “when” something happened. Users, however, may want to perform certain operations based on time instead of sequence number. For example, they may want to fetch a snapshot associated with a particular system time or issue a scan for all records committed before a certain time.
This RFC proposes tracking timestamps for sequence numbers so users can obtain a rough approximation of the mapping between sequence numbers and (system) timestamps.
## Goals
[Section titled “Goals”](#goals)
* Use a predictable, fixed amount of memory for the sequence tracker (therefore bounded storage space)
* Expose an API for determining the timestamp for a given sequence number
* Expose an API for determining the sequence number for a given timestamp
To reduce storage overhead, this RFC does not propose an exact mapping between sequence numbers and timestamps.
## Public API
[Section titled “Public API”](#public-api)
The API will be exposed through the CLI and involve changes to the manifest (which is implicitly a public API).
### CLI
[Section titled “CLI”](#cli)
```rs
/// Rounding behavior for non-exact matches in sequence-timestamp lookups.
#[allow(dead_code)]
#[derive(PartialEq)]
pub(crate) enum FindOption {
/// Round up to the next higher value when no exact match is found.
RoundUp,
/// Round down to the next lower value when no exact match is found.
RoundDown,
}
pub(crate) enum CliCommands {
// ...
/// Fetch an approximate (system) timestamp for a given sequence number.
/// SlateDB tracks a mapping between sequence numbers and timestamps and
/// maintains it with a lossy mechanism (over time the granularity of the
/// tracked mapping degrades).
GetTimestampForSeq {
/// The sequence number to fetch the timestamp for.
seq: u64,
/// The find option to use when fetching the timestamp.
find_opt: FindOption,
}
/// Fetch an approximate sequence number for a given timestamp.
/// SlateDB tracks a mapping between sequence numbers and timestamps and
/// maintains it with a lossy mechanism (over time the granularity of the
/// tracked mapping degrades).
GetSeqForTimestamp {
/// The timestamp to fetch the sequence number for.
ts: DateTime,
/// The find option to use when fetching the sequence number.
find_opt: FindOption,
}
}
```
The output of the CLI will include both the timestamp and the sequence number in order to indicate whether rounding happened.
### Manifest
[Section titled “Manifest”](#manifest)
```fbs
table ManifestV1 {
// ...
// The sequence tracker is a custom serialized type that is used to track
// a (lossy) mapping between sequence numbers and timestamps. The serialization
// format is described in RFC-0012.
seq_tracker: [ubyte];
}
```
We choose to represent the sequence tracker as a custom serialized type instead of a flatbuffer type to reduce manifest storage overhead. It uses Gorilla encoding (delta-of-deltas) for both keys (sequence numbers) and values (timestamps).
## Implementation
[Section titled “Implementation”](#implementation)
The high-level approach for implementing the tracker is to store the mapping in the manifest and update it whenever a memtable is flushed or SlateDB cleanly shuts down.
To reduce stored data we use two tactics:
1. The encoding uses [Gorilla Encoding](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf), which is an efficient encoding for timeseries data.
2. The granularity is decimated over time (older seq-ts pairs are downsampled)
### Serialization
[Section titled “Serialization”](#serialization)
Physically, the sequence tracker is serialized as two Gorilla-encoded arrays of equal length (one for sequence numbers and one for timestamps). The arrays are broken up logically (though not physically) into tiers, and within a tier the timestamps are expected to be evenly spaced (though this assumption is not enforced).
The Gorilla encoding scheme means sequence numbers and timestamps will take up between 1 and 32 bits depending on SlateDB insertion regularity (typically closer to 7 bits each).
The ultimate serialization format is:
```plaintext
+-------------------------------------+
| Version (u8) | Length (u32) |
|-------------------------------------|
| Sequence Numbers (Gorilla Encoding) |
|-------------------------------------|
| Timestamps (Gorilla Encoding) |
+-------------------------------------+
```
Since both sequence numbers and timestamps are monotonically increasing, we can represent this in memory as two sorted arrays and use simple binary search on either array for bi-directional lookup:
```rs
struct SequenceTracker {
sequence_numbers: Vec, // sorted
timestamps: Vec>, // sorted
capacity: u32,
}
```
### Tiering & Downsampling Strategy
[Section titled “Tiering & Downsampling Strategy”](#tiering--downsampling-strategy)
To meet the requirement of predictable, fixed storage for the sequence tracker, we need to downsample data over time.
When the sequence tracker is initialized, it will be configured with a fixed number of mappings to store (measured in key-value pairs). Once that number is reached, the sequence tracker will downsample by removing every other entry. To avoid aggressive downsampling, the sequence tracker will only track a timestamp mapping on a fixed interval.
The downsampling strategy will cause exponentially decreasing granularity over time. For example, if the sequence tracker is configured with 1024 mappings and a 30s reporting interval, the first time it fills up the data will represent a timeframe of `1024 * 30 seconds = 512 minutes`. When downsampling kicks in, this time window will cover the first 512 entries. The next time the sequence tracker fills up, this same data will be covered by the first 256 entries (and so on).
#### Concrete Example
[Section titled “Concrete Example”](#concrete-example)
Consider a sequence tracker with 8 entries recording timestamps every 30 seconds:
**Initial state (8 entries, full):**
```plaintext
12:00:00, 12:00:30, 12:01:00, 12:01:30, 12:02:00, 12:02:30, 12:03:00, 12:03:30
| . | . | . | . | . | . | . | . |
└──30s──┴────30s───┴───30s───┴────30s─┴────30s───┴───30s───┴────30s───┴──30s──┘
```
**After first downsampling (keep every other entry, 4 entries):**
```plaintext
12:00:00, 12:01:00, 12:02:00, 12:03:00
| . | . | . | . |
└──1m───┴────1m────┴───1m────┴───1m───┘
```
**After filling up again (8 entries):**
```plaintext
12:00:00, 12:01:00, 12:02:00, 12:03:00, 12:04:00, 12:04:30, 12:05:00, 12:05:30
| . | . | . | . | . | . | . | . |
└──1m───┴────1m────┴───1m────┴───1m───┴────30s───┴───30s───┴───30s───┴──30s──┘
```
**After second downsampling (4 entries):**
```plaintext
12:00:00, 12:02:00, 12:04:00, 12:05:00
| . | . | . | . |
└──2m───┴────2m────┴───1m────┴───1m───┘
```
#### Understanding Horizon Coverage
[Section titled “Understanding Horizon Coverage”](#understanding-horizon-coverage)
This pattern continues, with older entries becoming progressively sparser while maintaining coverage across the entire time range. Note that there’s an implicit “cutoff” when the buffer fills up that will downsample old entries so aggressively that there is no longer any retentition beyond a certain time.
To understand how much time a sampling reasonably covers you can reference the formula:
```plaintext
H ≈ (C * Δ / 2) * log2(C)
```
Where:
* H = horizon covered
* C = capacity (entries)
* Δ = base interval (30s in your case)
Here is a table of reasonable defaults for confiruation:
| Reporting Interval (Seconds) | Capacity | Horizon (Days) |
| ---------------------------- | -------- | -------------- |
| 30 | 2048 | 3.9 |
| 30 | 4096 | 8.5 |
| 30 | 8192 | 18.5 |
| 60 | 2048 | 7.8 |
| 60 | 4096 | 17.1 |
| 60 | 8192 | 37.0 |
| 300 | 2048 | 39.1 |
| 300 | 4096 | 85.3 |
| 300 | 8192 | 184.9 |
In the interest of simplicity, we will not make the retention period configurable, and instead choose a reporting interval of 60s with a capacity of 8192 entries.
## Rejected Alternatives
[Section titled “Rejected Alternatives”](#rejected-alternatives)
* **Storing timestamps in each row**: It is possible to store the system timestamp with each row in SlateDB alongside the logical timestamp. This would allow exact lookup of timestamps for sequence numbers. However, this would require significant additional storage space and would not be easy to retrieve using an admin API, limiting applicable use cases.
# Compaction State Persistence
Status: Accepted
Authors:
* [Sujeet Sawala](https://github.com/sujeetsawala)
## Background
[Section titled “Background”](#background)
Compaction currently happens for the following:
* L0 SSTs
* Various level Sorted Runs (range partitioned SST across the complete keyspace)
This RFC proposes the goals & design for compaction state persistence along with ways to improve current compaction mechanism by adding retries and tracking.
## Goals
[Section titled “Goals”](#goals)
* Provide a mechanism to track progress of a `Compaction`
* Allow retrying compactions based on the state of the `Compaction`
* Improve observability around compactions
* Separate out compaction related details from `Manifest` into a separate `Compactions` struct
* Coordination between `Manifest` and `Compactions`
* Coordination mechanism between externally triggered compactions and the main compaction process.
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Distributed compaction: SlateDb is a single writer and currently a single-compactor based database. With distributed compaction, we plan to further parallelise SST compaction across different compaction processes. This topic is out of scope of the RFC.
* Resuming partial compaction under MVCC depends on the sorted-run layout: with multi-versioned keys, how do we partition the keyspace into non-overlapping SSTs within a single SR?
## Constraints
[Section titled “Constraints”](#constraints)
* Changes should be backward compatible and extend the existing compaction structs
* State updates should be cost efficient
* Manifest can be eventually consistent with the latest view after compaction
## References
[Section titled “References”](#references)
* [Compaction RFC](https://github.com/slatedb/slatedb/blob/main/rfcs/0002-compaction.md)
* [Universal Compaction](https://github.com/facebook/rocksdb/wiki/universal-compaction)
## Problem Statement
[Section titled “Problem Statement”](#problem-statement)
This RFC extends discussions in the below github issue. It also addresses several other sub-issues.
[Issue #673](https://github.com/slatedb/slatedb/issues/673):
### Core Architecture Issues
[Section titled “Core Architecture Issues”](#core-architecture-issues)
1. **1:1 Compaction:Job Cardinality**: Cannot retry failed compactions - entire compaction fails if job fails
2. **No Progress Tracking**: `Compaction` state isn’t persisted, making progress invisible
3. **No State Persistence**: All compaction state is lost on restart
### Operational Limitations
[Section titled “Operational Limitations”](#operational-limitations)
5. **Manual Compaction Gaps**: No coordination mechanism for operator-triggered compactions ([Issue #288](https://github.com/slatedb/slatedb/issues/288))
6. **GC Coordination Issues**: Garbage collector needs better visibility into ongoing compactions ([Issue #604](https://github.com/slatedb/slatedb/issues/604))
7. **Limited Observability**: Limited visibility into compaction progress and failures
### Impact
[Section titled “Impact”](#impact)
* **Large compactions** (multi-GB) lose hours of work on failure
* **Engineering overhead** for debugging and manually restarting failed compactions
* **Customer impact** from extended recovery times during outages
* **Resource waste** from repeated processing of the same data
## Proposal
[Section titled “Proposal”](#proposal)
### Core Strategy: Iterator-Based Persistence
[Section titled “Core Strategy: Iterator-Based Persistence”](#core-strategy-iterator-based-persistence)
Rather than complex chunking mechanisms, we leverage SlateDB’s existing iterator architecture which provides natural persistence boundaries at **SST completion points**. This approach:
* **Builds on existing infrastructure**: Enhances current `execute_compaction_job` method
* **Uses natural boundaries**: SST completions provide \~256MB recovery granularity
* **Minimizes overhead**: Persistence aligns with existing I/O patterns
* **Scales cost-effectively**: Higher persistence frequency for larger, more valuable compactions
## Workflow
[Section titled “Workflow”](#workflow)
### Current Compaction Workflow
[Section titled “Current Compaction Workflow”](#current-compaction-workflow)
1. `Compactor` initialises the `CompactionScheduler` and `CompactionEventHandler` during startup. It also initialises event loop that periodically polls manifest, periodically logs and provides progress and handles completed compactions \[No change required]
2. The `CompactionEventHandler` refreshes the compaction state by merging it with the `current manifest`.
3. `CompactionEventHandler` communicates this compaction state to the `CompactionScheduler`(scheduler makes a call `maybeScheduleCompaction` with local database state).
4. `CompactionScheduler` is implemented by `SizeTieredCompactionScheduler` to decide and group L0 SSTs and SRs to be compacted together. It returns a list of `Compaction` that are ready for execution.
5. `CompactorEventHandler` iterates over the list of compactions and calls `submitCompaction()` if the count of running compaction is below the threshold.
6. The submitted compaction is validated that it is not being executed (by checking in the local `CompactorState`) and if true, is added to the `CompactorState` struct.
7. Once the `CompactorEventHandler` receives an affirmation, it calls the `startCompaction()` to start the compaction.
8. The compaction is now transformed into a `Compaction` and a blocking task is spawned to execute the `Compaction` by the `CompactionExecutor`
9. The task loads all the iterators in a `MergeIterator` struct and runs compactions on it. It discards older expired versions and continues to write to a SST. Once the SST reaches it’s threshold size, the SST is written to the active destination SR. Periodically the task also provides stats on task progress.
10. When a task completes compaction execution, the task returns the `{destinationId, outputSSTs}` to the worker channel to act upon the compaction terminal state
11. The worker task executes the `finishCompaction()` upon successful `CompactionCompletion` and updates the manifests and trigger scheduling of next compactions by calling `maybeScheduleCompaction()`
12. In case of failure, the compaction\_state is updated by calling `finishFailedCompaction()`
13. GC clears the orphaned states and SSTs during it’s run.
### Proposed CompactionState Structure
[Section titled “Proposed CompactionState Structure”](#proposed-compactionstate-structure)
The in-memory state contains the complete view of all compaction activity:
````rust
/// Container for compactions tracked by the compactor alongside its epoch.
pub(crate) struct Compactions {
// The current compactor's epoch.
pub(crate) compactor_epoch: u64,
pub(crate) core: CompactionsCore,
}
/// Represents an immutable in-memory view of .compactions file that is suitable
/// to expose to end-users.
pub struct CompactionsCore {
/// The set of recent compactions tracked by this compactor. These may
/// be pending, in progress, or recently completed (either with success
/// or failure).
recent_compactions: BTreeMap,
}
/// Canonical, internal record of a compaction.
///
/// A compaction is the unit tracked by the compactor: it has a stable `id` (ULID) and a `spec`
/// (what to compact and where).
pub struct Compaction {
/// Stable id (ULID) used to track this compaction across messages and attempts.
id: Ulid,
/// What to compact (sources) and where to write (destination).
spec: CompactionSpec,
/// Total number of bytes processed so far for this compaction.
bytes_processed: u64,
/// Current status for this compaction.
///
/// This is tracked only in memory at the moment.
status: CompactionStatus,
/// Output SSTs produced by this compaction, if any.
output_ssts: Vec,
}
/// Immutable spec that describes a compaction. Currently, this only holds the
/// input sources and destination SR id for a compaction.
pub struct CompactionSpec {
/// Input sources for the compaction.
sources: Vec,
/// Destination sorted run id for the compaction.
destination: u32,
}
/// Lifecycle status for a compaction.
///
/// State transitions:
/// ```text
/// Submitted <-> Running --> Completed
/// | |
/// | v
/// +----------> Failed
/// ```
///
/// `Completed` and `Failed` are terminal states.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
pub enum CompactionStatus {
/// The compaction has been submitted but not yet started.
Submitted,
/// The compaction is currently running.
Running,
/// The compaction finished successfully.
Completed,
/// The compaction failed. It might or might not have started before failure.
Failed,
}
````
### Proposed `compactor.fbs`
[Section titled “Proposed compactor.fbs”](#proposed-compactorfbs)
Backwards compatible FlatBuffers schema for persisting compaction state:
```fbs
union CompactionSpec {
TieredCompactionSpec,
}
enum CompactionStatus : byte {
/// The compaction has been submitted but not yet started.
Submitted = 0,
/// The compaction is currently running.
Running,
/// The compaction finished successfully.
Completed,
/// The compaction failed. It might or might not have started before failure.
Failed,
}
table TieredCompactionSpec {
// input L0 SSTable IDs
ssts: [Ulid];
// input Sorted Run IDs
sorted_runs: [uint32];
}
table Compaction {
// ID of compaction
id: Ulid (required);
// Specification defining the work to be done for this compaction
spec: CompactionSpec (required);
// Current lifecycle status for this compaction
status: CompactionStatus;
// Output SSTs already produced by this compaction.
output_ssts: [CompactedSsTable];
}
table CompactionsV1 {
// Fencing token to ensure single active compactor
compactor_epoch: uint64;
// In-flight compactions as of the last update
recent_compactions: [Compaction] (required);
}
```
### Coordination between Manifest and CompactionState
[Section titled “Coordination between Manifest and CompactionState”](#coordination-between-manifest-and-compactionstate)
A new `CompactorStateReader` and `CompactorStateWriter` are introduced to manage the dual persistence of `.manifest` and `.compactions` files. These encapsulate the ordering and fencing rules between `.manifest` and `.compactions` so the rest of the codebase can treat them as a single logical state.
* `CompactorStateReader`: loads `.compactions` before `.manifest` so GC/admin readers see a consistent view of in-flight compactions alongside a manifest that still references their inputs.
* `CompactorStateWriter`:
* Fences by incrementing `compactor_epoch` in the `.manifest` first and then the `.compactions`.
* Refreshes and merges remote state (including externally submitted compactions).
* Writes `.manifest` first, then `.compactions` to ensure completed compactions output is visible before the compaction is marked finished.
* On startup it rewrites `Running` to `Submitted` so they can be resumed.
* Trims older finished entries for GC.
### Persisting Internal Compactions
[Section titled “Persisting Internal Compactions”](#persisting-internal-compactions)
1. On each poll tick, `CompactorEventHandler` calls `state_writer.refresh()`, which loads `.compactions` first and then `.manifest`, merging any remote submitted compactions into the local `CompactorState`.
2. The scheduler proposes `CompactionSpec` values via `CompactionScheduler::propose`. The handler only schedules up to available capacity (`max_concurrent_compactions - running`).
3. For each proposal, the handler allocates a compaction id and calls `CompactorState::add_compaction`. This enforces destination conflict/overwrite rules. Accepted compactions are stored with status `Submitted`.
4. If any new compactions are added, the handler persists them with `write_compactions_safely()`, which writes the next `.compactions` file and retries on version conflicts. The stored view retains all active compactions plus the most recently finished one for GC (see [#1044](https://github.com/slatedb/slatedb/issues/1044)).
5. The handler then processes all `Submitted` compactions (internal and external) in `maybe_start_compactions`. It validates each spec (non-empty sources, L0-only destination must be above the highest existing SR, plus scheduler validation). Invalid specs are marked `Failed`. Valid specs are started in the executor and marked `Running`. The status changes are persisted with `write_compactions_safely()`.
6. The executor emits `CompactionJobProgress` messages with `bytes_processed` and `output_ssts`. The handler updates the in-memory compaction and persists to `.compactions` only when `output_ssts` grows (new output SST), which is the recovery boundary.
7. When a compaction finishes, the handler applies `finish_compaction`, writes a checkpointed manifest, then writes the updated compactions via `write_state_safely()` (manifest first, compactions second). The compaction status becomes `Completed`, and older finished entries are trimmed.
### Persisting External Compactions
[Section titled “Persisting External Compactions”](#persisting-external-compactions)
External requests are persisted into the same `.compactions` store and then handled by the regular compactor flow:
1. Client submits a `CompactionSpec` via `Admin::submit_compaction`, which calls `Compactor::submit` to write a new `Compaction` (status `Submitted`) into the `.compactions` store. This uses optimistic concurrency and retries on version conflicts. No scheduler validation happens at submit time.
2. On the next poll tick, `state_writer.refresh()` loads `.compactions` and merges newly submitted external compactions into the local state. `merge_remote_compactions` only accepts remote entries that are still in `Submitted` status.
3. The compactor then handles these submissions exactly like internal ones: `maybe_start_compactions` enforces capacity and validates each spec (non-empty sources, L0-only destination rule, scheduler validation). Invalid entries are marked `Failed`; valid ones are started and marked `Running`.
4. Progress and completion persistence is identical to internal compactions: new output SST boundaries are persisted via progress updates, and completion writes the manifest then compactions via `write_state_safely()`.
### Resuming Partial Compactions
[Section titled “Resuming Partial Compactions”](#resuming-partial-compactions)
1. The executor reports progress with `CompactionJobProgress` messages that include the full `output_ssts` list. The handler persists these updates to `.compactions` only when a new output SST is added.
2. On compactor startup, `CompactorStateWriter::new` flips any `Running` compactions back to `Submitted` and preserves their `output_ssts`, then writes this recovery state to `.compactions`.
3. When such a compaction is started, `StartCompactionJobArgs` carries the existing `output_ssts`. The executor derives a resume cursor by reading the last output SST and calling `last_written_key_and_seq` (last key + sequence from the final block).
4. Input iterators (L0 SSTs and sorted runs) are built as usual and wrapped in `ResumingIterator`, which seeks to the last written key and drains entries for that key while `seq >= last_seq` to avoid duplicates.
5. Compaction continues from that position and produces new output SSTs; each completed SST triggers another persisted progress update.
### Key Design Decisions
[Section titled “Key Design Decisions”](#key-design-decisions)
#### Persistence Boundaries
[Section titled “Persistence Boundaries”](#persistence-boundaries)
**Decision**: Persist state at the critical boundary: **Output SST Completion**: Every \~256MB of written data (always persisted) **Rationale**: Output SST completions provide the best recovery value per persistence operation. Each represents significant completed work that we don’t want to lose.
#### Enhanced Job Model
[Section titled “Enhanced Job Model”](#enhanced-job-model)
**Decision**: Keep 1:1 relationship between a logical `Compaction` and it physical execution attempt. **Rationale**: Simplifies tracking and state management. Resumes are handled by marking a `Compaction` as `Submitted` again. Retries are not handled. Once a `Compaction` is markfed `Failed`, it’s not retried. The number of resumes are not tracked in the `.compactions` file. Users can see resume activity in logs or by viewing previous `.compactions` files to see transitions from `Running` back to `Submitterd`. This approach simplifies the data model and state machine.
#### State Management Pattern
[Section titled “State Management Pattern”](#state-management-pattern)
**Decision**: Mirror the existing `ManifestStore` pattern with `CompactorStore`.
**Rationale**: Reuses proven patterns for atomic updates, version checking, and conflict resolution that are already battle-tested in SlateDB.
#### Recovery Strategy
[Section titled “Recovery Strategy”](#recovery-strategy)
* Resume from last completed output SST
The section below is under discussion here:
### Persistent State Storage
[Section titled “Persistent State Storage”](#persistent-state-storage)
#### Object Store Layout
[Section titled “Object Store Layout”](#object-store-layout)
The compaction state is persisted to the object store following the same CAS pattern as manifests, ensuring consistency and reliability:
```plaintext
/00000000000000000001.compactions # First compactor state
/00000000000000000002.compactions # Updated state after compactions
/00000000000000000003.compactions # Current state
```
*NOTE: The section below is discussed in detail [here](https://github.com/slatedb/slatedb/pull/695/files#r2239561471).*
### Protocol for State Management of Manifest and CompactionState
[Section titled “Protocol for State Management of Manifest and CompactionState”](#protocol-for-state-management-of-manifest-and-compactionstate)
This section describes the `CompactorStateReader` and `CompactorStateWriter` protocol in detail. The protocol is based on the following principals:
* .manifest should be source of truth for reader and writer clients (and should thus contain SRs)
* .compactions file should be an implementation detail of the compactor, not something the DB needs to pay any attention to.
The `.compactions` file also serves as an interface over which clients can build their custom distributed compactions say based on etcd (Kubernetes), [chitchat](https://github.com/quickwit-oss/chitchat), `object_store`, etc.
#### On startup…
[Section titled “On startup…”](#on-startup)
1. Compactor fetches the latest .manifest file (00005.manifest).
2. Manifest increments `compactor_epoch` and tries writing the `compaction_epoch` to the .manifest file (00006.manifest) File version check (in-memory and remote object store): if 00006.manifest exists,
* If latest .manifest compactor\_epoch > current compactor epoch, die (fenced)
* If latest .manifest compactor\_epoch == current compactor epoch, die (fenced)
* If latest .manifest compactor\_epoch < current compactor epoch, increment the .manifest file ID by 1 and retry. This process would continue until successful compactor write. (The current active compactor has updated the .manifest file)
3. Compactor fetches the latest .compactions file (00005.compactions).
4. Try writing above `compactor_epoch` to the next sequential .compactions position.(00006.compactions). File version check (in-memory and remote object store): if 00006.compactions exists,
* If latest .compactions compactor\_epoch > current compactor epoch, die (fenced)
* If latest .compactions compactor\_epoch == current compactor epoch, panic
* If latest .compactions compactor\_epoch < current compactor epoch, increment the .compactions file ID by 1 and retry. This process would continue until successful compactor write. (The current active compactor Job would have updated the .compactions file)
5. If compactor\_epoch in in-memory manifest (00005.manifest) >= `compactor_epoch`, older compactors are fenced now.
At this point, the compactor has been successfully initialised. Any updates to write a new .compactions (00006.compactions) or .manifest file (00006.manifest) by stale compactors would fence them.
#### On compaction job creation…
[Section titled “On compaction job creation…”](#on-compaction-job-creation)
(Manifest is polled periodically to get the list of L0 SSTs created. The scheduler would create a list of new compactions for these L0 SSTs as well)
1. Compactor fetches the latest .manifest file during the manifest poll. (00006.manifest)
2. Compactor writes to the next .compactions file with the list of compactions with empty `output_ssts` (00007.compactions in our example). If the file (00007.compactions) exists,
* If latest .compactions compactor\_epoch > current compactor epoch, die (fenced)
* If latest .compactions compactor\_epoch == current compactor epoch, reconcile the compactor state and go to step (2)
* If latest .compactions compactor\_epoch < current compactor epoch, panic (`compactor_epoch` went backwards)
#### On compaction job progress…
[Section titled “On compaction job progress…”](#on-compaction-job-progress)
1. Compactor writes to the next .compactions file the `Compactions` (persist when an SST is added to SR) with the latest progress (00008.compactions in our example). If the file (00008.compactions) exists,
* If latest .compactions compactor\_epoch > current compactor epoch, die (fenced)
* If latest .compactions compactor\_epoch == current compactor epoch, reconcile the compactor state and go to step (2)
* If latest .compactions compactor\_epoch < current compactor epoch, panic (`compactor_epoch` went backwards)
#### On compaction job complete…
[Section titled “On compaction job complete…”](#on-compaction-job-complete)
1. Update in-memory state:
* Remove compacted L0 SSTs and source SRs from `Manifest`
* Insert the output SR in `Manifest`
* Update `l0_last_compacted`
* Mark the compaction finished
* Remove finished compactions, retaining the most recent finished compaction for GC (see #1044).
2. Write .manifest using protocol described above
3. Write .compactions using protocol described above
If a failure occurs between (2) and (3), on restart, the `Compaction` is still in a `Running` state. The compactor will attempt to resume it. The inputs will have already been removed from the `Manifest` in step (2), so the compaction should fail during validation and be marked as `Failed`.
*NOTE: There is one pathological case where the compaction has a single SR input, which is also its output (e.g. input=\[SR1], output=SR1). In this case, the compaction will resume successfully and reprocess the remaining data in SR1. This is acceptable since an SR compaction of itself is a no-op.*
### External Process Integration
[Section titled “External Process Integration”](#external-process-integration)
**Administrative Commands**:
* `slatedb submit-compaction --request '' [--scheduler size-tiered]` - Submit a compaction request (`"Full"` or `{"Spec":{...}}` JSON).
* `slatedb read-compaction --id [--compactions-id ]` - Read a specific compaction by ULID.
* `slatedb read-compactions [--id ]` - Read the latest or a specific compactions file.
* `slatedb list-compactions [--start ] [--end ]` - List compactions files in an id range.
* `slatedb run-compactor` - Run the compactor in the foreground.
These commands map to `Admin::submit_compaction`, `Admin::read_compaction`, `Admin::read_compactions`, `Admin::list_compactions`, and `Admin::run_compactor`.
### Garbage Collection Integration
[Section titled “Garbage Collection Integration”](#garbage-collection-integration)
The garbage collection of .compactions file can leverage the existing logic of garbage collecting .sst files and .manifest files.
The .sst file is deemed to be garbage collected if it satisfies the following conditions:
* SST is older than the min age configured.
* SST is not referenced by any active manifest checkpoint.
Note: We would also need to handle the scenario mentioned in to avoid deletion of compacted SSTs and prevent data corruption.
The .manifest file is deemed to be garbage collected if it satisfies the following conditions:
* Avoid deletion of the latest manifest
* Delete manifest not referenced by active checkpoints
* Delete manifests that have passed the min\_age
The .compactions file can be cleaned by the garbage collector similar to .manifest file garbage collection conditions:
* Avoid deletion of the latest .compactions file
* Delete compactor files that have passed the min\_age
## Observability Enhancements
[Section titled “Observability Enhancements”](#observability-enhancements)
### Progress Tracking
[Section titled “Progress Tracking”](#progress-tracking)
* **Real-time progress**: Bytes processed, SSTs completed, estimated completion time
* **Phase tracking**: Reading inputs → Writing outputs → Updating manifest → Completed
* **Recovery metrics**: Work preservation percentage, recovery time
### Statistics
[Section titled “Statistics”](#statistics)
* **Performance**: Throughput, duration, success rates by compaction size
* **Recovery**: Jobs recovered, average recovery time, work preservation
* **Errors**: Categorized by type (network, memory, corruption) for retry decisions
* **Cost**: Persistence operations, overhead percentage
## Cost Analysis
[Section titled “Cost Analysis”](#cost-analysis)
#### Operation Count Breakdown
[Section titled “Operation Count Breakdown”](#operation-count-breakdown)
For a **typical 40GB compaction** (160 input SSTs → \~160 output SSTs):
**Baseline compaction operations:**
* **160 SST reads**: Reading input SST files
* **160 SST writes**: Writing output SST files
* **2 manifest updates**: Initial job start + final completion
* **Total baseline**: 322 operations
**With persistence enabled:**
* **160 SST reads**: Input SST files (unchanged)
* **160 SST writes**: Output SST files (unchanged)
* **160 state writes**: Persistence after each output SST (\~256MB intervals)
* **2 manifest updates**: Job lifecycle management
* **Total with persistence**: 482 operations
**Overhead calculation:**
* **Additional operations**: 160 (482 - 322)
* **Percentage increase**: +50% operations
* **Operations per GB**: \~4.0 additional ops/GB (160 ÷ 40GB)
#### Cloud Cost Analysis
[Section titled “Cloud Cost Analysis”](#cloud-cost-analysis)
Using **AWS S3 Standard** pricing:
* **PUT operations**: $0.0005 per 1,000 requests
* **GET operations**: $0.0004 per 1,000 requests
* **DELETE operations**: $0.0005 per 1,000 requests
**Baseline costs:**
* **160 PUTs** (SST writes): $0.000080
* **160 GETs** (SST reads): $0.000064
* **2 PUTs** (manifest): $0.000001
* **Total baseline**: $0.000145
**Additional persistence costs:**
* **160 PUTs** (state writes): $0.000080
* **Additional total**: $0.000080
**Cost impact:**
* **Additional cost**: $0.000080 (\~$0.00008)
* **Percentage increase**: +50% operations, but negligible absolute cost
* **Cost per GB**: \~$0.000002 per GB compacted
### Recovery Efficiency Analysis
[Section titled “Recovery Efficiency Analysis”](#recovery-efficiency-analysis)
#### Work Preservation Calculation
[Section titled “Work Preservation Calculation”](#work-preservation-calculation)
**Without persistence:**
* **Recovery strategy**: Restart compaction from beginning
* **Work preserved**: 0% (all progress lost)
* **Additional operations**: Full re-execution (322 operations repeated)
* **Time impact**: 100% of original compaction time
**With persistence:**
* **Average failure point**: 50% through compaction (statistical)
* **Work preserved**: \~50% of progress maintained
* **Recovery operations**: Resume from last checkpoint
* **Time impact**: \~50% of original compaction time saved
**Detailed recovery scenarios:**
| Failure Point | Without Persistence | With Persistence | Work Preserved | Operations Saved(work preserved percentage \* 322) |
| ---------------- | ---------------------- | ------------------------------- | -------------- | -------------------------------------------------- |
| **10% complete** | Restart (0% preserved) | Resume from 10% (10% preserved) | 10% | 32 operations |
| **25% complete** | Restart (0% preserved) | Resume from 25% (25% preserved) | 25% | 81 operations |
| **50% complete** | Restart (0% preserved) | Resume from 50% (50% preserved) | 50% | 161 operations |
| **75% complete** | Restart (0% preserved) | Resume from 75% (75% preserved) | 75% | 242 operations |
| **90% complete** | Restart (0% preserved) | Resume from 90% (90% preserved) | 90% | 290 operations |
**Average work preservation**: **50%** across all failure scenarios
#### Scaling Analysis
[Section titled “Scaling Analysis”](#scaling-analysis)
| Compaction Size | SSTs | Base Ops | +Persistence | Additional Cost | % Overhead |
| -------------------- | ----- | -------- | ------------ | --------------- | ---------- |
| **10GB** (40 SSTs) | 40 | 82 | 122 | $0.000020 | +49% |
| **40GB** (160 SSTs) | 160 | 322 | 482 | $0.000080 | +50% |
| **100GB** (400 SSTs) | 400 | 802 | 1,202 | $0.000200 | +50% |
| **1TB** (4,000 SSTs) | 4,000 | 8,002 | 12,002 | $0.002000 | +50% |
**Key observations:**
* **Operations increase by \~50%**, but absolute costs remain minimal
* **Cost scales linearly** with compaction size (\~$0.000002/GB)
* **Percentage overhead is consistent** at \~50% across all sizes
* **Total costs are negligible** compared to storage and compute costs
## Future Extensions
[Section titled “Future Extensions”](#future-extensions)
* Persistent state provides foundation for multi-compactor coordination and work distribution.
* Define a minimum time boundary between compaction file updates to prevent excessive writes to the file (see )
* Add last\_key to SST metadata to enable efficient range-based SST filtering during compaction source selection and range query execution.
# Persistence format for the write-ahead log (WAL)
Status: **Rejected**
In the discussion, we realized that having blocks that can be loaded separately makes also sense for the WAL. The separately loadable blocks need indices in the file format that point to the blocks. If we add blocks and indices to a new WAL format, the new format would be really similar to the existing SST format since also the new WAL format would need versioning and metadata similar to the one in the existing SST format. The only component of the existing SST format that is not needed for the WAL format is the filter. Not encoding the filter is allowed in the existing SST format when there are fewer keys in the SST than set in config `min_filter_keys`. Thus, not encoding filters for the WAL would also not justify a new format. Additionally, it is not impossible that filters for WAL files might be interesting for future use cases. Given all of these arguments, we decided to reject this RFC about a new persistence format for the WAL and re-use the existing SST format instead.
Authors:
* [Bruno Cadonna](https://github.com/cadonna)
[]()
## Summary
[Section titled “Summary”](#summary)
This RFC proposes a dedicated persistence format for the write-ahead log (WAL). The new format tries to minimize the overhead when adding incoming records to WAL and flushing the buffered WAL to object store. Tightly bound to the new persistence format is an in-memory data structure for batching and flushing WAL records. This RFC only specifies the persistence format and some desirable properties of the in-memory data structure. No exact implementation of the in-memory data structure is provided, so that the in-memory structure can be modified without the need of new RFC.
[]()
## Motivation
[Section titled “Motivation”](#motivation)
Currently, the WAL is buffered in memory in a `KVTable`. That is the same data structure that is used for the memtable. The `KVTable` is a skip map — an ordered map based on a skip list — that keeps all incoming key-value pairs ordered by key. While this is beneficial for the memtable for reading and flushing, it adds unnecessary overhead for the WAL. The WAL merely needs to maintain records in the order they are ingested to ensure durability and recovery.
In object storage, the WAL is stored as a sorted string table (SST). Encoding the SST from a `KVTable` also implies overhead that is unnecessary for the WAL. For example, filters and indices are not needed for a WAL object that stores records in the same order they were ingested and also reads those records sequentially in that order.
Separating the format of the WAL from the format of the SST allows the WAL format to model sequential writes in lock-step with the monotonically increasing sequence number. Sequence numbers are assigned to each ingested record according to the write snapshot they belong to. Having a WAL format model sequential writes is an advantage when records need to be read in the same order they were ingested. This kind of sequential read is the main read pattern of a write-ahead *log* by definition. The most prominent example for this sequential read pattern is the recovery of a database state. With the current SST format that is also used to store WAL objects, the records are sorted by key not by the sequence number making sequential reads by sequence number costly.
For these reasons, the WAL can be implemented with a first-in first-out (FIFO) data structure in memory and with a persistence format that sequentially stores and loads records without any specific indices or filters.
The overhead of using the skip map and the SST format for the WAL can be seen in the two flamegraphs included in GitHub issue [#1085](https://github.com/slatedb/slatedb/issues/1085). The overhead hinders higher ingest throughput into SlateDB.
[]()
## Goals
[Section titled “Goals”](#goals)
* Specify a queue-like format for WAL objects that can be flushed with minimal overhead.
[]()
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Specify the implementation of the in-memory data structure. Merely, some desirable properties will be mentioned.
[]()
## Design
[Section titled “Design”](#design)
[]()
### Format
[Section titled “Format”](#format)
The proposed new persistence format for the WAL objects starts with a list of variable-length records.
A record consists of:
* a sequence number represented by an 8-bytes unsigned integer,
* 8 flags in a 1-byte unsigned integer,
* optional the creation timestamp and the expiration timestamp of the record, each represented by 8 bytes signed integer,
* key length as a 2-bytes unsigned integer followed by the number of bytes set in the key length containing the actual key,
* value length as a 4-bytes unsigned integer followed by the number of bytes set in the value length containing the actual value.
All the integers are in little endian. The flags specify the type of the record and whether the records contain the expiration and/or creation timestamp. The timestamps are milliseconds since the unix epoch.
Record:
```plaintext
+----------------------------------------------------------------+
| sequence number (8-bytes unsigned integer, little endian) |
+----------------------------------------------------------------+
| flags (1-byte unsigned integer, little endian) |
+----------------------------------------------------------------+
| create_ts (8-bytes signed integer, little endian) |
+----------------------------------------------------------------+
| expire_ts (8-bytes signed integer, little endian) |
+----------------------------------------------------------------+
| key length (2-bytes unsigned integer, little endian) |
+----------------------------------------------------------------+
| key (variable length) |
+----------------------------------------------------------------+
| value length (4-bytes unsigned integer, little endian) |
+----------------------------------------------------------------+
| value (variable length) |
+----------------------------------------------------------------+
```
Flags:
```plaintext
b_0, b_1 = (0, 0) if the record is a value,
(0, 1) if the record is a tombstone,
(1, 0) if the record is a merge operand,
(1, 1) reserved
b_2 = 1 if the record has a creation timestamp, 0 otherwise
b_3 = 1 if the record has an expiration timestamp, 0 otherwise
b_4 - b_7 = 0 (reserved)
```
If the record is a tombstone, the value length and the actual values are omitted.
The list of records is followed by a compressed list of offsets, where each offset points to one record. Since the compressed list of offsets might be of variable length, it is followed by the size of the compressed list as a 4 bytes unsigned integer. After the size of the compressed list, the used compression codec is specified as a 1 byte unsigned integer. Initially, the available compression codecs are no compression and a delta encoding that still needs to be determined/specified. The record offsets in the list have the same order as the records. That is, the 0th offset points to the 0th record, the 1st offset points to the 1st record in the record list, and so on. Then, the format stores the CRC32 checksum of the records and the list of offsets including the size and the compression codecs as a 4 bytes unsigned integer. The specification of the compression codec for the offset list is followed by the specification of the compression codec used to compress records and offsets as a 1 byte unsigned integer. All the fields from the start to the compression codec for the offset list, inclusive are compressed with that compression codec. The available compression codecs for records and offsets are no compression, snappy, zlib, lz4, and zstd. The last two fields are the version of the format as a 2 bytes unsigned integer, and the type of the object as a 1 byte unsigned integer — 0x00 for `WAL` in this case. Future formats of data objects in SlateDB should use the same schema at the end of the footer to specify its type and the version of its format. All integers are in little endian.
```plaintext
+----------------------------------------------------------------+
| record 0 (variable length) |
+----------------------------------------------------------------+
| record 1 (variable length) |
+----------------------------------------------------------------+
| ... |
+----------------------------------------------------------------+
| record N (variable length) |
+----------------------------------------------------------------+
| compressed array of N record offsets |
| (variable length, |
| before compression each offset is |
| 8-bytes unsigned integer, little endian) |
+----------------------------------------------------------------+
| size of the compressed array of offsets |
| (4-bytes unsigned integer, little endian) |
+----------------------------------------------------------------+
| compression codec offsets |
| (1-byte unsigned integer, little endian) |
+----------------------------------------------------------------+
| CRC32 checksum (4-bytes, unsigned integer, little endian) |
+----------------------------------------------------------------+
| compression codec for offsets + records + checksum |
| (1-byte, unsigned integer, little endian) |
+----------------------------------------------------------------+
| version of format (2-bytes, unsigned integer, little endian) |
+----------------------------------------------------------------+
| type of object (1-byte, unsigned integer, little endian) |
+----------------------------------------------------------------+
```
Ideally, the in-memory data structure for the WAL stores the incoming records in ingestion order, so that the records do not need to be re-ordered before the flush. A simple FIFO data structure like a queue is recommended.
[]()
### Serialization
[Section titled “Serialization”](#serialization)
Records are serialized into the record format. All needed data to serialize the records into the format is present in the records. The serialized bytes of the records are added one after the other in ingestion order into the WAL object. For each serialized record, the offset from the beginning of the WAL object to the start of the serialized record is maintained in a list. Once all serialized records that should be stored in the WAL object are added, the list of offsets is compressed with a codec and the compressed list is added to the WAL object. Then the size of the compressed list is added to the WAL object followed by the identifier of the used compression codec. A CRC32 checksum is computed over the records, the offsets, the size of the offsets, as well as the compression codec and the checksum is added to the WAL object. Finally, the version of the format (i.e., `1`) and the type of the object (i.e., `0x00`) are added to the WAL object.
[]()
### Deserialization
[Section titled “Deserialization”](#deserialization)
The WAL object is read from the end. First, the type of the object and the version are deserialized and verified to know whether the object is indeed a WAL object and with what version the WAL object was serialized. Then, the CRC checksum is verified that the WAL object is valid. Next, the compression codec identifier for the list of offsets is deserialized. After that, the size of the compressed list of offsets is deserialized. The size is subtracted from the current offset of the WAL object to find the offset of the WAL object where the compressed list starts. The bytes from the starting point to the field that holds the size are decompressed with the compression codec found in the WAL object in the previous step. Now, all information are available to sequentially read and deserialize all records.
[]()
## Impact Analysis
[Section titled “Impact Analysis”](#impact-analysis)
SlateDB features and components that this RFC interacts with. Check all that apply.
[]()
### Core API & Query Semantics
[Section titled “Core API & Query Semantics”](#core-api--query-semantics)
* [ ] Basic KV API (`get`/`put`/`delete`)
* [ ] Range queries, iterators, seek semantics
* [ ] Range deletions
* [ ] Error model, API errors
[]()
### Consistency, Isolation, and Multi-Versioning
[Section titled “Consistency, Isolation, and Multi-Versioning”](#consistency-isolation-and-multi-versioning)
* [ ] Transactions
* [ ] Snapshots
* [ ] Sequence numbers
[]()
### Time, Retention, and Derived State
[Section titled “Time, Retention, and Derived State”](#time-retention-and-derived-state)
* [ ] Logical clocks
* [ ] Time to live (TTL)
* [ ] Compaction filters
* [ ] Merge operator
* [ ] Change Data Capture (CDC)
[]()
### Metadata, Coordination, and Lifecycles
[Section titled “Metadata, Coordination, and Lifecycles”](#metadata-coordination-and-lifecycles)
* [ ] Manifest format
* [ ] Checkpoints
* [ ] Clones
* [ ] Garbage collection
* [ ] Database splitting and merging
* [ ] Multi-writer
[]()
### Compaction
[Section titled “Compaction”](#compaction)
* [ ] Compaction state persistence
* [ ] Compaction filters
* [ ] Compaction strategies
* [ ] Distributed compaction
* [ ] Compactions format
[]()
### Storage Engine Internals
[Section titled “Storage Engine Internals”](#storage-engine-internals)
* [x] Write-ahead log (WAL)
* [ ] Block cache
* [ ] Object store cache
* [ ] Indexing (bloom filters, metadata)
* [ ] SST format or block format
[]()
### Ecosystem & Operations
[Section titled “Ecosystem & Operations”](#ecosystem--operations)
* [ ] CLI tools
* [ ] Language bindings (Go/Python/etc)
* [ ] Observability (metrics/logging/tracing)
[]()
## Operations
[Section titled “Operations”](#operations)
[]()
### Performance & Cost
[Section titled “Performance & Cost”](#performance--cost)
* **Latency (reads/writes/compactions):** The latency for writes with durability guarantee mainly depends on the configured flush interval (i.e., `flush_interval`) that is used to decide when the in-memory WAL data structure is flushed to object storage. The latency of writes might be less since fewer bytes need to be flushed to object store with the new WAL format. However, if there is a difference in latency, I expect that the difference will not be significant, in general. The latency for writes without durability guarantee might be lower if the in-memory data structure is a simple FIFO data structure as recommended in this RFC, since writes to the in-memory WAL are faster. A WAL object is read for recovery. If we define the latency of recovery as the time until the in-memory state is re-established then the new persistence format might decrease the latency of recovery because the records in the WAL do not need to be re-sorted according to the sequence number. The latency of compaction is not affected by the new persistence format.
* **Throughput (reads/writes/compactions):** Since the new persistence format has less space overhead without indices and filters, a WAL object can be sent faster to object storage for large WAL objects (small object are dominated by the first-byte latency). Thus, with the new format SlateDB can write more WAL records per time unit to object storage compared to the current format. The difference might not be significant, though. Additionally, the new persistence format might allow to recover more records per time unit since the records can be replayed faster (no re-sorting) and fewer bytes need to read from object storage (no filter and index blocks). The throughput of compaction is not affected by the new persistence format.
* **Object-store request (GET/LIST/PUT) and cost profile:** If the WAL is stored on S3 Express to reduce latency, the new format reduces transfer costs (i.e. data uploads, currently $0.0032 per GB).
* **Space, read, and write amplification:** All three amplifications are reduced.
[]()
### Observability
[Section titled “Observability”](#observability)
* Configuration changes: No
* New components/services: No
* Metrics: No
* Logging: No
[]()
### Compatibility
[Section titled “Compatibility”](#compatibility)
* Existing data on object storage / on-disk formats
* WAL objects in the old format are stored in the `\wal` directory and have the extension `.sst`.
* WAL objects in the new format are also stored in the `\wal` directory, but have the extension `.wal`.
* SlateDB decides which codec to use according to the extension of the WAL objects.
* SlateDB will only write WAL objects in the new format.
* Existing public APIs (including bindings): No
* Rolling upgrades / mixed-version behavior (if applicable)
* Clean shutdown (ideal rolling upgrade behavior): SlateDB flushed all data in the WAL to a L0 SST on object storage. No WAL object needs to be replayed. New WAL objects are written in the new WAL format with extension `.wal`.
* Dirty shutdown (erroneous rolling upgrade behavior): SlateDB did not flush all data in the WAL to a L0 SST on object storage. WAL objects need to be replayed. If SlateDB finds WAL objects with extension `.sst` it will decode them with the old format. New WAL objects are written in the new WAL format with extension `.wal`.
[]()
## Testing
[Section titled “Testing”](#testing)
* Unit tests:
* Adapt unit existing unit tests and add new unit tests where needed.
* Integration tests:
* Adapt existing unit tests if needed,
* Write integration tests for migration from the old to the new persistence format
* Fault-injection/chaos tests: No
* Deterministic simulation tests: No
* Formal methods verification: No
* Performance tests: Yes, to proof the improvements over the flamegraphs in [#1085](https://github.com/slatedb/slatedb/issues/1085)
[]()
## Rollout
[Section titled “Rollout”](#rollout)
* Milestones / phases:
* Develop new WAL format alongside the old format, guarded by an experimental flag
* Run performance tests and make adjustments
* Update docs and switch to the new format by default
* Remove obsolete code and experimental flag
* Feature flags / opt-in: No feature flag, new WAL format will become the default
* Docs updates:
* Update design documentation in SlateDB where the WAL is mentioned
* Check remaining documentation for mentions of the WAL
[]()
## Alternatives
[Section titled “Alternatives”](#alternatives)
* By keeping the current WAL format we would not fix the performance issues seen in the flamegraph in GitHub issue [#1085](https://github.com/slatedb/slatedb/issues/1085)
* Designing the new WAL format as version 2 of the SST format would blur the distinction between WAL objects that are used for writing and durability and SST objects that are optimized for reading the data.
* Using sizes of records instead of the offsets of records in the new WAL format would complicate the possible future addition of indices to the records in the WAL (e.g. for multi-writer support).
* Adding a configuration for the maximal size of the WAL instead of using the size of the L0 SSTs would increase the surface of the configuration without obvious benefits.
* Adding a magic number to the WAL format to distinguish it from files external to SlateDB did not seem to be necessary since the likelihood of having external files in SlateDB directories seems low.
* Coding the sequence number as a delta of a minimum sequence number instead of writing the full 8 bytes number in each record would save space, but it would not be straight-forward. All records in the same write batch have the same sequence number and a write batch can contain an arbitrary number of records. To use deltas of the sequence numbers, we need to maintain information about write batch boundaries which seemed not worth the effort.
[]()
## Open Questions
[Section titled “Open Questions”](#open-questions)
No
[]()
## References
[Section titled “References”](#references)
* GitHub issue [#1085](https://github.com/slatedb/slatedb/issues/1085)
[]()
## Updates
[Section titled “Updates”](#updates)
Log major changes to this RFC over time (optional).
# Custom Comparators
Status: Rejected
Authors:
* [Almog Gavra](https://github.com/agavra)
> \[!CAUTION] This RFC was rejected. See the [Rejection Analysis](#rejection-analysis) for more details.
[]()
## Summary & Motivation
[Section titled “Summary & Motivation”](#summary--motivation)
This RFC proposes a new API for customizing the ordering of keys in SlateDB. The ordering of keys in Slate is a critical component of the performance characteristics of the database and its ability to efficiently serve range queries.
The current implementation of SlateDB uses a lexicographic ordering of keys. This is a good default for many use cases, but it may not always be the best choice. Some examples of use cases where a different ordering is desirable include:
* Correct semantics for non-string keys (e.g. sorting floating point numbers)
* Sorting composite keys (e.g. a composite key of `[user_id, timestamp]` may sort ascending by `user_id` and descending by `timestamp`)
* Domain-specific ordering (e.g. Z-ordering for geographic data)
[]()
## Goals
[Section titled “Goals”](#goals)
* Provide a new API for customizing the ordering of keys in SlateDB
* Maintain backwards compatibility by defaulting to the existing lexicographic ordering
* Best-effort enforcing of the ordering invariant (e.g. a comparator doesn’t change after first deployment)
[]()
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Providing default implementations for non-lexicographic use cases
* Supporting changes to the ordering invariant after first deployment
[]()
## Design
[Section titled “Design”](#design)
[]()
### API
[Section titled “API”](#api)
We will introduce a new interface for customizing the ordering of keys in SlateDB, the `KeyComparator` trait:
```rust
use std::cmp::Ordering;
use std::sync::Arc;
/// Custom key comparison trait.
/// WARNING: Changing comparators on existing DB causes corruption.
pub trait KeyComparator: Send + Sync + std::fmt::Debug {
/// Compare two keys.
fn compare(&self, a: &[u8], b: &[u8]) -> Ordering;
/// Unique identifier stored in manifest for validation.
fn name(&self) -> &str;
}
pub type KeyComparatorType = Arc;
```
This will be passed in when constructing a new `Db` instance with the `DbBuilder`:
```rust
let db = DbBuilder::new(path)
// ...
.with_key_comparator(key_comparator)
.build();
```
[]()
### Manifest
[Section titled “Manifest”](#manifest)
We will introduce a new field to the manifest format to store the name of the comparator:
```proto
table ManifestV1 {
// ...
key_comparator_name: string;
}
```
This field will be used to validate that the comparator is not changed after first deployment. if it is set in the manifest, a comparator with the same name must be used to create the database in runtime (no comparator will also cause a panic). If it is not set and the manifest already exists, the database will panic if a custom comparator is supplied at runtime.
[]()
### Implementation
[Section titled “Implementation”](#implementation)
We’ll need to modify each of these places to use the new `KeyComparator` passed in from the `DbBuilder`. Note that because it’s a trait, we’ll need to use dynamic dispatch to call the comparator.
> \[!NOTE] Since comparisons happen on a hot path, we may want to avoid using the `dyn KeyComparator` if not necessary to avoid the overhead of dynamic dispatch. Instead, we can choose to use an `Option` to store the comparator and inline the default lexicographic comparator if not set.
>
> The alternative would be to always use the `dyn KeyComparator` and just implement a default implementation of it. This would be simpler, but it would add a small overhead to every comparison.
The following table was generated via an agent, but it seems to be accurate as far as I can tell:
| Location | Usage |
| ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `slatedb/src/bytes_range.rs:8` `slatedb/src/comparable_range.rs:8` | `BytesRange` and its `ComparableRange` backing types rely on the default `Ord` implementation for `Bytes` when comparing start/end bounds (via `cmp_bound`). Their logic powers `range.contains`, `intersect`, and conversions that every range-aware component (memtables, SST views, sorted runs, iterator seek validation) uses. |
| `slatedb/src/mem_table.rs:36,189` `slatedb/src/batch.rs:153,409` | `SequencedKey::cmp` keeps the memtable `SkipMap` and write-batch `BTreeMap` ordered by ascending user key and descending sequence number. `MemTableIterator::seek` and `WriteBatchIterator::seek` compare `RowEntry.key` / `key.user_key` to `next_key`, and `WriteBatch::remove_ops_by_key` issues `start..=end` range deletes that depend on this ordering. |
| `slatedb/src/merge_iterator.rs:21-119` | `MergeIterator` keeps a binary heap ordered by `RowEntry.key.cmp` (and reverse seq on ties) and short-circuits child seeks when `next_kv.key >= next_key`. This governs how memtable/WAL/SST iterators are merged for reads and compactions. |
| `slatedb/src/block_iterator.rs:73-135` `slatedb/src/sst_iter.rs:93-552` `slatedb/src/partitioned_keyspace.rs:13-80` | Block iterators skip entries with `kv.key < next_key`. Above the block layer, `SstView::contains`/`key_exceeds`, `InternalSstIterator::blocks_covering_view`, and `SstIterator::seek` compare requested keys to SST range bounds and block first-keys (via the partitioned keyspace helpers) to jump to the first block whose contents are `>= next_key`. |
| `slatedb/src/db_state.rs:137-303` `slatedb/src/sorted_run_iterator.rs:176-188` | `SsTableHandle::intersects_range`, `SortedRun::find_sst_with_range_covering_key`, and `table_idx_covering_range` compare SST start keys and requested ranges to decide which files overlap a read/compaction. `SortedRunIterator::seek` then drops whole SSTs whose `first_key < next_key` before seeking inside the current SST iterator. |
| `slatedb/src/db_iter.rs:56-69` `slatedb/src/db_iter.rs:341-357` | `DbIteratorRangeTracker` maintains the min/max key observed via `<`/`>` comparisons for SSI conflict detection, and `DbIterator::seek` enforces that callers only seek forward and stay inside the configured `BytesRange` by comparing `next_key` to `last_key` and the range bounds. |
| `slatedb/src/reader.rs:512-528` | The SST-building helper sorts buffered `RowEntry`s with `a.key.cmp(&b.key)` before emitting a table, modeling the requirement that compaction/flush writers provide keys in comparator order. |
For use of the comparator in the memtable, we will need to modify the `SequencedKey` type to include a reference to the comparator and override the `Ord` implementation to use it. All other locations can use the comparator reference directly.
[]()
## Impact Analysis
[Section titled “Impact Analysis”](#impact-analysis)
SlateDB features and components that this RFC interacts with. Check all that apply.
[]()
### Core API & Query Semantics
[Section titled “Core API & Query Semantics”](#core-api--query-semantics)
* [ ] Basic KV API (`get`/`put`/`delete`)
* [x] Range queries, iterators, seek semantics
* [x] Range deletions
* [ ] Error model, API errors
Range query semantics will be affected by this change since the comparator is used to determine the order of keys in the range.
[]()
### Consistency, Isolation, and Multi-Versioning
[Section titled “Consistency, Isolation, and Multi-Versioning”](#consistency-isolation-and-multi-versioning)
* [x] Transactions
* [ ] Snapshots
* [ ] Sequence numbers
`DbIteratorRangeTracker` maintains the min/max key observed via \ comparisons for SSI conflict detection. Since the comparator is used to determine the order of keys, the min/max key observed will be in the order of the comparator (this should be consistent with the query semantics so the change is mostly transparent to the user).
[]()
### Time, Retention, and Derived State
[Section titled “Time, Retention, and Derived State”](#time-retention-and-derived-state)
* [ ] Logical clocks
* [ ] Time to live (TTL)
* [ ] Compaction filters
* [ ] Merge operator
* [ ] Change Data Capture (CDC)
N/A - the change should not affect time, retention, and derived state.
[]()
### Metadata, Coordination, and Lifecycles
[Section titled “Metadata, Coordination, and Lifecycles”](#metadata-coordination-and-lifecycles)
* [x] Manifest format
* [ ] Checkpoints
* [ ] Clones
* [ ] Garbage collection
* [ ] Database splitting and merging
* [ ] Multi-writer
We will add a new field to the manifest format to store the name of the comparator to serve as a basic validation mechanism that ensures the comparator is not changed after first deployment (this is rudimentary and does not prevent changes if the comparator uses the same name).
[]()
### Compaction
[Section titled “Compaction”](#compaction)
* [x] Compaction state persistence
* [ ] Compaction filters
* [ ] Compaction strategies
* [ ] Distributed compaction
* [ ] Compactions format
Compactions will need to use the comparator to sort the keys in the SSTs.
[]()
### Storage Engine Internals
[Section titled “Storage Engine Internals”](#storage-engine-internals)
* [ ] Write-ahead log (WAL)
* [ ] Block cache
* [ ] Object store cache
* [x] Indexing (bloom filters, metadata)
* [x] SST format or block format
The indexes and SSTs will be sorted by the comparator
[]()
### Ecosystem & Operations
[Section titled “Ecosystem & Operations”](#ecosystem--operations)
* [ ] CLI tools
* [ ] Language bindings (Go/Python/etc)
* [ ] Observability (metrics/logging/tracing)
[]()
## Operations
[Section titled “Operations”](#operations)
[]()
### Performance & Cost
[Section titled “Performance & Cost”](#performance--cost)
There will potentially be a performance impact in situations that perform many comparisons, such as compactions due to the cost of dynamic dispatch. As part of implementation we will benchmark the performance impact and decide whether or not to use the `Option` approach outlined in the implementation section above.
[]()
### Observability
[Section titled “Observability”](#observability)
This will require a new configuration option to set the key comparator.
[]()
### Compatibility
[Section titled “Compatibility”](#compatibility)
Existing deployments will set an empty string for the key comparator name in the manifest the first time it is deployed with a version including this RFC. If the field does not exist but the manifest already does, a custom comparator will not be allowed.
[]()
## Testing
[Section titled “Testing”](#testing)
We should have comprehensive tests to ensure we don’t break the ordering invariant going forward. All main code paths should be covered, in particular queriyng and compactions.
[]()
## Rollout
[Section titled “Rollout”](#rollout)
* Milestones / phases:
* Feature flags / opt-in:
* Docs updates:
[]()
## Alternatives
[Section titled “Alternatives”](#alternatives)
N/A - the only alternative is to not use a custom comparator.
[]()
## Open Questions
[Section titled “Open Questions”](#open-questions)
The biggest option question is whether to use the `Option` approach or the `dyn KeyComparator` approach (see implementation section above).
[]()
## References
[Section titled “References”](#references)
* github.com/slatedb/slatedb/issues/663
* github.com/crossbeam-rs/crossbeam/issues/1180
[]()
## Updates
[Section titled “Updates”](#updates)
[]()
## Rejection Analysis
[Section titled “Rejection Analysis”](#rejection-analysis)
After discussion among maintainers, this RFC was rejected for the following reasons:
1. **Complexity vs. benefit**: Custom comparators add significant implementation complexity (dynamic dispatch on hot paths, validation logic, manifest changes) for marginal benefit (there are alternative ways to structure most keys without custom comparators).
2. **Bug potential**: RocksDB and Pebble have experienced bugs and regressions related to custom comparators. The additional code paths and invariants are difficult to test exhaustively.
3. **Industry precedent**: FoundationDB, DynamoDB, TiKV and CockroachDB intentionally do not expose custom comparators—bytewise comparison only, by design. Users encode keys to sort correctly using their well-documented tuple layer. This approach has proven successful at scale.
4. **Use cases are solvable with encoding**: All motivating use cases can be addressed through proper key encoding:
* Floating points: flip sign bit for positives, flip all bits for negatives
* Composite keys: use `0x00` delimiters or fixed-length components
* Descending order: XOR bytes with `0xFF` or subtract from max value
* Geospatial: Z-order and Hilbert curves encode to sortable integers
5. **Documentation over API complexity**: Providing clear documentation on key encoding patterns (with links to libraries like [storekey](https://github.com/surrealdb/storekey)) is preferable to exposing a complex API that enables subtle data corruption if misused.
See the [Data Modeling documentation](https://slatedb.io/docs/operations/data-modeling) for recommended key encoding patterns.
See the [Github Discussion](https://github.com/slatedb/slatedb/pull/1162) for more details.
# Compaction Filters
Table of Contents:
* [Summary](#summary)
* [Motivation](#motivation)
* [Goals](#goals)
* [Non-Goals](#non-goals)
* [Design](#design)
* [Limitations](#limitations)
* [Impact Analysis](#impact-analysis)
* [Operations](#operations)
* [Testing](#testing)
* [Rollout](#rollout)
* [Alternatives](#alternatives)
* [References](#references)
Status: Accepted
Authors:
* [Hussein Nomier](https://github.com/nomiero)
## Summary
[Section titled “Summary”](#summary)
This RFC introduces a public API for user-provided compaction filters in SlateDB. Users implement a `CompactionFilterSupplier` that creates a `CompactionFilter` instance for each compaction job. Each filter can inspect entries and decide to keep, drop, convert to tombstone, or modify values. The design makes existing internal types (`RowEntry`, `ValueDeletable`) public and uses `CompactionJobContext` to provide context to the filter.
Compaction filters are gated behind the `compaction_filters` feature flag. Enabling this feature may affect snapshot consistency. See [Limitations](#limitations) for details.
## Motivation
[Section titled “Motivation”](#motivation)
SlateDB has no public API for custom compaction filters. Users need this capability for:
1. **Custom TTL Logic**: Application-specific expiration beyond built-in TTL support.
2. **MVCC Garbage Collection**: Custom policies for user-defined versioning.
3. **Schema Migrations**: Data format conversions during compaction.
This design could also unify internal TTL handling (`RetentionIterator`) with user-provided filters in the future.
## Goals
[Section titled “Goals”](#goals)
* Provide a user-friendly API following established SlateDB patterns (`CompactionSchedulerSupplier`).
* Zero overhead when no filter is configured. Existing users are unaffected.
* Design that can potentially be extended to enable internal TTL filtering.
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Replacing the internal `RetentionIterator` immediately.
* Modifying the key bytes of an entry key (filters can only modify values).
* Emitting new entries during compaction (filters can only keep, drop, tombstone, or modify existing entries).
* Guaranteeing snapshot consistency when compaction filters are enabled (see [Limitations](#limitations)).
## Design
[Section titled “Design”](#design)
### Public Types
[Section titled “Public Types”](#public-types)
The following existing internal types are made public to support compaction filters:
```rust
/// Entry in the LSM tree (made public).
pub struct RowEntry {
pub key: Bytes,
pub value: ValueDeletable,
pub seq: u64,
pub create_ts: Option,
pub expire_ts: Option,
}
/// Value that can be a value, merge operand, or tombstone (made public).
pub enum ValueDeletable {
Value(Bytes),
Merge(Bytes),
Tombstone,
}
```
### CompactionJobContext
[Section titled “CompactionJobContext”](#compactionjobcontext)
```rust
/// Context information about a compaction job.
///
/// This struct provides read-only information about the current compaction job
/// to help filters make informed decisions.
pub struct CompactionJobContext {
/// The destination sorted run ID for this compaction.
pub destination: u32,
/// Whether the destination sorted run is the last (oldest) run after compaction.
/// When true, tombstones can be safely dropped since there are no older versions below.
pub is_dest_last_run: bool,
/// The logical clock tick representing the logical time the compaction occurs.
/// This is used to make decisions about retention of expiring records.
pub compaction_clock_tick: i64,
/// Optional minimum sequence number to retain.
///
/// Entries with sequence numbers at or above this threshold are protected by
/// active snapshots. Dropping or modifying such entries may cause snapshot
/// reads to return inconsistent results.
pub retention_min_seq: Option,
}
```
### CompactionFilterDecision
[Section titled “CompactionFilterDecision”](#compactionfilterdecision)
```rust
/// Decision returned by a compaction filter for each entry.
pub enum CompactionFilterDecision {
/// Keep the entry unchanged.
Keep,
/// Drop the entry entirely. The entry will not appear in the compaction output.
///
/// WARNING: Dropping an entry removes it completely without leaving a tombstone.
/// This means older versions of the same key in lower levels of the LSM tree
/// may become visible again ("resurrection"). Only use Drop when you are certain
/// there are no older versions that could resurface, or when that behavior is
/// acceptable for your use case.
Drop,
/// Modify the entry's value.
///
/// Pass `ValueDeletable::Tombstone` to convert the entry to a tombstone.
/// When converting to a tombstone, the entry's `expire_ts` is automatically cleared.
///
/// Pass `ValueDeletable::Value(bytes)` to change the value. Key and other
/// metadata remain unchanged.
///
/// Note: If `Value` is applied to a tombstone, the entry becomes a regular value
/// with the tombstone's sequence number, effectively resurrecting the key.
Modify(ValueDeletable),
}
```
### CompactionFilter Trait
[Section titled “CompactionFilter Trait”](#compactionfilter-trait)
```rust
/// Filter that processes entries during compaction.
///
/// Each filter instance is created for a single compaction job and executes
/// single-threaded on the compactor thread. The filter must be `Send + Sync`
/// to satisfy iterator trait requirements.
#[async_trait]
pub trait CompactionFilter: Send + Sync {
/// Filter a single entry.
///
/// Return `Ok(decision)` to keep, drop, or modify the entry.
/// Return `Err(FilterError)` to abort the compaction job.
///
/// This method is async to allow I/O operations (e.g., checking external
/// services, loading configuration). However, for best performance, prefer
/// doing I/O in `create_compaction_filter()` or `on_compaction_end()` when
/// possible, since this method is called for every entry.
async fn filter(
&mut self,
entry: &RowEntry,
) -> Result;
/// Called after processing all entries.
///
/// Use this hook to flush state, log statistics, or clean up resources.
/// Return `Err` to abort the compaction with an error.
///
/// Note: This is only called after all entries have been processed. If compaction
/// fails due to an earlier error, this method is not invoked.
async fn on_compaction_end(&mut self) -> Result<(), CompactionEndError>;
}
```
### CompactionFilterSupplier Trait
[Section titled “CompactionFilterSupplier Trait”](#compactionfiltersupplier-trait)
```rust
/// Supplier that creates a CompactionFilter instance per compaction job.
///
/// The supplier is shared across all compactions and must be thread-safe (`Send + Sync`).
/// It creates a new filter instance for each compaction job, providing isolated state per job.
#[async_trait]
pub trait CompactionFilterSupplier: Send + Sync {
/// Create a filter for a compaction job. Return Err to abort compaction.
///
/// This is async to allow I/O during initialization (loading config,
/// connecting to external services, etc.) before the filter processes entries.
///
/// # Arguments
///
/// * `context` - Context about the compaction job (destination, clock tick, etc.)
async fn create_compaction_filter(
&self,
context: &CompactionJobContext,
) -> Result, CreationError>;
}
```
### Error Types
[Section titled “Error Types”](#error-types)
```rust
/// Errors that can occur during compaction filter operations.
#[derive(Debug, Error)]
pub enum CompactionFilterError {
/// Filter creation failed in `create_compaction_filter`. This aborts the
/// compaction.
#[error("filter creation failed: {0}")]
CreationError(#[source] Box),
/// Filter failed while processing an entry. This aborts the compaction.
#[error("filter error: {0}")]
FilterError(#[source] Box),
/// Filter failed during `on_compaction_end`. This aborts the compaction.
#[error("compaction end error: {0}")]
CompactionEndError(#[source] Box),
}
```
These error types wrap the underlying cause, preserving error chains for debugging. The `#[source]` attribute enables `std::error::Error::source()` to return the wrapped error.
### Configuration
[Section titled “Configuration”](#configuration)
The `CompactionFilterSupplier` is configured on the component that runs compaction:
```rust
// In DbBuilder (db/builder.rs) - for embedded compactor
pub fn with_compaction_filter_supplier(
mut self,
supplier: Arc,
) -> Self {
self.compaction_filter_supplier = Some(supplier);
self
}
// In CompactorBuilder (db/builder.rs) - for standalone compactor
pub fn with_compaction_filter_supplier(
mut self,
supplier: Arc,
) -> Self {
self.compaction_filter_supplier = Some(supplier);
self
}
```
When running a standalone compactor (separate from the DB writer), user needs to ensure the `CompactorBuilder` is configured with the same `CompactionFilterSupplier` as the `DbBuilder`.
### Background: How SlateDB Compaction Works
[Section titled “Background: How SlateDB Compaction Works”](#background-how-slatedb-compaction-works)
For a comprehensive overview of SlateDB’s compaction design, see [RFC-0002: Compaction](/rfcs/0002-compaction).
SlateDB uses an LSM-tree architecture with two main storage layers:
1. **L0 (Level 0)**: Recently flushed SSTs from the memtable. These may have overlapping key ranges.
2. **Sorted Runs**: Compacted SSTs organized into sorted runs, each containing non-overlapping key ranges. Sorted runs are identified by ID, where lower IDs contain older data.
```plaintext
┌─────────────────────────────────────────────────┐
│ L0 (newest data) │
│ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │SST 4│ │SST 3│ │SST 2│ │SST 1│ │
│ └─────┘ └─────┘ └─────┘ └─────┘ │
├─────────────────────────────────────────────────┤
│ Sorted Runs (compacted, older data below) │
│ │
│ SR 10 (newest) ──► SR 5 ──► SR 0 (oldest) │
└─────────────────────────────────────────────────┘
```
**Compaction** merges entries from multiple sources (L0 SSTs and/or sorted runs) into a single destination sorted run. The compaction executor processes entries **one at a time** through an iterator pipeline.
The core compaction loop in `execute_compaction_job()` is straightforward:
```rust
while let Some(kv) = all_iter.next_entry().await? {
current_writer.add(kv).await?;
// ... handle SST size limits, progress reporting ..etc.
}
```
Each call to `next_entry()` retrieves the next entry from the iterator pipeline, which handles merging, deduplication, and filtering.
### Iterator Stack Integration
[Section titled “Iterator Stack Integration”](#iterator-stack-integration)
The compaction executor builds an iterator pipeline in `load_iterators()`. Each layer wraps the previous one, processing entries as they flow through:
```plaintext
MergeIterator (L0 + SortedRuns)
-> MergeOperatorIterator (resolve merge operands)
-> RetentionIterator (TTL, snapshot retention, tombstone cleanup)
-> CompactionFilterIterator (user filters)
```
**What each iterator does:**
1. **MergeIterator**: Combines entries from all input sources (L0 SSTs and sorted runs) into a single sorted stream. When the same key appears in multiple sources, entries are ordered by sequence number (newest first). This is where the actual “merge” in merge-sort happens.
2. **MergeOperatorIterator**: Resolves merge operands. If the user of the database uses a `MergeOperator`, this iterator combines consecutive merge operands into a single resolved value.
3. **RetentionIterator**: Applies built-in retention policies:
* Drops expired entries (TTL).
* Removes old versions not needed by snapshots.
* Cleans up tombstones at the bottommost level.
4. **CompactionFilterIterator** (this RFC): Applies user-provided filters. This is where `CompactionFilter::filter()` method is called for each entry.
This ordering ensures:
1. Merge operands are resolved before filtering.
2. Expired entries and old versions are already removed.
3. User filters only see “live” entries that would otherwise be written.
### Limitations
[Section titled “Limitations”](#limitations)
#### Feature Flag and Snapshot Consistency
[Section titled “Feature Flag and Snapshot Consistency”](#feature-flag-and-snapshot-consistency)
Compaction filters are gated behind the `compaction_filters` feature flag:
```toml
[dependencies]
slatedb = { version = "...", features = ["compaction_filters"] }
```
Enabling this feature may affect snapshot consistency.
**Why?**
Protecting snapshot data from arbitrary user filters adds significant complexity. Not all use cases require snapshot consistency guarantees, so we start simple with a feature flag to ensure users understand the trade-offs. This design can evolve if new use cases emerge that require snapshot protection.
RocksDB faced the same challenge and [removed snapshot protection from compaction filters in v6.0](https://github.com/facebook/rocksdb/wiki/Compaction-Filter), noting “the feature has a bug which can’t be easily fixed.”
When using compaction filters with snapshots, be aware that:
* Filters may modify or drop entries that snapshots expect to see
* Snapshot reads may return unexpected results if the filter altered the data
* Users who need consistent snapshots should carefully consider their filter logic
### Potential for Internal TTL Unification
[Section titled “Potential for Internal TTL Unification”](#potential-for-internal-ttl-unification)
The `CompactionFilter` trait is designed to be general enough that internal TTL filtering could potentially be refactored to use the same abstraction. However, the current `RetentionIterator` buffers all versions of a key before applying retention policies (e.g., keeping boundary values for snapshot consistency). Unifying these would require either refactoring `RetentionIterator` to work entry-by-entry, or extending the filter API to receive all versions of a key at once.
### Error Handling
[Section titled “Error Handling”](#error-handling)
| Method | Error Type | Behavior |
| ---------------------------- | -------------------- | --------------------- |
| `create_compaction_filter()` | `CreationError` | Aborts compaction job |
| `filter()` | `FilterError` | Aborts compaction job |
| `on_compaction_end()` | `CompactionEndError` | Aborts compaction job |
Creating a fresh filter instance per compaction provides:
1. **Isolation**: No shared mutable state between compaction jobs.
2. **Single-threaded execution**: Filter runs on the same thread as the compactor, no synchronization needed.
3. **State tracking**: Filters can safely accumulate statistics or state across all entries in a compaction.
4. **Simplified reasoning**: No concurrent access concerns within a filter.
### Performance Considerations
[Section titled “Performance Considerations”](#performance-considerations)
The `filter()` method is called for every entry during compaction. While the method is async to allow I/O when needed, frequent I/O per entry can significantly impact compaction throughput. For best performance:
* **Prefer batching I/O**: Load configuration or external state in `create_compaction_filter()` rather than per-entry in `filter()`.
* **Cache decisions**: If checking an external service, cache results to avoid repeated calls for similar entries.
**For CPU-intensive filters:**
If your filter performs expensive synchronous computation (e.g., complex parsing, cryptographic operations), consider using a dedicated compaction runtime to prevent blocking your application’s main runtime:
```rust
let compaction_runtime = tokio::runtime::Builder::new_multi_thread()
.worker_threads(2)
.thread_name("compaction")
.build()?;
let db = Db::builder("mydb", object_store)
.with_compaction_runtime(compaction_runtime.handle().clone())
.with_compaction_filter_supplier(Arc::new(MyCpuIntensiveFilter))
.build()
.await?;
```
### Usage Example
[Section titled “Usage Example”](#usage-example)
```rust
use slatedb::{
CompactionFilter, CompactionFilterSupplier, CompactionJobContext,
CompactionFilterDecision, CompactionFilterError, RowEntry, ValueDeletable,
};
use bytes::Bytes;
use std::sync::Arc;
use async_trait::async_trait;
/// A filter that converts all entries with a specific key prefix to tombstones.
struct PrefixDroppingFilter {
prefix: Bytes,
dropped_count: u64,
}
#[async_trait]
impl CompactionFilter for PrefixDroppingFilter {
async fn filter(
&mut self,
entry: &RowEntry,
) -> Result {
if entry.key.starts_with(&self.prefix) {
self.dropped_count += 1;
// Use Tombstone to shadow older versions in lower levels
return Ok(CompactionFilterDecision::Modify(ValueDeletable::Tombstone));
}
Ok(CompactionFilterDecision::Keep)
}
async fn on_compaction_end(&mut self) -> Result<(), CompactionFilterError> {
tracing::info!(
"Compaction dropped {} entries with prefix {:?}",
self.dropped_count,
self.prefix
);
Ok(())
}
}
struct PrefixDroppingFilterSupplier {
prefix: Bytes,
}
#[async_trait]
impl CompactionFilterSupplier for PrefixDroppingFilterSupplier {
async fn create_compaction_filter(
&self,
_context: &CompactionJobContext,
) -> Result, CompactionFilterError> {
Ok(Box::new(PrefixDroppingFilter {
prefix: self.prefix.clone(),
dropped_count: 0,
}))
}
}
// Usage
let db = Db::builder("mydb", object_store)
.with_compaction_filter_supplier(Arc::new(PrefixDroppingFilterSupplier {
prefix: Bytes::from_static(b"temp:"),
}))
.build()
.await?;
```
## Impact Analysis
[Section titled “Impact Analysis”](#impact-analysis)
SlateDB features and components that this RFC interacts with. Check all that apply.
### Core API & Query Semantics
[Section titled “Core API & Query Semantics”](#core-api--query-semantics)
* [ ] Basic KV API (`get`/`put`/`delete`)
* [ ] Range queries, iterators, seek semantics
* [ ] Range deletions
* [ ] Error model, API errors
### Consistency, Isolation, and Multi-Versioning
[Section titled “Consistency, Isolation, and Multi-Versioning”](#consistency-isolation-and-multi-versioning)
* [x] Transactions - **consistency may be affected** when compaction filters are enabled (see [Limitations](#limitations)).
* [x] Snapshots - **consistency may be affected** when compaction filters are enabled (see [Limitations](#limitations)).
* [ ] Sequence numbers
### Time, Retention, and Derived State
[Section titled “Time, Retention, and Derived State”](#time-retention-and-derived-state)
* [ ] Logical clocks
* [x] Time to live (TTL) - built-in TTL runs before user filters; expired entries are already removed
* [x] Compaction filters - this RFC
* [x] Merge operator - filters run after merge resolution
* [ ] Change Data Capture (CDC)
### Metadata, Coordination, and Lifecycles
[Section titled “Metadata, Coordination, and Lifecycles”](#metadata-coordination-and-lifecycles)
* [ ] Manifest format
* [ ] Checkpoints
* [ ] Clones
* [x] Garbage collection - Drop/Tombstone decisions remove entries
* [ ] Database splitting and merging
* [ ] Multi-writer
### Compaction
[Section titled “Compaction”](#compaction)
* [ ] Compaction state persistence
* [x] Compaction filters - this RFC
* [ ] Compaction strategies
* [ ] Distributed compaction
* [ ] Compactions format
### Storage Engine Internals
[Section titled “Storage Engine Internals”](#storage-engine-internals)
* [ ] Write-ahead log (WAL)
* [ ] Block cache
* [ ] Object store cache
* [ ] Indexing (bloom filters, metadata)
* [ ] SST format or block format
### Ecosystem & Operations
[Section titled “Ecosystem & Operations”](#ecosystem--operations)
* [ ] CLI tools
* [ ] Language bindings (Go/Python/etc)
* [ ] Observability (metrics/logging/tracing)
## Operations
[Section titled “Operations”](#operations)
### Performance & Cost
[Section titled “Performance & Cost”](#performance--cost)
* **Latency**: `filter()` is async but called per-entry - minimize I/O in hot path
* **Throughput**: For best performance, batch I/O in `create_compaction_filter()` or cache decisions
* **Object-store requests**: No direct impact; filters operate on in-memory data
* **Space amplification**: `Drop`/`Modify(Tombstone)` decisions reduce space; `Modify(Value)` may increase or decrease.
* **Zero overhead when disabled**: Users who do not configure a filter are not impacted.
Well-implemented filters have minimal overhead on compaction throughput.
### Observability
[Section titled “Observability”](#observability)
* **Metrics**: No new counters.
* **Logging**: Filter errors logged at WARN level.
* **Configuration changes**: New `compaction_filter_supplier` field in Settings.
### Compatibility
[Section titled “Compatibility”](#compatibility)
* **Existing data**: no change.
* **Public APIs**: New optional configuration in `Settings` and `DbBuilder`. `RowEntry` and `ValueDeletable` become public.
* **Rolling upgrades**: not needed.
## Testing
[Section titled “Testing”](#testing)
* **Unit tests**: Each decision type (Keep/Drop/Tombstone/Modify), error handling, lifecycle hooks.
* **Integration tests**: End-to-end compaction with custom filters, verify data correctness.
* **Fault-injection tests**: Filter errors, initialization failures.
* **Deterministic simulation tests**: Include filter behavior in DST.
* **Performance tests**: Benchmark compaction throughput with/without filters.
## Rollout
[Section titled “Rollout”](#rollout)
### Implementation
[Section titled “Implementation”](#implementation)
* Core traits and iterator integration.
* Make `RowEntry` and `ValueDeletable` public.
* Basic tests and documentation.
### Feature Flags
[Section titled “Feature Flags”](#feature-flags)
The `compaction_filters` feature flag gates the `CompactionFilterSupplier` trait. See [Limitations](#limitations) for why this is behind a feature flag.
### Docs Updates
[Section titled “Docs Updates”](#docs-updates)
* Add examples to API documentation.
* Update compaction documentation to describe filter integration point.
## Alternatives
[Section titled “Alternatives”](#alternatives)
### 1. Replace RetentionIterator entirely
[Section titled “1. Replace RetentionIterator entirely”](#1-replace-retentioniterator-entirely)
**Deferred**: Built-in retention handles subtle edge cases (snapshot barriers, merge operand expiration). Could be unified in future.
### 2. Batched filter API (all versions of a key at once)
[Section titled “2. Batched filter API (all versions of a key at once)”](#2-batched-filter-api-all-versions-of-a-key-at-once)
Instead of `filter(entry) -> Decision`, provide `filter(Vec) -> Vec` where all versions of a key are passed together. This would:
* Enable look-ahead logic (matching `RetentionIterator`’s current implementation)
* Allow filters to make decisions based on the full version history
**Deferred**: Adds API complexity and potential allocations. We’re starting simple with entry-at-a-time filtering, which covers most use cases. The API can evolve if batched filtering becomes necessary.
### 3. Single filter instance (no factory/supplier)
[Section titled “3. Single filter instance (no factory/supplier)”](#3-single-filter-instance-no-factorysupplier)
Supplier provides isolation between jobs and enables single-threaded execution without synchronization.
### 4. Define custom types instead of using RowEntry
[Section titled “4. Define custom types instead of using RowEntry”](#4-define-custom-types-instead-of-using-rowentry)
Using existing types (`RowEntry`, `CompactionJobContext`) reduces API surface and avoids per-entry allocations for wrapper types.
### 5. Built-in filter chaining
[Section titled “5. Built-in filter chaining”](#5-built-in-filter-chaining)
Users who need multiple filters can implement a single `CompactionFilter` that internally chains multiple filters. This keeps SlateDB simpler while still enabling advanced use cases.
## References
[Section titled “References”](#references)
* [GitHub Issue #225](https://github.com/slatedb/slatedb/issues/225) - Original feature request
* [GitHub PR #224](https://github.com/slatedb/slatedb/pull/224) - Previous draft implementation (closed)
* [Discord Discussion](https://discord.com/channels/1232385660460204122/1461435444934737940) - Design discussion thread
* [RFC-0003: Timestamps and TTL](/rfcs/0003-timestamps-and-ttl) - Built-in TTL implementation
* [RFC-0006: Merge Operator](/rfcs/0006-merge-operator) - Pattern for user-provided operators
* [RocksDB Compaction Filter](https://github.com/facebook/rocksdb/wiki/Compaction-Filter) - Reference implementation
## Updates
[Section titled “Updates”](#updates)
*Log major changes to this RFC over time.*
# Expose Row Information
Status: Approved
Authors:
* [marsevilspirit](https://github.com/marsevilspirit)
[]()
## Summary
[Section titled “Summary”](#summary)
This RFC proposes three related API improvements:
1. **Modify Write API Return Types**: Change the return type of `db.put()`, `db.delete()`, `db.merge()`, and `db.write()` (including their `_with_options` variants) from `Result<(), ...>` to `Result` to return a write handle containing the assigned sequence number and timestamps. This design allows for future extensions such as `await_durability()`.
2. **New Row Query Interface**: Introduce `get_row()` and `get_row_with_options()` for single-key row queries that return `RowEntry`. For scans, add `DbIterator::next_row()` which returns `RowEntry` (including metadata: sequence number, creation timestamp, expiration timestamp). The existing `DbIterator::next()` continues to return `KeyValue` for backward compatibility.
3. **Support Query by Version**: Add a `seqnum` option to `SnapshotOptions`, enabling users to create snapshots at specific sequence numbers and read historical versions of keys.
[]()
## Motivation
[Section titled “Motivation”](#motivation)
In practical applications, users often need to retrieve metadata information for keys. Currently, SlateDB has the following pain points:
**1. Unable to Query Row Information and Metadata**
Users need to obtain metadata such as TTL, sequence number, and creation time, but currently **no API provides this information**—even the `get()` method only returns the value itself.
**2. Lack of Feedback for Write Operations**
Currently, `db.put()` returns `Result<(), Error>`. Callers cannot know the sequence number assigned to the write operation, which is crucial for debugging, auditing, and MVCC scenarios.
**3. Lack of Versioned Query Capability**
Users cannot query a specific historical version of a key using a sequence number, which limits application scenarios related to Multi-Version Concurrency Control (MVCC).
[]()
## Goals
[Section titled “Goals”](#goals)
* Enable retrieval of complete row information including metadata (supporting both single-key queries and range scans).
* Return the sequence number assigned after a write operation (put, delete, merge, or batch write).
* Support reading historical versions of a key by specifying a sequence number.
[]()
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* No change/modification for underlying behavior of any API.
[]()
## Use Cases
[Section titled “Use Cases”](#use-cases)
[]()
### Durability Guarantees
[Section titled “Durability Guarantees”](#durability-guarantees)
Applications requiring strong durability guarantees can use the sequence number to verify that writes have been persisted:
```rust
// Write with durability tracking
let handle = db.put(b"critical_data", value).await?;
let write_seqnum = handle.seqnum();
// All writes with sequence numbers <= write_seqnum are guaranteed to be durable
// Applications can use this for replication, backup, or recovery checkpoints
checkpoint_manager.record_durable_seqnum(write_seqnum).await?;
```
[]()
### TTL Management
[Section titled “TTL Management”](#ttl-management)
Applications can query how long until a key expires using `get_row()`:
```rust
// Query remaining TTL for a key
if let Some(entry) = db.get_row(b"session:abc123").await? {
if let Some(expire_ts) = entry.expire_ts() {
let ttl_seconds = (expire_ts - current_time_micros()) / 1_000_000;
println!("Key expires in {} seconds", ttl_seconds);
}
}
```
[]()
## Design
[Section titled “Design”](#design)
[]()
### 1. Row Query Interface
[Section titled “1. Row Query Interface”](#1-row-query-interface)
> \[!NOTE] This change introduces new row query APIs that reuse the existing `RowEntry` type for returning complete row information. Since decoding metadata requires decoding the entire row, there is no performance penalty for returning the full row data.
**Public API**:
```rust
// Get complete row information for a single key
pub async fn get_row>(&self, key: K) -> Result, crate::Error>;
// Get row with options (new)
pub async fn get_row_with_options>(
&self,
key: K,
options: &ReadOptions,
) -> Result, crate::Error>;
// Scan API (unchanged) - returns DbIterator
pub async fn scan(&self, range: T) -> Result
where
K: AsRef<[u8]> + Send,
T: RangeBounds + Send;
// DbIterator methods
impl DbIterator {
// Returns KeyValue for normal scan usage (default behavior)
pub async fn next(&mut self) -> Result, crate::Error>;
// Returns complete row information including metadata
pub async fn next_row(&mut self) -> Result, crate::Error>;
}
```
> \[!NOTE] **Implementation Note**: Internally, the existing `next_key_value()` method is renamed to `next_row()` and modified to return `RowEntry`. The public `next()` method calls `next_row()` internally, extracts the `KeyValue` from the `RowEntry`, and returns it.
```rust
// RowEntry is already public; fields are pub(crate) with getter methods
pub struct RowEntry {
pub(crate) key: Bytes,
pub(crate) value: ValueDeletable,
pub(crate) seq: u64,
pub(crate) create_ts: Option,
pub(crate) expire_ts: Option,
}
impl RowEntry {
pub fn key(&self) -> &[u8] { &self.key }
pub fn value(&self) -> &ValueDeletable { &self.value }
pub fn seq(&self) -> u64 { self.seq }
pub fn create_ts(&self) -> Option { self.create_ts }
pub fn expire_ts(&self) -> Option { self.expire_ts }
}
// ValueDeletable type definition (unchanged)
pub enum ValueDeletable {
Value(Bytes), // Regular value
Merge(Bytes), // Merge operation value
Tombstone, // Deleted key marker
}
```
> \[!IMPORTANT] **ValueDeletable Behavior in `next_row()`**
>
> While `ValueDeletable` has three variants (`Value`, `Merge`, `Tombstone`), **neither `next()` nor `next_row()` will return entries with `Tombstone`** in practice:
>
> * **Tombstones are filtered out** by the underlying iterator
> * `next()` returns `KeyValue` (extracts the value from `ValueDeletable`, so users don’t need to match on it)
> * `next_row()` returns `RowEntry`, where users will only see `ValueDeletable::Value` or `ValueDeletable::Merge`
> * For merge operations: if a base value exists, returns `Value` with the computed result; otherwise returns `Merge` with the final merge value
**Usage Example**:
```rust
// Get complete row information
let row = db.get_row(b"key").await?;
if let Some(entry) = row {
println!("Key: {:?}", entry.key());
println!("Seq: {}, Created: {:?}, Expires: {:?}",
entry.seq(), entry.create_ts(), entry.expire_ts());
// Handle ValueDeletable variants (Tombstone will never appear here)
match entry.value() {
ValueDeletable::Value(v) => println!("Value: {:?}", v),
ValueDeletable::Merge(m) => println!("Merge value: {:?}", m),
ValueDeletable::Tombstone => unreachable!("get_row() never returns Tombstone"),
}
}
// Normal scan usage - returns KeyValue (no need to match on ValueDeletable)
let mut iter = db.scan(b"a"..b"z").await?;
while let Some((key, value)) = iter.next().await? {
println!("Key: {:?}, Value: {:?}", key, value);
}
// Scan with complete row information - returns RowEntry
let mut iter = db.scan(b"a"..b"z").await?;
while let Some(row_entry) = iter.next_row().await? {
match row_entry.value() {
ValueDeletable::Value(v) => {
println!("Key: {:?}, Value: {:?}, Seq: {}",
row_entry.key(), v, row_entry.seq());
}
ValueDeletable::Merge(m) => {
println!("Key: {:?}, Merge: {:?}, Seq: {}",
row_entry.key(), m, row_entry.seq());
}
ValueDeletable::Tombstone => unreachable!(),
}
}
```
**Implementation Details**:
1. **Add `DbIterator::next_row()` Method**:
* Rename the existing internal `next_key_value()` method to `next_row()` and make it public, with return type `Result, crate::Error>`.
* The public `next()` method calls `next_row()` internally, extracts `KeyValue` from the `RowEntry`, and returns it (maintaining backward compatibility).
* Since decoding metadata requires decoding the entire row, there is no performance penalty for providing both methods.
2. **`RowEntry.value` Behavior**:
* `RowEntry.value` remains as `ValueDeletable` type (no type changes needed).
* Both `next()` and `next_row()` will **never** return entries with `Tombstone` values:
* The underlying iterator already filters out tombstones.
* For **Merge Operations**:
* If merge operator is configured and merges are resolved, returns `ValueDeletable::Value` with the computed result.
* If no base value exists for merges, returns `ValueDeletable::Merge` with the final merge value.
* This is consistent with how `KeyValueIterator::next()` handles merges (see line 42-44 in iter.rs).
* `next()` extracts the value from `ValueDeletable` and returns it as `Bytes` in the `KeyValue` tuple, so normal scan usage doesn’t need to match on `ValueDeletable`.
3. **Implement `get_row()` Using `scan()`**:
* `get_row(key)` internally calls `scan(key..=key)` to get a `DbIterator`.
* Then calls `next_row()` on the iterator to get the `RowEntry`.
* This reuses existing scan logic and avoids code duplication.
4. **Multi-Version Behavior**: Both `next()` and `next_row()` return the **latest visible version** of each key. If a key has multiple versions at different sequence numbers, only the version visible at the current time (or specified snapshot sequence number) is returned.
[]()
### 2. Modify Put/Write Return Types
[Section titled “2. Modify Put/Write Return Types”](#2-modify-putwrite-return-types)
> \[!WARNING] **Breaking Change**: All existing code using `db.put`, `db.delete`, `db.merge`, and `db.write` (and their `_with_options` variants) must be updated to handle the new return type.
**WriteHandle Structure**:
```rust
/// Handle returned from write operations, containing metadata about the write.
/// This structure is designed to be extensible for future enhancements.
pub struct WriteHandle {
/// The sequence number of this entry.
seq: u64,
/// The creation timestamp (if set).
create_ts: Option,
}
impl WriteHandle {
/// Returns the sequence number assigned to this write operation.
pub fn seqnum(&self) -> u64 {
self.seq
}
/// Returns the creation timestamp assigned to this write operation.
pub fn create_ts(&self) -> Option {
self.create_ts
}
// Future extensions can be added here, for example:
// pub async fn await_durability(&self) -> Result<(), Error> { ... }
}
> [!NOTE]
> `WriteHandle` does not return `expire_ts` because each operation in a batch can have a different expiration time (or no expiration). Returning a single `expire_ts` for a batch write would be ambiguous or incorrect. Users who need `expire_ts` should query the row using `get_row()` or `next_row()`.
```
**Single Key Operations**:
```rust
// Old interface
pub async fn put(&self, key: K, value: V) -> Result<(), crate::Error>
pub async fn delete(&self, key: K) -> Result<(), crate::Error>
pub async fn merge(&self, key: K, value: V) -> Result<(), crate::Error>
// New interface. Returns WriteHandle with the assigned sequence number.
pub async fn put(&self, key: K, value: V) -> Result
pub async fn delete(&self, key: K) -> Result
pub async fn merge(&self, key: K, value: V) -> Result
```
**Batch Operations**:
```rust
// Old interface
pub async fn write(&self, batch: WriteBatch) -> Result<(), crate::Error>
// New interface. Returns WriteHandle with the batch commit sequence number.
pub async fn write(&self, batch: WriteBatch) -> Result
```
**Transaction Commit Operations**:
```rust
// Old interface
pub async fn commit(self) -> Result<(), crate::Error>
pub async fn commit_with_options(self, options: &WriteOptions) -> Result<(), crate::Error>
// New interface. Returns WriteHandle with the commit sequence number.
pub async fn commit(self) -> Result
pub async fn commit_with_options(self, options: &WriteOptions) -> Result
```
**Migration Example**:
```rust
// Single key operation - get sequence number and timestamps
let handle = db.put(b"key", b"value").await?;
println!("Seq: {}", handle.seqnum());
println!("Created at: {:?}", handle.create_ts());
// Batch operation
let mut batch = WriteBatch::new();
batch.put(b"key1", b"value1");
batch.put(b"key2", b"value2");
batch.delete(b"key1");
let handle = db.write(batch).await?;
println!("Batch committed at seq: {}", handle.seqnum());
// All operations in the batch share this sequence number and timestamps
// If you need the full RowEntry, use get_row with the seqnum
let row = db.get_row(b"key").await?;
if let Some(entry) = row {
assert_eq!(entry.seq(), handle.seqnum());
assert_eq!(entry.create_ts(), handle.create_ts());
}
// Option 2: Ignore return values (if you don't need the metadata)
let _ = db.put(b"key", b"value").await?;
let mut batch2 = WriteBatch::new();
batch2.put(b"another_key", b"another_value");
let _ = db.write(batch2).await?;
```
**Implementation Details**:
* Modify `DbInner::write_with_options` to return `WriteHandle` (containing the assigned `commit_seq` and `create_ts`).
* `put()`, `delete()`, and `merge()` return the `WriteHandle` received from `DbInner`.
* Both `put`, `delete`, and `merge` operations share the same underlying write pipeline.
* `WriteHandle` contains the sequence number and creation timestamp assigned to the write operation.
* The `create_ts` is assigned at the same time as the `commit_seq` during write batch processing.
* **Note on Transactions**: Within a transaction, the individual write operations (`put`, `delete`, `merge`) do not return `WriteHandle` because sequence numbers are not known during transaction execution. However, `DbTransaction::commit()` and `commit_with_options()` **do** return `WriteHandle`, allowing users to access the commit sequence number and timestamps assigned when the transaction is successfully committed.
[]()
### 3. Support Query by Version
[Section titled “3. Support Query by Version”](#3-support-query-by-version)
Add a `seqnum` field to `SnapshotOptions` to create snapshots at specific sequence numbers:
```rust
pub struct SnapshotOptions {
pub seqnum: Option, // New: create snapshot at specific sequence number
}
impl SnapshotOptions {
pub fn read_at(self, seq: u64) -> Self {
Self { seqnum: Some(seq), ..self }
}
}
// New snapshot API
pub fn snapshot_with_options(&self, options: SnapshotOptions) -> Result;
```
**Usage Example**:
```rust
// Write and get version number
let handle = db.put(b"key", b"value1").await?;
let seq1 = handle.seqnum();
// Subsequent update
db.put(b"key", b"value2").await?;
// Create snapshot at seq1
let snapshot = db.snapshot_with_options(SnapshotOptions::default().read_at(seq1))?;
// Current value
let current_value = db.get(b"key").await?;
assert_eq!(current_value, Some(Bytes::from("value2")));
// Read old version via snapshot
let old_value = snapshot.get(b"key").await?;
assert_eq!(old_value, Some(Bytes::from("value1")));
```
Snapshots created with `seqnum` can be used with all read operations:
**Usage Example**:
```rust
// Write some data
let handle1 = db.put(b"key1", b"value1_v1").await?;
let seq1 = handle1.seqnum();
db.put(b"key2", b"value2_v1").await?;
let handle2 = db.put(b"key1", b"value1_v2").await?;
let seq2 = handle2.seqnum();
db.put(b"key2", b"value2_v2").await?;
// Create snapshot at seq1
let snapshot = db.snapshot_with_options(SnapshotOptions::default().read_at(seq1))?;
// Normal scan at historical version via snapshot
let mut iter = snapshot.scan(b"key1"..=b"key2").await?;
while let Some((key, value)) = iter.next().await? {
// Will return value1_v1 and value2_v1
println!("Key: {:?}, Value: {:?}", key, value);
}
// Scan with complete row information at historical version
let mut iter = snapshot.scan(b"key1"..=b"key2").await?;
while let Some(row_entry) = iter.next_row().await? {
println!("Key: {:?}, Seq: {}, Value: {:?}",
row_entry.key(), row_entry.seq(), row_entry.value());
}
```
**Implementation Details**:
* **Snapshot-Based Versioning**: Extend the existing `Snapshot` mechanism to support creating snapshots at specific sequence numbers via `SnapshotOptions::seqnum`.
* **Version Retention and Availability**:
> \[!IMPORTANT] **Understanding Historical Version Availability**
>
> Historical versions are **not guaranteed to be available indefinitely**. Version retention is currently controlled by:
>
> 1. **Active Snapshots**: Historical versions are retained as long as there are open snapshots that reference them. The compactor uses `db_state.recent_snapshot_min_seq` to determine the minimum sequence number to retain.
>
> 2. **Per-Key Retention**: The `min_seq` is **per-key**, not global:
>
> * If key `k1` is written once with `seq=1` and never updated, it will remain available with `seq=1` indefinitely
> * If key `k2` is written with `seq=2` and later overwritten with `seq=123456`, the old version `(k2, seq=2)` will eventually be compacted away
> * There is no single global `min_seq` that applies to all keys
>
> 3. **Future Enhancement - Time Travel**: A planned feature (tracked in [#1263](https://github.com/slatedb/slatedb/issues/1263)) will allow configuring a retention window (e.g., 24 hours) to guarantee version availability within that time period, regardless of snapshot lifecycle.
>
> **Best Practices**:
>
> * For short-term version queries (e.g., debugging), use `snapshot_with_options()` and keep the snapshot open as needed
> * For long-term archival, use a separate backup mechanism rather than relying on SlateDB’s version history
> * Once time travel is implemented, configure an appropriate retention window for your use case
* **Snapshot Visibility**:
* Creating a snapshot with `seqnum` always succeeds, even if `seqnum < min_seq` or `seqnum > max_seq`
* Read operations on the snapshot use visibility checks:
* If data has been compacted (`seq < min_seq` for that key), it is not visible
* If data has not yet been written (`seq > max_seq`), it is not visible
* If no data satisfies the visibility condition, queries return empty results (`None` or empty iterator)
* **API Integration**:
* `snapshot_with_options(SnapshotOptions::default().read_at(seq))` creates a snapshot view at the specified sequence number
* All read operations on the snapshot (`get`, `scan`, `get_row`) will see data as of that sequence number
* The snapshot can be used with `next()` and `next_row()` methods on iterators to retrieve key-value pairs or complete row information respectively
[]()
## Impact Analysis
[Section titled “Impact Analysis”](#impact-analysis)
**Breaking Changes**:
1. **New Row Query APIs**:
* New APIs introduced: `get_row()`, `get_row_with_options()`, `DbIterator::next_row()`.
* `DbIterator::next()` maintains backward compatibility (still returns `Result, crate::Error>`).
* Internal `next_key_value()` method renamed to `next_row()` and made public, returning `RowEntry`.
* `RowEntry.value` remains as `ValueDeletable` type (no breaking changes to the type itself).
* Neither `next()` nor `next_row()` return `Tombstone` values (tombstones are filtered out by the underlying iterator).
2. **Put/Write Return Type Modification**:
* All `put()`, `put_with_options()`, `delete()`, `delete_with_options()`, `merge()`, `merge_with_options()`, `write()`, and `write_with_options()` calls must be updated to handle the new `Result` return type.
* To access the sequence number, call `.seqnum()` on the returned `WriteHandle`.
* Language bindings (Go/Python) must be updated accordingly. For example, in Go, consider exposing a `WriteHandle` type with a `Seqnum()` method, or alternatively return `(uint64, error)` directly. Note that for `write()`, all operations in the batch share the same sequence number in the current implementation.
**Backward Compatibility**:
* `get_row()`, `get_row_with_options()`, and `DbIterator::next_row()` are new APIs.
* They return the existing `RowEntry` type, which is already public.
* `DbIterator::next()` maintains backward compatibility (still returns `KeyValue`).
* `snapshot_with_options()` is a new API that extends existing snapshot functionality.
* `SnapshotOptions.seqnum` allows creating snapshots at specific sequence numbers.
* No impact on any storage formats (WAL/SST/Manifest).
**Migration Path**:
No migration needed for existing code. `DbIterator::next()` maintains backward compatibility:
```rust
// Existing scan code continues to work unchanged
let mut iter = db.scan(..).await?;
while let Some((key, value)) = iter.next().await? {
// No changes needed
}
// When you need complete row information, use next_row()
let mut iter = db.scan(..).await?;
while let Some(row_entry) = iter.next_row().await? {
// Access metadata: row_entry.seq(), row_entry.create_ts(), row_entry.expire_ts()
match row_entry.value() {
ValueDeletable::Value(v) => {
// Use row_entry.key() and v
}
ValueDeletable::Merge(m) => {
// Use row_entry.key() and m
}
ValueDeletable::Tombstone => unreachable!(),
}
}
// New row query APIs
let row = db.get_row(b"key").await?;
```
[]()
## Testing
[Section titled “Testing”](#testing)
**Unit Tests**:
* Row query: Normal cases, expired keys, tombstones, merge operations, sequence number monotonicity.
* Put/Write return values: Verify correctness of sequence numbers.
* Versioned query: Read historical versions, expired key handling, behavior after Compaction.
**Integration Tests**:
* End-to-end workflow: Combined use of row query and versioned query.
* Versioned query across multiple storage layers.
* Consistency of metadata during concurrent writes.
**Performance Testing**:
* `get_row()` performance: Since it returns complete row information, it decodes both metadata and value. Performance should be similar to existing `get()` operations.
* `scan()` with `next()` returning `KeyValue`: Performance should remain unchanged (backward compatible).
* `scan()` with `next_row()` returning `RowEntry`: Performance should be similar to `next()`, as metadata is already decoded during iteration.
* `get_with_options` with specified snapshot sequence number.
[]()
## Alternatives
[Section titled “Alternatives”](#alternatives)
**1. Dedicated Specialized Interfaces (e.g., `get_ttl`, `get_seqnum`, etc.)**
* **Reason for Rejection**:
1. **API Complexity**: This would rapidly expand the API surface area and increase user learning costs. A single generic interface `get_row` that returns complete row information (including metadata) maintains API simplicity.
2. **Atomicity**: Some use cases require retrieving both the key/value and its metadata atomically. Separate methods would prevent this (unless a snapshot is created). Returning the complete `RowEntry` naturally supports atomic access.
**2. Introduce `put_with_metadata()` Method**
* **Reason for Rejection**: Introducing a new API would lead to functional overlap. Modifying the existing `put()` return type, while a breaking change, maintains architectural simplicity and consistency.
**3. Return Only Metadata (RowMetadata) Instead of Complete Row (RowEntry)**
* **Reason for Rejection**: Since decoding metadata requires decoding the entire row, returning only metadata would not improve performance. Returning the complete `RowEntry` provides more value to users while utilizing an existing public type, reducing API complexity.
**4. No Change (Status Quo)**
* **Problem**: Users cannot access physically stored metadata (TTL, sequence number, etc.), limiting important scenarios like debugging, auditing, and MVCC implementations.
# Streaming and Change Data Capture (CDC)
Status: Accepted
Authors:
* [Chris Riccomini](https://github.com/criccomini)
## Summary
[Section titled “Summary”](#summary)
This RFC proposes change data capture (CDC) support in SlateDB. A `WalReader` struct is proposed, which allows to poll for available WAL files and read their contents. Users can poll repeatedly to assemble a stream of changes to the database. Bootstrapping and backfilling are outside the scope of this RFC, however some basic guidelines are discussed in future work.
## Motivation
[Section titled “Motivation”](#motivation)
CDC is the practice of capturing changes to data in a database. Change events are streamed to CDC consumers in write-order. CDC event consumers use the CDC feed to replicate data to other systems, trigger downstream processes, maintain an audit log, and so on. Many databases provide first-class CDC support, such as PostgreSQL’s logical replication, MySQL’s binlog replication, and MongoDB’s change streams. SlateDB CDC support will allow users to more easily integrate SlateDB into their data ecosystem.
### Background
[Section titled “Background”](#background)
CDC pipelines typically have two phases:
1. **Bootstrap / backfill**: initialize a consumer by reading the existing dataset (often as a consistent snapshot).
2. **Streaming**: continuously consume new mutations as they occur.
The key challenge is ensuring the cutover from snapshot → stream is correct (no missed data and, ideally, no duplicates). Most systems solve this by choosing a **high-watermark** in a database’s durability/replication log (LSN, binlog position, oplog timestamp, etc.), and then coordinating the snapshot and stream around that boundary. Common patterns include:
* **Snapshot-then-stream**: take a consistent snapshot, record the corresponding log position, then start streaming from that position.
* **Stream-then-snapshot**: start streaming first, then take a snapshot later and de-duplicate (or ignore) streamed events already covered by the snapshot.
* **Pure log replay**: if the log is retained long enough and contains sufficient information, rebuild state by replaying the log from an initial point.
CDC is most commonly implemented by reading an existing write/replication log (WAL/binlog/oplog/commitlog). Log-based CDC provides a natural ordering and low overhead on the write path, but introduces operational concerns around log retention and consumer lag. Depending on the database architecture, ordering may be global (single-leader log) or only per shard/partition (distributed logs), which affects whether consumers must merge multiple streams.
#### PostgreSQL
[Section titled “PostgreSQL”](#postgresql)
PostgreSQL CDC is commonly built on **logical decoding** / **logical replication**, which decodes row-level changes out of the WAL. Consumers use a **replication slot** (an LSN-based resume point) and a logical decoding output plugin to stream changes. Replication slots also act as backpressure on retention: the server must keep WAL needed by the slot until the consumer advances. PostgreSQL can pair streaming with an initial table copy so consumers can bootstrap from a consistent snapshot and then continue from the corresponding LSN.
#### MySQL
[Section titled “MySQL”](#mysql)
MySQL CDC is typically built on the **binary log (binlog)**. Consumers resume from a binlog file+offset or a GTID set, and stream transaction events in commit order. For bootstrapping, systems take a consistent snapshot and record the binlog position (or GTID) at the snapshot boundary, then stream binlog events after that point. Row-based binlogging is commonly required to reliably capture row-level mutations.
#### Cassandra
[Section titled “Cassandra”](#cassandra)
Apache Cassandra provides CDC by exposing a copy of **commitlog segments** for tables with CDC enabled. CDC output is **node-local** (each node emits mutations it applies), so consumers typically tail CDC per node and then de-duplicate/reconcile events across replicas. Retention is bounded by local disk and configured CDC space limits, so slow consumers can fall behind and lose events.
#### MongoDB
[Section titled “MongoDB”](#mongodb)
MongoDB exposes CDC via **change streams**, which are built on the replica set **oplog**. Consumers subscribe with an aggregation pipeline and receive a stream of change events plus a **resume token** that allows restarting without missing events (subject to oplog retention). Bootstrapping is typically done by taking an initial collection scan/dump and then starting the change stream from an operation time or resume token captured at the snapshot boundary.
### Current State
[Section titled “Current State”](#current-state)
SlateDB does not directly support CDC. Users that wish to receive near-realtime updates of their SlateDB have the following options:
1. Scan the database for changes
2. Dual write to SlateDB and an external message queue
3. Implement their own CDC solution by directly reading SlateDB’s WAL and/or compacted SSTs.
(1) and (2) are not optimal designs. (3) is the correct architecture, but it is not a simple implementation. This RFC proposes that we implement (3) in SlateDB so users can stream changes out of SlateDB without additional infrastructure.
## Goals
[Section titled “Goals”](#goals)
* Provide a durable, ordered stream of mutations to a SlateDB database.
* Single-digit second latency for consumers.
* Exposing WAL file boundaries so WAL files can be treated as distinct write operations. (These are not the same as transactions; a single WAL file may contain multiple transactions, which will have distinct seqnums.)
* Put, delete, and merge events should all be visible.
* The design should be modular enough to allow easy migration to a `slatedb-wal` package that does not depend on the `slatedb` package.
* Support for `wal_object_store_uri` when it is set to a different object store than the main object store.
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* WAL checkpointing and WAL retention policies. WAL garbage collection is independent of the `WalReader`. The `WalReader` must stay ahead of whatever mechanism is used (likely SlateDB’s garbage collector with WAL `min_age` configured). There is no guarantee that the WAL files are not garbage collected. Long-running consumers should copy WAL files to a separate location if they want to ensure they are not lost.
* TTL-based row-expiration events. Clients must implement their own logic to handle TTL expirations.
* Merge operator events. Merge writes will be exposed as-is; the consumer is responsible for applying the merge logic.
* Streaming directly from the writer client. Users will need to create a `WalReader`.
* Transaction boundaries. Though SlateDB writes transactions as a single WAL entry, a single WAL SST might have multiple transactions in it. Users will need to use RowEntry seqnums to track transaction boundaries.
* Message brokering. SlateDB will not provide message broker functionality such as consumer offset management, topic management, consumer groups, or partitioning. The goal of this RFC is not to provide a Kafka-style log broker or message queue.
* Support for `wal_enabled=false` databases (that only write to L0+).
* Support for parent DBs or external DBs. The `WalReader` will only read WAL files for the specific database it is instantiated for.
* Bootstrapping and backfilling. Users will need to implement their own bootstrapping and backfilling logic using `scan()`.
* Update events [similar to Debezium’s](https://debezium.io/documentation/reference/stable/transformations/event-changes.html#_change_event_structure). Users will only see the put/delete/merge operation for a given key.
## Design
[Section titled “Design”](#design)
SlateDB’s CDC design is based loosely on RocksDB’s [`getUpdatesSince`](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/db.h#L1959-L1974) API. In RocksDB, `getUpdatesSince` uses a [`TransactionLogIterator`](https://github.com/facebook/rocksdb/blob/feffb67303b7d8f38fd91acb9a5b6f6f06c068c3/include/rocksdb/transaction_log.h#L92) to read `write_batch`es. In SlateDB, we will use a `WalReader` to return `WalFile`s, which can be read to obtain `RowEntry`s. Unlike RocksDB, we do not provide a single logical iterator over the WAL; instead, we expose WAL file boundaries (`WalFile`) and a per-file iterator (`WalFileIterator`) so users can control polling intervals and parallelism without added configuration.
### `WalReader` API
[Section titled “WalReader API”](#walreader-api)
```rs
/// Iterator over entries in a WAL file.
pub struct WalFileIterator;
impl WalFileIterator {
/// Returns the next entry in the WAL file.
pub async fn next_entry(&mut self) -> Result, crate::Error>;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct WalFileMetadata {
/// The time this WAL file was last written to object storage.
pub last_modified_dt: DateTime,
/// The size of this WAL file in bytes.
pub size_bytes: u64,
/// The path of this WAL file in object storage.
pub location: Path,
}
/// Represents a single WAL file stored in object storage and provides methods
/// to inspect and read its contents.
pub struct WalFile {
/// The unique identifier for this WAL file. Corresponds to the SST filename without
/// the extension. For example, file `000123.sst` would have id `123`.
pub id: u64,
table_store: Arc,
}
impl WalFile {
/// Returns metadata for this WAL file.
///
/// Returns `Ok(None)` if the WAL file does not exist.
pub async fn metadata(&self) -> Result, crate::Error>;
/// Returns an iterator over `RowEntry`s in this WAL file. Raises an error if the
/// WAL file could not be read.
///
/// Returns `Ok(None)` if the WAL file does not exist.
pub async fn iterator(&self) -> Result, crate::Error>;
}
/// Reads WAL files in object storage for a specific database.
pub struct WalReader;
impl WalReader {
/// Creates a new WAL reader for the database at the given path.
///
/// If the database was configured with a separate WAL object store, pass that
/// object store here.
pub fn new>(path: P, object_store: Arc) -> Self ;
/// Lists WAL files in ascending order by their ID within the specified range.
/// If `range` is unbounded, all WAL files are returned.
pub async fn list>(&self, range: R) -> Result, crate::Error>;
/// Creates a [`WalFile`] handle for a WAL ID.
pub fn get(&self, id: u64) -> WalFile;
}
```
### Usage
[Section titled “Usage”](#usage)
Users can stream database changes by using `list()` to discover an initial set of WAL files, then using `get(last_seen_id + 1)` to poll for the next WAL file without re-listing.
```rs
// One-time discovery (optional). Consumers can also skip this if they are resuming
// from a persisted WAL ID.
let wal_files = wal_reader.list(..).await?;
let mut next_id = 0;
for wal_file in wal_files {
if let Some(mut iter) = wal_file.iterator().await? {
while let Some(row) = iter.next_entry().await? {
// Process the row (send it to a message queue, apply it to another DB, etc.)
}
next_id = wal_file.id + 1;
}
}
// Poll the next WAL ID without calling list() again.
loop {
let wal_file = wal_reader.get(next_id);
if let Some(mut iter) = wal_file.iterator().await? {
while let Some(row) = iter.next_entry().await? {
// Process the row.
}
next_id = wal_file.id + 1;
continue;
}
// The WAL file is not present (either not written yet, or GC'd). Sleep briefly and retry.
tokio::time::sleep(Duration::from_secs(1)).await;
}
```
*NOTE: This API uses file ID-based rather than seqnum-based ranges. This allows the reader to directly read WAL files rather than reading metadata and binary-searching for the appropriate file and seqnum offset in a WAL file. If users wish to resume from a specific seqnum, they will need to implement filtering logic themselves.*
This polling approach addresses the [operational concern](https://github.com/slatedb/slatedb/pull/634#discussion_r2791629993) that object store listings become expensive as the number of retained WAL files grows. By polling `get(next_id)` and checking `WalFile::iterator()` (or `WalFile::metadata()`), consumers can wait for future WAL files with a single object-store GET/HEAD per-polling interval, rather than repeatedly scanning the full WAL prefix via `list()`.
If a consumer falls behind (or suspects it missed an ID due to GC), it can use `list(next_id..)` to re-synchronize and find the next available WAL file, then resume polling with `get()`.
### Semantics and Guarantees
[Section titled “Semantics and Guarantees”](#semantics-and-guarantees)
The API provides exact row-for-row replication. The consumer will see all puts, deletes, and merges in the order they were written to the WAL.
Consumers that write their output and their WAL position atomically will achieve exactly-once processing semantics. Consumers that do not track their output atomically with their WAL position might see duplicate data if they crash and restart from a WAL position that they have already processed.
The `WalReader` will not interact with any SlateDB components (garbage collector, manifest, compactor, and so on). It will simply read WAL files from object storage as they are written. Consumers are responsible for ensuring that they read WAL files before they are garbage collected.
### WAL and Memtable Synchronization
[Section titled “WAL and Memtable Synchronization”](#wal-and-memtable-synchronization)
`batch_write.rs` currently writes to the `WalBuffer` and memtable `KVTable` in parallel. It is possible that the memtable might get flushed to L0 before the WAL file is written to object storage. If this happens, there is a brief window where an entry might be durably written but not end up in the WAL. Consider the following sequence of events:
1. t0: `batch_write.rs` calls self.wal\_buffer.append(\&entries)?.durable\_watcher();
2. t1: `batch_write.rs` calls self.write\_entries\_to\_memtable(entries);
3. t2: Memtable flush is triggered, and the memtable is written to L0.
4. t3: Writer crashes before the WAL file is written to object storage.
In this case, the CDC feed would not see data that is durably written to the database. We considered two options to address this:
1. Replay all writes in the database greater than the max seqnum in the last WAL file. This would require that we prevent the L0 writer and compactor from deleting seqnums that haven’t yet been written to the WAL. We would then need to scan all SSTs (and SRs) from top to bottom until we reach the max seqnum in the last WAL file.
2. Ensure that WAL files are written to object storage before memtables are flushed to L0. This would require that we block memtable flushes until the WAL write is confirmed.
This RFC proposes that we pursue option (2): blocking memtable flushes until the WAL write is confirmed. This is simpler to implement and makes the database semantics easier to reason about (any data in the DB is guaranteed to be in the WAL).
The performance impact on memtable flushes should be minimal. When `wal_enabled=true`:
* `await_durable=false` writes will not block on either WAL or memtable flushes, so low-latency writes are unaffected.
* `await_durable=true` writes will block on WAL writes as they do today (we always return `wal_watcher` in `batch_write.rs` if WALs are enabled).
* `flush_with_options` calls with `FlushOptions::flush_type=Wal` will block on WAL writes as they do today.
* `flush_with_options` calls with `FlushOptions::flush_type=Memtable` will now block on WAL writes as well.
When backpressure is applied, memtable flushes will be delayed until the WAL write is confirmed. This can add more latency to the write path since writes will be blocked for longer. Backpressure should be rare in practice, and when applied in this scenario, a WAL write will free memory just as a memtable flush would have, so the overall impact should be minimal.
To implement this, we will modify `mem_table_flush.rs` to check if `self.db_inner.oracle.last_remote_persisted_seq` is less than the memtable’s `KVTable::last_seq`. If it is, `flush_wals()` will be called and awaited before proceeding with the memtable flush. This guarantees that the last entry written to the WAL will be >= the last entry written to the memtable. This is not strictly the most efficient approach since WAL entries beyond the memtable’s last seqnum might get flushed, which could otherwise be ignored. However, this approach is simpler to implement and reason about.
If the WAL is disabled, `mem_table_flush.rs` will proceed as normal.
## Impact Analysis
[Section titled “Impact Analysis”](#impact-analysis)
SlateDB features and components that this RFC interacts with. Check all that apply.
### Core API & Query Semantics
[Section titled “Core API & Query Semantics”](#core-api--query-semantics)
* [ ] Basic KV API (`get`/`put`/`delete`)
* [ ] Range queries, iterators, seek semantics
* [ ] Range deletions
* [ ] Error model, API errors
### Consistency, Isolation, and Multi-Versioning
[Section titled “Consistency, Isolation, and Multi-Versioning”](#consistency-isolation-and-multi-versioning)
* [ ] Transactions
* [ ] Snapshots
* [ ] Sequence numbers
### Time, Retention, and Derived State
[Section titled “Time, Retention, and Derived State”](#time-retention-and-derived-state)
* [x] Time to live (TTL)
* [ ] Compaction filters
* [x] Merge operator
* [x] Change Data Capture (CDC)
### Metadata, Coordination, and Lifecycles
[Section titled “Metadata, Coordination, and Lifecycles”](#metadata-coordination-and-lifecycles)
* [ ] Manifest format
* [ ] Checkpoints
* [ ] Clones
* [ ] Garbage collection
* [ ] Database splitting and merging
* [ ] Multi-writer
### Compaction
[Section titled “Compaction”](#compaction)
* [ ] Compaction state persistence
* [ ] Compaction filters
* [ ] Compaction strategies
* [ ] Distributed compaction
* [ ] Compactions format
### Storage Engine Internals
[Section titled “Storage Engine Internals”](#storage-engine-internals)
* [x] Write-ahead log (WAL)
* [ ] Block cache
* [ ] Object store cache
* [ ] Indexing (bloom filters, metadata)
* [ ] SST format or block format
### Ecosystem & Operations
[Section titled “Ecosystem & Operations”](#ecosystem--operations)
* [ ] CLI tools
* [x] Language bindings (Go/Python/etc)
* [ ] Observability (metrics/logging/tracing)
## Operations
[Section titled “Operations”](#operations)
### Performance & Cost
[Section titled “Performance & Cost”](#performance--cost)
* Latency (reads/writes/compactions): See “WAL and Memtable Synchronization” section.
* Throughput (reads/writes/compactions): See “WAL and Memtable Synchronization” section.
* Object-store request (GET/LIST/PUT) and cost profile: Additional LIST requests will be made to poll for new WAL files. Each WAL file read will incur GET requests to read `RowEntry` data.
* Space, read, and write amplification: Consumers will read additional data from object storage, but there is no change to the data stored.
### Observability
[Section titled “Observability”](#observability)
* Configuration changes: `WalReaderOptions` struct to configure reader behavior.
* New components/services: `WalReader` component.
* Metrics: Metrics will be added to track `WalReader` list and read operations.
* Logging: Logs will be added to `WalReader` and `WalFile` methods to trace usage and errors.
### Compatibility
[Section titled “Compatibility”](#compatibility)
* Existing data on object storage / on-disk formats: No changes to existing data formats. WAL files will continue to be written as before.
* Existing public APIs (including bindings): New `WalReader` and `WalFile` API added; existing APIs remain unchanged.
* Rolling upgrades / mixed-version behavior (if applicable): No special considerations; new API can be used with existing databases.
## Testing
[Section titled “Testing”](#testing)
* Unit tests: `WalReader` and `WalFile` methods.
* Integration tests: Add `WalReader` thread to `slatedb/tests/db.rs` test.
* Fault-injection/chaos tests: `nightly.yaml` chaos test will inherit `WalReader` test from `slatedb/tests/db.rs`.
* Deterministic simulation tests: None
* Formal methods verification: None
* Performance tests: Manual one-time verification that WAL flushes do not significantly impact memtable flush latency.
## Rollout
[Section titled “Rollout”](#rollout)
* Milestones / phases: None
* Feature flags / opt-in: None
* Docs updates: Add CDC documentation to `website/src/content/docs/docs/design/change-data-capture.md`.
## Alternatives
[Section titled “Alternatives”](#alternatives)
Several alternatives were considered but rejected:
* Polling L0 rather than the WAL. This approach would allow us to support `wal_enabled=false` databases. However, it is more complex to implement correctly. To make the L0 reads work, we would need the CDC client to see every manifest so it could see every single L0 SST written. To do that, we would have to either depend on `min_age` to prevent manifests/compacted SSTs from being deleted, or we would have to use some kind of boundary in the manifest. This approach would also not expose all writes since some data is compacted prior to L0 flush.
* Reading the L0/sorted runs in reverse from oldest to newest. This would require the consumer to track data that’s written after the scan begins and read such data on subsequent polls. This was not considered in-depth due to its perceived complexity. This is similar to [this comment](https://github.com/slatedb/slatedb/pull/634#discussion_r2173603400), which proposes diffing database metadata at different points in time.
* Providing a CDC callback directly in the writer client. Writes would simply trigger a callback after WAL/L0 SSTs are written. This is similar to the [Notification on memtable flush](https://github.com/slatedb/slatedb/issues/1245) feature request.
* Attaching to the GC or compactor lifecycle. We could, for example, have GC trigger a callback or write data elsewhere when it removes WAL or L0 entries.
* Implementing a `ScanOptions::range` API that would allow scans to read only data in a specific seqnum range. This would allow users to implement their own CDC solution by repeatedly scanning starting from seqnum 0 and scanning in batches until they reach the latest seqnum. This was rejected due to performance concerns; scanning the entire database repeatedly would be inefficient for large databases.
Notably, this RFC does not preclude us from implementing any of the above alternatives. They are simply not proposed in this RFC.
### WAL and Memtable Synchronization
[Section titled “WAL and Memtable Synchronization”](#wal-and-memtable-synchronization-1)
This RFC proposes blocking memtable flushes on WAL flushes (when `wal_enabled=true`) to guarantee that any data visible in the DB is also present in the WAL.
During discussion, [an alternative was suggested](https://github.com/slatedb/slatedb/pull/634#discussion_r2789422085): filter the memtable during flush so we only flush rows with `seq <= last_remote_persisted_seq`, and carry forward rows with `seq > last_remote_persisted_seq` into a new or existing memtable so they can be flushed later once the WAL is durably persisted remotely. This avoids the explicit synchronization step.
A concern with this approach is that, in the degenerate case, the carry-forward set could grow large, though it won’t grow unbounded since `Db::maybe_apply_backpressure` includes `imm_memtables` in its memory usage calculation.
If we ever adopt this approach, one implementation would be to move `seq > last_remote_persisted_seq` rows into the oldest immutable memtable that is not currently flushing (falling back to the active memtable if no immutable memtables exist). This avoids creating a new memtable, but it requires mutating an “immutable” memtable and can cause L0 file sizes to deviate from the configured target (either small durable-only L0 files, or oversized L0 files if the filtered rows accumulate across multiple flushes). An alternative is to create a new memtable containing just the filtered rows and push it onto the `imm_memtables` `VecDeque`, but that can increase the number of very small L0 files.
For now, we keep the synchronization-based implementation; if it becomes a performance bottleneck, this filtering approach should be faster and can be revisited.
## Future work
[Section titled “Future work”](#future-work)
### Bootstraps and Backfills
[Section titled “Bootstraps and Backfills”](#bootstraps-and-backfills)
Consumers are responsible for bootstrapping or backfilling initial state outside of the CDC pipeline. SlateDB provides no built-in support for this at the moment.
Users can currently bootstrap data using `Db::scan()`, but there is no easy way to do a clean cut over to the WAL stream. Users may choose to simply accept duplicates, or they can use key/value payloads to de-duplicate data. For example, if a value contains a unique ID field, users can de-duplicate using this field.
Various fields in the manifest might be considered, but they are insufficient. The `wal_id_last_seen` field in the manifest is insufficient because subsequent WAL files might be written. Those writes are marked durable and exposed to `scan()` operations. The `last_l0_seq` contains the last sequence number written to L0, but writes with later sequence numbers might be persisted to the WAL (see [0011-transaction.md](/rfcs/0011-transaction) for sequence number details).
To provide a clean cutover, we need to expose the sequence number at the point the `scan()` is executed. [#1247](https://github.com/slatedb/slatedb/pull/1247) proposes exactly this. With `scan()`, `DbIterator` will be extended to have a `next_row()` function, which will return a `RowEntry`. The `RowEntry` will contain the sequence number of the row. Users can keep track of the highest sequence number seen during the scan, and then use that sequence number as a boundary when processing the WAL stream to avoid duplicates. This approach provides key-ordered rather than seqnum-ordered bootstrapping. If seqnum-ordered bootstrapping is required, users will need to implement their own logic to read all SSTs and sort the data by seqnum.
### Database diffs
[Section titled “Database diffs”](#database-diffs)
There was [some discussion](https://github.com/slatedb/slatedb/pull/634#discussion_r2778273686) in this RFC about periodic replication. The example cited was an hourly replication feed into Apache Iceberg:
> Consider the case of replicating SlateDB to Iceberg, once every hour or so. We wouldn’t need all the updates for this case, it is enough to only get the keys that were updated since previous replication, and just the most recent update for every key. The changes can be coalesced on the read side. Alternate approach for this scenario would be to compare two checkpoints and return the diffs. Wal based approach is more generally applicable, and is simpler. Do you see value in also adding the “only get the recent update since last time” approach for a future, different RFC?
This is an interesting idea, but it is outside the scope of this RFC.
## References
[Section titled “References”](#references)
* Githb issue [#249](https://github.com/slatedb/slatedb/issues/249) (CDC Streaming to support data sinks and event driven architectures)
* RocksDB’s [`getUpdatesSince`](https://github.com/facebook/rocksdb/blob/main/include/rocksdb/db.h#L1959-L1974) API
* SlateDB [CDC Discord thread](https://discord.com/channels/1232385660460204122/1467693163815768178)
## Updates
[Section titled “Updates”](#updates)
* 2025-06-22: WIP draft
* 2025-02-02: Re-write to focus on `WalReader` API.
# Range Metadata and Size Estimation
Status: Accepted
Authors:
* [FiV0](https://github.com/FiV0)
## Summary
[Section titled “Summary”](#summary)
This RFC proposes:
1. Adding a per-SST stats block to the SST file format containing aggregate statistics (`num_puts`, `num_deletes`, `num_merges`, `raw_key_size`, `raw_val_size`) and per-block statistics (`block_stats`), enabling efficient record counting within key ranges.
2. Exposing lower-level primitives — `DbReader::manifest()`, `SstReader`, and `SstFile` — that allow users to walk the manifest, open individual SSTs, and read per-SST stats and index data for size and cardinality estimation. Memtable stats are exposed via the existing `StatRegistry` (accessible through `Db::metrics()`).
Rather than providing a single high-level `Db::metadata(range)` function, this approach exposes modular building blocks. Users call `db.manifest()` to discover which SSTs exist, then use `SstReader` to open and inspect individual SSTs. This makes SlateDB more composable and avoids coupling the estimation logic to a specific API shape.
## Motivation
[Section titled “Motivation”](#motivation)
Users want to understand data distribution and storage usage for specific key ranges to support:
* **Cost estimation**: Calculating storage costs or scan costs for specific data partitions. For example, if a key range is an index in a query engine, it is good to know an approximate size when joining with other data.
* **Bulk operations**: Planning exports, migrations, or splits based on estimated sizes
* **Capacity planning**: Understanding storage requirements for different key ranges
* **Data modeling**: Understanding how data is distributed across the key space
Currently there is no better way than to scan the whole range to get an estimate or approximation of the data size.
Beyond size estimation, some workloads need to count records in a key range efficiently — for example, append-only logs computing consumption lag, query planners estimating `records_in_range()`, or systems deciding where to split data. Today this requires a full scan. Adding per-block record counts to the stats block enables block-level counting with at most two data block reads per boundary SST.
An earlier revision of this RFC proposed high-level functions (`estimate_size_with_options` and `estimate_key_count`) with configuration options like `error_margin`, `include_memtables`, and `include_files`. However, different use cases demand different knobs — compressed vs uncompressed sizes, varying accuracy vs I/O tradeoffs, inclusion or exclusion of memtables, etc. Baking all of these into the API through configuration options risks making the interface rigid and opinionated. Instead, we expose lower-level building blocks so users can compute whatever they need. For example, a user interested in key counts can derive `num_puts + num_merges - num_deletes` from per-SST stats, while a user interested in on-disk size can sum `SsTableView::estimate_size()` across covering SSTs, and one needing finer accuracy can use `SstFile::index()` to estimate at block granularity.
## Goals
[Section titled “Goals”](#goals)
* Add a per-SST stats block with aggregate and per-block statistics, referenced by `stats_offset`/`stats_len` in `SsTableInfo` and sst.fbs
* Expose `Db::manifest()` so users can discover SSTs covering a key range
* Provide `SstReader` and `SstFile` for opening individual SSTs and reading their stats (as `SstStats`) and index data
* Expose memtable stats as metrics via the existing `StatRegistry` / `Db::metrics()`
## Non-Goals
[Section titled “Non-Goals”](#non-goals)
* Exact size calculation or key counting (would require reading and processing a lot more data)
* High-level aggregation functions with configuration options (users derive these from the metadata)
* Providing a single `Db::metadata(range)` API that pre-computes which SSTs overlap a range (users do this themselves using the manifest)
* Exposing `SstFile::rows()` for reading row data (out of scope for size estimation)
## Design
[Section titled “Design”](#design)
### API
[Section titled “API”](#api)
#### `Db::manifest()`
[Section titled “Db::manifest()”](#dbmanifest)
```rust
impl Db {
...
pub fn manifest(&self) -> &ManifestCore;
...
}
```
Returns a reference to the current in-memory `ManifestCore`. No I/O. `ManifestCore` is already public (re-exported via `slatedb::manifest`) .
#### `SstReader` and `SstFile`
[Section titled “SstReader and SstFile”](#sstreader-and-sstfile)
```rust
pub struct SstReader {
object_store: Arc,
root_path: Path,
cache: Option>,
}
impl SstReader {
pub fn new(
root_path: impl Into,
object_store: Arc