Caching

SlateDB has two separate cache layers for SST reads. The block cache stores decoded SST blocks and metadata. The object-store cache stores raw bytes fetched from the object store. You can use either layer or both.

Block Cache

The block cache sits in the SST reader. It stores decoded data blocks, index blocks, filter blocks, and stats blocks. A hit here avoids the remote GET, and it also avoids decoding the block again.

DbBuilder installs a SplitCache by default. SplitCache keeps data blocks separate from SST metadata so large reads are less likely to evict indexes and Bloom filters.

You can replace the cache with DbBuilder::with_db_cache, or disable it with DbBuilder::with_db_cache_disabled. If you want to plug in your own implementation, use the DbCache trait.

FoyerHybridCache also belongs in this layer. It is a DbCache implementation, so it caches decoded SST entries, not raw object-store bytes. Its memory tier works like any other database cache. Its disk tier can still avoid remote I/O and decode work, but it adds a local disk read on a miss in memory. See Configuring Foyer Cache for tuning guidance.

Object-Store Cache

CachedObjectStore is a second cache layer for raw object-store bytes. It splits each object into fixed-size parts, stores those parts under a local root folder, and can serve later GET and HEAD requests from those local files when the needed parts are already present.

There are two ways to enable it. The first is configuration: set root_folder in ObjectStoreCacheOptions, and SlateDB builds the cache from those options and wraps the object store you passed to the builder in it. This is the only way to enable the cache from a settings file.

The second is to build the cache yourself with CachedObjectStore::builder and pass it to SlateDB as the object store:

let cache = CachedObjectStore::builder("/var/slatedb-cache", object_store)
    .with_cache_on_flush(true)
    .with_cache_on_compaction(true)
    .with_max_cache_size_bytes(Some(16 * 1024 * 1024 * 1024))
    .build()
    .await?;
let db = Db::builder(path, cache).build().await?;

Building it yourself lets you hold on to the cache instance, share it across several Db and DbReader instances in the same process, and warm it directly. Leave root_folder unset when you do this, otherwise SlateDB wraps your cache in a second one.

Either way the cache ends up as the innermost layer, closest to the object store you provided. SlateDB adds its own retry and metrics handling above it, so a read served from the cache is still counted as an object-store request. See Metrics for what that means for the slatedb.object_store.* metrics.

This cache stores object-store bytes, not decoded SST blocks. It helps when the block cache is cold because it can avoid a remote read even though SlateDB still needs to read from local disk and decode the block afterward. On a miss, SlateDB aligns the requested range to the configured part size, fetches that larger range from the object store, and saves the returned parts locally. The default part size is 4 MiB.

The built-in object-store cache is disk-backed today. The root folder is a filesystem path, and the current implementation stores parts under that directory. If you want an in-memory cache, use the block cache layer instead.

How Caches Get Filled

Reads consult the block cache first. On a miss there, SlateDB fetches bytes through the object-store layer, which may itself hit the object-store cache before going remote.

ReadOptions::cache_blocks and ScanOptions::cache_blocks decide whether SlateDB inserts a decoded data block after fetching it on a miss. They do not disable cache lookups. Reads and scans cache SST metadata, such as indexes, filters, and stats, independently of cache_blocks.

The defaults reflect the usual access patterns. Point reads default to cache_blocks = true because they are more likely to revisit hot data. Scans default to cache_blocks = false so a long sequential read does not fill the cache with data blocks that probably will not be reused soon. Scans can still benefit from entries that are already hot. Internal tasks follow the same idea: WAL replay and compaction read SSTs without populating the foreground cache.

Writes fill the block cache as well. BlockCachePolicy selects which components of an SST are inserted as that SST is written, and you install it with DbBuilder::with_block_cache_policy. The policy applies to the block cache only. Admission into the object-store cache is controlled separately, as described below.

The policy holds one target list per write source. The default inserts data blocks, the index, and the filters after a memtable flush, and inserts only the index and the filters as compaction output is written, so compaction does not evict blocks that reads have made hot. Replace either list with BlockCachePolicy::with_flush_targets or BlockCachePolicy::with_compaction_output_targets. An empty list disables insertion for that source.

Each list holds CacheTarget values:

• CacheTarget::Filters inserts the SST filter blocks, if any exist.
• CacheTarget::Index inserts the SST index.
• CacheTarget::Stats inserts the SST stats block, if one exists.
• CacheTarget::data(range) inserts the data blocks whose key span overlaps range.

CacheTarget::data accepts any key range, so CacheTarget::data::<&[u8], _>(..) selects every data block, while CacheTarget::data(b"user:".as_slice()..b"user;".as_slice()) selects only the blocks covering that key range. This policy caches all four components after a flush and nothing after compaction:

use slatedb::{BlockCachePolicy, CacheTarget, Db};

let policy = BlockCachePolicy::default()
    .with_flush_targets(&[
        CacheTarget::data::<&[u8], _>(..),
        CacheTarget::Index,
        CacheTarget::Filters,
        CacheTarget::Stats,
    ])
    .with_compaction_output_targets(&[]);

let db = Db::builder(path, object_store)
    .with_block_cache_policy(policy)
    .build()
    .await?;

The disk cache stays disabled unless you enable it, either by setting object_store_cache_options.root_folder or by passing your own CachedObjectStore to the builder.

By default, writes go straight to the upstream object store and do not populate the object-store cache. Each SST write source has its own admission flag: cache_on_flush stores SSTs written by memtable flushes locally, and cache_on_compaction does the same for compaction output. The builder exposes the same two flags as with_cache_on_flush and with_cache_on_compaction. Enabling them can help if readers are likely to touch freshly written SSTs soon afterward. WAL and manifest writes are never cached.

Warming the Object-Store Cache

If you want the disk cache warm before serving traffic, and you configured it through ObjectStoreCacheOptions, set preload_disk_cache_on_startup to PreloadLevel::L0Sst or PreloadLevel::AllSst. Db and DbReader load recent SSTs, or all SSTs, into the local cache on open until the cache size limit is reached.

That setting only applies to a cache SlateDB built for you. A cache you built yourself is warmed by you. Read the current SST set from DbMetadataOps::manifest, resolve each SST id to an object-store path with PathResolver::sst_path, and pass the paths to CachedObjectStore::load_files_to_cache. Construct the resolver with PathResolver::new, which takes the manifest as well as the database path because a manifest can reference SSTs owned by another database after a clone. The max_bytes budget is applied in path order, so order paths by priority. Fetches are best-effort, and failures are logged and skipped.

Sharing Between Instances

The two cache layers behave differently when you open multiple Db or DbReader instances against the same cache.

For the object-store cache, sharing is straightforward. If multiple instances on the same machine use the same root folder, whether they set root_folder or pass it to CachedObjectStore::builder, they reuse the same local cache directory. Instances in the same process can also share one CachedObjectStore instance directly. For the same database path, that lets one instance benefit from parts fetched by another. When you use Db::resolve_object_store and provide different path arguments to builder methods like DbBuilder::new, SlateDB keeps the cached files under different path prefixes, so they do not clobber one another. Using the same part size gives the best reuse. Different part sizes can coexist, but they will not reuse the same cached part files.

If you provide an object store directly with PrefixStore or other custom wrapping instead of using Db::resolve_object_store, you must configure different root folder values for each instance to prevent cache collisions. SlateDB no longer automatically resolves root prefixes from metadata locations.

The block cache is different. Both DbBuilder::with_db_cache and DbReaderBuilder::with_db_cache let you pass in your own cache object, so you can choose to reuse the same process-local cache implementation across builders. For Db, that mainly gives you a shared memory or disk budget, not shared hits: SlateDB scopes each instance’s entries so one Db does not read another Db’s cached blocks by accident. If your main goal is cross-instance warming, the object-store cache is the better fit.

A cache you pass in stays owned by you: Db::close() and DbReader::close() never close it, so closing one instance does not affect the others sharing it. Once every instance using the cache is closed, call DbCache::close yourself so hybrid implementations can flush their memory tier to disk.

For DbReader, treat a caller-managed block cache as an advanced optimization. It can make sense to keep one reader-side cache per database in a single process, but if you want a cache that is naturally shared between writers and readers, or across many short-lived instances, the object-store cache is the simpler mechanism to share.

Tradeoffs

The block cache saves decode work and can also save remote I/O. With a memory-only implementation, it avoids both. With FoyerHybridCache, a hit in the disk tier still avoids remote I/O and re-decoding, but it adds a local disk read.

The object-store cache only saves the remote I/O. SlateDB still has to read from local disk and decode the block. Many deployments use both layers: a block cache for the hot decoded working set, and an object-store cache for colder bytes and range-aligned fetches. Block size matters here too, because it controls cache granularity and therefore affects hit rate and read amplification. Tuning covers that knob.

If you enable FoyerHybridCache and also enable the object-store cache, SlateDB may store the same SST data twice on local disk. That can still be a reasonable trade if the object-store cache’s range prefetching saves enough remote reads, but the cost is more local storage traffic.