SlateDB
← all posts

Benchmarking SlateDB

Chris

SlateDB has been around for a few years now. We’ve begun to pay more attention to performance lately. As Kent Beck (purportedly) said, “Make It Work, Make It Right, Make It Fast.” We feel we’ve earned the right to make SlateDB fast. Of course, before we improve performance, we must measure it. That means benchmarks.

Benchmarks are tricky. It’s easy to do something that is perceived as marketing, hyperbole, or outright dishonesty. It’s questionable whether public benchmarks are even useful anymore. A frontier LLM is capable of building excellent bespoke benchmarks for each user.

Yet we found ourselves rebuilding SlateDB’s benchmark suite a few weeks back. We—the SlateDB developers—still need them to measure performance. We also believe the data will help users gauge whether SlateDB meets their performance needs.

This post covers the benchmarking philosophy we landed on, how we benchmarked SlateDB, and some things we learned along the way.

(You can head over to benchmark.slatedb.io if you just want the results.)

Legacy benchmarks

We have had benchmarks for quite a while: both Criterion microbenchmarks and a benchmark-db.sh script. We also run a nightly benchmark-db.sh job against a Tigris object store bucket. (A special thanks to them for donating the bucket to us free of charge.)

These benchmarks were largely ignored. The output was buried in a GitHub Actions summary. It didn’t alert when significant regressions occurred, either. If you managed to find it, the output was hard to read.

Our goal was to make SlateDB’s benchmarks easier to find and understand. We also wanted better diagnostic information so developers could visualize SlateDB’s behavior under different workloads.

Benchmark suite

We drew inspiration from ClickBench, DuckDB’s benchmark suite, Lucene’s nightly benchmarks, and Apache Iggy’s public benchmark dashboard. Tectonic: Bridging Synthetic and Real-World Workloads for Key-Value Benchmarking provides a good overview of the space, for those looking to get up to speed.

Table 1 in the Tectonic paper is particularly relevant. It compares YCSB, KVBench, RocksDB’s db_bench, and Tectonic’s own workloads.

Feature coverage across YCSB, KVBench, db_bench, and Tectonic
Capability YCSBKVBenchdb_benchTectonic
Operations Insert
Update
Read-modify-write
Point query
Empty point query
Range query
Point delete
Empty point delete
Range delete
Distributions Uniform
Normal
Beta
Zipfian
Exponential
Log normal
Poisson
Weibull
Pareto
Properties Dynamic workload shifts
Context-aware shifting
Data sortedness
Variable query selectivity
Variable key-value length
Temporality-based access
Customizable key prefix
Composite keys
* Requires significant manual intervention.

Many of the YCSB and db_bench workloads cover the behavior users are likely to care about. We chose to adopt a subset of those. If a user needs a more specialized workload, they should run it themselves. The benchmark repository is available for that purpose.

We also added cost metrics. Throughput and latency matter, but SlateDB exists in part because object storage changes the economics of persistence. Databases should be cheap to run. If it is not, we need to know.

The suite is made up of three components:

  • A standalone benchmark repository with the suite and its configuration.
  • GitHub Actions workflows that run it.
  • A public benchmark website where the results are easier to inspect.

Methodology

We follow YCSB and db_bench workloads where possible, but SlateDB does not map perfectly to either. SlateDB is designed around object storage. A cache miss is far more costly, both in terms of money and latency. Misses incur a metered object store API call and a remote network hop. Rather than adjust our configurations to accommodate this, we opted to keep configuration close to SlateDB’s defaults.

The workloads run on AWS Graviton machines managed by WarpBuild in us-east-1. They talk to an Amazon S3 bucket in the same region and to Tigris through its us-east-1 on-ramp. We use a relatively large machine instance (m8g.2xlarge) because compaction is CPU-intensive.

We configured SlateDB’s cache to hold roughly 10% of the database. The rest lives in object storage. A db_bench run of RocksDB keeps the entire database on local disk. This is clearly an apples to oranges comparison. These settings are closer to the way many users configure SlateDB, though.

Another tradeoff applies to mixed read/write workloads. A read that misses the cache can take tens or hundreds of milliseconds while data comes back from object storage. If one task does both reads and writes, those misses hold back its write throughput. We could separate readers and writers or add more parallelism to report a stronger aggregate throughput number. We kept the configuration closer to RocksDB’s workload instead.

Findings

Our first discovery was that routing over the public internet is unpredictable. At one point, we ran in Hetzner and expected a nearby path to our Tigris object store bucket. The traffic took a longer route through a different on-ramp, which showed up in latency. We also found local routing behavior that caused tail-latency problems in us-east-1. This led to very sawtoothed ingestion charts as SlateDB throttled writes. We eventually worked through these with our cloud provider.

The ingestion benchmark also pushed us toward a simple form of trivial moves. When ingested data has no overlapping key ranges, SlateDB can move it directly from L0 into the next sorted run without rewriting it. RocksDB supports this, but we hadn’t bothered to implement it. Our sustained-ingest workload showed it was worth doing. We’ve opted to disable the setting in the benchmark so we continue to represent baseline performance. We have a more sophisticated trivial move implementation in the works, too.

Another result came from our new cost performance metrics. We recently added distributed compaction support. Compaction can now run on one or more remote machines. This implementation involves periodically checking persisted compaction state in object storage to see if new work is scheduled.

Most SlateDB deployments keep the writer, compactor, and garbage collector in one process, though. We were still using remote-style object store polling in that scenario. The benchmark showed that an idle database with default settings cost roughly $40 per-month in API polling fees. We made the local coordination path in-memory while preserving the state needed for remote compactors to participate. The idle cost fell to under $5. In practice, you would tune your compactor settings or close idle databases, but $40 per-month is clearly excessive.

The benchmarks also changed how we think about disk caching. SlateDB has a block cache and an object-store cache. The block cache holds pieces such as data blocks, index blocks, filters, and metadata. The object-store cache works at a larger granularity; its default partition size is 4 MiB. That granularity is expensive when a cache miss occurs. A request for a small key can trigger a 4 MiB fetch from object storage. This appeared as significant latency spikes in our P99 metrics.

We are now planning a more capable object-store mirror cache with read-through, write-through, and write-back modes. The division of labor is clearer, too. If you want to keep the whole database local, an object-level cache makes sense. If you only want partial local caching, the Foyer hybrid cache is usually a better fit. It already supports eviction and works with much smaller units of data so a cache miss is less costly. ZeroFS has shown us that a prefetching cache for spatial locality is also possible.

More to come

Performance tuning is a never-ending endeavor. We will continue driving SlateDB’s cost and latency down while improving its throughput. We now have a means to evaluate that progress. The results are available at benchmark.slatedb.io if we’ve piqued your curiosity.