Performance
SlateDB’s performance characteristics are primarily determined by the object store being used. This page provides guidance on expected performance and how to tune SlateDB for your use case.
Latency
Section titled “Latency”SlateDB’s write latency is dominated by object store PUT operations. The following table shows expected latencies for different object stores:
| Object Store | Expected Write Latency | Notes |
|---|---|---|
| S3 Standard | 50-100ms | Network latency dominates |
| S3 Express One Zone | 5-10ms | Single-digit millisecond latency |
| Google Cloud Storage | 50-100ms | Network latency dominates |
| Azure Blob Storage | 50-100ms | Network latency dominates |
| MinIO | 5-20ms | Depends on network and disk |
Read latency is primarily determined by:
- Bloom filter misses: If a bloom filter indicates a key might exist in an SST, SlateDB must read the SST from object storage.
- Object store GET latency: The time it takes to download an SST from object storage.
- SST size: Larger SSTs take longer to download and parse.
Throughput
Section titled “Throughput”SlateDB’s write throughput is limited by:
- Object store PUT rate limits: Most object stores limit PUT operations to 3,500 requests per second per prefix.
- Network bandwidth: The time it takes to upload SSTs to object storage.
- SST size: Larger SSTs provide better compression but take longer to upload.
Read throughput is limited by:
- Object store GET rate limits: Most object stores limit GET operations to 5,500 requests per second per prefix.
- Network bandwidth: The time it takes to download SSTs from object storage.
- SST size: Larger SSTs provide better compression but take longer to download.
Tuning
Section titled “Tuning”SlateDB provides several configuration options to tune performance:
Write Performance
Section titled “Write Performance”flush_ms: The time to wait before flushing the mutable WAL to object storage. Lower values reduce write latency but increase object store PUT frequency.l0_sst_size_bytes: The size of L0 SSTs. Larger SSTs provide better compression but take longer to upload.max_unflushed_memtable: The maximum number of unflushed memtables. Lower values reduce memory usage but may increase write latency.
Read Performance
Section titled “Read Performance”bloom_filter_false_positive_rate: The false positive rate for bloom filters. Lower values reduce unnecessary SST reads but increase bloom filter size.block_size: The size of blocks within SSTs. Larger blocks provide better compression but take longer to read individual keys.
Memory Usage
Section titled “Memory Usage”max_memtable_size_bytes: The maximum size of the in-memory memtable. Lower values reduce memory usage but may increase write latency.max_unflushed_memtable: The maximum number of unflushed memtables. Lower values reduce memory usage but may increase write latency.
Benchmarks
Section titled “Benchmarks”SlateDB currently has two benchmarking tools: bencher and microbenchmarks.
It comes with a bencher tool to benchmark put/get operations against an object store. You can configure the tool with a variety of options. See bencher/README.md for details. benchmark-db.sh also serves as an example.
Nightly
Section titled “Nightly”We run bencher nightly and publish the results here. The benchmark runs on WarpBuild’s warp-ubuntu-latest-x64-16x runners, which use Hetzner machines in Frankfurt. We use Tigris for object storage with the auto region setting, which resolves to Frankfurt as well. Bandwidth between WarpBuild (Hetzner) and Tigris seems to be about 500MiB/s down and 130MiB/s up. We routinely max out the bandwidth in the nightly tests.
Microbenchmarks
Section titled “Microbenchmarks”We use Criterion to run microbenchmarks (located in benches) for specific internal SlateDB functions. A comment is left on all PRs when a > 200% slowdown is detected.
Nightly
Section titled “Nightly”Microbenchmarks also run nightly with the pprof-rs profiler. The resulting profiler protobuf files are published here. The microbenchmarks run on standard Linux Github action runners.
We highly recommend using pprof.me to view the <microbenchmar>.pb files, though any pprof compatible tool may be used.