# Metrics

> How to collect and export SlateDB metrics using the MetricsRecorder trait

{/* Metric reference tables have long names; horizontal scroll avoids wrapping. */}
<style>{`
  .sl-markdown-content table {
    display: block;
    overflow-x: auto;
    white-space: nowrap;
  }
`}</style>

SlateDB uses a recorder-based metrics system. You provide an implementation of the
`MetricsRecorder` trait and SlateDB pushes metrics to it. If no recorder is configured,
all metric operations are silently discarded with nearly no overhead.

## Configuring a recorder

Pass your recorder to `DbBuilder`:

```rust
use slatedb::{config::MetricLevel, Db, Settings};
use slatedb::object_store::memory::InMemory;
use slatedb_common::metrics::MetricsRecorder;
use std::sync::Arc;

let recorder: Arc<dyn MetricsRecorder> = /* your implementation */;
let settings = Settings {
    metric_level: MetricLevel::Debug,
    ..Settings::default()
};

let db = Db::builder("my_db", Arc::new(InMemory::new()))
    .with_settings(settings)
    .with_metrics_recorder(recorder)
    .build()
    .await
    .unwrap();
```

### Metric levels

`MetricLevel` controls which metrics are registered. Metrics below the configured
level are replaced with no-op handles when they are registered.

`MetricLevel::Info` is the default and registers standard operational metrics.
`MetricLevel::Debug` also registers debug-level metrics when SlateDB defines
them. Configure the level with `Settings::metric_level`.

## The MetricsRecorder trait

`MetricsRecorder` has four registration methods, one per metric type:

```rust
pub trait MetricsRecorder: Send + Sync {
    fn register_counter(&self, name: &str, description: &str, labels: &[(&str, &str)])
        -> Arc<dyn CounterFn>;
    fn register_gauge(&self, name: &str, description: &str, labels: &[(&str, &str)])
        -> Arc<dyn GaugeFn>;
    fn register_up_down_counter(&self, name: &str, description: &str, labels: &[(&str, &str)])
        -> Arc<dyn UpDownCounterFn>;
    fn register_histogram(&self, name: &str, description: &str, labels: &[(&str, &str)], boundaries: &[f64])
        -> Arc<dyn HistogramFn>;
}
```

Each method returns a handle that SlateDB calls on the hot path:

| Trait | Method | Value type | Use case |
|-------|--------|------------|----------|
| `CounterFn` | `increment(u64)` | Monotonic | Request counts, bytes written |
| `GaugeFn` | `set(i64)` | Absolute | Memory usage, queue depth |
| `UpDownCounterFn` | `increment(i64)` | Additive (positive or negative) | In-flight compactions |
| `HistogramFn` | `record(f64)` | Distribution | Latency, I/O sizes |

`GaugeFn` and `UpDownCounterFn` are separate following OpenTelemetry semantics:
gauges represent point-in-time snapshots (`set`), while up-down counters track
additive changes (`increment` with positive or negative values).

Labels are fixed at registration time as `&[(&str, &str)]` key-value pairs. SlateDB
uses labels to collapse related metrics into a single name (e.g. `slatedb.db.request_count`
with an `op` label instead of separate counters per operation).

## Available metrics

All metric names use dot-separated notation: `slatedb.<subsystem>.<metric>`.

### Database (`slatedb.db.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.db.request_count` | counter | `op`: get, scan, flush | Number of DB requests |
| `slatedb.db.write_ops` | counter | | Write operations |
| `slatedb.db.write_batch_count` | counter | | Write batches |
| `slatedb.db.backpressure_count` | counter | | Backpressure events |
| `slatedb.db.l0_stall_count` | counter | `type`: num_ssts, num_ssts_per_key | L0 flush dispatch stalls by cause |
| `slatedb.db.immutable_memtable_flushes` | counter | | Immutable memtable flushes |
| `slatedb.db.total_mem_size_bytes` | gauge | | Total memory usage |
| `slatedb.db.l0_sst_count` | gauge | | L0 SST count summed across all segment trees |
| `slatedb.db.segment_max_l0_sst_count` | gauge | | Maximum L0 SST count across all segments (useful for detecting backpressure on specific segments) |
| `slatedb.db.sorted_run_count` | gauge | | Number of sorted runs across all trees (root tree plus each named segment tree per RFC-0024) |
| `slatedb.db.sst_view_count` | gauge | | Total number of SST views including L0 views and every sorted-run SST view across all trees |
| `slatedb.db.sst_count` | gauge | | Number of distinct physical SSTables (deduplicates sst_view_count by SsTableId, so sst_count is at most sst_view_count) |
| `slatedb.db.external_db_count` | gauge | | Number of external databases referenced in the manifest |
| `slatedb.db.sst_filter_false_positive_count` | counter | | Bloom filter false positives |
| `slatedb.db.sst_filter_positive_count` | counter | | Bloom filter positives |
| `slatedb.db.sst_filter_negative_count` | counter | | Bloom filter negatives |

### Write-Ahead Log (WAL) (`slatedb.wal.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.wal.wal_buffer_flushes` | counter | | WAL buffer flushes |
| `slatedb.wal.wal_buffer_flush_requests` | counter | | WAL buffer flush requests |
| `slatedb.wal.wal_buffer_estimated_bytes` | gauge | | Estimated WAL buffer size |

### Block cache (`slatedb.db_cache.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.db_cache.access_count` | counter | `entry_kind`: filter, index, data_block, stats; `result`: hit, miss | Cache accesses |
| `slatedb.db_cache.error_count` | counter | | Cache errors |

### Compactor (`slatedb.compactor.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.compactor.bytes_compacted` | counter | | Total bytes compacted |
| `slatedb.compactor.epoch` | gauge | | Current compactor epoch |
| `slatedb.compactor.last_compaction_timestamp_sec` | gauge | | Last compaction time (epoch seconds) |
| `slatedb.compactor.running_compactions` | up_down_counter | | Currently running compactions |
| `slatedb.compactor.total_bytes_being_compacted` | gauge | | Bytes in active compactions |
| `slatedb.compactor.total_throughput_bytes_per_sec` | gauge | | Compaction throughput |

### Garbage collector (`slatedb.gc.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.gc.deleted_count` | counter | `resource`: manifest, wal, compacted, compactions | Deleted resources |
| `slatedb.gc.count` | counter | | GC runs |

### Memtable flush (`slatedb.memtable_flush.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.memtable_flush.memtable_freeze_count` | counter | | Frozen memtables |
| `slatedb.memtable_flush.checkpoint_request_count` | counter | | Checkpoint requests |
| `slatedb.memtable_flush.flush_request_count` | counter | | Flush requests |
| `slatedb.memtable_flush.l0_upload_count` | counter | | Completed L0 uploads |
| `slatedb.memtable_flush.l0_flush_count` | counter | | Completed L0 flushes |
| `slatedb.memtable_flush.manifest_refresh_count` | counter | | Manifest refreshes |

### Object store (`slatedb.object_store.*`)

All object store metrics carry four labels:

| Label | Values |
|-------|--------|
| `component` | db, reader, gc, compactor |
| `store_type` | main, wal |
| `op` | get, put, delete |
| `api` | get, get_range, get_ranges, head, list, list_with_offset, list_with_delimiter, put, multipart_init, multipart_part, multipart_complete, delete |

| Name | Type | Description |
|------|------|-------------|
| `slatedb.object_store.request_count` | counter | Total API calls (success and error) |
| `slatedb.object_store.error_count` | counter | Failed API calls |
| `slatedb.object_store.request_duration_seconds` | histogram | Per-request latency |

The instrumented store sits beneath the retrying layer, so each retry attempt
is counted separately. It sits above the optional object-store cache, so these metrics
count the calls SlateDB makes into the cache: a cache hit is counted as one
request even though it never reaches the remote store, and the requests the
cache issues against the remote store to fill a miss are not counted.
Use the cache metrics below to tell hits from misses.

### Object store cache (`slatedb.object_store_cache.*`)

| Name | Type | Labels | Description |
|------|------|--------|-------------|
| `slatedb.object_store_cache.part_hit_count` | counter | | Cache part hits |
| `slatedb.object_store_cache.part_access_count` | counter | | Cache part accesses |
| `slatedb.object_store_cache.cache_keys` | gauge | | Cached keys |
| `slatedb.object_store_cache.cache_bytes` | gauge | | Cached bytes |
| `slatedb.object_store_cache.evicted_keys` | counter | | Evicted keys |
| `slatedb.object_store_cache.evicted_bytes` | counter | | Evicted bytes |

When the cache comes from `object_store_cache_options`, it records to the
recorder configured on the database builder. When you build a
`CachedObjectStore` yourself, it records to a no-op recorder unless you pass
one with `with_metrics_recorder`, so these metrics stay empty until you do.

These are passed to `register_histogram` as the `boundaries` parameter.

## Using DefaultMetricsRecorder

`slatedb-common` ships a `DefaultMetricsRecorder` backed by atomics. It's useful
in tests and in some production scenarios that don't have a dedicated metrics
backend (e.g. periodic logging):

```rust
use slatedb_common::metrics::DefaultMetricsRecorder;
use std::sync::Arc;

let recorder = Arc::new(DefaultMetricsRecorder::new());
let db = Db::builder("test_db", object_store)
    .with_metrics_recorder(recorder.clone())
    .build()
    .await
    .unwrap();

// ... perform operations ...

let snapshot = recorder.snapshot();
for metric in snapshot.all() {
    println!("{}: {:?} {:?}", metric.name, metric.labels, metric.value);
}
```

## Using the metrics-rs facade

The [`metrics`](https://crates.io/crates/metrics) crate is a lightweight facade
(similar to `log` for logging). Install any compatible exporter
(e.g. `metrics-exporter-prometheus`) and wire it up with a thin adapter:

```rust
use metrics::{counter, describe_counter, describe_gauge, describe_histogram, gauge, histogram};

struct MetricsRsCounter(metrics::Counter);

impl CounterFn for MetricsRsCounter {
    fn increment(&self, value: u64) {
        self.0.increment(value);
    }
}

// Gauge, UpDownCounter, and Histogram wrappers follow the same pattern.

pub struct MetricsRsRecorder;

impl MetricsRecorder for MetricsRsRecorder {
    fn register_counter(
        &self, name: &str, desc: &str, labels: &[(&str, &str)]
    ) -> Arc<dyn CounterFn> {
        let labels: Vec<(String, String)> = labels.iter()
            .map(|(k, v)| (k.to_string(), v.to_string())).collect();
        describe_counter!(name.to_string(), desc.to_string());
        Arc::new(MetricsRsCounter(counter!(name.to_string(), &labels)))
    }
    // ... other methods follow the same pattern
}
```

`MetricsRsRecorder` is stateless since the `metrics` facade manages all state
globally.

## Implementing a Prometheus recorder

A Prometheus recorder maps each `register_*` call to a labeled time series
within a `Family`:

```rust
use prometheus_client::metrics::counter::Counter as PromCounter;
use prometheus_client::metrics::family::Family;
use prometheus_client::registry::Registry;

type Labels = Vec<(String, String)>;

// Thin wrapper that bridges prometheus-client's Counter to SlateDB's CounterFn.
struct PromCounterHandle(PromCounter);

impl CounterFn for PromCounterHandle {
    fn increment(&self, value: u64) {
        self.0.inc_by(value);
    }
}

struct PrometheusRecorder {
    registry: Mutex,
    counters: Mutex>>,
    // ... gauges, histograms
}

impl MetricsRecorder for PrometheusRecorder {
    fn register_counter(
        &self, name: &str, desc: &str, labels: &[(&str, &str)]
    ) -> Arc<dyn CounterFn> {
        let mut families = self.counters.lock().unwrap();
        let family = families.entry(name.to_string()).or_insert_with(|| {
            let f = Family::::default();
            self.registry.lock().unwrap().register(name, desc, f.clone());
            f
        });
        let labels: Labels = labels.iter()
            .map(|(k, v)| (k.to_string(), v.to_string())).collect();
        let counter = family.get_or_create(&labels).clone();
        Arc::new(PromCounterHandle(counter))
    }
    // ... other methods follow the same pattern
}
```

## Implementing an OpenTelemetry recorder

A recorder that maps directly to the OpenTelemetry SDK instruments:

```rust
use opentelemetry::metrics::MeterProvider;
use opentelemetry::KeyValue;
use opentelemetry_sdk::metrics::SdkMeterProvider;

struct OtelRecorder {
    meter: opentelemetry::metrics::Meter,
}

impl OtelRecorder {
    fn new(provider: &SdkMeterProvider) -> Self {
        Self { meter: provider.meter("slatedb") }
    }
}

impl MetricsRecorder for OtelRecorder {
    fn register_counter(
        &self, name: &str, desc: &str, labels: &[(&str, &str)]
    ) -> Arc<dyn CounterFn> {
        let attrs: Vec = labels.iter()
            .map(|(k, v)| KeyValue::new(k.to_string(), v.to_string()))
            .collect();
        let counter = self.meter.u64_counter(name.to_string())
            .with_description(desc.to_string()).build();
        Arc::new(OtelCounter { counter, attrs })
    }
    // ... other methods follow the same pattern
}
```