# Segmented Compaction

> Split a dataset into independent LSM trees, each with its own compaction and retention policy

Segmented compaction splits a database into independent LSM trees keyed by a
prefix.  Segments let each tree independently configure compaction policy to
match its access pattern needs, and let a whole segment be dropped cheaply
once it ages out. All segments share a WAL and writes across segments can
be done transactionally/atomically.

A [`PrefixExtractor`](https://docs.rs/slatedb/latest/slatedb/trait.PrefixExtractor.html)
derives a segment prefix from each key, and each segment becomes its own
logical LSM tree. Because segments own disjoint key ranges, they are compacted
and retired independently, and reads/scans automatically prune to the segments
overlapping the query.

Common fits:

- **Timeseries / append-ordered data**: one segment per time bucket or log
  segment; old segments
  freeze and age out as a unit.
- **Metadata/data separation**: isolate a churny, frequently-read metadata
  keyspace from bulky, write-once data so each gets its own compaction and
  caching policy.
- **Column-family-like isolation**: give small, frequently-overwritten
  structures an aggressive policy while slow-moving data uses a
  low-write-amplification one.

See [RFC 0024](/rfcs/0024-segment-oriented-compaction) for the full design.

## Enabling segments

Configure an extractor at creation with
[`DbBuilder::with_segment_extractor`](https://docs.rs/slatedb/latest/slatedb/struct.DbBuilder.html#method.with_segment_extractor).
The application encodes segment boundaries in the key (e.g. an hour bucket, a log segment id);
the extractor returns the prefix length that identifies the segment. The example below routes
every key to the segment named by its first three bytes.

### Rust

```rust
use std::sync::Arc;
use slatedb::{Db, PrefixExtractor, PrefixTarget};

struct FixedThreeByteExtractor;

impl PrefixExtractor for FixedThreeByteExtractor {
    fn name(&self) -> &str { "fixed_three_byte" }
    fn prefix_len(&self, target: &PrefixTarget) -> Option<usize> {
        let (PrefixTarget::Point(key) | PrefixTarget::Prefix(key)) = target;
        (key.len() >= 3).then_some(3)
    }
}

let db = Db::builder(path, object_store)
    .with_segment_extractor(Arc::new(FixedThreeByteExtractor))
    .build()
    .await?;
```

### Go

```go
type fixedThreeByteExtractor struct{}

func (fixedThreeByteExtractor) Name() string { return "fixed_three_byte" }

func (fixedThreeByteExtractor) PrefixLen(target slatedb.PrefixTarget) *uint64 {
    var key []byte
    switch t := target.(type) {
    case slatedb.PrefixTargetPoint:
        key = t.Key
    case slatedb.PrefixTargetPrefix:
        key = t.Prefix
    }
    if len(key) < 3 {
        return nil
    }
    n := uint64(3)
    return &n
}

builder := slatedb.NewDbBuilder("example-db", store)
if err := builder.WithSegmentExtractor(fixedThreeByteExtractor{}); err != nil {
    panic(err)
}
db, err := builder.Build()
```

### Java

```java
class FixedThreeByteExtractor {
  name() {
    return "fixed_three_byte";
  }

  prefix_len(target) {
    const key = target.tag === "Point" ? target.key : target.prefix;
    return key.length < 3 ? undefined : 3;
  }
}

const builder = new DbBuilder("demo-db", store);
builder.with_segment_extractor(new FixedThreeByteExtractor());
const db = await builder.build();
```

### Python

```python
from slatedb.uniffi import DbBuilder, PrefixExtractor, PrefixTarget

class FixedThreeByteExtractor(PrefixExtractor):
    def name(self) -> str:
        return "fixed_three_byte"

    def prefix_len(self, target: PrefixTarget) -> int | None:
        key = target.key if target.is_point() else target.prefix
        return None if len(key) < 3 else 3

builder = DbBuilder("demo-db", store)
builder.with_segment_extractor(FixedThreeByteExtractor())
db = await builder.build()
```

:::caution
Whether segmentation is enabled is fixed for the life of the database. The
extractor's `name()` is persisted in the manifest; opening with a different
name, or adding or removing an extractor, fails. This is primarily as a soft
check against accidental misuse.
:::

List the segments that currently exist (in the manifest or in memtables) with
[`DbStatus::list_segments()`](https://docs.rs/slatedb/latest/slatedb/struct.DbStatus.html#method.list_segments).

## Evolving the key schema

An extractor's name is fixed, but its implementation may evolve
when the change is backward-compatible. For example, a schema byte can
reserve disjoint key ranges for successive segmentation policies:

```text
# all records below use a schema marker as the first part of the key prefix

# initially use weekly partitions
[01][week]... -> [01][week]  # v1: weekly segments
# later migrate to daily partitions
[02][day]...  -> [02][day]   # v2: daily segments
```

The new implementation must keep extracting the original weekly prefixes for
all v1 keys while adding daily prefixes only for v2 keys. In general:

- Keep the same stable name and preserve the behavior of previous PrefixExtractor
  implementations.
- Ensure segment prefixes from all versions form an antichain: no
  prefix may be a prefix of another (a schema discriminator at the
  start of the key is a simple way to reserve disjoint ranges).

SlateDB checks the name and, on open, asks the configured extractor to
recognize every persisted segment prefix. It also rejects new writes that
would introduce a nested prefix. These are guardrails rather than complete
compatibility verification: SlateDB does not persist the implementation's
version or revalidate every existing key. Maintaining backward-compatible
routing under a stable name is therefore the application's responsibility.

## How it compacts

Each segment is compacted on its own schedule. The default size-tiered
[`CompactionScheduler`](/docs/design/compaction) applies per segment, and
compactions in different segments are parallel-safe (disjoint keys, no
cross-segment ordering). Embed a semantic hint in the prefix (e.g. a time
bucket) and a custom scheduler can vary policy per segment.

:::note
Writing to N segments at once produces up to N× the L0 objects (smaller each);
the WAL is shared, so it is unaffected. Most data models keep one active
segment, so this rarely bites in practice.
:::

## Deleting Segments Wholesale

The default scheduler never drops data; segment retention is the application's
job. To retire a whole segment, schedule a `CompactionSpec::drain_segment`
compaction. Unlike a merge, a drain reads and rewrites nothing. Instead it
detaches the segment's L0s and sorted runs from the manifest, leaving a drain
marker (`l0=[]`, `compacted=[]`) that the writer prunes, after which the
garbage collector reclaims the files.

A drain is a `CompactionSpec::drain_segment` submitted through
[`Admin.submit_compaction`](https://docs.rs/slatedb/latest/slatedb/admin/struct.Admin.html), built
from the segment's L0 SSTs and sorted runs read from `read_compactor_state_view`. Every binding
mirrors this API:

### Rust

```rust
use bytes::Bytes;
use slatedb::admin::Admin;
use slatedb::compactor::{CompactionSpec, SourceId};

let admin = Admin::builder(path, object_store).build();
let view = admin.read_compactor_state_view().await?;

let segment = Bytes::from_static(b"hour=10/");
if let Some(seg) = view.manifest().segment(&segment) {
    // Drain every L0 SST and sorted run currently visible in the segment.
    let sources: Vec = seg
        .l0()
        .iter()
        .map(|v| SourceId::SstView(v.id))
        .chain(seg.compacted().iter().map(|sr| SourceId::SortedRun(sr.id)))
        .collect();
    admin.submit_compaction(CompactionSpec::drain_segment(segment, sources)).await?;
}
```

### Go

```go
admin, err := slatedb.NewAdminBuilder("example-db", store).Build()
if err != nil {
    panic(err)
}
view, err := admin.ReadCompactorStateView()
if err != nil {
    panic(err)
}

var sources []slatedb.SourceId
for _, seg := range view.Manifest.Segments {
    if !bytes.Equal(seg.Prefix, []byte("hour=10/")) {
        continue
    }
    for _, v := range seg.L0 {
        sources = append(sources, slatedb.SourceIdSstView{Field0: v.Id})
    }
    for _, r := range seg.Compacted {
        sources = append(sources, slatedb.SourceIdSortedRun{Field0: r.Id})
    }
}
_, err = admin.SubmitCompaction(slatedb.CompactionSpecDrainSegment{
    Segment: []byte("hour=10/"),
    Sources: sources,
})
```

### Java

```java
const admin = new AdminBuilder("demo-db", store).build();
const view = await admin.read_compactor_state_view();

const prefix = Buffer.from("hour=10/");
const seg = view.manifest.segments.find((s) => prefix.equals(Buffer.from(s.prefix)));
const sources = [
  ...seg.l0.map((v) => SourceId.SstView(v.id)),
  ...seg.compacted.map((r) => SourceId.SortedRun(r.id)),
];
await admin.submit_compaction(CompactionSpec.DrainSegment(seg.prefix, sources));
```

### Python

```python
from slatedb.uniffi import AdminBuilder, CompactionSpec, SourceId

admin = AdminBuilder("demo-db", store).build()
view = await admin.read_compactor_state_view()

seg = next(s for s in view.manifest.segments if s.prefix == b"hour=10/")
sources = [SourceId.SST_VIEW(v.id) for v in seg.l0]
sources += [SourceId.SORTED_RUN(r.id) for r in seg.compacted]
await admin.submit_compaction(CompactionSpec.DRAIN_SEGMENT(segment=seg.prefix, sources=sources))
```

To retire segments automatically (e.g. by a TTL), implement
[`CompactionScheduler`](https://docs.rs/slatedb/latest/slatedb/compactor/trait.CompactionScheduler.html)
and return `drain_segment` specs from `propose`, wired in with
`CompactorBuilder::with_scheduler_supplier`.