Block Transforms
BlockTransformer lets SlateDB apply a reversible byte transform to SST blocks, usually for encryption. SlateDB applies it to data blocks and to filter, index, and stats blocks in both WAL SSTs and compacted SSTs.
SlateDB does not transform manifest files, compaction-state files, or the trailing SsTableInfo record and final offset/version footer in an SST. It reads that footer first so it can find the stored blocks and learn which compression codec to use.
On write, SlateDB encodes the block, optionally compresses it, passes the bytes to BlockTransformer::encode, computes CRC32 over the transformed bytes, and writes the result to object storage.
On read, SlateDB fetches the stored bytes, verifies the CRC32, calls BlockTransformer::decode, optionally decompresses the decoded bytes, and then decodes the block contents.
The checksum covers the transformed payload, not the original plaintext block. Corruption in storage is detected before SlateDB calls decode. A mismatched transformer usually fails later, during decode, decompression, or block decoding.
If you combine a transformer with Compression, the transformer sees compressed bytes, not raw row encodings.
Contract
Section titled “Contract”The trait has two async methods, encode and decode. decode must invert encode for every block SlateDB writes. The methods are async so implementations can offload CPU-heavy work or call an external service.
SlateDB does not record transformer identity, key ID, or format version in SsTableInfo. If you need key rotation or format changes, make the stored block self-describing or keep compatible configuration outside SlateDB.
Configuration
Section titled “Configuration”Every process that reads or writes SST blocks needs compatible transformer logic.
- Writers use
DbBuilder::with_block_transformer. - Standalone compactors use
CompactorBuilder::with_block_transformer. - Read-only handles use
DbReaderBuilder::with_block_transformer. - SST inspection uses
SstReader::new.
If a writer used a transformer and a reader does not, SlateDB can still open the footer metadata, but reads of the filter, index, stats, or data blocks fail. Readers covers the reader-side configuration in more detail.
WalReader does not expose a BlockTransformer setting today. WAL SSTs written with a custom transformer are not inspectable through that API.
