Durability & Consistency

ZeroFS makes each filesystem mutation visible atomically, orders mutations through one commit path, and persists them through explicit or periodic flush barriers.

Overview

File contents live as compressed, encrypted frames inside immutable segment objects. Metadata—including inodes, directory entries, and one 32-byte FrameLoc per 32 KiB extent—lives in an object-backed LSM tree. The recovered state after a crash is an atomic prefix of the mutations that were visible before the crash. An explicit durability barrier determines which prefix must survive.

Startup reads the latest durable manifest; it does not replay a journal or run a filesystem repair pass.

Write Atomicity

Each filesystem mutation executes in one database transaction. A write transaction contains its extent pointers, per-segment accounting changes, inode metadata, directory changes, and global statistics updates. The extent bytes are compressed, encrypted, and appended to the in-memory open segment while the transaction is staged; the foreground write does not issue an object-store PUT.

The transaction's metadata changes commit as one WriteBatch. The possible outcomes are:

Frames left in a segment without committed pointers are unreachable and are reclaimed later. A transaction never exposes only some of its metadata or extent pointers.

Operation Ordering

All write transactions pass through one commit worker. It drains pending transactions, merges them into an atomic batch, and submits batches in order. Per-inode locks serialize mutations that inspect the same file or directory; the commit worker is the sole writer of global statistics and per-segment counters.

If operation A completes before operation B begins, B is submitted in a later batch. Therefore:

visible(B) → visible(A)

After a crash, the recovered durable state preserves a prefix of that order. It cannot contain B while omitting an earlier completed A. Mutations in the unflushed suffix may all be absent.

Crash Recovery

SSTs and segment objects are immutable. Each segment frame has an authenticated-encryption tag, while the segment footer includes a CRC32C for torn-write detection.

A flush uploads the open data segment before flushing metadata that refers to it. A durable manifest therefore never points to an unuploaded segment. A crash between those steps can leave an unreferenced segment object, which garbage collection later removes, but not a durable dangling extent pointer.

On startup, ZeroFS reads the latest manifest and opens the referenced SSTs. It does not scan the namespace, replay a filesystem journal, or run an fsck-style repair step.

Durability Semantics

Ordinary writes can return while file data remains in the in-memory open segment and metadata remains in the LSM memtable. The open segment is also sealed in the background when it reaches 256 MiB; at most four uploads run concurrently.

A flush runs under one barrier: it seals and uploads the open segment, then flushes the metadata memtable. If the segment seal or upload fails, metadata is not flushed. A flush occurs when:

• a client requests durability (fsync, NFS COMMIT, or NBD FLUSH/FUA);
• the periodic interval elapses (flush_interval_secs, 30 seconds by default, 5 seconds minimum);
• a batch completes with sync_writes = true;
• segment garbage collection begins; or
• the process shuts down gracefully.

There is no independent memtable-capacity flush. Applications that require a particular write to survive a crash must use the synchronization operation provided by their access protocol, subject to the protocol differences below.

Verified fsync

A successful fsync is verified: it means every mutation recorded for the target inode before it is durable on object storage, including mutations issued through another handle for the same inode. Acknowledged file data lives in the in-RAM open segment buffer, and its metadata in the memtable, until the next flush, so a crash before that flush loses them. ZeroFS does not let that loss pass silently. A fsync issued after a restart, over writes made before it, returns a stale-handle error (ESTALE) rather than a false success.

Each acknowledged write is tagged with a durability lineage token naming the current unbroken durable lineage. A restart reopens the database and, with no way to prove it inherited the un-flushed buffers, regenerates the token. A fsync presents the token its writes were made under; if the database has since regenerated, the tokens differ and the fsync fails. The first fsync after a write is the durability boundary, so a failure reports real loss: redo the writes and fsync again.

This holds on a single node and under replication alike. With a replicated standby a clean failover can prove it inherited the writes (the standby receives un-flushed extent frames over replication and materializes their segments at takeover) and keeps the token, so the fsync succeeds transparently; the failover cases are diagrammed in High availability. The check is per inode and matches POSIX: a file fsync covers that file's data and metadata across every handle, while a directory fsync covers its directory-entry mutations, including links, unlinks, and renames. syncfs uses a filesystem-wide scope.

Eliminating the Durability Window

By default, writes between fsync calls live in the open segment buffer and memtable and become durable only when the next periodic flush or explicit fsync persists them. Committed-but-unflushed file data is bounded by the 256 MiB open buffer plus up to 4 × 256 MiB of in-flight segment uploads. For workloads that need every write durable on return, set sync_writes = true in the [lsm] section.

When enabled, the commit coordinator forces a flush after every coalesced write batch before returning success to the caller. Concurrent writes still merge into a single batch and a single flush.

The trade-off is per-operation latency. Each batch waits for the full durability barrier: the open segment is sealed and uploaded, then the memtable flushes to an L0 SST. This is expensive for chatty workloads.

[lsm]
sync_writes = true

Protocol Considerations

The durability guarantees available to applications depend on the access protocol.

Verification Through Crash Testing

ZeroFS verifies its consistency guarantees through systematic crash simulation using failpoints injected throughout the data path.

verify_all()
  enumerate_inodes()
  enumerate_tombstones()
  enumerate_orphans()
  walk_directory_tree()
  verify_directory_counts()
  verify_nlink_counts()
  verify_directory_nlinks()
  find_orphaned_inodes()
  verify_orphan_set_drained()
  verify_stats_counters()
  verify_tombstones()
  verify_file_extents()
  verify_inode_counter()
  verify_orphaned_extents()
  verify_dir_entry_scan_consistency()
  verify_orphaned_directory_metadata()
  verify_dir_cookie_counters()

Test Methodology

For each failpoint, the test suite:

1. Performs a filesystem operation with the failpoint enabled
2. Aborts the operation at the injection point and tears down the filesystem instance, discarding all in-memory state
3. Restarts the filesystem from object storage
4. Runs the full consistency checker
5. Verifies either complete rollback or complete commit, never partial state

This verifies that the atomicity and ordering guarantees hold at each injected crash point.

Deterministic Simulation Testing

The data path, namespace mutations, garbage collection, compaction, and crash recovery also run under deterministic simulation. Each run uses a single-threaded runtime and virtual time. A seed assigns object-store latency and therefore the order in which concurrent tasks wake; replaying that seed reproduces the schedule. Fresh seeds accumulate coverage, while a companion test verifies that replay produces the same execution trace.

Each simulated world runs several kinds of work concurrently:

• file writes, truncates, hole punches, reads, scans, and fsyncs;
• writers updating separate regions of one inode;
• namespace changes, including rename, links, special nodes, and open-unlink cleanup; and
• live garbage-collection passes.

The simulation uses the production retry and durability paths over an in-memory object store. Some runs inject transient failures whose mutation succeeds but response is lost, forcing retries to meet an already-applied write. At a seeded point, the process crashes: tasks are cancelled, in-flight storage requests stop, and a new instance opens the surviving objects.

At every quiesced point and after every crash, the state must satisfy a model-based oracle:

1. Each ordinary file and shared-inode region equals the replay of an operation prefix no older than the last acknowledged fsync. One in-flight operation of unknown fate is permitted per actor at the cut, and the shared file's durable size is fixed.
2. The namespace tree is structurally isomorphic to a prefix of its operation log: paths, node kinds and values, file contents, and hardlink identity must all match.
3. Every referenced extent still resolves: the filesystem is re-read in full, so a dangling frame pointer or a wrongly deleted segment surfaces immediately. Modeled data files also require a key for every nonzero extent and reject keys beyond EOF.
4. Segment accounting reconciles through typed snapshots: each counter's live bytes equal the byte sum of the extent pointers referencing it, live bytes never exceed appended bytes, and every referenced segment has a counter.
5. All incremental footprint gauges (segment count, appended bytes, live bytes, and reclaimable bytes) equal an authoritative scan.
6. The full consistency checker passes, including directory and link invariants, extent readability and EOF bounds, orphan metadata, inode and directory-cookie counters, and the global stats behind statfs.

Garbage-collection passes run with seeded per-pass tuning so the rarely-taken paths execute too: checkpoint-pinned passes, the write-cold gates (tail scrub, chain compaction), small round budgets that exercise the gather caps, and multi-batch drains.

A companion mode combines the simulation with failpoints for crash windows that contain no await point. It can pause between a garbage-collection decision and its irreversible action, allow concurrent commits to land, and then crash at that exact code location. The pointwise failpoint suite remains as a separate guard for each window.

CI runs the simulation as a dedicated test target with the required SlateDB and Tokio configuration flags. A scheduled job additionally soaks fresh seeds for 50 minutes every hour.

export DST_RUSTFLAGS="--cfg dst --cfg tokio_unstable --cfg io_uring_skip_arch_check"
RUSTFLAGS="$DST_RUSTFLAGS" cargo test --test dst

# A timed soak: fresh random seeds, fanned across all cores, until the
# budget elapses. Scale depth with DST_ROUNDS / DST_OPS / DST_CRASH_PCT.
DST_WALL_CLOCK_SECS=3600 RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --test dst -- --nocapture seeds

# Failpoint-placed crashes and widened windows
RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --features failpoints --test dst crash_points

# Reproduce a failure from its printed seed
DST_SEEDS=16039913183294875603 RUSTFLAGS="$DST_RUSTFLAGS" \
  cargo test --test dst -- --nocapture

Continuous Integration Testing

In addition to crash simulation, CI runs pjdfstest, stress-ng, Linux kernel builds, Jepsen's local-fs suite, and a ZFS integration test on every commit.

Model-Based Testing

Jepsen (local-fs): Generates random sequences of filesystem operations and checks each result against a reference model, shrinking any divergence to a minimal failing case. It runs over a 9P mount. A crash mode injects the loss of un-fsynced writes: it kills the server mid-run, dropping the in-memory open segment buffer and memtable, then recovers from the object store and verifies the surviving state is consistent with the last fsync.

Layered Filesystem Testing

The CI suite includes a test that creates a ZFS pool on a ZeroFS NBD block device:

1. Create a 3GB block device file via 9P
2. Connect via NBD and create a ZFS pool
3. Extract the Linux kernel source (~80,000 files)
4. Compute checksums of all files
5. Export the ZFS pool and restart ZeroFS
6. Reimport the pool and verify all checksums match

This test validates data integrity through multiple filesystem layers and across process restarts.

Test Matrix

All tests run on every pull request and merge to the main branch.

Conditional Writes and Fencing

ZeroFS uses conditional writes (put-if-not-exists) for fencing to prevent split-brain scenarios when multiple instances access the same storage backend. This ensures that only one writer can be active at a time.

AWS S3, Azure Blob Storage, and Google Cloud Storage support conditional writes natively. For S3-compatible object stores that do not support conditional puts, ZeroFS can use Redis as a coordination backend:

[aws]
conditional_put = "redis://localhost:6379"

When configured, ZeroFS uses Redis to coordinate conditional write operations, providing the same fencing guarantees as native conditional put support. See the configuration reference for details.

Backend Durability

ZeroFS delegates storage durability to the object storage backend.

ZeroFS considers a flush durable after the backend acknowledges the required object writes. The failures that acknowledgment protects against depend on the provider, storage class, region, and replication configuration; ZeroFS does not add another remote replica outside that configuration.