Garbage Collection
Deleting a file removes it from the filesystem namespace immediately. Its bytes are reclaimed later by background stages. File contents live in immutable objects under segments/; storage usage decreases only when segment garbage collection deletes or repacks those objects.
What Deletion Does
Unlink removes the inode in a single transaction. What happens to the file's extents depends on the file's size:
- Files of 10 extents or fewer (320 KiB at the 32 KiB extent size) have their extent keys deleted in the same transaction.
- Larger files leave a tombstone, a record of the file's remaining size. The extents stay in place until the garbage collector processes the tombstone.
If the file is still open, the unlink only detaches the inode from the namespace and records it as an orphan; the same small-file/tombstone decision runs when the last handle closes.
Deleting an extent key removes a 32-byte pointer (a FrameLoc) from the metadata store and debits the live-byte counter of the segment that holds the extent's frame. It does not touch the segment object; the data bytes remain in object storage until segment GC removes or repacks the segment.
Renaming over an existing file deletes the replaced file through the same path. In both cases the operation returns before any space is reclaimed.
Deletion decision
file size <= 10 extents (320 KiB)
-> extent keys deleted in the
unlink transaction
file size > 10 extents
-> tombstone written;
extents remain until GC
file still open at unlink
-> inode orphaned; decision
deferred to last close
The Reclamation Pipeline
File contents are packed as encrypted frames into immutable segment objects; the metadata LSM holds one 32-byte pointer per extent and a live-byte counter per segment. Space returns in two stages for file data and one for metadata. Each is eventual, and none is triggered by the deletion itself:
- Tombstone GC deletes the file's extent pointer keys and debits each source segment's live-byte counter. No object is deleted; the segment objects still hold the frames.
- Segment GC deletes segment objects whose live-byte counter has reached zero and repacks fragmented ones. This is where billed object-store usage for file data decreases.
- Metadata compaction and object GC rewrite and delete metadata SSTs. The SSTs hold only metadata (inodes, directories, extent pointers, counters), so this stage reclaims metadata space, not file data.
A delay between deleting files and object-store usage decreasing is normal, not a fault. Each stage runs in the background on its own schedule.
Stage 1: Tombstone GC
The read-write instance runs a continuous background task that drains the tombstone queue. Each pass processes the queue in rounds of up to 10,000 extents across up to 10,000 tombstones; rounds repeat back to back until the queue is empty, and the task then sleeps 10 seconds before the next pass.
Large files are collected tail-first: each round deletes extents from the end of the file and updates the tombstone's remaining size. A partially collected file is a valid intermediate state, and an interrupted pass resumes from the recorded remaining size.
Each delete runs under the per-inode write lock and, in the same transaction, debits the live-byte counter of every removed frame's source segment. Those counters are how the next stage finds dead and fragmented segments.
After this stage, the extent pointers are gone from the metadata store. The object store still holds the same segment objects.
GC pass
loop:
round:
process up to 10,000 tombstones
delete up to 10,000 extent keys
(tail-first per file)
debit segment live-byte counters
if queue not empty: next round
else: sleep 10 s, start next pass
Stage 2: Segment GC
The writer runs one segment GC pass at startup, then selects the delay before each subsequent pass from three configurable tiers:
| Tier | Default interval | Used when |
|---|---|---|
| Base | 60 seconds | The backlog is drained, an error occurred, or neither faster condition applies. |
| Drain | 15 seconds | The store is active and dead space is at least 20%, or reserve-deferred seams remain. |
| Fast | 5 seconds | The previous pass exhausted its work budget, actionable work remains, and no reads, writes, mutations, or seals occurred since that pass. |
A pass runs at least four compaction batches by default, even under client load. After that floor it continues only while the store remains idle, up to 32 batches, checking for foreground work between batches. The flush barrier and segment-counter scan run once for the entire pass.
Before selecting candidates, the pass seals the open segment buffer and flushes metadata under the flush barrier. Segment GC is therefore also a flush trigger. Each pass logs the selected cadence, its reason, and the work completed or left queued.
The pass then reads every per-segment live-byte counter in one scan; it lists nothing on the object store. Only segments sealed before the pass are eligible. Two actions follow:
Dead-segment deletion. A segment whose counter is zero is a delete candidate. It is deleted only after a per-segment delete horizon has passed, and only after a verification step reads the segment's directory and confirms that no extent pointer still references it. Verification is fail-closed: any read error keeps the segment, since a leaked segment is preferred to lost data. The object DELETE is the point where billed usage decreases.
Segment compaction. Segments below 50% live or smaller than 1 MiB are repacked. Their live frames are fetched with coalesced ranged GETs, sorted by inode and extent, and sealed into new segments of up to 256 MiB.
Drained source segments become dead but are not deleted in the same pass, so in-flight reads of their old locations remain valid. Recent read fan-out supplies candidates for up to half the budget; remaining capacity is filled most-fragmented-first. Read-directed compaction handles dense hot-seam chains, while the tail scrub handles write-cold segments above its dead-space floor.
| Parameter | Value |
|---|---|
| Pass interval | adaptive: 60 s base / 15 s while active with a large dead backlog / 5 s while saturated and idle (all configurable) |
| Delete-horizon floor | now + 60 s |
| Fragmentation threshold | live bytes < 50% of object size |
| Small-segment threshold | < 1 MiB |
| Tail-scrub band | dead fraction in (floor, 50%], write-cold only; floor default 5%, configurable |
| Repack target size | 256 MiB |
| Compaction budget | 64 segments / 256 MiB live per batch (byte budget configurable via compact_round_max_mib, 64–4096 MiB); a floor of 4 batches per pass under load, up to 32 while the store stays idle |
| Minimum payoff to compact | 1 MiB freed, or 64 MiB of live bytes gathered (a quarter of the seal threshold), or at least one chain packed |
| Segment deletes per pass | 1,024 |
| Orphan sweep interval | 24 h wall-clock, persisted across restarts |
A pass issues a 64-byte footer GET plus a directory GET per segment it verifies or repacks, and one DELETE per dead segment; a pass that repacks additionally issues the coalesced ranged GETs for the live frames and one PUT per packed segment. No pass lists the object store, so steady-state GC traffic is proportional to the garbage produced, not to the data owned.
Tail scrubbing. Normal compaction leaves a band of partially dead segments above its fragmentation threshold. The scrub spends leftover budget on that band, most-dead-first, once a segment is write-cold.
A segment is write-cold after 30 seal rotations in the current process, or after the whole store has had no writes for five minutes. Because the scrub uses only budget left after deletion and ordinary compaction, it primarily runs during idle periods.
Orphan sweep. A crash can leave a segment object without a live-byte counter, making it invisible to the normal scan. Once every 24 hours, a persisted schedule lists segments/ by shard and checks whether each object has a counter.
Counter-less objects pass the same fail-closed directory verification and deletion horizon as dead segments, with at most 1,024 deletions per sweep. This is the only path that lists the segment namespace.
Checkpoints Delay Reclamation
Checkpoints pin segment objects:
- An ephemeral checkpoint (including a read replica's auto-renewed reader checkpoint) pushes the delete horizon to its expiry time plus a 30-second clock-skew margin, never below the 60-second floor.
- A persistent checkpoint cannot be timed out. While one exists, segment deletion and compaction are paused entirely; the counters keep tracking garbage and reclamation catches up once the checkpoint is deleted. The orphan sweep is unaffected, because no manifest, and therefore no checkpoint, ever references an orphan.
- If the checkpoint list cannot be read, the pass is skipped. Reclamation fails closed.
Stage 3: Metadata Compaction and Object GC
The metadata LSM holds only inodes, directory entries, tombstones, extent pointers, and segment counters. Its compaction merges metadata SSTs; a coordinator in the writer process polls every 5 seconds and schedules work size-tiered, executed by an embedded worker. There is no standalone compactor process; metadata compaction always runs inside zerofs run.
The metadata store's object garbage collector, also in the writer process, scans its manifest, compacted-SST, and compaction-state objects every 1 minute, and deletes objects that the current manifest no longer references and that are at least 1 minute old. These deletions reclaim metadata space only; file data is reclaimed by segment GC.
Where Reclamation Runs
| Instance | Tombstone GC | Segment GC | Metadata compaction + object GC |
|---|---|---|---|
| Read-write | yes | yes | yes |
| Read-only mount | no | no | no |
| Checkpoint mount | no | no | no |
All reclamation runs in the writer process. Read-only mounts and checkpoint mounts open the database through a reader and run no stage, but their checkpoints delay segment reclamation as described above. Object-store usage can decrease only while the read-write instance is running.
Crash Safety
- A crash after a compaction seal but before the repoint leaves the source segments in place and readable; the packed segment is an orphan with no references, and a later pass deletes it.
- A crash between a segment DELETE and the drop of its counter key leaks one stale counter key, nothing else.
- A counter that under-counts leads to a skipped delete (the fail-closed directory verification), never to a deleted live segment.
These paths are exercised by failpoint crash tests that abort the operation at each injection point, reopen the filesystem, and verify consistency.
Observability
The Prometheus endpoint exports four counters for tombstone GC:
| Metric | Meaning |
|---|---|
zerofs_tombstones_created_total | Tombstones created by deletions |
zerofs_tombstones_processed_total | Tombstones fully collected |
zerofs_gc_extents_deleted_total | Extent keys deleted by GC |
zerofs_gc_runs_total | GC passes started |
All four counters count from process start. Within one run, zerofs_tombstones_created_total minus zerofs_tombstones_processed_total approximates the tombstone backlog; a gap that grows over time means deletions are outpacing collection. Files of 10 extents or fewer never create tombstones, so a small-file workload shows no tombstone activity. The monitor dashboard (zerofs monitor) and the web UI display the same counters.
Segment GC exposes footprint, work-completed, and backlog metrics. The Prometheus reference lists every series and its units; the terminal monitor and Web UI show the main footprint and cadence values.
Each pass also emits an info-level summary with its deletes, compactions, relocated frames, remaining backlog, deferred-chain reasons, and next cadence. The object count and total size under segments/ show the resulting change at the backend.
NBD TRIM
TRIM on an NBD device skips tombstones: the discard transaction deletes the pointer keys of fully covered extents and zeroes the covered portion of partially covered extents (an extent that becomes all zeroes is also deleted). The same transaction debits the source segments' live-byte counters; segment GC then reclaims the space on its own schedule.
Configuration
Values are configurable through the [gc] section (see Configuration): the base pass interval, the idle drain interval, the busy-backlog drain interval and its dead-space trigger, the per-pass batch floor, the per-round compaction budget, the tail-scrub floor, and the read-directed switch (read_directed, default true).