Daniil Medovich

One mmap for you, or two SIGBUSes for someone else?

EN RU

TL;DR. Amber can run embedded inside another process, and that raises the bar for failure semantics: any I/O error that cannot be handled safely must stop the system cleanly, locally, and visibly, instead of letting it continue silently, corrupt data, and leave the system in an undefined state. This matters. A lot.

1. mmap in brief

mmap takes a file and makes its bytes available as an ordinary slice in the address space. After that we "read the file" by accessing memory, while the kernel lazily loads pages as needed and keeps them in the page cache.

import "syscall"

f, _ := os.Open("segment.fidx")
defer f.Close()
fi, _ := f.Stat()

// data is a []byte as long as the file, but nothing is really in RAM yet:
data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()),
    syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil { /* ... */ }
defer syscall.Munmap(data)

// This is the "read". It looks like nanosecond memory access:
v := binary.BigEndian.Uint64(data[off : off+8])

The same operation through explicit I/O. pread (in Go, ReadAt) is positional reading: one syscall, reads n bytes at offset off, does not touch the shared file offset, and returns (n, err). That last part is everything we need: failure is a value, and it is visible.

var buf [8]byte
if _, err := f.ReadAt(buf[:], off); err != nil {
    return err            // error is a value, and it is visible
}
v := binary.BigEndian.Uint64(buf[:])

The temptation is obvious: binary search over an mmap slice is indistinguishable from binary search over a regular slice. No syscalls in the hot loop, no buffer management, OS page cache for free.

What we really get at data[off]

The line v := data[off] lies about its cost. If the page containing off is not resident:

  1. The CPU raises a page fault.
  2. Control enters the kernel. The kernel checks whether the page is in page cache.
  3. If not, the kernel synchronously reads it from disk. The thread waits.
  4. The page is mapped, the instruction is restarted, and data[off] "returns a value".

All of that is invisible in the source code. What looks like a memory access can become blocking disk I/O measured in milliseconds, and we do not control when it happens because it depends on the page cache state controlled by the kernel.

Thanks, scheduler

When we call file.ReadAt, the goroutine enters a syscall through entersyscall and moves to _Gsyscall; its P moves to _Psyscall. The P is not detached immediately: the runtime optimistically keeps it in case the syscall returns quickly. But if the read really blocks on disk, sysmon sees a P stuck in a syscall and hands it to another M via handoffp. Other goroutines can continue on that P. sysmon will not hand it off instantly, but it can hand it off because the goroutine state is visible to the runtime.

A page fault on mmap memory is not a Go syscall. It is a CPU exception handled transparently by the kernel. The runtime does not know about it: no entersyscall, no goroutine state transition. From the runtime's point of view the goroutine is still _Grunning on a _Prunning P, while the M is stuck in the fault handler. Here sysmon is powerless: it cannot hand off the P, because it is not _Psyscall, and it cannot preempt the M because it is stuck in the kernel and will not reach a safepoint until the fault resolves.

So mmap does not merely hide I/O. It hides it from the Go runtime too, depriving the scheduler of information it is built around.

2. Why I wanted mmap in amber

In the FTS index we have a "unique section": sorted 64-bit hashes of df==1 tokens, around 13 MB per segment. There are dozens of segments. Keeping all of that resident means hundreds of megabytes of RAM for data that is rarely touched.

mmap would solve this "for free": the whole section is addressable, only the pages actually touched become resident, cold pages are evicted by the kernel, and binary search over sorted hashes stays trivial code with zero syscalls per probe.

This is a textbook mmap case: a large read-only sorted file, point random lookups, and a desire to use the OS page cache. LMDB is built on this and works. So the question is honest: why not us?

The CIDR 2022 tradeoff frame

mmap has four classes of problems:

  1. Transactional safety. mmap can flush dirty pages at arbitrary times; msync does not provide strict ordering and can break WAL guarantees. Our case is read-only, so this mostly does not apply.
  2. I/O stalls. Any access can fault and block implicitly; we lose control over when I/O happens and cannot prefetch meaningfully.
  3. Error handling. I/O errors during a page fault arrive as SIGBUS, not as return values. For us this is fatal.
  4. Performance under eviction. Page eviction in the kernel is single-threaded; on many cores, each eviction can require TLB shootdowns through inter-processor interrupts. Transparent huge pages can add latency spikes.

It is important to be fair: mmap is not evil in general. For read-mostly, fits-in-RAM, single-node data on stable local storage, it is a great tool. Even if we set fail-stop aside, mmap is not an obvious win for our profile. But that is context. The deciding factor is our own invariant.

3. The argument amber requires: fail-stop

fail-stop is a contract: after detecting an error that cannot be handled safely, the system stops cleanly, locally, and visibly, instead of continuing in a distorted or ambiguous state. amber is built around this.

mmap is incompatible with this contract. SIGBUS is the opposite of fail-stop.

fail-stop says: detect the error, stop cleanly, return it as a value. SIGBUS says: the process dies on an arbitrary instruction, and turning that into a carefully handled error is not realistically viable. In C you can try signal handlers and sigsetjmp/siglongjmp; in the Go runtime, with goroutine stacks and its own signal handling, this is not a serious option. In an embedded library it is even worse: we would be intercepting signals in someone else's process.

And amber is embedded. It lives inside the user's process. A SIGBUS from a flaky disk kills not only amber, but the host application. You cannot ship an embedded library that takes down somebody else's process because a network disk blinked.

So mmap is categorically excluded by contract: mmap is an abstraction that makes I/O invisible. We built the storage around the invariant that I/O and I/O failures must be visible and returned as values.

Where that led

It led to an index format where the hot section is read through explicit pread on demand: every probe is (n, err). A bad block becomes a Go error that we raise upward; the reader is refcounted; the failure is localized; the host process lives. This format is AFT2.

The important point: pread does not take page cache away from us. File pages are cached by the kernel with ReadAt just as they are with mmap. We do not lose the main thing mmap was supposed to give us. We only pay syscalls: O(log n) probes per lookup instead of zero under mmap. For the df==1 section that is roughly 20 reads on a cold, rare path: a tiny price for making every error a returned value and making I/O visible to the scheduler again.

4. AFT2, our index format

AFT2 replaced the radix snapshot from fts-engine.

The road to the format

  1. We started with the fts-engine radix snapshot. It stored each posting entry as a decimal DocID string inside serialized tree nodes. On a 100k segment the result was 38 MB on disk, 150 MB in RAM, and 400 ms to parse. .fidx files were 92% of amber storage and dominated RSS.
  2. The token distribution insight: in logs, about 80% of tokens have df==1, appearing in exactly one record. These are UUID stems, trace IDs, and identifiers. Keeping them in a normal string dictionary means around 40 bytes of string/slice headers per token, or about 50 MB of headers per segment in the executor LRU.
  3. Split by document frequency. Two populations, two representations.

The .fidx structure (AFT2)

[ "AFT2" magic ]
[ df>=2 section: sorted token dictionary + delta-varint uint64 posting lists ]
[ df==1 section: two flat arrays of 8-byte values:                           ]
[                fnv64a(token), sorted ascending                             ]
[                || parallel full entry IDs                                  ]
[ CRC32 ]

df>=2 is tens of thousands of tokens, mostly message-template words. It is resident and searched through binary search over the dictionary. There are not many of them, so it is cheap.

df==1 is the 80%. No string headers: only hashes and IDs, 8 bytes each. Collisions do not lose records: equal hashes sit next to each other, and lookup returns all entries with that hash. A 64-bit collision inside a segment (~1M tokens gives about 3e-8 birthday probability) can at worst add one unrelated record, but it never loses the right one; result-count equality gates catch this in benchmarks.

Mechanics

In loaded mode, the df==1 section is not resident. It is searched with pread over the file. After Save(), the index is demoted to file-backed: resident hash arrays (~13 MB per segment) are cleared, and only path + offset + count remain. A freshly sealed index costs the executor cache the same couple of megabytes as a loaded one.

The binary search itself, from lookupUnique in file-backed mode:

h := tokenHash(token)
var buf [8]byte
readU64 := func(off int64) (uint64, bool) {
    if _, err := file.ReadAt(buf[:], off); err != nil {
        return 0, false          // bad block is a value, not SIGBUS
    }
    return binary.BigEndian.Uint64(buf[:]), true
}

lo, hi := 0, n
for lo < hi {
    mid := (lo + hi) / 2
    v, ok := readU64(uniqOff + int64(mid)*8)
    if !ok { return nil }        // fail-stop in miniature
    if v < h { lo = mid + 1 } else { hi = mid }
}

The whole point is the line if !ok. With mmap this probe would be data[mid*8], and there would be no !ok; the only failure mode would be process death. pread turns "the process died" into "the function returned an error".