Why Lucene Can Use mmap and Most Databases Can't

We know Lucene reads its index through mmap. The file is mapped into memory and reads are served straight from the OS page cache. No system call per read, no copy, and a page is loaded only when someone touches it. It’s a big reason Lucene is fast.

So the natural question is: why don’t databases like Postgres or MongoDB do the same? They also jump to specific pages through indexes instead of reading whole files. The read pattern is basically the same.

But mmap is not a magic solution that fits every storage engine. The difference is not in how they read. It’s in whether their data ever changes after it’s written.

1. Write-once vs in-place updates

• Most databases edit existing pages in place. If a balance changes, the same bytes on the same page are overwritten.
• Lucene never modifies a file after writing it. Even its writes don’t go through mmap. New files are written with ordinary file writes and only then mapped read-only.

Everything else in this post follows from this one difference.

2. Write-once by design: Lucene’s packed format

Lucene’s files are packed so tightly that editing them in place is not practical.

Let’s take the word laptop. Its posting list is the sorted list of documents that contain it: 3, 7, 42, 108. Lucene doesn’t store these numbers as they are. It stores the gaps between them, 3, 4, 35, 66, because small numbers take fewer bytes. The lists for other words sit right behind it in the same file.

Now let’s say doc 42 is edited and no longer mentions “laptop”:

One word, one document. "lemon" and "light" were never touched but both of them moved. (Each box here is one byte. Real Lucene packs gaps in blocks of 128, which makes it worse, not better.)

One gap changed and every byte after it shifted. A real document has hundreds of words, so one edit would mean rewriting hundreds of these lists.

So Lucene doesn’t edit. An update is a delete + add:

Product P42 changes from "gaming laptop" to "gaming phone". Nothing is edited. The old copy is flagged in a small new file, the new copy goes into a new segment, and a merge later writes a clean segment. Even the delete marker is a new file.

In other words, every file Lucene has ever written is only read from then on.

3. In-place updates under mmap

Reading through mmap is not the issue. The issue is writing. With mmap, the OS decides when a changed page is saved to disk and which pages stay in memory. A database that edits pages in place needs to make those decisions itself. If it hands them to the OS, it runs into two problems. Let’s see them one by one.

3.1 Half-saved transactions

A transaction usually changes more than one thing, and the database promises all or nothing. After a crash you either see every change or none of them. To keep that promise, the database has to decide when changed pages reach the disk. With mmap, the kernel decides that. Below are two examples of the same problem, one across pages and one inside a single page.

Example A: across pages

Let’s say we transfer ₹500 from account A (on page 1) to account B (on page 7).

Every database keeps a write-ahead log (WAL) so it can finish or undo a transaction after a crash. It works only because of one rule: the log reaches the disk before the changed pages do. A database with its own buffer pool writes the pages itself, so it simply flushes the log first.

On mmap the database still flushes the log to the disk. But it can’t stop the kernel from writing a changed page earlier. If the kernel does that and the machine crashes, the log on disk has never heard of the change. So recovery has nothing to undo, and the ₹500 is gone.

Both databases have a log. The difference is who writes page 1 and whether the log gets there first.

The database can force a write to the disk any time it wants. What it can’t do is stop the kernel from writing a page early. The only way around it is to edit a private copy and move it into the mapped file once the log is safe. But that is the same as building a buffer pool again.

Example B: inside one page

Now zoom in. Even a single edit takes more than one step. Let’s say B’s row on page 7 holds a balance and a version number. The transaction has updated the balance but not yet the version.

In a database with its own buffer pool, the transaction holds a lock on the page while editing it. The background process that writes pages to disk has to take the same lock first, so it waits until the edit is done. The kernel takes no lock. It can copy the page in the middle of the edit, and if the machine crashes before the next write, that half-edited page is what stays on disk.

Same edit, same moment. The buffer pool gives the database a "not now" button. mmap doesn't.

Lucene has neither of these problems. It never edits a page, so there is never a half-finished change for the OS to save. A new segment that’s only partly on disk is not referenced by any commit yet, so a crash just leaves an unused file behind.

3.2 Cache eviction without query context

This one is not about correctness and it doesn’t come from editing in place. It’s about how well the cache gets used.

At the application level we have control over what is stored in application buffers. We can customize what goes into those buffers based on query patterns. But at the OS cache level we lose that control. Let’s see one example.

The OS evicts pages based only on how recently they were touched. Assume that the cache holds the hot index pages I1–I6 and someone runs a full scan of a 100 GB table. The database knows those scan pages will never be read again, so it pushes them through a small ring of reused slots. The OS has no way of knowing that. To the OS the scan pages are just the most recently used ones, so the index pages go out. Classic cache pollution problem.

The database knows the scan is a one-time read. The OS can only guess from recency.

Lucene has the same limitation. It just accepts it because its workload is mostly index lookups. Databases serve different use-cases with mixed workloads, so they want the control.

4. Write-once implications for Lucene

Write-once is not free either. Until a merge runs, deleted documents still sit inside the segment files. Let’s say a 10 GB segment has 4 GB of deleted docs. Please find the breakdown below.

Cost	What happens
Wasted page cache	The OS caches pages, not documents. 40% of the cached bytes for that segment hold dead docs, so less live data fits in RAM.
Wasted CPU	Lucene still decodes the deleted doc IDs from the posting lists and then skips them.
Skewed scoring	BM25’s IDF counts how many docs contain a word. If 1,000 docs have “laptop” and 300 of them are deleted, IDF still uses 1,000 until the merge. So scores can shift slightly after it.

That’s why merge policies prioritize segments with a lot of deletes, and why an index that stops receiving writes is often force-merged.

5. Summary

mmap is safe only when the data never changes after it’s written. Lucene designed its file format around that: tightly packed, write-once segments where an update is a delete plus an add. Databases that edit pages in place can’t hand that control to the OS without giving up the guarantees their applications depend on. So they keep their own cache.

• Lucene is a library of printed books. The librarian (the OS) can shelve, fetch and remove copies however she likes. The books never change, so nothing can go wrong.
• A database is a whiteboard people are actively writing on. You can’t let the janitor photograph and archive it whenever he walks by. He might capture half a sentence.

So before reaching for mmap in your own storage layer, knowing your application is important.

Ask one question: does any byte change after it’s written? If no, let the OS do the caching. If yes, keep your own buffer pool.

6. References

• MongoDB MMAPv1. MongoDB’s original storage engine was built on mmap. It was later replaced by WiredTiger, which manages its own cache, and removed in version 4.2.
• Crotty, Leis, Pavlo. “Are You Sure You Want to Use MMAP in Your DBMS?” CIDR 2022. Covers these problems in much more detail.
• LMDB. A database that does use mmap successfully. It never edits a page in place. A changed page is written as a new copy and one pointer flips at commit. By default it maps the file read-only and writes with normal file writes.