the page that was not there

mmap returned a pointer. None of the memory behind it was resident.

page-size=16384 pages=256 bytes=4194304

mmap-minor-faults=0
resident-after-mmap=0

first-read-minor-faults=256
first-read-major-faults=0
resident-after-read=256
zero-sum=0

second-read-minor-faults=0
second-read-major-faults=0

The mapping covered 4 MiB. The first loop loaded one byte from every 16 KiB page. It caused exactly 256 minor faults, and all 256 pages became resident. The second loop executed the same source loads without another page fault.

Every byte returned zero. The program never initialized them.

The pointer was valid before the first loop. The pages were not present in the process’s physical map. The first load made each missing page necessary, so the processor stopped the instruction, XNU found the mapping, obtained a zero-filled page, installed a translation, and retried the load.

The word “memory” had been hiding several states behind one pointer.

the pointer named an address in a map

The opening allocation is ordinary C:

volatile unsigned char *mapping = mmap(
    NULL,
    256 * page_size,
    PROT_READ | PROT_WRITE,
    MAP_PRIVATE | MAP_ANON,
    -1,
    0);

MAP_ANON says there is no file descriptor supplying the initial bytes. MAP_PRIVATE says later modifications are private to this mapping. The protection permits loads and stores.

A successful return establishes a range in the process’s virtual address map. The range has a start, an end, current permissions, maximum permissions, inheritance behavior, and backing-object rules.

It does not promise that a separate physical page already exists for every page-sized part of the range.

The distinction begins with the pointer bits. Suppose mapping prints:

0x7000000000

That number is a virtual address in this process. Another process can use the same numerical address for unrelated storage. The kernel can map two virtual pages in one process to one physical page. It can map one file page into several processes. It can remove a mapping while the numerical pointer value remains in a local variable.

The earlier address experiment separated an object from its storage location. Virtual memory inserts another relationship:

C++ or C object
    occupies bytes in a virtual address range

virtual address range
    has an entry in a task VM map

virtual page
    may have a current hardware translation

physical page
    contains resident bytes

backing object
    explains how absent bytes can be produced again

These relationships change at different times.

A pointer does not carry the physical page number, residency bit, file offset, access permission, or object generation. The load instruction supplies the address and access type. Hardware and the operating system resolve the rest.

sixteen low bits selected a byte inside the page

The measured page size was 16,384 bytes:

16,384 bytes = 2^14 bytes

For a page-aligned virtual address, the low 14 bits select a byte within the page. The remaining address bits identify which virtual page contains it.

For this address:

0x7000012345

the split is:

virtual page base: address with low 14 bits cleared
offset in page:    address & (16384 - 1)

The offset is not translated. Once the virtual page is associated with a physical page, the same byte offset selects the final byte there.

The opening loop advances by exactly page_size:

for (size_t i = 0; i < page_count; ++i) {
    sum += mapping[i * page_size];
}

Every iteration has offset zero in a different virtual page. That is why a one-fault-per-page prediction is possible.

Changing the stride changes the prediction. Reading every byte still faults at page boundaries, not once per byte. Reading one byte every 32 KiB skips every other 16 KiB page. Reading addresses 0 and 16,383 stays within one page. Address 16,384 begins the next.

The page is the mapping and protection unit observed here. Cache lines are smaller. The measured machine reports a 128-byte cache-line choice in the earlier false-sharing experiment, while its virtual pages are 16 KiB:

16,384 bytes/page / 128 bytes/cache line = 128 cache lines/page

A page fault and a cache miss therefore cannot be synonyms. One missing page can contain 128 independently cached lines. A resident page can miss every relevant CPU cache.

residency was a snapshot, not a physical address

Before touching the mapping, the probe called mincore with one output byte per virtual page.

Apple’s mincore manual describes it as a query of whether pages are currently core resident. The low bit in each output byte reported:

before first access: 0 resident pages
after first access:  256 resident pages

That supports a narrow statement: at each query, the kernel reported whether each mapped page was resident.

It does not reveal:

the physical page number
which CPU cache contains a line
whether a translation is in a TLB
which DRAM bank holds the bytes
whether the page will still be resident one instruction later
whether another virtual address maps the same page
whether a resident file page has warm data-cache lines

Residency is also not ownership. A clean file page can be resident in the page cache and mapped by many processes. An untouched anonymous range can have a valid VM map entry and no resident page. A copy-on-write page can be physically shared until one process stores to it.

The output array is observational. It does not pin the pages. The kernel remains free to reclaim pageable memory after the call.

This makes mincore useful for falsifying “all pages were already resident” in the opening. It is inadequate for proving a complete memory-performance model.

sixty-four gibibytes cost no resident bytes

I asked for a much larger anonymous range:

reserve-bytes=68,719,476,736
touch-bytes=67,108,864
touched-pages=4,096

The process’s Mach task accounting changed like this:

after mapping 64 GiB:
    virtual delta:   68,719,476,736 bytes
    resident delta:               0 bytes
    footprint delta:              0 bytes

after touching 64 MiB:
    virtual delta:                 0 bytes
    resident delta:      67,125,248 bytes
    footprint delta:     67,158,088 bytes
    minor faults:              4,096

The virtual delta after mmap equals the requested 64 GiB exactly. Resident size and physical footprint did not grow at that measurement point.

The touch loop stored one byte per page across the first 64 MiB:

64 MiB / 16 KiB per page = 4,096 pages

The fault counter increased by exactly 4,096. Resident and footprint accounting grew by roughly 64 MiB. Their small differences from the logical touch size are process-wide accounting effects, not extra payload bytes in the mapping.

This is why an allocator can reserve an address range larger than available RAM. Reservation consumes virtual address space and kernel metadata. Physical backing is acquired as pages become necessary.

It does not follow that every write into an arbitrarily large overcommitted mapping will succeed. Eventually the system must supply physical memory, compressed-memory capacity, swap backing, or a failure policy. The 64 GiB probe touched only 64 MiB. It did not test exhaustion.

Virtual size answers “how much address range exists?” Resident size answers a different current-state question. Physical footprint follows platform accounting rules. Logical objects, allocator requests, resident pages, compressed pages, and disk swap are not interchangeable columns.

one mapping became five hundred and twelve map entries

The 64 GiB request appeared contiguous to the program. A Mach region walk found:

mapping-region-count=512
mapping-region-covered=68,719,476,736
first-region-size=134,217,728

The first region was 128 MiB. The complete virtual span contained 512 such regions:

64 GiB / 128 MiB = 512

This was not a hardware page size. It came from XNU’s anonymous mapping policy.

In the installed SDK, ANON_CHUNK_SIZE is:

#define ANON_CHUNK_SIZE (128ULL * 1024 * 1024)

The tagged XNU vm_map_enter chooses that chunk size for ordinary anonymous mappings. When a null object covers more than one chunk and the mapping has access permissions, the loop from line 3452 creates successive pieces.

The implementation comment in the exported header explains that the split limits the cost of copying when a small part of a large anonymous object is wired and copy-on-write cannot be used for that part.

Three granularities now exist:

one user request:       64 GiB
one XNU anonymous chunk: 128 MiB
one hardware page:        16 KiB

The user pointer traverses all 64 GiB contiguously. The VM map can represent the range with several entries and backing objects. Each entry covers many hardware pages.

An API returning one continuous interval does not require one kernel metadata record or one physical allocation.

mmap installed the rule, not every translation

The current POSIX mmap specification defines the relationship between a process address range and a memory object. The Darwin syscall path adds platform details.

In XNU tag xnu-11417.140.69, mmap validates the address, size, flags, file, and protection.

For an anonymous mapping, the path reaches:

result = mach_vm_map_kernel(
    user_map,
    &user_addr,
    user_size,
    0,
    vmk_flags,
    IPC_PORT_NULL,
    0,
    FALSE,
    prot,
    maxprot,
    inheritance);

The null port means no external memory object supplies file bytes. The VM layer selects an address and creates map entries with the requested protection and inheritance.

For a file mapping, kern_mman.c line 928 instead calls vm_map_enter_mem_object_control with the vnode pager’s memory-object control and the file offset.

Both calls establish a mapping rule. Neither source path contains a loop that must read or zero every page before returning.

The opening measurements agree:

mmap minor-fault delta: 0
resident pages:         0

The absence of faults during this mmap call does not prove that no kernel memory was allocated. Map entries and VM objects are kernel metadata. It proves that the process’s 256 data pages were not made resident through counted soft faults.

the load first asked a translation cache

The compiled loop eventually contains a load using a virtual address. On arm64, the memory-management unit translates that address according to the current translation regime.

Arm’s memory-management documentation separates translation tables from the translation lookaside buffer.

The translation lookaside buffer, or TLB, caches recently used translations and their attributes. A hit can provide the physical address and permission information without reading the page-table hierarchy again.

On a TLB miss, hardware can walk translation tables in memory. Each table descriptor selects the next level. A final page descriptor supplies an output address and attributes such as access permissions and memory type.

Two outcomes are easy to confuse:

TLB miss, valid page-table entry:
    hardware walks the tables and completes the load

invalid translation or disallowed access:
    hardware raises a synchronous exception

A TLB miss is not automatically a page fault delivered to the operating system. It can be resolved entirely by hardware when the page-table entry is valid.

The opening first access had no usable translation because XNU had not populated the physical mapping for that anonymous page. Hardware could not finish the load from existing translation state. It raised a data abort.

the exception recorded the address and access type

An arm64 synchronous exception preserves enough state for the kernel to understand what failed:

exception syndrome:
    class and fault status

fault address:
    the virtual address used by the instruction

saved processor state:
    registers and the instruction return point

access type:
    read, write, or execute as decoded from the syndrome

In the tagged XNU sleh.c, handle_user_abort receives the saved state, exception syndrome, fault address, decoded fault status, and requested protection.

For a serviceable VM fault, it obtains the current thread’s VM map. Some non-translation faults first try arm_fast_fault. If that does not resolve the event, the path calls:

result = vm_fault(
    map,
    vm_fault_addr,
    fault_type,
    false,
    VM_KERN_MEMORY_NONE,
    THREAD_ABORTSAFE,
    NULL,
    0);

If vm_fault succeeds, handle_user_abort returns without sending a user exception. Returning from the kernel retries execution from the saved user state. The original source-level load then completes.

This explains why no C error code appeared in the opening loop. The fault was an internal demand event that the kernel resolved.

A fault the kernel cannot resolve follows another path. The later PROT_NONE experiment makes that visible as a signal.

the map lookup converted an address into an object offset

The fault handler begins with a virtual address and access type. It needs to answer:

Does a map entry contain this address?
Does the entry permit the requested access?
Which VM object backs it?
What offset inside that object corresponds to the address?
Is a resident page already available?
If not, how can the page be produced?

Tagged vm_fault_internal first truncates the fault address to the map’s page boundary. It retains the page offset separately when map page size and hardware page size differ.

It then reaches vm_map_lookup_and_lock_object, which returns the top-level object, object offset, current protection, wiring state, map version, and physical-map target.

The map version matters because other threads can change mappings while fault handling drops locks to obtain or fill a page. Before installing the final translation, the code verifies that the mapping still describes the same object and offset. If it changed, the fault retries its lookup instead of installing a page from a stale map snapshot.

This resembles the queue’s source-validation loop. The data structures differ, but the question is the same:

does the shared mapping still mean what it meant before work happened
without the map lock?

The faulting thread can wait for a busy page, a pager, free memory, or another fault. Correctness requires revalidation when it resumes.

three coordinates identified one byte

The address used by the load was only the first coordinate.

Let a four-page mapping begin at virtual address M, and let the faulting load use:

virtual address = M + 2P + 137

where P is the 16 KiB page size. The map lookup separates that expression into:

map entry:          the entry containing M + 2P
page within entry:  2
byte within page:   137

The map entry supplies a backing-object offset. Conceptually, the VM layer adds the distance from the entry’s virtual start to that backing offset:

object byte offset
    = entry backing offset
    + (fault address - entry virtual start)

It then separates the object byte offset into an object page and the same byte offset 137. The pmap layer eventually associates virtual page M + 2P with a physical page. Hardware adds 137 to the physical page base when the load retries.

One byte therefore has three useful identities:

virtual coordinate:
    process map plus virtual address

object coordinate:
    VM object plus object offset

resident coordinate:
    physical page plus byte offset

They are related, but none can replace the other.

Two mappings of the same file region can use different virtual addresses and the same vnode-object offset. A private child and its parent can begin with the same resident page at one object offset, then use different resident pages after one of them writes. An evicted clean file page can lose its resident coordinate while both the virtual and object coordinates remain meaningful.

Aliasing follows the same separation. If two virtual pages refer to one object page, changing the resident bytes through a shared writable mapping can become visible through both addresses. If one mapping carries a private copy obligation, its first store can create a new resident answer at that object coordinate instead. Comparing pointer bits alone cannot distinguish those contracts. The map entries and their backing relationships decide whether the two loads still converge on one page.

The separation also explains why map operations do not all do the same work:

mprotect:
    changes allowed access at the virtual mapping and pmap layers

munmap:
    removes the virtual rule even if an object and page survive elsewhere

fork:
    preserves virtual contents while adding a copy obligation

page reclaim:
    can remove resident data while leaving a way to reproduce it

remapping the same address:
    gives old virtual bits a new map and object meaning

The opening anonymous mapping had an object coordinate even before it had a private resident page for each offset. The zero-fill path supplied the missing resident coordinate on demand.

This is also why printing a pointer cannot reveal the object offset or physical page. The same numerical pointer can acquire a different answer after unmap and remap. The later controls force exactly that change.

the missing anonymous page was manufactured as zeroes

The anonymous mapping has no file page to read. C still promises that new anonymous bytes begin as zero on this interface.

XNU searches the VM object’s resident pages and any shadow chain. When an absent page reaches the bottom without a supplying object, the tagged source labels the path a zero-fill fault.

At vm_fault.c line 5634, the fast path obtains a page for the original object. Later it calls vm_page_zero_fill and increments vm_statistics_zero_fill_count.

Zeroing is a security boundary. Reassigning a physical page without clearing it could reveal bytes from another process or an earlier object in this process.

For 256 pages of 16 KiB:

256 pages * 16,384 bytes/page
= 4,194,304 bytes zeroed

The source loop asked for 256 bytes, one per page. The operating system initialized 4 MiB because protection and mapping operate at page granularity.

The measured sum was zero:

zero-sum=0

That value alone would not prove the kernel zeroed full pages. The documented interface and source path establish the rule; the measurement confirms the sampled offsets.

the page entered the physical map before the load retried

After vm_fault_page returns a page, XNU must connect it to the faulting virtual address.

The tagged path at vm_fault.c line 6475 calls vm_fault_enter. That function prepares the page and reaches the physical-map layer:

VM map:
    high-level address ranges, objects, offsets, inheritance, protection

pmap:
    architecture-dependent virtual-to-physical mappings

vm_fault_enter calls vm_fault_pmap_enter_with_object_lock, which installs the translation with the allowed protection and fault type.

The public pmap interface takes:

pmap
virtual address
physical page number or address
protection
fault access type
wiring state
mapping options

After successful insertion and cleanup, the arm64 abort handler returns. The processor retries the user instruction. This time a page-table walk can find a valid descriptor, and the translation may then be cached in a TLB.

The complete slow path is not one unconditional straight-line function list. XNU has fast faults, resident-page hits, zero-fill paths, copy-on-write paths, compressed pages, vnode pagers, retry branches, and errors.

For the measured untouched anonymous page, the supported outline is:

load from virtual address
data abort
handle_user_abort
vm_fault
vm_fault_internal
map lookup to anonymous VM object and offset
obtain and zero a page
enter translation through pmap
return from exception
retry load

The running kernel identifies itself as xnu-11417.140.69.710.16. The public source tag is xnu-11417.140.69. The inspected source matches the public base version, not Apple’s additional patch suffix. Claims about exact private patch behavior remain bounded by that mismatch.

the first write found permission already waiting

After the first read populated all 256 pages, the probe stored one byte in each:

first-write-minor-faults=0
second-write-minor-faults=0

One convenient explanation would be “writes to anonymous pages always fault after reads.” This mapping disproved it.

The map was created with PROT_READ | PROT_WRITE. XNU’s zero-fill path knew the mapping permitted writes. It could install a writable translation on the first read after satisfying any copy obligations. The later store needed no protection upgrade or new page.

This is not a universal rule for every mapping.

A read-only mapping cannot acquire write permission from a store. A copy-on-write mapping can deliberately install read access to a shared page and fault on the first write. A dirty-tracking system can revoke write permission so the next store becomes observable. Executable pages can have separate code-signing and write policies.

The access requested by the current fault and the maximum rights of the map entry both matter.

eleven rounds made the average less convincing

The opening total hides the distribution across individual batches. I mapped 4,096 new pages in each of eleven rounds and timed groups of sixteen first loads. Every round recorded:

first pass:
    4,096 minor faults
    0 major faults

second pass:
    0 minor faults
    0 major faults

Across 2,816 first-touch batches, one run reported:

minimum:  492 ns/page
median:   802 ns/page
p90:    1,581 ns/page
p99:    2,398 ns/page
maximum: 2,883 ns/page

The second-touch batches had a 2.60 ns/page median, but a 622 ns/page p90 and a maximum above 3 microseconds per page.

That wide second-pass tail cannot be page faults because the counters stayed at zero. The batch timer includes scheduler interruption, timer overhead, changing core placement, frequency, cache state, translation misses, and loop execution. Sixteen-page batches amortize timer calls but do not isolate those mechanisms.

The first-touch distribution likewise measures the complete demand-zero path as experienced by this process. It is not the cost of one XNU function or one hardware exception instruction.

The median gap is still useful:

first touch:  about 802 ns/page
second touch: about 2.6 ns/page at the median

The exact values move between processes. The one-fault-per-page counts and zero values are the stable causal evidence.

no page fault did not mean no translation work

After first touch, a load can still be delayed by:

a data-cache miss
a TLB miss and hardware page-table walk
a branch or dependency stall
memory-controller queueing
scheduler preemption
another core changing translation state

The process fault counter observes kernel-handled faults. A valid page-table walk need not increment it.

To separate demand faults from later translation and cache costs, I constructed dependent pointer chains. Every node contained the index of the next node, so the processor could not issue the next load before the current one returned.

One chain placed each node at the start of a different 16 KiB page. A control placed the same number of nodes one 64-byte cache line apart. Both used the same randomized cycle and number of dependent loads.

Every page and line was written before timing. Every timed run reported:

timed-minor-faults=0

The memory was resident before measurement. Demand paging was removed.

sixteen pages were already different

One complete process reported median nanoseconds per dependent access:

nodes   page-spaced   line-spaced   ratio
    1          0.48          0.48    1.00
    4          1.96          1.95    1.00
   16          6.64          1.82    3.64
   64          6.61          1.82    3.63
  256          8.91          1.82    4.89
1,024         20.00          1.82   10.99
2,048         29.08          1.82   15.98
4,096         35.27          6.71    5.26
8,192         35.40          6.81    5.20

A second process preserved the shape while moving several absolute values:

16 pages:    5.07 versus 1.82 ns
1,024 pages: 18.83 versus 1.82 ns
8,192 pages: 32.75 versus 5.73 ns

The page-spaced chain slows at a much smaller node count. Translation capacity is a plausible contributor because each node occupies another virtual page while the control uses another cache line within a compact span.

It would be wrong to read exact M4 TLB sizes from the ledges.

The page-spaced chain also has:

more physical distance between useful cache lines
different cache-set mapping
different page-table working set
different DRAM locality
different prefetch opportunities

At 4,096 nodes, the line control itself grows to 256 KiB and slows sharply. Cache capacity is visibly part of the result.

Without architecture counters for TLB refills, page walks, cache misses, and cycles, the experiment shows a page-spacing penalty after prefaulting. It does not assign every nanosecond to a TLB.

a page table had its own memory working set

Translation tables are ordinary memory structures maintained by the operating system and consumed by hardware.

With a 16 KiB translation granule and eight-byte descriptors, one table page can contain:

16,384 bytes/table / 8 bytes/descriptor
= 2,048 descriptors/table
= 2^11 descriptors/table

Each fully used level can therefore select 11 address bits. A final page offset selects 14 bits.

The number of active levels depends on the configured virtual-address range and translation regime. Arm permits several starting levels and block descriptors. I cannot read the process’s translation-control registers from user mode, so I do not claim the exact level count used for every address on this M4.

The arithmetic still exposes why sparse address patterns have a cost. Mapping one byte in many distant regions can require page-table pages at several levels even when little payload is resident.

The 64 GiB reservation did not need one final descriptor per page immediately. Final translations appeared as pages were faulted in. Kernel VM map metadata existed first; page-table population followed demand.

Page tables consume physical memory too. They can occupy caches, miss in memory, and require synchronization when changed.

a translation cache needed process identity

Two processes can both use virtual address 0x7000000000 for different pages. A cached translation therefore needs an address-space identity in addition to a virtual page number.

Arm uses address space identifiers, or ASIDs, to tag relevant cached translations. Entries from different application address spaces can coexist without one process using another process’s result. Arm’s virtualization documentation distinguishes these ASIDs from VMIDs used for stage-two virtual-machine translation.

This is one reason a context switch does not necessarily require discarding every cached translation.

ASIDs are finite and operating systems manage their reuse. Global mappings and architecture-specific invalidation rules add more detail. The article does not infer XNU’s exact ASID allocation policy from timing.

The important prediction is narrower:

same virtual bits in two processes
do not imply same translation

The current thread’s address-space context decides which translation tables and tags apply.

permission failure reached the process instead of returning to the load

I mapped one page with no permitted access:

void *page = mmap(
    NULL,
    page_size,
    PROT_NONE,
    MAP_PRIVATE | MAP_ANON,
    -1,
    0);

The mapping call succeeded. Reading the first byte produced:

prot-none-read=refused
signal=10
code=1
fault-address-matches=1

On this macOS process, signal 10 is SIGBUS. The siginfo_t address equaled the attempted pointer.

I changed the same map entry to PROT_READ:

mprotect(page, page_size, PROT_READ);

The next read succeeded, returned zero, and produced one minor fault:

prot-read=succeeded value=0 minor-faults=1

Then I removed access again:

protection-removed-read=refused
signal=10
code=1
fault-address-matches=1

The first anonymous demand fault was resolvable. The map entry permitted the requested load, so XNU could create and map a zero-filled page.

The PROT_NONE access was not resolvable under the mapping contract. Installing a readable translation would violate the current permission. XNU converted the failed Mach exception into a process signal.

Both events are hardware faults. Only one is visible to normal application control flow as an error.

changing permission changed more than metadata

mprotect must update the process’s high-level map protection. It must also ensure that hardware cannot keep using a cached writable or readable translation that contradicts the new rights.

Suppose a page has a writable TLB entry and the process changes it to read-only:

old cached permission: read and write
new map permission:    read only

Changing only the VM map entry would be unsafe. A processor could continue storing through its old cached result.

The pmap layer changes page-table permissions and performs architecture-required translation invalidation and synchronization. Other cores that might cache the affected translation have to observe the new rule before the protection change can be treated as complete.

This family of cross-core invalidation is called a TLB shootdown. The name describes the coordination goal, not one universal implementation. Arm supplies local and shareable invalidation operations plus ordering instructions. XNU’s pmap and platform configuration choose the concrete sequence.

Permission relaxation has different constraints. Adding write permission may still require replacing a cached read-only translation. Copy-on-write may prevent granting it until a private page exists.

The syscall return means the protection operation completed according to its interface. It does not make mprotect a cheap field assignment.

unmap removed the rule while the pointer bits survived

I mapped one writable page, stored 41, saved the pointer value, and called munmap.

The local pointer variable still contained the same bits. Its next load produced:

after-unmap-read=refused
signal=11
address-matches=1

Here signal 11 was SIGSEGV. XNU found no usable map entry for the address.

In this single-threaded probe, I then installed a new anonymous mapping at that exact now-empty range using MAP_FIXED:

same-address=1
old-value=41
new-value=0
new-read-minor-faults=1

The virtual address returned. The old mapping did not.

The first load from the replacement mapping caused a new zero-fill fault and returned zero. Nothing in the pointer bits distinguished:

old map entry, old VM object, value 41

from

new map entry, new VM object, untouched zero page

MAP_FIXED can destroy an existing mapping at its target, so using it with an arbitrary address is dangerous. The probe used only the exact page it had just unmapped, in one thread, and installed the replacement immediately.

The experiment extends the address-reuse result from C++ object storage to a process VM map. A stale pointer can outlive both an object and the virtual mapping that once made its address usable.

fork made every writable page conditional

Before fork, I filled 64 anonymous pages with the byte 7.

The child changed every even page to 9:

for (size_t page = 0; page < 64; page += 2) {
    mapping[page * page_size] = 9;
}

The output:

pages=64
child-changed=32
child-unchanged=32
child-first=9
child-second=7
child-minor-faults=32

parent-unchanged=64
parent-first=7
parent-second=7

The parent and child began with the same contents. Copying one MiB eagerly during fork would preserve isolation, but it would spend time and memory even if the child immediately called exec.

Copy-on-write delays the copy. Both processes initially refer to shared physical pages with write access withheld. A load can use the shared page. The first store by one process faults.

XNU sees that the mapping permits a logical write but carries a copy obligation. It allocates a private page, copies the old 16 KiB contents, maps the new page writable in the faulting process, then retries the store.

The other process keeps the old page.

The child’s 32 stores touched 32 distinct pages and produced exactly 32 minor faults. All 32 odd pages remained 7 in the child. All 64 pages remained 7 in the parent.

Copy-on-write has page granularity:

useful bytes modified:       32 bytes
private page bytes created:  32 * 16,384
                           = 524,288 bytes

Changing one byte can require copying a full page.

the region stayed copy-on-write after half the pages copied

The Mach region query added another view:

child-share-mode-before=1
child-share-mode-after=1
child-private-pages-after=32
child-resident-pages-after=64

The SDK names share mode 1 SM_COW. The region remained classified as copy-on-write because half its pages were still shared under that contract.

The pages_shared_now_private count became 32, matching the stores and soft faults.

Region metadata describes a range while individual pages inside it can have different physical states:

32 pages:
    private child copies containing 9

32 pages:
    still shared with the parent containing 7

One region label does not imply one physical-sharing state for every page.

The tagged fault source makes the first-write trap deliberate. When a read reaches a page with a pending copy obligation, XNU can withhold write access. The comment around vm_fault.c line 5715 says the first write will fault to push the page into a copy object or create a shadow.

The later copy-on-write branch obtains a new page in the top object and copies data up the shadow chain. The pmap mapping for the faulting process then changes, while the source page remains available to the other process.

The map-level needs_copy rule and the hardware read-only translation work together. One records semantic obligation. The other traps the store that makes the obligation immediate.

a shadow was a sparse record of disagreement

The phrase “copy-on-write object” can sound as though fork secretly copied a complete second object. The measurements rule that out as a useful model.

Before the child wrote, all 64 pages already contained 7. After the child wrote the even pages, its logical contents were:

object offset 0 pages:  9, 7, 9, 7, 9, 7, ...
parent contents:        7, 7, 7, 7, 7, 7, ...

Only 32 child pages became private. A VM object layer can represent that state sparsely. The child’s top object needs resident pages for the offsets where its contents disagree with the older shared state. For an unchanged offset, lookup can continue through a backing or shadow relationship and find the original page.

For four pages, the logical lookup resembles:

child top object:
    offset 0 -> private page containing 9
    offset 1 -> absent
    offset 2 -> private page containing 9
    offset 3 -> absent

older shared object:
    offset 0 -> page containing 7
    offset 1 -> page containing 7
    offset 2 -> page containing 7
    offset 3 -> page containing 7

A child read at offset 0 stops at its top page. A child read at offset 1 descends and finds the shared page. The parent continues to see pages containing 7. The logical object is complete even though no single layer contains all of its current pages.

The first write to offset 0 cannot alter the lower page in place because the parent still observes it. The fault path obtains a page in the child’s top object, copies the 16 KiB source contents, and then permits the store of 9. After that, the top page is the child’s answer for the whole offset.

The byte store changed this:

one byte of logical value
one page of physical ownership
one object layer's resident-page set
one process's pmap translation

It did not need to change the child’s virtual address. It did not need to change the parent’s mapping. It did not copy the other 32 pages merely to keep the object description rectangular.

This representation also makes the first-write fault necessary. Before the child disagrees with the shared state, there is no private top page to map writable. A read-only hardware translation catches the exact store that creates disagreement. After copying and installing the top page, later stores to that page can proceed without repeating the copy.

Shadow depth is another possible cost. Repeated snapshots can create several layers to search, while page faults, object collapse, and other VM policies can change that structure. The current Mach query exposed SM_COW and the count of pages that became private. It did not expose a complete object graph, and the probe did not measure lookup time as a function of shadow depth.

The narrow conclusion is enough to predict the 32-fault result. Copy-on-write records difference at page granularity. The object relationship supplies old contents, the fault creates a private disagreement, and the pmap makes the new answer local to the child.

copy-on-write moved work out of fork

The optimization changes where costs appear.

Eager copying would make fork proportional to the parent’s copied memory. Copy-on-write makes initial fork work closer to map and object metadata, then charges pages to later writes.

For a parent with 8 GiB of writable resident pages:

eager copy:
    potentially copy 8 GiB before child proceeds

copy-on-write:
    share initially
    copy only pages written before exec or exit

If the child writes 1 percent of those pages, page data copied is roughly:

8 GiB * 0.01 = 81.92 MiB

That is a mechanism estimate. It omits page-table work, object metadata, huge-page splitting, kernel bookkeeping, and parent writes that also trigger copies.

Copy-on-write can also move a latency surprise into the first mutation. Code that benchmarks only fork and ignores the child’s writes measures the optimization’s setup, not its total cost.

Memory accounting becomes conditional. Two processes can each report large virtual ranges while sharing physical pages. Summing their resident sizes can double-count shared memory. Physical footprint metrics use platform-specific accounting rather than simply adding mappings.

private and shared file mappings diverged on the first store

I created a one-page file filled with A, then mapped it twice:

private_mapping = mmap(
    NULL, page_size, PROT_READ | PROT_WRITE,
    MAP_PRIVATE, descriptor, 0);

shared_mapping = mmap(
    NULL, page_size, PROT_READ | PROT_WRITE,
    MAP_SHARED, descriptor, 0);

Both first bytes began as A.

The private mapping changed byte zero to P. The shared mapping changed byte one to S and called msync(MS_SYNC).

The resulting two-byte prefixes were:

private mapping: PA
shared mapping:  AS
file via pread:  AS

The private store did not change the file. It created or used a private page for that mapping.

The shared store changed the page associated with the file mapping. After synchronous msync, pread observed AS.

MAP_PRIVATE does not mean the original file bytes were eagerly copied during mmap. The mapping can initially use the file-backed page and copy when written.

MAP_SHARED does not mean every store is instantly durable on storage media. The dirty page can live in memory before writeback. msync requests synchronization according to its interface; storage devices and filesystems still have ordering and durability semantics beyond a CPU store.

The two flags answer what later stores mean, not whether the first read needs a fault.

Anonymous zero fill has no external source. A file mapping has a vnode pager and object offset.

The first load from a nonresident file-backed page follows the same opening stages:

virtual-address load
translation fault
VM map lookup
backing object and file offset

The page-production branch differs. If the page is already in the unified page cache, XNU can map that resident page. If it is absent, the pager and filesystem may initiate storage I/O, possibly reading a cluster rather than one page.

The earlier file experiment showed that an on-disk format has pages, encodings, checksums, and read ranges unrelated to the hardware VM page size. Mapping a file adds another page system:

hardware virtual-memory page
filesystem cache page
storage block
application file-format page

They can have different sizes and boundaries.

One hardware fault can prompt readahead of neighboring file data. Later pages may already be resident when their addresses are touched. A page-fault count therefore need not equal the number of file pages read from storage.

The current file-sharing probe wrote the file immediately before mapping it. Those bytes were already in the system’s cache. It does not measure a cold disk fault.

minor and major named the source of delay too coarsely

The opening used getrusage:

ru_minflt: page reclaims that did not require I/O
ru_majflt: page faults that required I/O

The untouched anonymous pages produced 256 minor faults and zero major faults. XNU could allocate and zero memory without reading a file from storage.

The names do not define exact latency:

minor fault:
    may allocate and zero a page
    may map a resident file-cache page
    may perform copy-on-write
    may update page tables and translation state

major fault:
    waits for backing data through an I/O path

A minor fault can cost much more than an ordinary load. A major fault can be influenced by storage cache, readahead, queueing, compression, and other processes.

The current machine did not permit a controlled global page-cache purge without privileged action. Creating a new file is not identical to forcing durable media service, and the filesystem can satisfy data from memory after the write that created it.

I therefore did not publish a fresh “SSD page-fault latency” from this article. The major-fault column stayed zero in the controlled anonymous experiment.

the second pass could lose every TLB entry and still show zero faults

The second opening loop reported:

minor faults: 0
major faults: 0

That proves no counted process page fault occurred in its measured interval.

It does not prove:

all 256 translations remained in one TLB
all data remained in L1 cache
no page-table walk occurred
the loop used the same CPU core

If a TLB entry was evicted but the page-table descriptor remained valid, hardware could walk the tables and refill the TLB without XNU. The process fault count would stay zero.

If the thread migrated to another core, the destination core could lack the translation and refill it. Again, no process page fault is required.

This corrects a common benchmark shortcut:

zero page faults
does not imply
zero virtual-memory translation cost

Fault counters and TLB counters answer different questions.

locking forced pages to exist before the loop

The POSIX mlock interface requests that a mapped range remain resident.

I mapped 4 MiB without touching it, then called mlock:

bytes=4,194,304
pages=256
resident-before=0
resident-after-lock=256
lock-minor-faults=512
read-minor-faults=0
sum=0

After mlock returned, all 256 pages were resident. The later read loop faulted zero times and found zero bytes.

The tagged XNU mlock calls vm_map_wire_kernel. The wire path marks the physical mapping nonpageable and simulates faults across the range to obtain pages and enter translations.

The current run counted 512 minor faults for 256 pages, exactly two per page. This repeated, but I did not establish which two accounting sites produce the count. The source contains fast wire faults, fallback vm_fault_internal, wiring transitions, and trace/accounting paths. “mlock faults once per page” would contradict the measurement.

The justified result is:

mlock populated and wired the complete range before returning
the subsequent loop incurred no page fault
the process counter increased twice per page during this path
the exact double count remains unresolved

Keeping the unexplained factor of two is better than assigning it to a convenient internal branch.

wiring changed a resource guarantee

An ordinary resident page can later become pageable. Wiring asks the kernel not to reclaim it through the normal pageable path while the lock remains.

That matters for operations whose external device or timing contract cannot tolerate disappearance:

DMA from a device
real-time or low-latency buffers
cryptographic secrets that should avoid swap
kernel interfaces requiring stable resident pages

The guarantee consumes a scarce system resource. A process that wires all memory can prevent the kernel from reclaiming pages for other work.

Operating systems therefore apply permissions and resource limits. The probe inspected RLIMIT_MEMLOCK and selected only 4 MiB. Its printed limit value of zero represented RLIM_INFINITY in the probe’s output convention, not a literal zero-byte limit.

Wiring also moves cost to setup. The later loop looks fast because allocation, zeroing, translation insertion, and wiring happened inside mlock.

A benchmark that excludes setup can make pinned memory look free. A service that allocates and pins per request pays the setup on every request.

pinned did not mean visible to every device

Keeping CPU pages resident is necessary for some DMA paths but may not be sufficient.

A device can have its own input-output memory-management unit and translation tables. Registering a buffer for RDMA can require:

pin or otherwise stabilize CPU pages
validate access rights
create device-visible translation metadata
obtain local and remote keys
synchronize with the device and driver

A GPU with a unified virtual-address model can still fault, migrate, or require explicit registration depending on platform and allocation type.

“Pinned” is also used differently across APIs. CUDA pinned host allocation, operating-system wiring, RDMA memory registration, and language-level immovability do not promise identical properties.

The current M4 probe exercised mlock and CPU residency. It did not register the range with a GPU or NIC. Device visibility remains unmeasured.

This boundary matters because a pointer valid to the CPU is not automatically a complete device address contract.

large pages traded translation reach for allocation constraints

If one TLB entry describes a larger block, the same number of entries can cover more bytes.

For illustration:

512 entries * 16 KiB = 8 MiB reach

512 entries * 2 MiB = 1 GiB reach

The entry count is illustrative, not an M4 measurement. The ratio follows from page size.

Larger mappings can reduce:

TLB pressure
number of final page descriptors
some page-walk traffic

They can increase:

internal fragmentation
zeroing and copy-on-write granularity
contiguous allocation difficulty
cost of splitting or changing protection
memory retained for sparse use

XNU has superpage flags and pmap support, but the public interfaces and actual allocation policy vary by platform. The current probe did not obtain and verify a superpage mapping. It cannot claim a huge-page performance result.

On Linux, explicit huge pages and transparent huge pages have distinct allocation, collapse, split, and fallback behavior. Those mechanisms were not run on this macOS machine.

The translation argument predicts why large pages can help. Only a verified mapping size and workload measurement can say whether they did.

eight threads faulted one page more than once

The opening equality tempted another shortcut:

minor-fault count == newly resident page count

That was true for one thread walking distinct pages. It was not a general identity.

I started eight threads, let all of them reach a barrier, then released them to read the same byte in one untouched anonymous page.

Four process runs reported:

run 1: same page, 6 minor faults
run 2: same page, 8 minor faults
run 3: same page, 8 minor faults
run 4: same page, 8 minor faults

Every thread returned zero. Only one virtual page became resident. Several threads entered fault handling before the first fault completed the mapping.

XNU coordinates faults through busy pages, object locks, map validation, and retry paths. Multiple fault events can converge on one final resident page. The process counter counts fault activity, not unique physical allocations.

A control assigned the same eight threads disjoint parts of 256 pages:

disjoint pages:       256
minor-fault count:    256
sum:                    0

The control setup created the threads before taking the baseline so thread-stack creation faults did not inflate the measured interval.

This matters when a service starts many workers on the same cold model or mapping. A spike of fault events can exceed the number of newly resident pages because callers collide on the same missing state.

Fault coalescing prevents eight independent final copies of the anonymous page. It does not make seven arriving threads free.

one busy page made other faults wait

Inside vm_fault_page, a page under construction can be marked busy. A second fault for the same object and offset must not independently publish conflicting contents.

The waiting path protects several invariants:

one object-offset pair resolves to one current page
zero fill completes before readers consume bytes
pager I/O does not install two competing results
copy-on-write publishes the correct top-object page
errors wake every waiter with a consistent outcome

The exact synchronization changes by fast path, page state, and pager. The public result is visible: eight source loads all returned the same zero byte even though several fault events were counted.

This waiting is not a user mutex, but it is still queueing. A page-fault storm can serialize on VM objects, free-page acquisition, zeroing, pager work, or pmap updates.

The 256 disjoint-page control permits more independent work, but it can contend on allocator and VM-global structures.

That distinction explains a production trace shape:

many threads blocked on one page:
    duplicate demand or synchronized cold start

many threads faulting distinct pages:
    broad working-set activation

The same total fault count can produce different completion time because arrival topology differs.

truncating the file invalidated an apparently mapped page

A file mapping can remain in the VM map while its backing object changes.

I created a two-page file:

first page begins with  A
second page begins with B

I mapped both pages and read AB. Then I truncated the file to one page without unmapping the two-page range.

The result:

before=AB
after-first=succeeded:A
after-second=refused
signal=10
address-matches=1

The first page remained backed by valid file contents. Access to the second mapped address produced SIGBUS at that address because the file no longer had data for the offset.

The pointer was:

inside the original mmap length
inside a surviving VM map range
page aligned and readable by its original protection

Those facts were insufficient. The backing object’s valid extent had changed.

This is a different failure from PROT_NONE and munmap:

PROT_NONE:
    entry exists, requested access is forbidden

munmap:
    entry no longer exists

file truncation:
    mapping exists, pager cannot supply the requested file offset

All three can stop an ordinary load before it returns a byte.

a mapped file needed a lifetime protocol

Closing the file descriptor does not by itself destroy a successful mapping. The mapping holds its relationship to the memory object until munmap or another mapping operation changes it.

That does not make file shape immutable.

Another thread or process can:

truncate the file
extend it
modify shared bytes
replace a pathname with another file
unlink the directory entry
change storage availability

Pathname replacement does not retarget an existing mapping. The mapping refers to the object opened at mapping time, not whatever a later pathname lookup finds.

Truncation can make a previously valid offset unsatisfiable, as the probe showed.

An application that memory-maps a growing data file needs an agreement about:

who owns file length
when readers learn a new committed length
whether writers can truncate in place
how mappings are replaced
what happens to readers during rotation

A manifest pointing to immutable files avoids several races. Publishing a new name or object identity after the bytes are complete lets old readers finish against old mappings.

The mapping API supplies address access. It does not supply a multi-process version protocol.

shared bytes did not create C++ synchronization

MAP_SHARED lets processes observe modifications to common mapped storage. It does not make arbitrary C++ objects in that storage safe for concurrent access.

If two threads or processes concurrently write one ordinary integer without a synchronization protocol, language and inter-process rules still matter.

For C++ threads in one process, conflicting non-atomic accesses can form a data race and undefined behavior.

For separate processes, std::atomic support in shared storage requires properties of the implementation, object construction, alignment, and process-shared use. A mutex stored in shared memory needs process-shared attributes and recovery policy if an owner dies.

Page coherence is too coarse to implement a queue:

the processes can map the same page
does not imply
they agree on operation ordering or object lifetime

msync concerns mapping and file synchronization. It is not a substitute for an atomic happens-before relation between threads.

Virtual memory provides shared bytes. A concurrent data structure provides rules for changing them.

page protection could implement observation

The failed PROT_NONE load shows that permission can turn a memory access into a controlled event.

Systems use this mechanism for:

guard pages around stacks
copy-on-write
dirty tracking
garbage-collector barriers
snapshotting
demand loading
watchpoints and debuggers
lazy restoration

A dirty tracker can mark pages read-only. The first store faults, the handler records that page as dirty, then restores write access. Later stores proceed without faults until tracking begins again.

The granularity is a page. One store to one object can mark unrelated objects on the same page dirty.

A user-level signal handler has strict safety limits. It cannot call arbitrary allocation, locking, or I/O functions safely. Production userfault mechanisms differ by operating system, and macOS does not expose Linux userfaultfd.

The protection probe used sigsetjmp only to make a deterministic demonstration. It did not build a safe general fault-driven runtime.

The mechanism is valuable because it changes a hidden first access into an observable control transfer. It is dangerous because every missed edge becomes a process crash or recursive fault.

stack growth used the same illusion with a boundary

A thread stack is usually represented by a virtual range larger than the pages currently active.

As function calls and local objects move the stack pointer into fresh mapped pages, the operating system can supply backing on demand. That avoids populating every thread’s maximum stack range at creation.

A guard region near the stack boundary has no permitted access. Crossing it converts unbounded growth into a fault rather than silently writing an adjacent mapping.

This separates:

reserved stack address range
currently committed stack pages
guard page or guard region
configured maximum

Creating thousands of threads can therefore reserve enormous virtual space without making all of it resident. Each thread still consumes kernel scheduling state and at least some live stack pages.

Large recursive frames or variable-length stack allocations can jump across a guard if the platform and generated code do not probe intermediate pages. Compilers and operating systems use stack-clash mitigations on relevant targets.

The current article did not create a crash-by-recursion probe because the process-limit and compiler-mitigation result would need its own controlled signal stack. The mapping model predicts why a stack has both a virtual maximum and a smaller current footprint.

heap allocation stopped at the same page boundary

malloc can satisfy small requests from allocator-managed regions already mapped into the process.

For larger regions, an allocator can obtain virtual memory from the operating system, divide it into size classes or extents, and populate pages as callers touch them.

This makes several measurements differ:

bytes requested by application
bytes reserved from virtual address space
bytes committed by first touch
bytes resident now
bytes retained by allocator after free
bytes returned to the operating system

Calling free ends the application’s ownership of an allocation. It does not require the allocator to call munmap immediately. Keeping pages in an arena can make a later allocation fast while resident size remains high.

Conversely, reserving a large arena need not make it resident. The 64 GiB mapping established that.

An allocator benchmark that touches every byte measures page population as well as allocator metadata. A benchmark that allocates and frees without touching can measure mostly reservation and bookkeeping.

The earlier vector allocation counter recorded requested bytes. It explicitly did not call that value physical memory. The VM experiment supplies the missing reason.

prefaulting moved latency into admission

A service can touch pages before admitting traffic:

for (size_t offset = 0; offset < bytes; offset += page_size) {
    checksum += mapping[offset];
}

This moves demand faults from request execution into initialization.

It does not make the bytes permanently resident unless another mechanism wires them or memory pressure never reclaims them.

The trade:

without prefault:
    lower startup work
    first requests can fault unpredictably

with prefault:
    longer startup
    known pages and translations are established once
    later reclaim can still reintroduce faults

with mlock:
    startup population plus a residency resource commitment

For a latency service, the correct phase depends on the SLO and rollout protocol. A new worker can load, fault, validate, and warm its model before receiving requests. Readiness should mean more than “the process accepted a socket.”

Prefaulting an entire maximum context cache can waste memory if most capacity remains unused. Touching only expected working regions reduces setup but leaves a tail when rare paths expand.

The operating decision requires a working-set distribution, not only a mapping size.

memory pressure could reverse the residency result

After first touch, mincore reported all pages resident. That state is not permanent.

Under memory pressure, XNU can choose among pages based on state and backing:

clean file-backed page:
    discard mapping residency
    reproduce bytes from file later

dirty shared file page:
    write back according to filesystem policy

anonymous or private dirty page:
    keep resident, compress, or move to swap backing

wired page:
    not eligible for ordinary pageable reclamation

The actual policy includes queues, reference information, compressor state, task policy, and system pressure.

If a clean file page is discarded, a later load faults and finds or reloads file data. If an anonymous page is compressed, a later fault can decompress it. If it reached swap, service can require storage I/O.

The same source load can therefore have different paths at different times:

TLB hit
hardware table walk
resident-page soft fault
compressor fault
swap-in or file page-in
unresolvable error

The opening establishes only the untouched anonymous case on an otherwise functioning local system.

compression preserved contents without preserving latency

Compressing an inactive anonymous page trades CPU work and compressed-memory space for avoiding or delaying storage I/O.

The virtual address and logical bytes remain part of the process. The original uncompressed physical page can be reclaimed.

On a later access, the VM fault path can obtain the compressed representation, allocate a page, decompress bytes, install a translation, and retry.

From the application:

pointer value: unchanged
logical value: unchanged
load latency: changed
resident accounting: changed
compressed accounting: changed

The tagged vm_fault.c contains separate compressor and compressor-swapin fault classifications. Their existence in source does not show that the opening pages entered either path.

The probe did not induce controlled compressor pressure. Creating system-wide pressure to force compression would disturb unrelated applications and would not guarantee which pages XNU selected.

The article therefore describes the source-supported mechanism without presenting a compressor timing.

This is another reason “resident after warmup” must be measured near the workload rather than assumed from an earlier phase.

advice was not a command to the future

Interfaces such as madvise let a process describe expected access patterns or discardability.

Examples include sequential access, random access, near-term need, and pages that can be discarded. Exact flags and effects vary by operating system.

Advice can influence:

readahead
page clustering
retention
prefaulting
discard policy

It does not generally promise that a page will be resident at a future instruction. “Will need” is not the same contract as wiring.

Likewise, “do not need” can have different observable consequences for anonymous and file-backed pages. Some discard interfaces allow the old bytes to survive until reuse, while others define zero or file contents on the next access.

I did not add an advice timing because a one-process result without memory pressure would mainly show the current heuristic response. A useful experiment would vary access order, pressure, file cache state, and advice while recording fault type and I/O.

Hints are part of a policy conversation with the kernel. Protection and mapping validity are correctness contracts.

first touch did not choose a NUMA node here

On a multi-socket NUMA machine, physical memory is closer to some CPUs than others. An operating system can place a newly faulted anonymous page near the CPU that first writes it.

That creates a consequence:

initialization thread on socket 0
worker threads on socket 1

can produce

resident pages physically attached to socket 0
remote accesses from socket 1

Parallel first touch can distribute pages according to their eventual owners. numactl, CPU affinity, and page-placement queries can test that on Linux NUMA hardware.

The measured Apple M4 system is not a multi-socket NUMA test bed, and macOS does not expose the same Linux placement controls. I cannot measure a remote-socket penalty here.

Apple silicon still has a physical memory hierarchy, CPU clusters, GPU access, caches, and controllers. Calling it “unified memory” does not mean every requester sees identical latency or bandwidth.

The portable idea is that virtual contiguity does not promise one physical locality. The NUMA-specific placement result remains a prediction for appropriate hardware.

cpu affinity was the missing variable twice

The page-stride benchmark and protection benchmark both exposed scheduling sensitivity.

Without verified affinity:

a round can begin on one core and finish on another
P and E cores can have different latency and frequency
translation caches differ by core and level
reader threads can share cores
load can change scheduler decisions

The protection benchmark became faster as more readers were created. That is consistent with a scheduling or frequency regime change, but the current evidence cannot choose one.

macOS offers thread-affinity hints rather than the Linux taskset style hard CPU ID binding used by many microbenchmarks. A hint does not establish exact physical placement.

The article keeps process replication and medians, but it does not hide this missing control.

On Linux, a stronger campaign would record:

taskset placement
numactl memory policy
page migration state
CPU model and frequency
perf TLB refill and page-walk events
minor and major faults
context switches and migrations

Cross-hardware validation is necessary before turning one ledge into a cache-capacity constant.

page-table invalidation was a correctness protocol

Changing or removing a translation races with cores that might already have cached it.

The required high-level sequence is:

change the page-table entry or make it invalid
issue architecture-required ordering
invalidate matching cached translations in the required scope
wait for completion as required
only then reuse a page or rely on the new permission

If physical page P was removed from virtual address V and immediately reused for another process, a stale TLB entry for V could expose the new owner’s bytes. Translation invalidation is therefore part of memory isolation and lifetime, not only performance.

Unmapping can also interact with in-flight memory operations and page-table walks. Architecture rules define break-before-make requirements for some descriptor changes.

The operating system owns this protocol because application threads cannot know which cores and translation caches contain the mapping.

This echoes safe memory reclamation:

queue node:
    do not reuse storage while a thread can dereference an old pointer

physical page:
    do not repurpose it while a CPU can use an old translation

The mechanisms differ, but both require evidence that stale reachability has ended.

four million final descriptors were only the upper layer

A fully populated 64 GiB range with 16 KiB pages contains:

64 GiB / 16 KiB
= 4,194,304 pages

At eight bytes per final descriptor:

4,194,304 * 8 bytes
= 33,554,432 bytes
= 32 MiB

That is the space for final descriptors if represented without larger block mappings and packed into full table pages. Higher-level tables add more.

The probe touched only 4,096 pages:

4,096 * 8 bytes = 32 KiB of final descriptor values

Actual table allocation occurs by page, not by individual eight-byte values. Sparse distribution can require many partially filled table pages. Contiguous pages share upper paths and pack final tables efficiently.

The calculation does not measure XNU’s page-table footprint. It provides an upper-layer scale and assumptions.

Virtual reservation can be cheap relative to full page-table population. Touch shape decides how much of the hierarchy becomes necessary.

This is why two 64 MiB resident working sets can have different translation metadata:

one contiguous 64 MiB range
one page in each of thousands of distant regions

Logical resident bytes alone miss sparsity.

an inference cache could be virtually large and physically conditional

A serving process can reserve address space for a maximum KV cache while committing pages as requests arrive.

That can avoid relocating existing cache entries when the logical capacity grows. It also creates operational questions:

Which request first touches each page?
Does first-token latency include page population?
Can idle cache pages be reclaimed or compressed?
Are pages pinned for GPU transfer?
What happens when virtual capacity exceeds feasible backing?
How is admission tied to resident and device memory?

Reporting only reserved cache bytes can exaggerate current physical use. Reporting only resident bytes can understate the maximum commitment accepted from active requests.

An admission controller needs a model of future touches. Once a request is admitted with a promised context length, the service may be obligated to back pages later.

Paged KV cache designs use their own logical blocks, free lists, and device allocations. Those blocks need not equal the CPU’s 16 KiB VM pages.

Virtual paging is a mechanism below the serving allocator, not a substitute for cache admission and eviction policy.

a mapped model could make startup look complete too early

Memory-mapping model weights can make process startup return before all weight pages are resident.

The first inference then becomes the first sequential or layer-ordered traversal of those mappings. It can fault pages from the cache or storage while GPU submission waits.

A health endpoint that checks only:

file opened
mmap succeeded
model object constructed

can declare readiness before the working set is available.

A stronger readiness probe can:

touch required weight pages
validate checksums
perform one representative forward path
record major and minor faults
confirm device transfers and kernels complete

That probe moves cold costs into rollout. It also consumes bandwidth and can interfere when many replicas warm at once.

Staggered warming and admission avoid a fleet-wide page-in storm. Immutable file mappings let replicas share clean cache pages on one host, but concurrent first access can still contend on pager and storage work.

The eight-thread one-page result is a small version of that duplicate arrival.

zero copy still permitted faults and translations

Mapping a file avoids an explicit application read into a separate buffer. Sharing a tensor through a buffer protocol can avoid a payload copy. Neither promise removes virtual-memory work.

A zero-copy path can still incur:

minor faults to map resident file pages
major faults to obtain absent file data
TLB misses and table walks
copy-on-write after a private store
device registration
cache maintenance
page pinning

“No payload copy at this interface” remains a useful claim. It should not be expanded to “no bytes moved anywhere.”

The first anonymous load caused the kernel to zero 16 KiB for one requested byte. That is data movement without an application copy call.

The file mapping can move bytes from storage into the page cache without a read in the application trace.

The device can DMA from pinned pages after registration without CPU copying, while the IOMMU still translates addresses.

Each layer needs its own movement and ownership statement.

the fault path could become the hidden serial section

Consider a request handler that touches 2,048 cold anonymous pages:

2,048 pages * 16 KiB/page = 32 MiB

Using the measured 802 ns/page first-touch median as an illustrative local value:

2,048 * 802 ns
= 1,642,496 ns
= about 1.64 ms

This arithmetic is not a production prediction. The measured distribution, concurrent contention, page source, scheduler, and memory pressure can change it substantially.

It shows why thousands of sub-microsecond events become visible at request scale.

If the service has a 10 ms latency budget, 1.64 ms is 16.4 percent:

1.64 ms / 10 ms = 0.164

Optimizing a 100 microsecond compute kernel would target the smaller component.

The correct fix may be warmup, admission, mapping policy, file layout, request placement, or keeping pages resident. “Optimize memory access” is not specific enough.

fault counters needed a time boundary

Process-wide counters accumulate activity from:

application mappings
thread stacks
dynamic libraries
allocator metadata
the benchmark's output buffers
runtime initialization
other threads

The concurrent probe initially measured thread creation and reported 25 faults for an eight-thread one-page test. Moving the baseline after every worker reached the barrier reduced the interval to 6 or 8.

That edit changed the question from:

what did this complete threaded operation fault?

to:

what faulted after all worker stacks and synchronization state were ready?

Neither boundary is universally correct. A service cold-start measurement should include thread creation. A page-handler microbenchmark should exclude it.

Counters need a named interval and competing activity control. A total printed at process exit cannot automatically be assigned to the code line nearest it.

a profiler sample could miss every fault

A fault lasting hundreds of nanoseconds may be shorter than a sampling profiler’s interval. Thousands of faults can still create aggregate delay.

Sampling can attribute time to:

the faulting load
kernel VM functions
zeroing
pager work
the scheduler

depending on profiler capabilities and call-chain capture.

Event counters provide counts but not necessarily causal latency. Tracing every fault provides detail but adds overhead and can produce enormous data.

A practical investigation combines:

wall time around the operation
minor and major fault deltas
residency before and after
sampled stacks
targeted trace for a smaller reproduction
hardware counters where available

The opening was small enough that exact page arithmetic falsified several explanations before a profiler was necessary.

On this machine, unprivileged kernel tracing access is restricted. The tagged source supplies mechanism, but no live trace proves the exact branch taken by each opening fault.

memory metrics described different obligations

For one mapping, production telemetry may expose:

virtual bytes
resident bytes
physical footprint
compressed bytes
swap-ins
minor faults per second
major faults per second
wired bytes
dirty file pages
page-in latency

No single metric means “memory used.”

Virtual bytes can be huge and cheap now. Resident bytes can fall under pressure while future latency risk rises. Wired bytes consume capacity that cannot be reclaimed normally. Compressed bytes consume less DRAM but imply decompression work. Major faults can expose storage dependence.

An alert on virtual size would have fired on the safe untouched 64 GiB reservation. An alert only on resident size could miss a service that accepted commitments it cannot later back.

Capacity planning needs both current state and future obligations:

current resident working set
maximum live request growth
copy-on-write breakage under child writes
warm replica count
wired or device-registered pools
pressure and reclaim behavior

The unit is not only bytes. Fault rate and latency connect memory state to request behavior.

the economic result depended on which bytes stayed shared

Suppose ten worker processes map the same 20 GiB immutable weight file.

If clean pages are shared through the host page cache, ten virtual mappings do not require 200 GiB of distinct physical weight pages. If every worker modifies private pages, copy-on-write can erode that sharing.

The difference affects:

replicas per host
startup I/O
memory pressure
fault rate
host count
cost per served request

An optimization that changes a mapping from shared read-only weights to private transformed weights may improve compute and increase host memory. The net value depends on both.

If shared residency permits four replicas on a host where private copies permit two, the capacity effect can dominate a modest kernel speedup.

That scenario needs actual model size, sharing measurements, workload, and SLO data. The article does not attach a dollar value to the mechanism alone.

The operational question is precise:

which physical pages remain shared after the real initialization and
request path?

Virtual mapping counts cannot answer it.

the original load could now be predicted

The opening pointer was valid because mmap had installed VM map entries for its range. mincore found zero resident pages because no data page had yet become necessary.

The first load from each 16 KiB page:

missed a usable translation
raised an arm64 data abort
reached XNU with its virtual address and read access
found a permitted anonymous map entry
resolved the object offset
obtained and zeroed a physical page
installed a pmap translation
returned from the exception
retried and completed the same load

The second load could use the installed page-table entry, perhaps after a hardware TLB refill, without another process page fault.

The first write did not fault because the original map allowed writes and the zero-fill resolution had no pending copy obligation.

After fork, write permission was deliberately withheld until the child copied a shared page. After PROT_NONE, no legal resolution could grant the read. After munmap, no entry described the address. After file truncation, the entry survived but the pager could not provide the old offset. After mlock, the faults happened during wiring instead of the later loop.

The address never contained those distinctions.

mmap had made the pointer meaningful. The load made the page real.