three hundred processes waited for one pid

The first description of HAMi issue 1662 blamed a file called /tmp/vgpulock/lock. Two hundred to three hundred processes were starting at once. Container logs repeated unified_lock locked, waiting 1 second, and a CUDA initialization that should have been quick took roughly a minute.

That account was accurate for the deployed release. It became misleading while the issue remained open.

On current HAMi-core main, try_lock_unified_lock() still exists and now uses flock(), but postInit() never calls it. The active gate is an unnamed, process-shared POSIX semaphore named sem_postinit, stored inside a memory mapped cache file. Patching the unused flock helper can’t shorten the path. Swapping the semaphore for something fancier makes the waiting tidier, but it can’t remove the work that every process performs one at a time. And most of that work is a CUDA primary-context retain and release whose purpose is surprisingly indirect: make the process briefly appear in an NVML list so HAMi-core can discover its PID as seen by the host.

I didn’t arrive at that description by reading the word lock and stopping. The lock name had changed, the synchronization primitive had changed, and the expensive operation sat several calls below it. I have been working on this issue, so I needed a description that would survive a patch review. The only stable way I found was to carry one process from Kubernetes allocation to its first cuInit(0), account for every owner it met, and then release 300 copies of the same process together.

The source in this article is frozen at 52f33fc7fa1fbb3f08148ab076d0e7447bec7f2a for HAMi-core and 8fdd92813b07f3bdb749ee88779a44c373315bb6 for the HAMi control plane. Three pieces of open work sit at separate revisions: the benchmark suite, which is not mine and whose published numbers I use with that caveat attached, and the GPU-free concurrency test and host-PID RFC, which are mine. I will say main, open PR, or prototype when crossing those boundaries.

I could run the source assertions, wire-protocol tests, PID parsers, regression models, and a 300-slot shared-state model on an arm64 machine without CUDA. The A100 timings came from the public benchmark artifacts linked by the RFC, not from this machine. Keeping those two evidence classes separate matters more here than adding another decimal place.

the container is prepared before the process exists

HAMi is not one library. The repository named HAMi contains the Kubernetes control plane: scheduling logic, a device plugin, monitors, configuration, and deployment charts. HAMi-core contains libvgpu.so, the C shared library loaded inside a workload container. Issue 1662 crosses both repositories even though the visible delay happens inside the second one.

A Kubernetes device plugin is a process on each node that advertises special hardware resources to the kubelet. The kubelet is the node agent that asks a container runtime to create containers. A plugin registers a Unix socket under /var/lib/kubelet/device-plugins/, reports healthy devices, and implements an Allocate remote procedure call. Allocate does not allocate device memory for the eventual application. It returns instructions the runtime should apply while constructing the container: device nodes, environment variables, mounts, annotations, or CDI device names. Kubernetes documents that boundary in its device plugin contract.

HAMi’s NvidiaDevicePlugin.Allocate reads the device assignment already made for the pending Pod and translates it into a ContainerAllocateResponse. For a non-MIG shared GPU request, a simplified response looks like this:

environment:
  CUDA_DEVICE_MEMORY_LIMIT_0=4096m
  CUDA_DEVICE_SM_LIMIT=25
  CUDA_DEVICE_MEMORY_SHARED_CACHE=/usr/local/vgpu/<random>.cache

mounts:
  host libvgpu.so       -> /usr/local/vgpu/libvgpu.so       read-only
  per-container cache  -> /usr/local/vgpu                  read-write
  /tmp/vgpulock        -> /tmp/vgpulock                    read-write
  ld.so.preload        -> /etc/ld.so.preload               read-only

CUDA_DEVICE_MEMORY_LIMIT_0=4096m says that visible CUDA device zero receives a 4096 MiB budget. CUDA_DEVICE_SM_LIMIT=25 supplies a percentage-like target to the core-utilization controller. The randomly named shared cache gives the processes in one allocated container a common file. The libvgpu.so mount supplies the interceptor. The preload file makes the Linux dynamic loader load it early. /tmp/vgpulock is node-shared and survives as a compatibility and coordination path.

The ordering kills a tempting fix on its own. Why not have Allocate write a PID translation file for the container? Because Allocate runs before the runtime creates the container and before its application calls fork(). At that moment there is no application PID to translate. A Pod with five later child processes needs five translations, and those children may be created and destroyed repeatedly. Maintaining the file would require a live host-side observer plus a race-free update protocol. It would no longer be a static allocation result.

The device-plugin DaemonSet (a Pod that Kubernetes runs on every node) defaults to hostPID: true. A Pod normally gets a private PID namespace, so PID 37 inside it may be PID 8011 on the node. hostPID: true places the plugin in the host’s PID namespace. That fact later makes the plugin a plausible identity broker, but it does not mean workload Pods should receive the same privilege.

There is a second coupling hidden in the cache. HAMi’s Go monitor maps the same file and casts its bytes to a Go sharedRegionT with unsafe.Pointer, which is Go’s reinterpret_cast in a language that normally refuses to have one. The C compiler lays out shared_region_t; the Go code independently promises the same field order, sizes, padding, semaphore width, 16-device arrays, and 1024 process slots. No serializer mediates between them. The raw bytes are the interface.

That is an ABI, an application binary interface. A source-level API says which functions may be called. This ABI says byte 0 is an initialized flag, some later bytes hold a sem_t, and a much later span holds exactly 1024 process records. If C adds an eight-byte field and Go does not, the monitor can read a PID as memory usage without either compiler reporting an error. Nothing in either toolchain will warn you. The open benchmark PR includes abi_check, which compares sizeof and offsetof values between the two sides, because a failing assertion is the only warning this interface can ever give.

The control plane has now finished its part. It has arranged the environment and filesystem view. Only then does the runtime execute the application.

a shared library stands between the program and CUDA

Suppose the program contains this C code:

#include <cuda.h>

int main(void) {
    CUresult result = cuInit(0);
    return result == CUDA_SUCCESS ? 0 : 1;
}

The compiler records an undefined reference to cuInit. At program startup, the ELF dynamic loader maps the executable and its shared libraries, resolves symbols, applies relocations (patches that write real addresses into the loaded code), and transfers control to the program. Linux’s /etc/ld.so.preload is a system-wide list of shared objects the loader should place before ordinary dependencies. The behavior is documented in ld.so(8). Because libvgpu.so appears first in the search order and exports a function named cuInit, the program’s call reaches HAMi-core rather than the NVIDIA driver’s entry directly.

This is symbol interposition: supply a definition with the same binary name and arrange for the loader to choose it first. A wrapper can inspect arguments, enforce a policy, update accounting, then call the real function. It avoids modifying PyTorch, TensorFlow, a Python CUDA extension, or a closed application.

Interposing one direct call is easy. Production CUDA code also asks for function pointers. It can call dlsym(handle, "cuMemAlloc_v2"), or newer CUDA libraries can use cuGetProcAddress to request a driver entry compatible with a CUDA version. If HAMi-core wrapped only the executable’s relocations, either lookup would return the real driver pointer and all later calls through that pointer would escape the limiter.

So HAMi-core interposes the resolver as well. Its dlsym wrapper first finds the real glibc dlsym with dlvsym(RTLD_NEXT, ...), where RTLD_NEXT means “whichever library comes after me in the search order”. It loads libvgpu.so and checks names beginning with cu; __dlsym_hook_section maps supported names to wrapper addresses. Its cuGetProcAddress and cuGetProcAddress_v2 wrappers perform the same substitution for the CUDA driver’s own lookup API.

That resolver wrapper has a recursion problem. To implement dlsym, it needs to call the real dlsym; resolving that function can itself pass through the wrapper. The current code tries known glibc symbol versions and keeps a 100-entry thread-and-pointer ring to notice recursive RTLD_NEXT results. This is one of the places where a normal C function pointer becomes systems code: the wrapper is active while assembling the mechanism it will use to escape itself.

load_cuda_libraries() opens libcuda.so.1 with RTLD_NOW | RTLD_NODELETE and fills a table of real function pointers. A macro such as

CUDA_OVERRIDE_CALL(cuda_library_entry, cuInit, Flags)

selects the saved pointer rather than resolving the public name again. Calling cuInit(Flags) by name from inside the wrapper would simply recurse back into the wrapper.

I generated an inventory from the pinned tree: 572 C function definitions, 558 unique names. Memorizing that list would be pointless. The grouping that actually helped me navigate is smaller:

• entry and symbol-resolution code owns loading, dlsym, cuGetProcAddress, preInit, and postInit;
• context, device, stream, event, module, and graph wrappers preserve the CUDA control surface;
• memory and allocator wrappers enforce and account for allocations;
• shared-region code coordinates processes;
• the utilization watcher observes NVML and throttles kernel launches;
• 216 definitions in nvml_entry.c preserve the broad NVML surface, while a small selected subset changes the view exposed to the container.

An API catalogue would bury the design. The wrappers exist because every way of obtaining or using a CUDA entry must either pass through policy or be deliberately allowed through.

one new hook has to appear in four places

The most common beginner mistake in an interposition library is to implement a function and assume the loader will find it. HAMi-core’s hook chain has four representations of the CUDA surface:

1. cuda_library_entry[] in cuda/hook.c stores symbol names and real function pointers.
2. An enum in libcuda_hook.h assigns stable table indices used by CUDA_OVERRIDE_CALL.
3. DLSYM_HOOK_FUNC(...) entries in libvgpu.c return wrapper addresses to dynamic lookups.
4. A C definition in src/cuda/*.c implements the wrapper.

Order matters between the string table and enum. If enum member 72 says cuMemAlloc_v2 while string-table slot 72 contains cuMemFree_v2, the allocation wrapper can call a real free function through a type cast. Both files compile. The failure appears only at runtime, far from the edit.

The repository’s hack/check_cuda_hook_consistency.py strips comments, extracts all four sets, checks the table and enum count and order, reports duplicates, and requires each non-optional entry to have both a dlsym route and a wrapper definition. That script encodes architectural knowledge a reviewer should not have to reconstruct for every CUDA release.

Versioned symbols add another dimension. CUDA preserves compatibility by shipping names such as cuCtxCreate_v2, cuCtxCreate_v3, and cuCtxCreate_v4. The unversioned name requested through cuGetProcAddress may need to map to a particular implementation for the caller’s CUDA version. HAMi-core’s function map selects the real name for a version interval; the fallback lookup tries _v3, then _v2, then the base wrapper. If a driver does not export the newest name, load_cuda_libraries walks backward through prior suffixes for selected entries.

The C preprocessor can also rewrite names before HAMi-core sees them. Its nvml_prefix.h maps older spellings such as nvmlDeviceGetCount to v2. CUDA 13 changes the signature behind cuMemAdvise; the source conditionally omits the older wrapper to avoid defining two incompatible functions after header macro expansion. Supporting a new toolkit is an ABI exercise before it is a matter of adding another case.

Not every acquired function should be redirected. Current find_symbols_in_table deliberately returns null for names beginning with cuGraph, allowing graph functions obtained through cuGetProcAddress to fall through to the driver. Directly linked graph wrappers still exist for a broad set of nodes and execution functions. Whether that asymmetry is desired policy or compatibility debt has to be decided from tests; pretending every listed graph entry is always intercepted would be false.

The boundary also explains the build. CMake combines object libraries from multiprocess, allocator, cuda, and nvml into one shared libvgpu.so, then links CUDA and NVML. A non-debug build strips symbols, but public wrapper symbols still need enough dynamic visibility for interposition. The library has to load in arbitrary processes, so a constructor that eagerly touched CUDA would be risky. Initialization is instead pulled by first symbol lookup or first wrapped call and guarded with once flags.

There are three useful hook tests, each catching a different escape:

direct relocation
  executable calls cuMemAlloc_v2 by name

dlsym
  executable asks dlsym for "cuMemAlloc_v2", then calls the pointer

CUDA resolver
  executable asks cuGetProcAddress for "cuMemAlloc", then calls the pointer

All three should hit the same accounting assertion. Add old and new CUDA version requests to test the version map, and ask for an unsupported symbol to prove clean pass-through. A test that calls only ordinary PyTorch allocation can pass while a plugin has already escaped through a stored driver pointer.

Interposition is also not a security boundary against a hostile workload. A process with enough filesystem and device access can load a differently named driver, issue ioctls, use a statically bound path, or tamper with its own address space. HAMi-core provides transparent resource virtualization for cooperating CUDA applications under a controlled container deployment. The container runtime, device-node permissions, Linux security controls, and scheduler remain part of isolation.

one cuInit has three beginnings

NVIDIA defines cuInit(0) as initialization of the Driver API for the current process. The Driver API documentation also notes that it may preload libraries needed for JIT compilation. HAMi-core wraps that one public beginning with three local phases:

CUresult cuInit(unsigned int flags) {
    pthread_once(&pre_cuinit_flag, preInit);
    ENSURE_INITIALIZED();

    CUresult result = real_cuInit(flags);
    if (result != CUDA_SUCCESS)
        return result;

    pthread_once(&post_cuinit_flag, postInit);
    return CUDA_SUCCESS;
}

The actual source uses casts and macros, but this preserves the order. pthread_once pairs a flag with a function and guarantees that concurrent threads in one process complete that function once. It is not a cross-process lock.

Three hundred processes have three hundred independent pre_cuinit_flag objects and may all enter preInit together.

preInit initializes logging, obtains the real resolver, loads the CUDA symbol table, calls ensure_initialized(), and registers a child handler with pthread_atfork, the libc facility that runs callbacks around every fork(). ensure_initialized() has another process-local pthread_once; its callback opens and maps the shared cache and registers the process slot. Calling it from both preInit and the public wrapper looks redundant. It is, and it is also safe, because the once flag owns idempotence. Then HAMi-core calls the real NVIDIA cuInit. Only after the driver reports success does postInit initialize the private allocation lists, map visible CUDA indices to NVML indices, discover the host PID, set the utilization policy, and possibly launch a watcher thread.

That order protects two different truths. The shared region must exist before accounting begins. The host-PID detector needs a functioning CUDA driver because its current method briefly retains a primary context. Moving the whole detector before real cuInit would violate its own dependency. Moving all of shared-region initialization after it would leave nowhere to record the answer.

The fork handler is easy to miss and expensive to ignore. fork() gives the child a copy of the parent’s address space, including once flags that say initialization already happened. The child is a new process with a new PID and needs its own slot. HAMi-core’s at-fork callbacks reset the shared-region initialization status and post_cuinit_flag; childReinitPostInit also clears pidfound. The next CUDA use in the child rebuilds process-specific state instead of impersonating its parent.

Python readers encounter the same path. torch.cuda.init() is Python syntax, but PyTorch eventually crosses its C++ boundary and invokes the CUDA runtime or Driver API. Once the call reaches an ELF symbol or acquired function pointer, the language that originated it is irrelevant to libvgpu.so.

the cache file is a tiny multiprocess database

CUDA_DEVICE_MEMORY_SHARED_CACHE names a regular file. HAMi-core opens it with O_RDWR | O_CREAT, extends it to sizeof(shared_region_t), and maps it with

mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

mmap asks the kernel to associate a range of virtual addresses with file bytes. MAP_SHARED means writes become visible through other mappings of the same file. Each process receives a different pointer value, but all pointers refer to the same underlying pages. This is shared memory implemented through a file-backed mapping.

The top-level structure contains node and container policy plus process state:

typedef struct {
    _Atomic int32_t initialized_flag;
    uint32_t major_version;
    uint32_t minor_version;
    _Atomic int32_t sm_init_flag;
    _Atomic size_t owner_pid;
    sem_t sem;
    uint64_t device_num;
    uuid uuids[16];
    uint64_t limit[16];
    uint64_t sm_limit[16];
    shrreg_proc_slot_t procs[1024];
    _Atomic int proc_num;
    _Atomic int utilization_switch;
    _Atomic int recent_kernel;
    int priority;
    _Atomic uint64_t last_kernel_time;
    sem_t sem_postinit;
} shared_region_t;

The first process takes a short lockf file lock while creating this format. It initializes both semaphores with a pshared argument of 1. A POSIX semaphore is a counter that processes can sleep on; pshared makes the one living inside the mapping usable by every process that maps the file. The creator fills the limits from environment variables, publishes version 1.2, and finally stores the magic initialized flag with release ordering. A later process sees the flag with acquire ordering and checks version and limit consistency.

Acquire and release are C atomic memory orders. A release store says earlier writes in this thread must become visible before the published value. An acquire load that observes it prevents later reads from moving before it. The pair lets the initialized flag mean more than “some integer equals 19920718”, a magic constant that reads suspiciously like somebody’s birthday; it means the entire structure written before that publication is ready to read.

Each process slot contains two identities and three classes of GPU state:

pid       PID in the workload process's namespace
hostpid   PID that host-side NVML reports

used[device]
  context_size, module_size, data_size, offset, total
monitorused[device]
device_util[device]
  decoder, encoder, SM utilization

status, seqlock

Registration modifies the compact slot array, so init_proc_slot_withlock() waits on sem, clears a stale slot if the same PID has looped around, otherwise initializes slot proc_num, increments the count, removes some dead entries, and posts the semaphore. It caches my_slot in process-private memory. Normal allocation accounting can then update its own slot without scanning 1024 records.

The array is compact: deleting slot 17 moves the final live slot into position 17 and decrements proc_num. Copying a C structure containing _Atomic members with plain assignment can tear or discard atomic semantics, so current main has copy_proc_slot_atomic(), which loads and stores each member. It must also repair my_slot if the moved record belongs to the current process.

Memory totals change more often than membership. A global semaphore around every allocation would turn the data path into the startup problem again. HAMi-core instead keeps membership serialized but updates per-process totals with atomic fetch-add and fetch-sub, single operations the hardware completes without letting another core slip in between. A slot’s seqlock becomes odd while a writer changes a group of counters and even afterward. A reader loads the sequence, reads the total, then loads the sequence again. It retries if the values differ or the first was odd.

A seqlock favors readers when writes are brief. It does not block a writer or make an arbitrary collection of multiple writers safe by magic. Here the discipline is local: a process updates its cached slot, and readers need a consistent total for an out-of-memory decision. The retry loop backs off from CPU pause or yield instructions to microsecond sleeps if a writer remains active.

This database has no server process, no transaction log, and no schema migration engine. Its schema is the C/Go ABI, its row lock is a process-shared semaphore, its hot counters are atomics, and its crash recovery is code that inspects PIDs. Somewhere around here I stopped writing “cache file” in my notes.

The ABI deserves a closer look because several issue 1662 fixes sit next to it. On linux/amd64, the open abi_check records these landmarks:

sizeof(shared_region_t) = 2,008,952
offsetof(limit)         = 1,600
offsetof(sm_limit)      = 1,728
offsetof(procs)         = 1,856
sizeof(limit[0])        = 8

The Go mirror appears to lack both the C slot’s named seqlock and the final sem_postinit. It preserves their bytes through reserved fields: three 64-bit words after Go’s slot status occupy the same span as C padding, seqlock, and two unused words; four 64-bit words at the end occupy the same 32 bytes as glibc’s sem_t on the tested platform. That is deliberate binary padding, but the names still matter. Go cannot safely operate a field it calls unused, and sem_t size is a libc and architecture property rather than a C language constant.

Major/minor version checks currently log disagreement after mapping the file. A migration that changes size has an earlier problem: the opening process extends and maps according to its local sizeof, while an old monitor may map a different length. Safe evolution needs an explicit compatibility rule:

• adding meaning inside already reserved bytes can preserve size and offsets, if every reader tolerates the new version;
• changing a hot slot field requires updating C and Go together and retaining an ABI test for both targets;
• changing sem_t representation across libc or architecture cannot be made portable by matching one captured number;
• replacing the synchronization primitive may require a new cache version or a self-describing header rather than silently reusing 1.2.

Atomics do not solve the cross-language half automatically. C’s _Atomic uint64_t tells the C compiler which operations are indivisible and what memory ordering they have. The Go mirror currently reads ordinary uint64 fields through an unsafe cast. If C writes concurrently while Go reads, language-level data-race and visibility contracts still need examination even when aligned 64-bit hardware loads happen to be atomic on amd64. A monitor snapshot should either follow the seqlock protocol, use corresponding Go atomic loads, or read under a synchronization boundary both languages define.

The current C seqlock also assumes one writer per slot. Consider two threads in one process updating different allocation categories at the same time:

initial sequence 0
writer A increments -> 1
writer B increments -> 2
reader sees even 2 and may accept while both writers are active
writer A increments -> 3
writer B increments -> 4

The odd/even rule only encodes “some writer is active” when writers serialize their sequence transitions. The process-private allocator mutex serializes many tracked allocation updates, but context and other paths must be checked against that assumption.

A seqlock is a protocol shared by every writer, not a property gained from adding a counter to the struct.

This is also why the CPU model I kept for this article deliberately does less than a fake CUDA benchmark would. It registers 300 unique PIDs, assigns host PIDs, removes a subset through compact-array moves, and asserts survivor uniqueness. Post-init owner death stays marked unresolved in it, because I could not find a recovery in the source and refused to invent one for the model.

one process has two correct PIDs

Linux gives a process an identity in each PID namespace that contains it. The initial namespace sees every ordinary process on the host. A container runtime usually creates a child namespace for a Pod. Inside that namespace the first process can be PID 1 even though the host calls it PID 8011. A child forked inside the Pod might see itself as PID 6 while the host sees PID 8127.

Neither number is intrinsically false. A PID only has meaning together with the namespace from which it is interpreted.

HAMi-core needs both. getpid() returns the container-visible number and is a convenient key for finding the process’s own shared-region slot. NVML is a host-side management library. The process arrays returned by nvmlDeviceGetComputeRunningProcesses use the host-visible number. The utilization watcher reads an NVML sample and needs hostpid to join that sample to the right slot.

The current set_task_pid() learns the join key through an experiment:

1. Ask NVML which processes currently have compute contexts on a visible GPU.
2. Retain this process’s primary CUDA context on device zero.
3. Ask NVML for the process list again.
4. Find the PID present in the second set but absent from the first.
5. Record that PID and the context’s reported GPU memory.
6. Release the primary context.

If the first set is {410, 921} and the second is {410, 921, 8127}, the extra PID is inferred to be the caller’s host PID.

A CUDA context is driver state that owns or names resources required to execute work on a device. A device’s primary context is a retainable context shared by runtime users in the process. cuDevicePrimaryCtxRetain returns a handle and increases its use; release drops the retain. HAMi-core is not retaining it here because the application requested a context. It is causing an externally visible event so the process can recognize itself in NVML.

That distinction explains both the cost and the context accounting. The NVML record contains usedGpuMemory. set_task_pid() stores it in the global context_size. Later, the wrapper for cuDevicePrimaryCtxRetain adds that size to the process slot once per device, and the release wrapper subtracts it. The startup probe therefore discovers two values in one maneuver: identity and an initial estimate of context memory.

There are awkward edge cases. The code loops over NVML devices but breaks after the first mapped one. Visible CUDA indices may not line up with host NVML indices, so map_cuda_visible_devices() constructs a mapping from CUDA_VISIBLE_DEVICES. The RFC reports that CUDA_VISIBLE_DEVICES=1 and 1,0 still produce SET_TASK_PID FAILED on the tested eight-A100 node, even after an earlier fix stopped leaking the probe context. A before-and-after set difference also assumes exactly one new relevant PID appears. Serialization makes that assumption less bad by allowing only one detector to perturb the list at a time.

Here is the current postInit critical section, the stretch of code that runs while the lock is held, reduced to its ownership:

int acquired = lock_postinit();

if (acquired) {
    result = set_task_pid();
    unlock_postinit();
} else {
    result = NVML_ERROR_TIMEOUT;
}

pidfound = result == NVML_SUCCESS;
init_utilization_watcher();

The semaphore is necessary for the current inference algorithm. Two processes taking the “before” snapshot together and retaining contexts together could each see two new PIDs and choose the wrong one. The lock does not protect the short set_host_pid() store. It protects the entire active measurement.

This also corrects the word contention. A trace can show 299 processes waiting at sem_timedwait, but that does not make sem_timedwait the service being serialized. The queue forms because one holder is executing NVML and CUDA work. Replacing a polling file lock with a blocking semaphore saves retry sleeps and CPU activity. It leaves service time inside the critical section. For a burst, tail latency still accumulates approximately one service time per position in line.

lock_postinit() uses a fresh absolute timeout for each 30-second attempt and gives up after more than ten timeouts. That can mean roughly 300 seconds of waiting. On failure, postInit sets pidfound = 0. The kernel-launch wrappers call rate_limiter() only when pidfound == 1. Time out often enough, then, and SM limiting is simply off, silently, for the remaining life of the process. That worries me more than the minute of startup latency ever did, and the RFC treats it as the more serious defect.

The other semaphore, sem, has an owner_pid next to it. On a timeout, lock_shrreg() tests whether the recorded owner is dead, then uses compare-exchange, the atomic “replace this value only if it still holds what I just read” operation, so that exactly one waiter clears it, posts the semaphore, and retries. sem_postinit has no equivalent owner field. POSIX unnamed semaphores are not robust mutexes, robust in the technical sense where the kernel tells the next locker that the owner died. If a process dies after decrementing sem_postinit and before posting, nobody restores the count. The code eventually times out, but the invariant remains broken.

One source tree, then, contains three synchronization stories at once. try_lock_unified_lock() uses flock and has no caller on main. sem serializes shared-region membership, with explicit but imperfect owner-death recovery. sem_postinit serializes the CUDA/NVML identity experiment, with a timeout escape and no ownership recovery at all. A patch claiming to fix “the lock” has to say which of the three it means.

releasing three hundred workers together

The open bench_init PR turns the production symptom into a repeatable density experiment. The harness is not mine, and I have leaned on it constantly. The parent creates an anonymous shared file with memfd_create, maps one result slot per worker, and creates a pipe. It then fork()s and exec()s the same binary for each worker.

The exec is important. A plain child after fork inherits the parent’s loaded libraries and possibly CUDA state. A real workload child starts a fresh program image and goes through the ELF loader and libvgpu.so initialization. The benchmark parent never touches CUDA. Each worker increments an atomic ready count and blocks in read(pipefd[0]). The parent waits until all workers reach that barrier, records a monotonic timestamp, and closes the pipe’s write end. Every blocked read observes EOF and wakes. Closing one descriptor becomes a broadcast without a spin loop.

The worker records entry time, calls cuInit(0), records return time and the CUresult, then writes its own slot. The parent waits for children, sorts only successful latencies, and reports nearest-rank p50, p95, and p99 along with wall time. It initializes every result to failure before launch so a worker that dies before writing cannot masquerade as a zero-millisecond success.

Nearest-rank p99 at 300 workers is sorted sample 297 when counting from one. It tells us about a process near the back of the line, not necessarily the single worst process. With 128 workers, it is sample 127. The exact percentile definition belongs in the harness because different interpolation rules can move small-sample results.

The current public result table is:

workers	baseline p99	guarded NStgid p99	broker p99
128	78.37 s	27.85 s	27.01 s
240	165.20 s	52.89 s	53.56 s
300	228.04 s	68.73 s	70.08 s

Baseline and broker entries are the mean p99 of three runs on the same eight-A100 node. NStgid entries are single runs. Across the three broker runs, all 2,004 workers started, with no broker miss, procfs hit, post-init lock wait, or PID failure. The RFC reports sample coefficients of variation, standard deviation as a fraction of the mean, of 1.106 percent at 240 and 0.093 percent at 300 for that campaign.

Those results falsify two easy headlines.

First, the original issue said roughly one minute, but 300 workers in the controlled baseline reached 228.04 seconds at p99. The production observation was a real workload symptom, not a bound on the mechanism.

Second, the broker does not reduce cuInit p99 at 300 workers to microseconds. The broker query took about 183 and 206 microseconds p99 in two 500-sample prototype runs. Whole cuInit still took 70.08 seconds p99 because the CUDA driver and shared-region registration remain.

I fitted straight lines to the three public worker counts, as a compact description rather than a physical law:

baseline: intercept -33.864 s, slope 858.087 ms/process, R² 0.99353
broker:   intercept  -5.162 s, slope 248.706 ms/process, R² 0.99847
NStgid:   intercept  -2.699 s, slope 235.878 ms/process, R² 0.99810

The negative intercept warns against extrapolating to one worker. Three high-N points are describing the burst regime. Within that regime, removing the probe cuts the fitted slope by about 71 percent. The RFC expresses the same effect directly: 64.5 to 69.8 percent lower p99 across 128 to 300 workers.

A minimal queue model explains why the slope is more revealing than one endpoint. If each process holds a serialized resource for average service time $s$, and process $k$ is roughly the $k$th served, then its waiting component is

$$W_k \approx (k - 1)s.$$

Removing an operation from the critical path reduces $s$ for nearly every worker behind it.

Making only the first worker faster changes an intercept. Changing the serialized service changes the slope.

four workers make the queue visible

Take four workers released at time zero. Give each 20 ms of independent loader and driver work before the identity gate, 50 ms inside the serialized probe, and 10 ms after it. Ignore operating-system noise for the moment.

time (ms)       0       20      70      120     170     220
worker A        parallel [probe A] post
worker B        parallel wait    [probe B] post
worker C        parallel wait            [probe C] post
worker D        parallel wait                    [probe D] post

All four can spend their first 20 ms simultaneously because no shared owner forbids it. Worker A then occupies the detector from 20 to 70 ms. B can finish at 130 ms, C at 180 ms, and D at 230 ms. The batch wall time is not four times the whole 80 ms single-process path. It is roughly

$$20 + 4(50) + 10 = 230\text{ ms}.$$

The critical fraction, not total single-worker latency, multiplies by worker count. If the broker changes the serialized 50 ms probe to a 1 ms independent request, all workers can overlap it. Their idealized completion moves near 31 ms. If a separate shared-region gate still costs 12 ms per registration, the new batch becomes roughly

$$20 + 4(12) + 1 + 10 = 79\text{ ms}.$$

The first bottleneck disappeared; the second now determines the tail.

Real cuInit is not that clean. Workers do not reach the gate at identical times after the pipe closes. Linux chooses which runnable process gets CPU, the loader faults different pages, the driver may serialize internal work, and the first user of a library pays cold initialization. The semaphore order is also not a promised FIFO service discipline. A later arrival can acquire before an earlier waiter depending on scheduling and implementation. For p99, we care about the distribution of completion ranks, not a promise that worker 297 literally arrived 297th.

This makes three timestamps distinct:

• release offset: when the parent closed the barrier until the worker entered cuInit;
• call latency: time from the worker entering cuInit until it returned;
• batch wall span: time from release until the final relevant worker ended.

A scheduler delay before cuInit increases release offset but not call latency. Waiting inside sem_postinit increases call latency. The parent wall span sees both. bench_init retains worker start offsets and reports a span so these effects need not be folded into one number.

Now suppose a detector times out and returns failure. If the percentile code sorts only successful calls, the slowest or most contended workers may vanish from the latency array. A patch could apparently improve p99 by failing the tail. That is why ok, failed, and failure reason must be read beside the percentiles, and why the RFC calls out 849 timeout cycles in the 300-worker baseline.

Mean per-worker latency answers a different cost question. For the four-worker example, successful call latencies are roughly 80, 130, 180, and 230 ms, with a mean of 155 ms. Summing them gives 620 process-milliseconds of occupied or waiting time even though only 230 ms passed on the wall clock. In a cluster, that sum corresponds to worker capacity unavailable to useful application work. Wall span describes readiness; summed latency helps explain resource waste.

Throughput after the burst also needs care. If every process initializes once and then serves for an hour, shaving initialization can greatly improve readiness without changing steady-state tokens per second. In the reported queue-driven deployment, Pods repeatedly spawn child processes per task, so startup lies on the recurring task path and queue throughput can fall. The same latency number has different operational weight under a long-lived model server and a process-per-task service.

The line fit can be interpreted through that timeline. Its high-N slope is the additional p99 cost associated with one more competing process across all serialized and density-sensitive components. It is not identical to the duration of one set_task_pid call. Removing the context probe changes both direct work and interference with driver initialization. The remaining 236 to 249 ms per-process fitted slope can include shared-region serialization, driver-internal serialization, and CPU density. Assigning it all to sem without another phase trace would repeat the original mistake at a new lock.

An ideal next campaign records a phase vector for every worker:

loader_ready
cuInit_wrapper_enter
shared_region_enter/acquired/released
real_cuInit_enter/exit
identity_tier_enter/exit
postInit_exit
cuInit_wrapper_exit

Sort workers by final latency, then inspect the phase composition of p50, p95, p99, and max. If the p99 worker spends 60 seconds before real_cuInit returns, optimizing a later semaphore cannot help it. If the broker tier completes in 200 microseconds but shared registration waits 50 seconds, the new target is clear. If release offsets dominate, the benchmark host is CPU-saturated before the library path even begins.

Run this way, the burst stops being a stress test and becomes an instrument. The barrier controls arrival pressure, per-phase clocks locate service, failure counts stop the tail from quietly disappearing, worker-count sweeps reveal which cost scales, and component tests remove the GPU when the GPU is not the question.

The baseline is not perfectly linear. CUDA driver initialization, CPU scheduling, NVML calls, cache-page faults, slot cleanup, and hardware activity are not a single deterministic server. A fit with $R^2$ near one says the linear component dominates these measured high-density points; it does not grant a universal 858.087 ms constant to every GPU and kernel version.

The benchmark suite contains two useful controls. phase_probe measures NVML initialization, process-list snapshots, and primary-context operations separately. nested_retain asks whether retaining an already held primary context becomes cheap, while warm_holder keeps a primary or non-primary context alive in another process. Their job is to distinguish “CUDA is slow” from “this particular identity probe is slow.” The PR description reports a rough 80/20 split between primary-context work and NVML, with nested retains around a microsecond after establishment. These are reported A100 results from an open PR, not measurements I reproduced locally.

One more metric deserves its own column in future runs: failures by path. A fast p99 calculated after silently excluding PID failures can reward a broken implementation. The harness already reports ok and failed; the proposed telemetry needs broker miss, procfs miss, fallback use, post-init waits, and SM-limiter-disabled outcomes as well.

what the library protects after initialization

Before deleting the probe I had to answer a prior question: what is the host PID actually for? Removing it would be pointless if libvgpu.so then failed to enforce the resources supplied by Allocate. The rest of the codebase is the answer.

Start with ordinary device memory:

CUdeviceptr pointer;
CUresult result = cuMemAlloc_v2(&pointer, bytes);

A CUdeviceptr is an integer-sized handle representing an address in a CUDA device address space. HAMi-core’s wrapper calls allocate_raw, which reaches add_chunk. That function finds the current device, calls oom_check(dev, size), performs the real GPU allocation outside its private allocator mutex, then checks the budget again under the mutex before recording the chunk.

The two checks close different windows. A pre-check avoids an allocation known to exceed the limit. While the driver is allocating, another process may consume shared budget. The second check notices the race; if the budget has been crossed, HAMi-core unlocks, frees the just-created driver allocation, and returns CUDA_ERROR_OUT_OF_MEMORY.

The private linked list maps addresses to sizes and devices so cuMemFree_v2 can remove the right amount later. The shared slot stores aggregate bytes so other processes and the monitor see container-wide consumption. Allocation types distinguish context, module, and data usage, although the current total is the main limiter input.

Synchronous and stream-ordered allocations need different accounting. cuMemAllocAsync draws from a CUDA memory pool. A pool can reserve more memory than one requested chunk and reuse it after a logical free. HAMi-core queries CU_MEMPOOL_ATTR_RESERVED_MEM_HIGH and accounts the increase in the pool’s high-water reservation, rather than blindly treating every request size as new physical pressure. cuMemAllocFromPoolAsync must query the caller’s pool, not the default pool. An unrecognized async free is passed to the real driver instead of being leaked merely because the interceptor lacks a list entry.

Managed memory, pitched allocations, host registration, arrays, virtual-memory reservation and mapping, IPC handles, external resources, and graph nodes expand the surface. Not every wrapper changes accounting. Some perform an OOM check, some track a precise size, some forward directly, and some contain a TODO where the byte estimate cannot include driver alignment. Correct coverage means matching the semantics of each allocation family, not putting used += bytesize before every function whose name contains Mem.

cuMemGetInfo_v2 is part of the virtual view. An application inside a 4 GiB slice should not necessarily be told that the physical 48 or 80 GiB GPU is available. NVML calls need a similar view: device count and handles must match visible devices, and memory information should reflect the slice rather than disclose or invite use of the whole card. Keeping that second window consistent with the first is the entire reason HAMi-core maintains an NVML hook table beside the CUDA one.

The memory path is mostly admission and accounting. The SM path is feedback control.

An NVIDIA GPU contains streaming multiprocessors, abbreviated SMs, that run thread blocks from kernels. CUDA_DEVICE_SM_LIMIT=25 does not carve out a fixed physical quarter of every SM. HAMi-core starts a utilization watcher, samples per-process SM utilization through NVML, compares observed utilization with the configured target, and adjusts a process-local token quantity. Before selected launch wrappers call the real driver, rate_limiter consumes tokens proportional to the grid size and waits when the balance is too low. A watcher replenishes or contracts the share based on feedback.

The code computes an internal capacity from

SM count * maximum threads per SM * FACTOR

and updates it with compare-and-swap loops. cuLaunchKernel and cuLaunchKernelEx call the limiter when pidfound == 1. pre_launch_kernel also updates a shared last_kernel_time, at most once per configured interval, using an atomic max-like compare-exchange loop.

This controller is approximate by construction. NVML samples utilization over a window; a launch grid is not identical to occupied GPU cycles; kernels vary in block size, registers, shared memory, and duration. The implementation is a rate regulator driven by observation, nothing like MIG, which carves a GPU into fixed slices at the hardware level. That makes host-PID identity operationally central: without the join between NVML’s PID and the slot, the feedback loop cannot attribute utilization.

Signals add a coordination channel. Shared slots carry a status; handlers for SIGUSR2 and SIGUSR1 switch it. suspend_all, resume_all, and wait_status_all use those fields to coordinate processes. Signals are asynchronous notifications, so the handlers deliberately perform a small atomic status store rather than take a normal mutex.

The word free is another place where physical and virtual views diverge. Suppose a physical GPU has 80 GiB, HAMi gives a container a 10 GiB limit, and the shared ledger says its processes use 3 GiB. The application’s useful answer to “how much can I still allocate?” is about 7 GiB, not the driver’s physical free count.

When HOOK_MEMINFO_ENABLE is compiled, cuMemGetInfo_v2 obtains the real driver totals and computes a bounded view. For nonzero configured limit $L$, ledger usage $U$, and physical total $P$:

$$T_{visible} = \min(L, P)$$ $$F_{visible} = \max(0, T_{visible} - U).$$

If usage already exceeds the configured limit, current code returns CUDA_ERROR_INVALID_VALUE rather than underflow an unsigned subtraction. If no virtual limit is configured, it keeps the driver’s physical total and subtracts ledger usage from it. cuDeviceTotalMem_v2 is simpler and returns the configured limit directly at the pinned revision, which makes a zero-limit configuration worth a focused compatibility test.

NVML’s memory-info wrapper starts with the physical response, finds the device’s CUDA-visible index, reads usage, monitor, and limit, and rewrites free, total, and used when a limit exists. usage is the interceptor ledger. monitor is the last host observation written by the watcher. They can disagree because allocations are asynchronous, samples are periodic, contexts and imported memory complicate attribution, or an API path escaped accounting. Logging both is a diagnostic opportunity; choosing one without naming its semantics would blur admission state and observation.

The OOM decision in allocator.c uses the shared ledger:

new_allocated = get_gpu_memory_usage(device) + requested_bytes;
if (limit != 0 && new_allocated > limit)
    reject;

MEMORY_LIMIT_TOLERATION_RATE exists in the header but the displayed current check compares directly with the limit. When rejection occurs, the code tries to clear stale process slots and recursively checks once the dead usage is gone. That repair calls clear_proc_slot_nolock from a function whose name explicitly says it expects the caller to own the membership lock, so call-site ownership deserves a regression test. “Atomic totals” do not make array compaction lock-free.

ENSURE_RUNNING() combines initialization with process suspension. It calls ensure_initialized, then sleeps while the current slot’s status is not 1. Many memory, copy, advise, and launch wrappers use it. A HAMi signal can pause new intercepted operations without stopping arbitrary CPU instructions in the process. Calls that only query configuration may use ENSURE_INITIALIZED instead. The distinction controls whether a suspended workload can inspect state and whether a forgotten status transition can hold a thread forever.

The launch path has a similarly precise order:

ensure the process is RUNNING
ensure postInit has completed
publish recent-kernel time
apply rate limiter when host PID is known
call the real CUDA launch

cuLaunchCooperativeKernel performs the first three but does not call rate_limiter at the pinned revision. Graph launches obtained through a resolver may also follow a pass-through path. Those differences belong in a coverage matrix. A core-limit test should exercise ordinary, extended, cooperative, and graph launches and assert which policy each is designed to receive.

The watcher itself separates slow observation from short shared updates. get_used_gpu_utilization asks NVML for running-process memory and utilization without holding sem, then takes the region lock only while matching host PIDs and writing slots. Holding sem across NVML would reproduce the same critical section error as host-PID discovery. Once the samples are stored, each process uses cached SM-limit and policy values on its launch fast path to avoid mapping and aggregating shared state for every kernel.

By this point the code has pulled the phrase “GPU memory” apart into at least four different quantities:

• physical total and free bytes reported by the driver;
• configured virtual limit supplied by the device plugin;
• intercepted allocations recorded in the shared ledger;
• sampled per-process use observed by NVML.

A bug report saying “memory is wrong” is underspecified until it says which pair differs, on which device mapping, at what point relative to asynchronous work. The host PID is the key for joining the fourth meaning to the third.

At this point the architecture can be stated without brand names:

Kubernetes policy
    -> environment, mounts, visible device set
interposed CUDA and NVML APIs
    -> admission, altered view, accounting, launch pacing
file-backed shared region
    -> container-wide membership and totals
host PID
    -> join key from management samples to process slots

Issue 1662 sits on the last arrow, but a safe repair must preserve every arrow above it.

/proc/self/status gave the right-looking wrong answer

Linux exposes process metadata as text under /proc. A line in /proc/<pid>/status can look like this:

NStgid:	8011	1

NStgid lists thread-group IDs, which ordinary user space calls process IDs, across nested PID namespaces. The leftmost value corresponds to the oldest ancestor namespace represented by the procfs mount, and the rightmost value to the namespace of the process reading that view. This looked like the whole fix: read the first number and never create a CUDA context at all.

The phrase represented by the procfs mount is where that shortcut breaks. I found out by running it, not by reading it.

A normal Pod with hostPID: false receives a procfs view rooted in its private PID namespace. In the k3s/containerd setup I tested, the process read

NStgid:	6

while a node-side check identified it as 8011 1. The container’s procfs had no obligation to reveal namespace ancestry above the namespace it represents.

Parsing the first number succeeds and returns 6, which is worse than a parse failure because it looks plausible.

My guarded prototype at b7a4c0e has no implicit /proc default because of that test. It checks LIBVGPU_HOST_PROCFS; only an administrator who mounted a procfs from the initial PID namespace can opt in. It reads <root>/self/status, locates the exact NStgid: field, parses positive decimal values with bounds checks, and selects the first.

That parser is mundane C with security consequences. It rejects a null pointer, zero capacity, the wrong field name, an empty list, negative or zero PIDs, integers above INT_MAX, junk attached to a number, and more namespace levels than the output buffer can hold. strtol alone is not validation; the caller must inspect errno, the end pointer, range, and delimiters.

The prototype then performs a separate NVML lookup for the known PID to learn context memory. Identity no longer requires creating a context merely to observe a set difference. The design can later make context_size lazy or derive it through a less intrusive path.

Mounted host procfs works in trusted bare-metal, HPC, or Apptainer-style deployments where the operator accepts the exposure. It is a poor Kubernetes default. A broadly mounted /proc can reveal other tenants’ command lines, environment variables, file descriptors, and process relationships. Giving the workload hostPID: true exposes more and can affect whom it may signal. The RFC keeps the procfs path explicit and opt-in for that reason.

The benchmark column labelled NStgid measures the benefit of deleting the context probe under a valid guarded setup. It does not show that a default Pod can read its host PID from its ordinary /proc. Mine could not.

the socket asks the kernel who connected

A Unix domain socket carries a byte stream between local processes. Unlike a TCP socket, its address is a filesystem path or an abstract local name, and no network packet needs to leave the machine. On Linux, the listening side can ask getsockopt(SOL_SOCKET, SO_PEERCRED, ...) for the connecting peer’s PID, UID, and GID as represented in the listener’s PID namespace.

Place the listener in HAMi’s device-plugin Pod, which already uses the host PID namespace, and connect from a workload container. The kernel translates the client into the listener’s namespace. The returned PID is the host-visible identity needed by NVML. An independent reproducer reported the same behavior in the issue thread after the original prototype.

This moves identity from an inference to an authenticated kernel observation:

current path
  snapshot NVML -> create observable CUDA context -> snapshot NVML -> diff

broker path
  connect -> kernel reports peer credentials -> return peer PID

My prototype client at 7953d5a does more than connect, read(4), and hope. Its request is eight bytes:

offset  bytes  value
0       4      ASCII "HPID"
4       2      protocol version, big-endian
6       2      command, GET_PID

The twelve-byte response carries the same magic, version, a status, and a 32-bit PID. Fixed sizes make the first protocol easy to bound. Big-endian encoding gives each multi-byte integer one portable wire order.

SOCK_STREAM does not preserve application message boundaries. A successful send(fd, buffer, 8, 0) may send fewer than eight bytes, and one recv may return only the first part of a response. write_full and read_full advance their pointers until the entire frame has moved, retry EINTR, treat zero as a closed connection, and prevent SIGPIPE where the platform permits. My CPU-only protocol probe deliberately fragmented both sides of a socket pair to exercise that property.

Availability failures must be short because fallback exists. The client makes the connect nonblocking, waits with poll for at most 500 ms, reads the pending socket error with SO_ERROR, restores descriptor flags, and also applies send and receive timeouts. A missing socket returned ENOENT in about 9 microseconds in the reported prototype run, after which the old NVML method took over.

Trust is separate from speed. A malicious container able to replace the socket could return another process’s PID and corrupt accounting. Before connecting, validate_trusted_socket uses lstat so symlinks are not silently followed. It verifies that the parent is a directory, the endpoint is a socket, both are owned by UID 0, and the directory is not group- or world-writable. It also refuses arbitrary configured paths: the prototype accepts only the compiled broker socket location.

That last check collides with the current /tmp/vgpulock mode. Main creates /tmp/vgpulock with 0777, while a trusted broker directory must not be world-writable. The RFC uses a protected subdirectory such as /tmp/vgpulock/hostpid/, mounted through the already existing parent path. Deployment code, client validation, and server ownership have to agree on those permissions. A secure client cannot be bolted onto an insecure directory layout afterward.

Why make the device plugin the server instead of adding a mandatory monitor? HAMi-core also runs outside Kubernetes. Maintainers had already required that the library not depend on a monitor being present. The plugin is available in the Kubernetes configuration that needs the fast path; it already participates in Allocate, already has hostPID: true, and can arrange the socket mount. Outside that environment, the client sees no broker and falls back. Old plugin plus new core, or new plugin plus old core, continues through the existing path.

The server still needs a resource model. One thread or goroutine can accept connections and hand each to a bounded worker. Returning four payload bytes is cheap, but an unbounded goroutine per connection lets a workload create memory and file-descriptor pressure in a privileged node daemon. The implementation needs connection deadlines, maximum frame size, concurrency bounds, request counters, and cancellation during shutdown.

The protocol should return only the caller’s own PID. A general “translate this PID” command would expose a namespace oracle. The current request contains no claimed PID at all; the kernel supplies identity. The service learns that a process connected, replies with the credential it received, and closes.

Not every sandbox has this namespace relationship. gVisor implements much of the guest process model in its Sentry, and a socket observation may not map to the actual host process that owns GPU state. The RFC requires a runsc test and a fail-closed detection path. Kata Containers places Pods inside virtual machines; cross-Pod GPU sharing there is a different architecture and remains out of scope until a supported configuration is available. “Works on containerd” is evidence for containerd, not for every runtime called a container.

The tiered decision is consequently narrow:

deployment	first identity mechanism	exposure	fallback
Kubernetes with gated HAMi plugin support	plugin-owned SO_PEERCRED socket	caller’s own host PID	NVML probe
trusted HPC or bare metal, explicit opt-in	NStgid through trusted host procfs	selected host procfs mount	NVML probe
bare Docker, unsupported sandbox, no contract	none	no new host view	hardened NVML probe

The broker comes first, trusted procfs second, and the current detector last. Each fast path is disabled unless deployment sets its environment variable. Failure means “try the next supported mechanism,” never “invent a PID.”

The client is only half of the contract. The broker server does not exist in HAMi main at the source pin, and designing it still means reading the current Go ownership path, because a correct C client with no securely mounted endpoint is dead code.

Go syntax places the function name before its parameter types:

func (plugin *NvidiaDevicePlugin) Allocate(
    ctx context.Context,
    request *AllocateRequest,
) (*AllocateResponse, error)

The receiver plugin *NvidiaDevicePlugin is a pointer to the plugin object, roughly analogous to a C++ member function’s this. The two return values are the response and an error. A nil error means success. The context.Context carries cancellation and deadlines from the gRPC request.

The broker listener has different lifetime from Allocate. One listener belongs to the node plugin process and should start once, not once per workload. Allocate should only decide whether a particular container gets the protected socket directory mounted and whether the client feature flag is set. If every Allocate tried to bind the same path, concurrent container creation would race and plugin restart would leave ambiguous ownership.

The server startup sequence needs to be explicit:

create protected parent directory, owned by root
remove a stale socket only after verifying its type and ownership
bind AF_UNIX listener to the fixed path
chmod the socket according to the client UID contract
start bounded accept/serve loops
publish readiness

The current parent /tmp/vgpulock is mode 0777 because older coordination uses it. The broker should create a child directory, for example /tmp/vgpulock/hostpid, with no group or other write permission. Removing “whatever exists at the socket path” is unsafe in a writable parent: a hostile user could substitute a symlink or unrelated file. Use lstat, reject the wrong owner or type, and keep creation inside the protected child.

Socket access policy has two plausible forms. A mode that lets every local process connect is simple, but then any host process can ask for its own PID, which it generally already knows; the server still must reveal nothing else. A dedicated group narrows access but requires the device plugin response to give the container matching group credentials. Either way, directory execute permission, socket write permission, container user IDs, user namespaces, and SELinux or AppArmor labels must be tested together. Unix mode bits are not the only authorization layer on a Kubernetes node.

For each accepted descriptor, the Go server would obtain peer credentials before trusting request bytes. On Linux this uses unix.GetsockoptUcred or an equivalent SYS_GETSOCKOPT wrapper. The server then reads exactly eight bytes, validates magic, version, and command, encodes a bounded positive PID into the twelve-byte response, writes all bytes, and closes. The C client and Go server need shared golden frames even though they cannot import one common header:

valid v1 GET_PID request
valid v1 success response for PID 8011
bad magic
unsupported version
unsupported command
truncated request
extra bytes according to the chosen framing rule

Network byte order must be implemented explicitly in Go with binary.BigEndian, matching the C shift helpers. Writing a Go struct directly would import Go padding and native endianness into the protocol, the same class of mistake the shared-region ABI is already vulnerable to.

Backpressure deserves a decision before the 300-process test. The listener has an accept backlog maintained by the kernel. Accepted connections consume file descriptors and memory. A fixed worker semaphore can cap simultaneous handlers; the accept loop can reject or briefly queue beyond that cap. The bound should comfortably exceed an expected initialization burst or the broker merely creates a new serialized slope. At the same time, “one goroutine per connection without limit” lets a local workload exhaust a privileged daemon.

Useful server metrics have finite labels:

accepted connections
successful PID replies
bad magic/version/command
peer-credential failures
read/write timeouts
active and rejected handlers
request duration histogram

Never label a metric by PID, Pod UID, socket error string, or arbitrary client data; those create unbounded cardinality. The HAMi-core client needs counters for broker_success, broker_failure, procfs_success, and nvml_fallback, plus the reason category and duration. With both sides, an operator can tell “the server rejected us” from “the mount was absent” without turning on per-call debug logs across 300 processes.

Kubelet restart and plugin restart are distinct. Kubernetes expects a device plugin to recreate its registration socket and register again after kubelet deletes the device-plugin sockets. The host-PID endpoint lives in HAMi’s own protected path and must be rebound on plugin restart. Existing workload containers keep the directory mount; a pathname created after restart becomes visible through that mount. Clients connecting during the gap fall back.

Rollout order follows that behavior:

1. Deploy a plugin capable of serving the endpoint, but do not mount or enable it for workloads.
2. Verify ownership, labels, readiness, restart, and bounded-burst behavior on canary nodes.
3. Deploy a HAMi-core client that understands the broker but leaves it off.
4. Gate both the mount and LIBVGPU_HOSTPID_BROKER for selected Pods.
5. Compare whole-init latency and fallback counters before expanding.

If the feature is rolled back between steps 3 and 4, clients use the old path. If the server dies after step 4, clients use the old path. If an old client sees the new socket, it ignores it. That is compatibility by absence of obligation, not by having both versions interpret a new shared-memory layout.

The server patch must not claim to solve the legacy fallback. Bare Docker still uses the NVML detector; a missing broker during an outage exercises it; gVisor may require it. The sem_postinit owner-death problem and the permanent pidfound = 0 behavior remain release blockers even when the Kubernetes happy path no longer waits there.

removing one semaphore reveals another

Once host-PID discovery no longer needs the before-and-after experiment, sem_postinit can leave the successful fast path. All 300 workers may ask the broker independently. They still converge on the per-container shared-region file and call init_proc_slot_withlock, where sem serializes membership.

The residual p99 near 70 seconds says this remaining path deserves measurement, not speculation. That is why I wrote the shared-region concurrency PR: the first production-source test that needs CUDA headers to compile but no GPU, CUDA library, NVML library, or device node at runtime.

Its parent forks 128 workers by default, with an allowed maximum of 256 in the current patch. A pipe barrier releases them into the real ensure_initialized() together. They remain alive while the parent maps the actual cache file and verifies:

• proc_num equals the worker count;
• every worker PID occupies exactly one slot;
• no duplicate PID exists;
• sequence-point timestamps exist and appear in acquisition order;
• every worker exits, with an outer deadline catching deadlock.

The sequence-point macro expands to nothing in the normal library build. The test target defines a callback that records times before and after selected production lines. Instrumentation therefore observes the same multiprocess_memory_limit.c instead of copying its algorithm into a toy.

On the reported 128-CPU Rostam node, ten 128-worker runs took 92.243 to 107.744 ms wall time. A 256-worker run took 355.600 ms wall, with 348.926 ms p99 total and 335.880 ms p99 semaphore wait. Those numbers are not release assertions; slow CI hardware should still pass a correctness test. An optional environment ceiling lets a performance job fail deliberately, while default CTest checks deadlock and state integrity.

Put those two results side by side and they look like they cannot both be true. The CPU-only shared-region test finishes 256 registrations in hundreds of milliseconds, yet the broker campaign leaves tens of seconds at whole-cuInit p99. They are measuring different boundaries. The test excludes the NVIDIA driver, loader variation, post-driver work, and production machine scheduling; it isolates shared-region code. The A100 benchmark includes the entire fresh process path. Both numbers are needed precisely because neither implies the other.

The test also exposes where failure injection should go. A worker can be killed:

• before waiting on sem;
• after acquiring it but before writing owner_pid;
• after publishing owner_pid but before slot creation;
• midway through compacting slots;
• after slot publication but before unlock;
• while holding sem_postinit in the legacy fallback.

Current sem recovery covers a dead recorded owner after a timeout. There is still a window between semaphore decrement and owner publication. Current sem_postinit has no recorded owner. A real repair can use a process-shared robust pthread mutex, robust being POSIX’s own word here, whose next locker receives EOWNERDEAD and repairs protected state before calling pthread_mutex_consistent. Or it can use a broker/server ownership model that removes shared owner death from the fast path entirely. Either choice changes the shared ABI and requires mixed-version planning.

Timeout is not recovery by itself. It limits how long a waiter blocks; it does not restore a semaphore count, finish a half-written compact array, or prove which invariant the dead owner last established.

the patch stack follows the dependency stack

A single large patch could add a broker, replace PID parsing, alter the shared layout, change fallbacks, and claim a speedup. It would be nearly impossible to review because a failed test would not identify which contract moved. The sequence I proposed in the RFC starts with pieces that change no production behavior at all:

1. Land the GPU-free shared-region concurrency regression test.
2. Land a strict NStgid parser and CUDA-free tests, with no default path.
3. Add the broker client behind LIBVGPU_HOSTPID_BROKER, including telemetry for selected path and fallback.
4. Add the Go broker server and a chart-gated socket mount in the device plugin.
5. Add trusted procfs behind LIBVGPU_HOST_PROCFS.
6. Make context_size discovery lazy, now that identity no longer provides it as a side effect.
7. Repair timeout behavior and define sem_postinit owner death for the remaining NVML fallback.
8. Document the contract, gates, metrics, upgrade order, and rollback.

The first patch is currently PR 244, open at 063db856316b495b0ff5b9de5f1fd31034721e2a. The CUDA benchmark is PR 239, open at 9c233954f9acf65396f228ff491dd55f7a408964. The NStgid and broker branches are prototypes, not merged features. Main at the article pin still calls set_task_pid() behind sem_postinit. That status is part of code comprehension. Reading a prototype and then describing it in the present tense would teach a system users cannot run. Reading only main would hide the measured design work the LFX project is meant to complete.

Patch order also controls benchmark interpretation. PR 244 must pass before and after the performance changes; it proves that registration stayed correct, not that the change is fast. PR 239 should run unchanged against baseline, client-only fallback, broker success, broker missing, trusted procfs, and fault-injected server states. Changing the harness at the same time as the implementation makes before and after less comparable.

A convincing result row needs more than p99:

revision and dirty state
HAMi and HAMi-core versions
GPU model, count, driver and CUDA versions
container runtime and hostPID mode
visible-device mapping
worker count and launch barrier
success/failure count
identity path counts
post-init wait and timeout counts
p50, p95, p99, max, wall span
raw per-worker records

The worker counts should include 1, 2, 8, 32, 128, 240, and 300. Low counts show the fixed floor and whether a fast path harms ordinary startup. High counts expose serialized slope. Repeating each condition and randomizing condition order reduce the chance that GPU warmth, background load, or cache state is mistaken for the patch.

Correctness matrices are larger than performance matrices. At minimum:

• containerd with private and host PID namespaces;
• nested PID namespaces;
• broker present, absent, slow, malformed, wrong owner, wrong permissions, and restart during a burst;
• trusted procfs correct, private procfs mistakenly supplied, missing NStgid, malformed lines, excess nesting, and PID reuse;
• CUDA_VISIBLE_DEVICES=0, 1, 1,0, UUIDs, idle GPUs, and busy GPUs;
• graceful exit, SIGTERM, SIGKILL, fork after initialization, and exec;
• memory-only policy, active SM limiting, multiple devices, and monitor restart;
• gVisor/runsc detection and explicit fallback;
• old/new plugin and HAMi-core combinations.

The security tests should assert the negative result, not merely see an error in a log. A world-writable broker directory must never supply a trusted PID. A symlink endpoint must be rejected. A response with valid magic but PID zero must fail. A partial frame followed by EOF must not reuse bytes from an old response. An unprivileged client must not ask about an arbitrary third party.

The rollout follows the compatibility boundary. Ship the server dark, gated off. Enable its protected socket and metrics on a canary node. Ship clients that still default to the legacy path. Enable the broker variable for selected workloads, compare path and failure counters, then widen. Rollback unsets the variable or chart gate; it does not require rewriting the mapped shared region. Only after the fast path is boring should maintainers consider removing old machinery.

reading a failure without chasing the wrong lock

I chased the wrong lock in this codebase, in order: the flock helper named in the issue, which turned out to have no caller on main; then sem_postinit; then set_task_pid(); and only then the primary-context retain that actually costs the time. Each correction narrowed the patch instead of invalidating the production report. What follows is the recording discipline that would have made that chase much shorter.

First, establish which code is running. Container images may carry a release whose try_lock_unified_lock still uses O_CREAT | O_EXCL, a later version whose helper uses flock, or current main where postInit uses sem_postinit. Search the exact built source and inspect exported symbols:

git rev-parse HEAD
rg -n 'postInit|lock_postinit|try_lock_unified_lock' src
readelf -Ws build/src/libvgpu.so | rg 'cuInit|dlsym|cuGetProcAddress'

Then prove interposition. LD_DEBUG=libs,bindings can show loader search and binding activity, although it is noisy and changes timing. /proc/<pid>/maps shows whether the intended libvgpu.so, libcuda.so.1, shared cache, and runtime libraries are mapped. ldd is useful for declared dependencies but does not prove a later dlopen or actual call path.

Next, separate phases. Add monotonic timestamps at entry and exit of wrapper cuInit, real cuInit, ensure_initialized, slot registration, lock_postinit, each NVML snapshot, primary-context retain and release, set_host_pid, and watcher startup. A monotonic clock measures elapsed time without jumping when wall time is adjusted. Include PID in both namespaces if known, thread ID, cache path, device mapping, wait attempts, and selected identity tier.

Do not infer service time from waiter logs. If a process logs “waiting 30 seconds,” identify which PID holds the resource and what that holder executes. For sem, read owner_pid and test liveness from the namespace that gave that PID meaning. For sem_postinit, current main cannot name the owner, which is itself evidence for the recovery patch.

Check the cache as an ABI, not as printable text. Run abi_check; compare file size with sizeof(shared_region_t); inspect major and minor versions; validate proc_num <= 1024; find duplicate PIDs; and ensure every slot total is consistent with its subcategories. Do this with a tool compiled against the same headers, or with the Go mirror at the matching HAMi revision.

Then reproduce density with a barrier. Launching 300 processes in a shell loop measures shell scheduling and creates a long arrival ramp. bench_init waits until all fresh execs are ready, then broadcasts. Retain raw results so a bimodal distribution, failed tail, or staggered start does not disappear into one percentile.

Finally, run controls that can disprove the favored theory:

• a build that bypasses only host-PID probing but preserves registration;
• the GPU-free registration test;
• one worker and 300 workers;
• a warm primary-context holder;
• broker success and immediate ENOENT fallback;
• core limiting enabled and disabled, while checking whether launch wrappers actually call rate_limiter;
• a killed lock owner.

The controls exist to let the favored theory lose. Mine lost more than once, and the production report survived every correction. The report was never wrong; my model of which code was running was.

when the process leaves

Startup code is incomplete until its ownership has an exit path.

HAMi-core registers exit_handler with atexit when it first creates or maps the region. On an ordinary return from main or a call to exit, the handler checks whether this process owns the shared-region semaphore. If so, it uses a compare-exchange to clear owner_pid and posts sem. It finds its process slot, atomically writes PID and status to zero, and leaves physical compaction for a later process holding the membership lock. That lazy cleanup makes the exit handler shorter and avoids trying to compact a shared array while process teardown is already underway. The next registration removes PID-zero slots and a limited number of dead PIDs. Moving the last slot keeps the live prefix compact.

SIGKILL never runs user-space handlers, which is why lock_shrreg() also checks a timed-out recorded owner, and why no owner-death protocol can depend on atexit alone. A crash can also occur between two writes, so recovery must specify which states are valid and how to repair an incomplete one.

Fork creates a different cleanup hazard. The child inherits mappings and file descriptors but should not execute the parent’s process-specific atexit logic as if it owned the same slot. The at-fork callback resets initialization state; the child remaps and registers itself before accounting its CUDA work. An exec then replaces its address space, and the loader begins the interposition path again.

The broker adds another lifecycle. Its socket path must not be left pointing to a dead listener during an upgrade. A server should bind a new protected endpoint, start accepting, expose readiness, and remove it on controlled shutdown. The client treats connect, framing, timeout, and validation failures as unavailable and falls back. A stale “updating” marker is weaker than the socket’s actual readiness and needs an expiry if used at all.

At the end of a clean process lifetime the ledger should read zero everywhere: no live slot carrying the container PID, no context, module, or data bytes still accounted to it, no semaphore count consumed without a matching post, and no broker connection or worker still owned by a request that ended.

The 300-process problem looked at first like one bad lock. It was an identity experiment: expensive because the experiment creates a CUDA context just to make its own process observable, serialized because a set difference cannot tell two simultaneous arrivals apart, and necessary because two namespaces gave one process two correct names. Three hundred processes were never waiting for a four-byte integer. They were waiting for the system to prove which four bytes belonged to each of them.

Two things stay open as I write this. The fitted 236 to 249 milliseconds per process that remain after the probe is gone have not been attributed to phases yet; the phase-vector campaign is the next thing I want to run. And sem_postinit still has no answer for a dead owner, which means the fallback path everyone keeps until the broker becomes boring still cannot survive a SIGKILL at the wrong instruction.