The Physics Tax — The Coherency Bill Your Hardware Runs Before the Protocol Speaks

20 March 2026 • Yuriy Polyulya •

#distributed-systems #engineering-principles #trade-offs #formal-methods

The Cost of Distribution

The platform team added nodes to the rate limiter’s counter-shard service every time quota enforcement fell behind peak traffic. At 18 nodes the throughput curve had been climbing. At 22 nodes it plateaued. They added 8 more. Throughput dropped 12% within an hour of the rollout completing.

A two-hour perf lab run before any of those rollouts would have predicted the drop: run a CO-free, open-loop load generator at N = 1, 2, 4, 8, 16, 24, 32 in a staging cluster, record the saturation throughput at each point, fit the USL curve, compute N_max. N_max characterization is a lab measurement, not a production discovery. Production cannot run this experiment cleanly — co-tenant noise, live traffic variance, and the impossibility of holding all variables constant while changing N make the USL fit unreliable. The lab can. A staged load test at commissioning produces a receipt the team never had; the receipt tells you the cluster’s N_max before the first production scale-out decision. Production’s role is to monitor whether the actual throughput-vs-N curve deviates from the lab-characterized model — not to be the place where N_max is discovered by accident.

The on-call engineer’s first instinct was to look for the broken node. CPU was healthy across the fleet. Network utilization was nominal. Error rates were flat. The load balancer showed even distribution. No alert fired. Dashboards reported a healthy cluster that was, inexplicably, slower than it had been before the scale-out. The team reverted the last deployment — no change. They blamed the load balancer, filed a bug, and rolled the cluster back to 22 nodes. Throughput recovered. They concluded that 30 nodes was somehow “too many” and moved on.

What no dashboard showed them: the number of node pairs. At 22 nodes there are 231 pairs. At 30 nodes there are 435 — 88% more coordination channels for 36% more nodes. Every state-sharing event in the cluster — counter synchronization, quorum acknowledgment, invalidation broadcast — touches some fraction of those pairs. The cost grows with $N(N-1)$ , not $N$ . The throughput formula’s denominator gained a quadratic term faster than its numerator gained a linear one. At 22 nodes they were sitting just below the peak of the scaling curve. At 30 nodes they were past it. Every node they added past that peak was actively degrading throughput, not contributing to it.

This is not a misconfiguration. It is a provable property of distributed systems that share state. The Universal Scalability Law (USL) makes it precise: throughput as a function of node count has a single maximum — $N_{\max}$ — beyond which each additional node reduces total throughput. The team’s cluster had an $N_{\max}$ of approximately 22. They had no receipt for it because they had never run the two-hour measurement that produces one, and their dashboards were built to detect failure, not to describe the shape of the scaling curve.

The physics taxes in this post do not care about your cloud provider, your language runtime, or your architecture diagram. Coherency overhead grows quadratically in node count regardless of how the nodes were provisioned. Fan-out amplification compounds geometrically at every level of call depth regardless of whether each service was “optimized.” These are not engineering failures — they are the physics of distribution. The failure is measuring averages and believing them. Every number below is extractable from a standard load test — and until you have measured them, you do not know whether your last scaling operation moved you toward the frontier or past it. The following table organizes all four physics taxes by what you pay, the design consequence each imposes, and how to measure each in a standard load test:

Tax	What You Pay	Design Consequence	How to Measure
Storage Tax ( WAL fsync)	Single-node baseline $\gamma$ capped at $1/t_{\text{fsync}}$ per writer thread — 500 ops/sec at 2ms fsync P99	Durable consensus (Raft, Paxos, Zab) serializes every committed entry through storage before $\alpha$ or $\beta$ contribute anything; group commit is the primary Pareto movement to recover $\gamma$	Run a single-threaded synchronous write benchmark: one write at a time, each flushed to durable storage before the next begins. Divide 1s by P99 flush latency to get the raw $\gamma$ ceiling
Contention Tax ($\alpha$ )	Throughput ceiling at $1/\alpha$ regardless of node count	Serial fraction $\alpha > 0$ whenever operations share a coordinator; Amdahl’s ceiling is hard	Profile leader CPU and lock contention under load; fit USL at $N$ = 1, 2, 4
Coherency Tax ($\kappa$ )	Throughput peaks at $N_{\max}$ nodes, then declines	$\kappa > 0$ is inevitable when nodes share state; the question is how large	Fit USL to throughput vs. $N$ load test; extract $\kappa$ from the curvature
Geometric Tax (fan-out)	P99 of a composite request grows geometrically with fan-out depth	Adding fan-out depth for coverage multiplies the tail latency floor	Run a CO-free, open-loop load generator with high-resolution latency histogram output at fan-out $N$ = 1, 10, 100; plot P99 vs. $N$

Both taxes contract the Pareto frontier inward, excluding operating points that a naive analysis would consider reachable. For the formal treatment of the impossibility results that define the frontier’s shape, see The Impossibility Tax.

The taxes apply in sequence, not in parallel. The Storage Tax constrains the single-node baseline $\gamma$ before multi-node coordination costs enter the picture at all. A consensus leader serializing writes to cloud-attached block storage hits a throughput ceiling of 500–1,000 ops/sec per writer thread at 1–2ms fsync P99 — before a single packet crosses the network and before $\alpha$ or $\kappa$ contribute anything. Measuring only coordination costs on a system that is already storage-bound produces a confident-but-wrong USL fit. The right diagnostic order: measure $\gamma$ first via a single-threaded synchronous write benchmark, then fit $\alpha$ and $\kappa$ against that ceiling. Group commit is the primary intervention to recover $\gamma$ ; reducing $\kappa$ on a storage-bound system returns nothing.

---

The Coordination Tax — Universal Scalability Law

The coordination tax has a precise formula: the Universal Scalability Law ( USL ) [1] establishes that throughput peaks at a specific node count $N_{\max}$ and declines beyond it — adding nodes past that point actively reduces performance. This section develops the mathematics and shows how to measure the two coefficients that determine where the peak falls.

The root error in most capacity planning is the linear scaling assumption: adding N nodes delivers N times the single-node throughput. This assumption is mathematically refuted by the USL coherency term. Every system that shares state pays a per-node-pair coordination cost that grows quadratically; past $N_{\max}$ , throughput actively declines. The magnitude of $\kappa$ determines how much horizontal scaling headroom the system has before the investment turns negative.

Three numbers make this precise. $\alpha$ (contention) is the serial fraction — think of it as friction: it slows progress from the first step because some fraction of every workload cannot parallelize regardless of how many nodes you add. $\kappa$ (coherency) is the per-node-pair synchronization overhead — think of it as gravity: weak at small node counts, growing quadratically as the cluster expands, pulling throughput back down past $N_{\max}$ . $N_{\max}$ is the summit: the node count where friction and gravity together peak throughput, and past which every additional node actively degrades it — under the measurement conditions used when $\kappa$ was fitted. The summit’s elevation depends on which $\kappa$ you use; bare hardware, instrumented production, and worst-case jitter yield values spanning a 30% range (The Reality Tax quantifies the three-tier model). All three are extractable from a standard load test; the definitions below give them formal precision so they can be measured and fitted. To see how $\kappa$ manifests at observable scale before the formal treatment, consider a concrete example.

Three nodes, ten nodes — the connection pool version. A connection pool manager coordinates which worker holds which database connection. At three workers, there are three pairs: worker 1 and 2, worker 1 and 3, worker 2 and 3. When worker 1 checks out a connection, it broadcasts state to the other two. Three messages, trivially fast. Now scale to ten workers. There are 45 pairs. Every checkout coordinates across 45 potential contention points — each worker must reconcile its view of available connections against nine others. The per-message cost is unchanged. But every checkout now sends nine times as many coordination messages as at three workers, while adding only 3.3x as many workers. At some node count in this range, the coordination overhead overtakes the throughput benefit. Adding more workers past that point makes the pool slower, not faster — not because connections ran out, but because knowing what everyone else is holding costs more than the marginal connection is worth. That is $\kappa$ : the per-node-pair synchronization overhead. The numbers make the asymmetry concrete: at 3 nodes, $\kappa \cdot 3 \cdot 2 = 6\kappa$ total overhead. At 10 nodes, $\kappa \cdot 10 \cdot 9 = 90\kappa$ — fifteen times more overhead for 3.3 times more workers.

Directional estimate before load tests exist. Formal USL fitting requires purpose-built load generation infrastructure. Teams without it can extract a directional $\kappa$ estimate from production APM data as a bootstrap baseline — explicitly caveated and sufficient to determine whether a full measurement campaign is warranted.

Pull P99 write latency for the past 30 days and find the date of the last significant traffic spike. Three diagnostic questions reconstruct the frontier signal from data you already have: Did P99 stay roughly flat as throughput climbed? The system was interior during that event — free headroom existed. Did P99 spike sharply while throughput plateaued or fell? The frontier was being approached. During the spike, did adding replicas immediately reduce per-instance latency? If yes, $\kappa$ is low and the cluster was interior on the throughput axis. If latency held or rose despite new replicas, the cluster may have been past $N_{\max}$ .

This is not a USL fit — it is a memory of a natural experiment that production already ran. Record it before the team rotates and the memory is lost. A partial measurement, explicitly caveated, beats a precise assumption, silently trusted. The full measurement recipe follows; return to it once you have the tooling.

With that intuition established, the formal definitions give it measurable precision.

Definition 10 -- Contention Tax alpha: the serial fraction that caps throughput gain regardless of node count, with two distinct physical sources

Axiom: Definition 10: Contention Tax $\alpha$

Formal Constraint: The contention coefficient $\alpha \in [0,1]$ is the fraction of operations that must serialize through a shared resource — a lock, a leader node, a write coordinator. When $\alpha = 0$ , operations are perfectly parallelizable; when $\alpha = 1$ , all operations serialize and adding nodes provides no benefit. The USL treats $\alpha$ as uniform across all $N$ nodes (equipartition). Two physically distinct mechanisms contribute: logical contention (lock acquisition, write coordinator serialization — software-restructurable) and physical contention (WAL fsync serialization, cache-line ping-pong, NIC interrupt coalescing — hardware-bound). The USL formula does not distinguish them; the denominator grows identically regardless of cause.

Engineering Translation: A consensus leader on cloud-attached general-purpose block storage (~1–2ms fsync P99 on a provisioned network-attached volume) hits ~500–1,000 ops/sec per writer thread at $N = 1$ before $\alpha$ or $\kappa$ are relevant — WAL fsync is the first serialization barrier. Logical contention yields to protocol redesign; physical contention yields only to hardware topology changes or load reduction. Measure $\alpha$ by profiling lock contention and leader queue depth; profile CPU cache miss rates and NIC interrupt affinity separately to distinguish the two sources.

Definition 11 -- Physical Coherency Floor: the hardware-determined minimum coordination cost paid per node-pair for shared state, below which no protocol optimization can reach

Axiom: Definition 11: Physical Coherency Floor $\kappa_{\text{phys}}$

Formal Constraint: The physical coherency coefficient $\kappa_{\text{phys}} \geq 0$ is the per-node-pair overhead of hardware-level state synchronization — cache invalidation, NIC contention, memory bus coordination. A load test measures the total observed coherency — hardware floor plus a protocol-dependent component $\beta$ :

$$\kappa_{\text{total}} = \kappa_{\text{phys}} + \beta$$

$\kappa$ ’s contribution grows quadratically: each of the $N(N-1)/2$ node pairs contributes $\kappa$ to total overhead. Sharding into $K$ domains reduces total coherency cost by $K$ : a structural reset of $N$ in the quadratic term. The effective coefficient is $\kappa_{\text{eff}} = (1 - f)\,\kappa/K + f\,\kappa$ where $f$ is the fraction of cross-shard operations.

Engineering Translation: No protocol choice can reduce coherency below $\kappa_{\text{phys}}$ . Sharding helps exactly as much as you can keep operations shard-local: a distributed transaction spanning all $K$ shards restores the full $\kappa N^2$ term for that operation. Measure $\kappa_{\text{total}}$ from a load test per coherency domain, not fleet-wide — run the measurement at the telemetry configuration that will be active in production; observability pipelines (trace exporters, metrics scrapers, structured logging) are themselves contention sources and shift the measured $\kappa$ upward. The value your load test produces is the operationally relevant number for that telemetry footprint.

Topology note on $\beta$ decomposition: The additive form $\kappa_{\text{total}} = \kappa_{\text{phys}} + \beta$ models $\beta$ as a coherency-term penalty. This mapping is exact for leaderless protocols (EPaxos, Dynamo) whose all-to-all cross-talk scales as $O(N^2)$ . For leader-based protocols (Raft, Multi-Paxos), the leader’s serialization queue acts as an Amdahl bottleneck: write overhead manifests primarily as an inflation of $\alpha$ (contention), with $\kappa$ picking up the follower fan-out component. Measuring $\kappa_{\text{total}}$ from a USL fit automatically captures both contributions in the fitted coefficients; the decomposition matters when choosing between leader-based and leaderless topologies.

NUMA topology — two-level coherency. Definition 11 treats $\kappa_{\text{phys}}$ as uniform across all node pairs in the coherency domain. This holds when all processes share a CPU socket. Multi-socket servers and tiered datacenter topologies create a two-level coherency hierarchy: $\kappa_{\text{intra}}$ for intra-socket communication (shared L3 cache, sub-100ns invalidation latency) and $\kappa_{\text{inter}}$ for cross-socket communication (QPI/UPI/Infinity Fabric, typically $3\text{--}5\times$ $\kappa_{\text{intra}}$ ). A process spanning CPU sockets experiences an effective $\kappa_{\text{eff}}$ set by the fraction of accesses crossing the socket boundary — not the intra-socket floor. Named failure mode: NUMA-optimistic $N_{\max}$ — a team runs USL fits with all benchmark threads pinned to one NUMA node, measures $\kappa_{\text{intra}}$ , and derives an $N_{\max}$ that is $3\text{--}5\times$ above what production workloads, which span sockets, will reach. Fix: run USL fits under production thread placement, explicitly spread across sockets. The resulting $\kappa_{\text{eff}}$ is the number that belongs on the birth certificate — not the single-socket benchmark value.

The node-pair count is what makes $\kappa$ dangerous: 45 pairs at 10 nodes, 4,950 at 100 nodes, 499,500 at 1,000. The $N$ that matters for this count is the coherency domain size, not the total cluster size — which is precisely why sharding is a structural intervention, not a tuning knob. $\kappa$ is not a theoretical parameter; measure it per coherency domain from a load test.

Definition 12 -- Scalability Bound N_max: the node count at which throughput peaks and adding nodes begins to degrade performance

Axiom: Definition 12: Scalability Bound $N_{\max}$

Formal Constraint: For a system with USL parameters $\alpha$ and $\kappa$ , the node count at which throughput peaks is:

$$N_{\max} = \sqrt{\frac{1 - \alpha}{\kappa}}$$

For $N > N_{\max}$ , adding nodes decreases throughput — the quadratic denominator grows faster than the linear numerator. The formula assumes $\alpha$ and $\kappa$ uniform across the coherency domain. The derivation of this formula — showing that $N_{\max}$ is the unique maximum of the USL throughput curve — is established in Proposition 7 immediately following.

Engineering Translation: A hot shard with $\alpha_{\text{hot}} \gg \alpha_{\text{avg}}$ has $N_{\max}^{\text{hot}} \ll N_{\max}^{\text{cluster}}$ — it enters the retrograde regime while aggregate metrics still show headroom. Cluster-wide USL fits average across shards and return a healthy-looking $\alpha_{\text{avg}}$ , masking a per-shard frontier that has already crossed into the retrograde region. The operationally relevant fit is per-shard under production key distribution, not on fleet-wide aggregates.

For $\kappa = 0.001$ and $\alpha = 0$ : $N_{\max} \approx 32$ . For $\kappa = 0.0001$ : $N_{\max} \approx 100$ . For $\kappa = 0.00001$ : $N_{\max} \approx 316$ . Most production Raft clusters with strong consistency and cross-shard transactions operate with $\kappa \approx 0.002$ –$0.005$ (author’s estimate from USL fitting on production load tests; no published empirical benchmark exists for this range), placing $N_{\max}$ between 14 and 22 nodes. Teams scaling past 30 nodes observe throughput plateau or decline and attribute it to noisy neighbors or hardware issues. The actual cause is coherency overhead reaching the quadratic regime.

Proposition 7 -- USL Throughput Curve: throughput rises to a peak at N_max then declines as coherency overhead outpaces the parallelism gain from additional nodes

Axiom: Proposition 7: USL Throughput — Gunther 2007

Formal Constraint: For a system with $N$ nodes, serial fraction $\alpha$ , coherency coefficient $\kappa$ , and single-node throughput $\gamma$ :

$$X(N) = \frac{\gamma N}{1 + \alpha(N - 1) + \kappa N(N - 1)}$$

For $\kappa > 0$ , $X(N)$ peaks at $N_{\max}$ and strictly declines for $N > N_{\max}$ . When $\kappa = 0$ , reduces to Amdahl’s Law — throughput saturates at $\gamma / \alpha$ but never declines. [1]

Engineering Translation: Stateless services (web tiers, API gateways, sharded databases with no cross-shard transactions) operate with $\kappa \approx 0$ — they plateau, they do not collapse. The “summit then decline” shape applies only to components that share state across nodes. Measure $\kappa$ first: if it is effectively zero, Amdahl’s serial fraction $\alpha$ is your actual constraint, not USL retrograde.

Proof sketch -- USL throughput ceiling (Gunther 2007): why the quadratic coherency term overtakes the linear parallelism term past N_max, making throughput decline

Axiom: USL Throughput Ceiling — Gunther 2007

Formal Constraint: Differentiate $X(N)$ with respect to $N$ treating it as continuous. Setting $dX/dN = 0$ yields $N_{\max} = \sqrt{(1-\alpha)/\kappa}$ . The second derivative is negative, confirming a maximum. For $N > N_{\max}$ , the quadratic denominator grows faster than the linear numerator — throughput declines monotonically. When $\kappa = 0$ , reduces to Amdahl’s Law, which saturates but never declines; the $\kappa$ term turns saturation into regression. [1]

Engineering Translation: The peak is not a wall — it is a summit you begin descending the moment you pass it. With $\kappa = 0.001$ and $\alpha = 0.02$ ($N_{\max} \approx 31$ ), a cluster at $2\times N_{\max}$ delivers 84% of peak throughput; the hardware investment doubled, the extracted throughput shrank. Measure $\kappa$ from a load test before provisioning past your estimated $N_{\max}$ .

The storage floor — $\gamma$ before $\alpha$ and $\beta$ . For durable consensus protocols (Raft, Multi-Paxos, Zab), the single-node baseline $\gamma$ is bounded by a constraint that precedes the USL denominator entirely: WAL fsync latency. Every committed entry must be durably appended to persistent storage before the leader acknowledges it; at 2ms fsync P99 on general-purpose cloud block storage, a strictly unbatched, single-threaded serial WAL caps $\gamma$ at 500 ops/sec per writer thread at $N = 1$ , before $\alpha$ or $\kappa$ contribute anything.

The industry-standard Pareto movement to recover $\gamma$ is group commit: batch multiple WAL entries into a single fsync via a deliberate wait window (typically 1–10ms). Twenty entries batched behind a 2ms fsync pay 2ms total instead of 40ms — restoring $\gamma$ toward 2,000–10,000 ops/sec at the cost of a raised P50 write latency equal to the wait window. Group commit is movement along the single-node Pareto frontier: latency floor rises, throughput ceiling expands — not a free improvement but a deliberate exchange.

The blast radius extends beyond the single node. A wait window of $w$ ms raises P50 write latency by $w$ ms at the storage layer. That shift propagates up the call graph in two structurally distinct ways. In a serial call chain of depth $D$ — service A calls B, B calls C, each synchronously waiting on the one below — every hop has the storage wait in its critical path, so the composite P50 floor accumulates $D \times w$ across the full chain. In a parallel scatter-gather fan-out — $N$ downstream services dispatched concurrently from a single coordinator — all $N$ callers see the storage wait simultaneously; the composite bottlenecks on the slowest concurrent caller, so the P50 floor rises by $w$ ms once, not $N \times w$ . The geometric fan-out tax (Proposition 8) then operates on this already-raised baseline: Dean & Barroso’s max-order-statistic amplification geometrically shifts the effective tail percentile that each individual server must satisfy to meet the composite SLA — amplification of tail probability on the raised distribution, not additive stacking of $w$ across the parallel branches.

A 5ms group-commit window adds 5ms to the composite P50 of any request with storage acknowledgment in its serial critical path — once per serial level in the call graph, not multiplied across concurrent branches at a given fan-out level. Teams that measure group commit’s benefit in isolation — single-node throughput recovery — without re-measuring the composite P50 misattribute the trade-off. They see the throughput gain and miss the latency shift that propagates up every serial level of the service graph above it.

A second blast radius is tenant isolation. When group commit batches 10ms of writes into a single fsync, it produces I/O spikes on the underlying block device that can induce latency jitter in other services sharing the same cloud storage network — neighbor-induced jitter that does not appear in the service’s own P50 but manifests as intermittent P99 degradation in co-resident workloads. On cloud block storage with shared I/O credit pools, a group-commit-induced I/O burst from one tenant can exhaust the shared credit, introducing latency noise into neighbors that have no group commit of their own and no instrumentation pointing at the source. If your system uses group commit, the $\gamma$ in your USL fit reflects that protocol choice; the raw storage ceiling is $1/t_{\text{fsync}}$ per writer thread — the strictly unbatched, single-thread serial ceiling — measurable independently via a single-threaded synchronous write benchmark: one write at a time, each flushed to durable storage before the next begins. In high-throughput architectures with multiple concurrent writer threads, a second physical constraint precedes CPU contention entirely: the block device IOPS ceiling. General-purpose provisioned network storage delivers a fixed baseline IOPS cap regardless of per-fsync latency; once the aggregate write rate across all writer threads reaches that ceiling, additional threads queue at the storage layer rather than the CPU, and $\alpha$ never fully engages. Measure this ceiling with a multi-threaded synchronous write benchmark at your provisioned IOPS tier before interpreting USL fits. The binding floor on $\gamma$ is therefore $\min(1/t_{\text{fsync}},\; \text{IOPS}_{\text{device}} / N_{\text{writers}})$ — whichever limit the architecture reaches first.

Physical translation. Your 300-node cluster is slower than your 200-node cluster. Not because you added bad nodes — because the coherency tax $\kappa$ compounds quadratically. At $N = 200$ with $\kappa = 0.0001$ and $\alpha = 0.02$ ($N_{\max} \approx 99$ ), throughput is 88% of peak; at $N = 300$ , it is 75%. For $\kappa = 0.001$ ($N_{\max} \approx 32$ ), $N = 200$ is already at 37% — the degradation is catastrophic well before 200 nodes.

Mental Model: The USL as a Topographical Map. Picture throughput as elevation on a mountain whose shape is set by $\alpha$ and $\kappa$ . Scaling horizontally means walking up this mountain. $\alpha$ is friction — it slows ascent from the first step because some fraction of work is always serial. $\kappa$ is gravity — it grows stronger the higher you climb, and past $N_{\max}$ it actively pulls throughput back down. The mountain peak is $N_{\max}$ ; it is not a wall you cannot cross but a summit you descend from the moment you pass it. An interior operating point means you are still climbing — adding nodes still gains elevation. A past-peak operating point means every additional node descends. The practical question “how much horizontal scaling headroom remains?” translates directly: how far is the current node count from the summit? Two clusters with identical hardware but different coordination protocols climb different mountains — same friction, different gravity, different summit heights.

The gravitational pull has measurable force: with $\kappa = 0.001$ and $\alpha = 0.02$ ($N_{\max} \approx 31$ ), a cluster at twice the summit node count delivers 84% of peak throughput — the hardware investment doubled, the throughput dividend shrank by 16%. At three times the summit, throughput falls to 66% of peak: the hardware bill tripled, the extracted throughput declined by approximately one-third. This is not linear decay — it is gravitational acceleration, steepening with each additional node past $N_{\max}$ . The Pareto frontier on the throughput axis is not a flat ceiling but a slope, and $\kappa$ is the incline angle.

Sharding restructures the problem: each shard is a separate, smaller mountain with its own $N_{\max}$ . The cluster does not escape the USL — it resets the $N$ in the quadratic term, climbing a shorter mountain that is easier to summit before the descent begins.

The Topographical Map — Case Study: Empirical $\kappa$ Derivation for Counter-Shard Scaling. The rate limiter’s counter-shard service is the mountain. The team plans to scale to 30 nodes — “more nodes means more quota throughput.” The load test (CO-free, open-loop, production key distribution) maps the terrain:

$N$	$X(N)$ ops/sec	Efficiency $X(N)/(N \cdot \gamma)$
1	1,000	1.00
2	1,887	0.94
4	3,226	0.81
8	4,348	0.54

Full USL regression yields $\gamma = 1{,}000$ ops/sec, $\alpha = 0.04$ , $\kappa = 0.010$ . Summit: $N_{\max} = \sqrt{0.96/0.010} \approx 10$ nodes; peak throughput $X(10) \approx 4{,}425$ ops/sec.

The team’s 30-node target — computed from the fitted USL parameters (gamma=1,000, alpha=0.04, kappa=0.010):

$$X(30) = \frac{1{,}000 \times 30}{1 + 0.04 \times 29 + 0.010 \times 30 \times 29} = \frac{30{,}000}{1 + 1.16 + 8.70} = \frac{30{,}000}{10.86} \approx 2{,}762 \text{ ops/sec}$$

Three times the nodes delivers 62% of peak — 2,762 versus 4,425 ops/sec. The team is 20 nodes past the summit, descending the far slope while paying for the ascent. $\kappa = 0.010$ is high for a counter service because every quota increment routes through a single-region Raft quorum, serializing on the quorum leader. Sharding to $K = 4$ independent Raft groups eliminates cross-shard coordination for 95% of increments; with 5% cross-shard traffic, $\kappa_{\text{eff}} = 0.95 \times (0.010/4) + 0.05 \times 0.010 \approx 0.0029$ , raising the summit to $N_{\max} \approx 18$ — nearly doubling the throughput ceiling without adding hardware.

Watch out for: The measured coherency term in the USL fit — the total $\kappa_{\text{total}} = \kappa_{\text{phys}} + \beta$ from Definition 11 — is not fixed. $\kappa_{\text{phys}}$ is the hardware floor: cache invalidation, NIC contention, memory bus, irreducible regardless of protocol. The protocol-overhead component $\beta$ does vary with protocol choice: synchronous Raft replication at every write sets $\beta$ directly from consensus round-trip time. Conflict-free merge eliminates per-write consensus (lowers $\beta$ significantly on writes) but relocates the coordination cost to two hidden paths: the background GC path (tombstone compaction, requiring at-least-once delivery and scheduled merges) and the read path (merge at read time, scaling with state divergence and tombstone count). Named failure mode: “protocol-overhead budget invisibility” — systems that migrate from Raft to conflict-free merge report a $\beta$ drop on the write-path load test but observe read latency inflation and GC pressure emerge as a new frontier constraint. The benchmark measured the wrong axis. The frontier in the coordination-throughput space shifts depending on which protocol drives $\beta$ , but $\kappa_{\text{phys}}$ — the hardware floor — does not disappear regardless of protocol choice.

Boundary condition — Equipartition Assumption. The USL formula treats all $N$ nodes as interchangeable load-carrying units sharing work uniformly. Real deployments violate this whenever a hot shard concentrates disproportionate traffic on a subset of nodes. Under skewed load, the serial fraction $\alpha$ measured from a cluster-wide load test reflects the average; the bottleneck node operates at a local $\alpha_{\text{hot}} \gg \alpha_{\text{avg}}$ . The Amdahl ceiling for the hot shard — $1/\alpha_{\text{hot}}$ — is the actual throughput limit, not the cluster-wide $1/\alpha_{\text{avg}}$ . Similarly, the coherency cost for the hot shard is not $\kappa \cdot N(N-1)$ applied uniformly but $\kappa \cdot N_{\text{hot}}(N_{\text{hot}}-1)$ where $N_{\text{hot}}$ is the number of nodes that must synchronize with the bottleneck. If one shard receives 80% of writes in a 100-node cluster, the effective $N$ for the coherency penalty is not 100 but the replication factor of that shard — typically 3 to 5. The quadratic blowup is avoided only because the hot shard’s coherency domain is small; the throughput ceiling is hit because its $\alpha$ is high. Named failure mode: “hot shard equipartition collapse” — a load test at uniform traffic measures a healthy $\alpha$ and $\kappa$ ; production skew moves the binding constraint to $\alpha_{\text{hot}}$ , which never appeared in the fit. The symptom is a P99 tail that diverges at production key distributions while load tests show headroom.

The USL curve shows what this looks like. Three values of $\kappa$ produce three fundamentally different scaling profiles — the same architecture, the same hardware, but different coordination costs. The peak of each curve is $N_{\max}$ : the node count beyond which horizontal scaling is a net loss.

The dashed gray line is linear scaling — the theoretical ideal where doubling nodes doubles throughput. The green curve ($\kappa = 0.0001$ ) tracks close to linear up to about 100 nodes before peaking. The blue curve ($\kappa = 0.001$ ) diverges earlier, peaking around 32 nodes. The red curve ($\kappa = 0.01$ ) peaks at roughly 10 nodes and then falls steeply — at 50 nodes, throughput is lower than at 5. Each dot marks $N_{\max}$ . The contention parameter $\alpha = 0.02$ is held constant; only $\kappa$ varies. The same hardware, the same workload, three fundamentally different ceilings — set entirely by how much state nodes must share.

The third physics tax: fan-out tail amplification. The USL governs throughput — how many requests the cluster can sustain as nodes are added. A second, independent physical constraint governs latency: when a composite request fans out to $N$ backend nodes and waits for all of them, the composite response time is the maximum of $N$ independent latency samples, not their average. If each node delivers latency below threshold $T$ with probability $p$ — equivalently, if $T$ is the single-node $p$ -th percentile — the probability that all $N$ nodes respond within $T$ is:

$$P(\max(X_1, \ldots, X_N) \leq T) = p^N$$

Equivalently, the probability that at least one node exceeds $T$ — that the composite request misses its latency target — is $1 - p^N$ . Both quantities are exact under the independence assumption; both are pessimistic lower bounds when infrastructure correlation is present (shared switches, shared hypervisors — all nodes slow simultaneously). The miss probability grows monotonically with fan-out and is insensitive to per-node optimizations once $N$ is large.

The practical consequence is brutal. Take a JVM service with P99 = 10ms — a server that is fast, well-tuned, meeting its tail-latency SLA 99% of the time. Route a scatter-gather request through $N = 50$ such servers. The probability that every one of the 50 servers answers within 10ms is $0.99^{50} \approx 0.605$ — the P99 threshold of a single node becomes the median of the composite request: slightly worse than a coin flip that the 50-server scatter-gather finishes within what looked like a safe tail budget. Stated directly: adding fan-out depth converts what was a 1-in-100 slow event into a 1-in-2 slow event, without changing a single line of server code. To deliver a composite P99 guarantee at fan-out $N$ , each individual server must instead hit $0.99^{1/N}$ — the target percentile is the $N$ -th root of the composite SLA. At $N = 50$ that is P99.98; at $N = 100$ it is P99.99. For a JVM service whose P99.99 may be $50\text{--}200\times$ its P99 due to stop-the-world GC pauses, the composite SLA is structurally unachievable regardless of how many nodes are added.

The USL and the fan-out law are independent taxes on the same system, paid simultaneously. The USL bounds throughput as a function of node count; the fan-out law bounds latency as a function of scatter-gather depth. A microservice architecture that adds fan-out depth for coverage breadth is tightening both constraints at once — shrinking $N_{\max}$ by growing the coherency domain, and raising the required per-server percentile by growing the scatter-gather width. Both mechanisms are irreducible hardware realities.

Measurement Recipe — Fitting $\kappa$ from a Load Test

Positioning a system on the Pareto frontier requires measured values for $\kappa$ and $\alpha$ — the numbers that bound $N_{\max}$ and determine whether the operating point falls in the viable or retrograde regime. Six steps extract those values from a load test. Before starting, two tool-selection caveats determine the outcome: ignore them and the fit will be wrong in opposite directions.

Before you begin — two separate runs, two separate tools. The recipe requires two fundamentally different measurements that cannot be collapsed into a single load test. The Capacity Run pushes the system to saturation to extract $X(N)$ for the USL fit — it needs throughput numbers, not latency numbers, and it must reach saturation deliberately. The Latency Run measures coordinated-omission-free tail latency at a fixed sub-saturation load — it needs accurate P99 numbers, which means open-loop request scheduling so the tool never pauses under slow responses. The two runs use different tools precisely because they serve opposite goals.

The first tool-selection constraint governs the Latency Run; the second governs the Capacity Run. Both are disqualifying if ignored.

The open-loop saturation trap. An open-loop generator issues requests on a fixed arrival-rate clock regardless of whether prior responses have arrived. This is essential for accurate latency measurement. But if you push that arrival rate past the system’s actual capacity, the in-flight queue grows without bound — latency diverges, connections drop, and throughput numbers become garbage. Open-loop tools do not saturate gracefully. Never use an open-loop generator to find the saturation throughput. Use it only at a known sub-saturation rate after the saturation point has been established by a Capacity Run.

The inverse error applies when collecting throughput data for the USL fit — a closed-loop tool is required there precisely because open-loop tools fail under saturation.

Coordinated Omission in the Capacity Run. A closed-loop load generator pauses issuing new requests whenever the previous request is slow. That means it stops generating load precisely during the overloads you need to measure — latency is under-reported and the throughput plateau appears higher and later than it really is. Fitting $\kappa$ from such a measurement gives an optimistic curve: $N_{\max}$ is too high, the saturation point looks further away, and the architecture will underperform the model in production. This systematic error is Coordinated Omission. For the Capacity Run it does not matter — you are collecting throughput at saturation, not latency. For the Latency Run it is fatal: use a CO-free, open-loop load generator at a fixed rate below saturation, with high-resolution latency histograms.

1. Run your service at $N = 1$ node (single-node baseline). Capacity Run: use a closed-loop load generator (fixed concurrency ceiling, not fixed arrival rate) and ramp request rate until throughput plateaus. Record $X(1) = \gamma$ — this is your per-node saturation capacity. Critical: run with production fsync durability settings enabled. Disabling fsync generates a fictitious $\gamma$ that invalidates every downstream USL calculation.
2. Capacity Run — find $X(N)$ for each N point. For each of $N$ = 2, 4, 8, 16, 32, 64 nodes: use a closed-loop tool and ramp offered load until throughput stops increasing — that plateau is $X(N)$ , the saturation throughput. Record only the saturation value. Run at least 5 minutes at saturation to suppress queueing transients. Discard all latency numbers from this run — they are meaningless under queue-saturation conditions. Common mistake: fixing offered load at a constant RPS across all $N$ values reveals sub-saturation behavior, not the USL curve — the resulting fit will underestimate $\kappa$ and overestimate $N_{\max}$ .
3. Plot $X(N)/N$ (efficiency) vs. $N$ . A perfectly parallelizable system has flat efficiency at every $N$ . Steeply falling efficiency before $N = 8$ indicates high $\alpha$ (serialization bottleneck). Concave-downward bending past $N = 16$ indicates $\kappa > 0$ . Efficiency below 50% before $N = 32$ means $\kappa$ is in the 0.001–0.01 range and you are near or past $N_{\max}$ .
4. Fit the USL formula using two-point closed-form estimates. From the $N = 2$ measurement (with $\kappa \approx 0$ approximation): $\alpha = 2 \cdot X(1)/X(2) - 1$ . From the $N = 4$ measurement: $\kappa = (4 \cdot X(1)/X(4) - 1 - 3\alpha) / 12$ . Derivation: $X(4)/X(1) = 4/(1 + 3\alpha + 12\kappa)$ , rearranging gives $12\kappa = 4 \cdot X(1)/X(4) - 1 - 3\alpha$ .
5. Compute $N_{\max} = \sqrt{(1 - \alpha)/\kappa}$ . This is where your architecture peaks. Operating past it is a net loss.
6. Sanity check: Plot $X(N)$ for your measured $\alpha$ and $\kappa$ against all six data points. If they diverge by more than 15%, something is wrong. Signals: $\alpha < 0$ means load imbalance or measurement error (check for coordinated omission in the load test); $\kappa < 0$ means super-linear scaling, likely a cold-cache artifact in early data points; $\alpha > 0.5$ means a single serialization bottleneck is dominating — identify it before scaling further.
7. Latency Run — measure coordinated-omission-free P99 at your operating point. Now that $N_{\max}$ is known, switch to a CO-free, open-loop load generator with high-resolution latency histograms. Fix the offered load at approximately 80% of $X(N)$ for your target node count — well below saturation, so the arrival-rate clock never overruns the queue. Run for at least 10 minutes to populate the histogram tail; the P99 from this run is the latency the architecture actually delivers at a sustainable operating point. Never report P99 from the Capacity Run — a saturating queue inflates tail latency by orders of magnitude and the number is not reproducible.

Watch out for: you are probably operating past $N_{\max}$ and do not know it. The diagnostic is a single plot: throughput per node (efficiency) vs. node count. A monotonically decreasing curve past $N \approx 20$ is coherency penalty, not hardware. The fix is to reduce $\kappa$ — weaken consistency, shard state independently, or move to leaderless coordination — rather than add more nodes. Adding nodes past $N_{\max}$ is spending money to make your system slower.

Named failure mode: synthetic traffic mismatch. Step 1 instructs running the load generator at “production-representative load.” Accurately reproducing production key distribution, cache-hit ratios, and burst patterns in a synthetic harness is an engineering project in its own right — one that takes weeks to months for teams doing it for the first time. A load test run with uniform-random keys produces a uniformly distributed access pattern; production may have 1% of keys receiving 40% of requests. Under hot-key access, the hot counter shards face higher lock contention, worse cache behaviour, and more inter-node replication traffic per logical key — raising the effective $\kappa$ on the hot path well above the cluster-wide average. A fit from uniform-synthetic traffic underestimates $\kappa$ and overestimates $N_{\max}$ for exactly the traffic distribution that will saturate the system first.

Fix: record the traffic distribution assumption explicitly on the birth certificate alongside $\kappa$ . Three options in ascending accuracy order: (a) uniform synthetic — the simplest starting point; record “uniform-synthetic baseline; hot-key correction unknown” as an Assumed Constraint; (b) histogram-sampled — export a key-access histogram from production APM and replay it proportionally through the load generator’s scripting interface; achievable in a day and removes the most egregious systematic error; (c) trace replay — sample a 60-minute production key-access trace and replay it at target RPS. The gap between (a) and (c) is the measurement investment that realistically takes months. An explicitly caveated (a) measurement is more useful than an implicit assumption of (c) accuracy — the caveat is visible in the birth certificate six months later when the team reviews Assumed Constraints before a capacity event.

Named failure mode: treating the commissioning $\kappa$ as permanent. Every $\kappa$ produced by this recipe is a point-in-time measurement, not a stable architectural constant. The fitted value reflects the traffic distribution, co-tenant load, and deployment environment conditions present during the load test. In production, each of these factors varies: hot-key patterns shift with user behavior, co-tenant workloads contend for shared network and memory buses, and the deployment environment changes as the system scales. These variations do not add to $\kappa$ ; they multiply it — each compounding the realized coherency cost above the commissioning baseline. A birth certificate entry should state the measurement conditions as Assumed Constraints and flag the $\kappa$ value as requiring re-measurement whenever those conditions change materially. Teams that treat the commissioning fit as a permanent ceiling are measuring the architecture they tested, not the system they operate.

The living-contract extension. The governance layer introduced in The Governance Tax formalizes this principle: the birth certificate becomes a living contract, with explicit update triggers (E7: $\kappa_{\text{eff}}$ exceeds $\kappa_{\max}$ by 15%; E8: traffic distribution shifts by more than 0.3 Zipf exponent units) that enforce re-measurement when production conditions diverge from commissioning assumptions. The Assumed Constraints recorded here are the preconditions those triggers check against.

The Bootstrap Protocol — From Vibes to Vectors

The Bootstrap Protocol gives teams without load-test infrastructure or APM tooling a starting position — rough but directional estimates of $\hat{\kappa}$ and $\hat{\alpha}$ — before the Measurement Recipe’s full instrumentation is in place. The Measurement Recipe above requires coordinated-omission-free load generation and a staging cluster sized to production; the Degraded Measurement Guide requires at least APM lock-wait reporting. Some teams have neither: a new service on a three-engineer oncall rotation, a platform team handed a legacy system with no telemetry budget, or an application running entirely on managed cloud components that report only aggregate metrics. The hardware envelope can be established before the application is deployed, while the coordination fingerprint is already embedded in signals every running system emits.

The protocol divides into two phases with a structural reason for the division. Pre-deployment, the system does not yet exist in production — but the infrastructure it will run on does. That infrastructure has fixed physical properties: a durable-commit rate for a single writer thread, and a round-trip cost between node pairs. These two numbers bound the achievable region before any protocol choice is made. Post-deployment, the system is running but may lack formal instrumentation — yet it is already generating three signals that carry the coordination fingerprint: inter-node traffic volume, per-node compute costs, and end-to-end tail latency response to node count changes. The two phases answer different questions from different evidence; combining them produces the bootstrap Pareto Ledger entry.

Phase 1 — Infrastructure Baseline (pre-deployment).

Two quantities define the hardware floor of the achievable region. Neither requires the application to be running. Both are properties of the infrastructure class being commissioned.

Storage ceiling $\gamma_{\text{raw}}$ . The single-node durable-commit rate — how many synchronous writes per second can one node sustain before the storage device is the bottleneck. The measurement requires a single-threaded synchronous-write workload against the specific instance type and disk class that production nodes will use: no request batching, no asynchronous buffering, no parallel writers — conditions that isolate the fsync serialization barrier. The result is $\gamma_{\text{raw}}$ in operations per second per writer thread. Any system whose measured end-to-end throughput is below $\gamma_{\text{raw}}$ is storage-bound: the bottleneck is disk serialization, not coordination overhead. Fitting USL coefficients on a storage-bound system absorbs disk contention into $\kappa$ and produces a misleading frontier map — inflated $\kappa$ , underestimated $N_{\max}$ , invisible real bottleneck. Establish $\gamma_{\text{raw}}$ first; only proceed to USL fitting if end-to-end throughput exceeds it.

Infrastructure class matters significantly. Single-threaded fsync throughput varies by more than an order of magnitude across storage classes:

Storage class	Single-threaded fsync P99 ops/sec
Cloud network-attached (AWS EBS gp3)	400–800
Provisioned IOPS (io2)	1,000–3,000
Local NVMe (i3/i4, bare-metal)	5,000–20,000+

Two systems with identical $\alpha$ and $\kappa$ but different storage classes occupy different starting points in the achievable region — the NVMe system’s frontier sits at $10\times$ higher absolute throughput before coherency overhead becomes the binding constraint. Always measure $\gamma_{\text{raw}}$ on the infrastructure class the system will actually run on; never transfer a value from a different storage class.

Network floor $L$ . The P99 round-trip time between node pairs in the planned topology. The critical properties of this measurement: P99, not mean (mean hides variance that compounds across quorum rounds); taken repeatedly over a sufficient sample (a single measurement reveals nothing about the distribution tail); taken between production-representative node pairs (cross-AZ is a different number than same-rack). This measured P99 RTT is $L$ — the unit in which every consistency guarantee in this series is priced. Every Raft quorum write costs at minimum $1 \times L$ in round-trip time; every cross-shard distributed transaction costs at minimum $3 \times L$ . An architecture whose SLA requires a latency that is a small multiple of its deployment topology’s $L$ has consumed most of its budget before the application processes a single request.

Pre-deployment invariant. $\gamma_{\text{raw}}$ and $L$ are the hardware floor of the achievable region on the infrastructure the system will actually run on — not the vendor’s benchmark configuration. A replication topology commitment made before these two numbers are established is a commitment made against assumed, not measured, constraints. The achievable region has a different shape on different infrastructure classes; the bootstrap protocol establishes which shape this system lives in.

Phase 2 — Coordination Fingerprint (post-deployment, low-instrumentation environment).

For a system already running with no APM, no lock-wait reporting, and no custom telemetry, three cloud-native signals yield rough but directional estimates of $\alpha$ and $\kappa$ . These are bounds, not measurements — sufficient to answer “does the physics tax warrant a proper measurement campaign?” before committing to a full instrumentation project.

Signal 1 — Inter-node traffic volume as a coordination overhead indicator.

For a Raft cluster with $N$ nodes, each committed write generates $2(N-1)$ inter-node messages: one leader-to-follower append per replica and one acknowledgement per replica back to the leader. The coordination overhead ratio expresses how much of the observed inter-node traffic is protocol overhead versus application payload:

$$R_{\text{coord}} = \frac{\text{inter-node egress bytes per day}}{\text{writes per day} \;\times\; \text{median write payload bytes}}$$

$R_{\text{coord}}$	Interpretation	Implied $\kappa$ range
0.5–2	Expected for Raft at $N = 3\text{--}5$	$[0.0001,\; 0.001]$
2–5	Coordination traffic exceeds payload	$[0.001,\; 0.005]$ — investigation warranted
>5	Coordination dominates	$>0.005$ — likely past $N_{\max}$

Signal 2 — Leader-to-replica compute cost ratio as an $\alpha$ proxy.

In a leader-based system, the leader serializes all write coordination — it processes every write proposal, drives quorum, and applies the result before handing off to replicas. If the monitoring environment reports per-instance compute costs, the normalized cost ratio $R_{\text{CPU}}$ between leader and replica reveals how much of the compute bill is serialization overhead:

$$\hat{\alpha}_{\text{billing}} \approx 1 - \frac{1}{R_{\text{CPU}}}$$

When $R_{\text{CPU}} = 2.0$ : $\hat{\alpha} \approx 0.50$ — serial bottleneck is binding, a single node is carrying coordination cost for the cluster. When $R_{\text{CPU}} = 1.1$ : $\hat{\alpha} \approx 0.09$ — contention is low; if throughput is plateauing, coherency overhead is the more likely constraint.

Signal 3 — Tail latency response to node count as a $N_{\max}$ boundary probe.

The retrograde throughput region has a distinctive fingerprint: adding nodes raises latency at constant offered load. This probe must be run in staging, not by routing live traffic to a production shadow cohort. Production traffic carries distribution variance, co-tenant noise, and bursty arrival patterns that corrupt the N comparison: if P99 differs between N and N+k nodes in production, you cannot distinguish the USL coherency penalty from a transient arrival-rate spike or a noisy neighbor. In a staging environment, you control both variables. Spin up a staging cluster at N nodes and at N+k nodes, drive both with a CO-free, open-loop load generator at identical per-node offered load, and hold long enough for the latency distribution to stabilize. The staging N+k-versus-N P99 comparison is a direct probe of the USL curve’s slope at the current node count:

Staging N+k vs. N P99	Interpretation	Action
N+k P99 lower	Interior — additional nodes reduced per-node queue depth	Continue scaling
N+k P99 within 10%	Near $N_{\max}$ — additional nodes provide no benefit	Escalate to Degraded Measurement Guide
N+k P99 higher	Past $N_{\max}$ — coherency penalty active	Stop adding nodes; reduce $\kappa$ first

The third case — P99 rising with no load increase, only with node addition — is the operational fingerprint of the $\kappa N^2$ term dominating in the USL denominator.

Combining the signals — bootstrap Pareto Ledger entry.

The three post-deployment signals each produce an independent lower bound on $\kappa$ . Take the maximum as the bootstrap estimate — the tightest constraint is the binding one:

$$\hat{\kappa}_{\text{bootstrap}} = \max\!\left(\hat{\kappa}_{\text{egress}},\; \hat{\kappa}_{\text{LB}}\right), \qquad \hat{\alpha}_{\text{bootstrap}} = \hat{\alpha}_{\text{billing}}$$ $$\hat{N}_{\max}^{\text{bootstrap}} = \sqrt{\frac{1 - \hat{\alpha}_{\text{bootstrap}}}{\hat{\kappa}_{\text{bootstrap}}}}$$

These estimates carry a 40–60% uncertainty budget — treat the output as directional, not precise. Apply the sufficiency threshold from the Degraded Measurement Guide: if $|N_{\text{current}} - \hat{N}_{\max}| / N_{\text{current}} > 0.30$ , the bootstrap estimate is conclusive for the current capacity decision. If the margin falls within 30%, escalate to the Degraded Measurement Guide’s proxy signals, then to the full Measurement Recipe if the decision is load-bearing.

What the bootstrap protocol does and does not do. It does not replace instrumentation. It answers a bounded question: given only infrastructure-class properties and native platform signals, is there enough evidence to determine whether the physics tax warrants a proper measurement campaign? The protocol recognizes that teams without instrumentation still have two things: a physical infrastructure that can be characterized before deployment, and a running system whose coordination behavior is already visible in billing and routing telemetry. That combination is enough to determine whether the gap between current node count and $N_{\max}$ is safe to ignore or demands immediate attention — and discovering the latter from billing signals alone is sufficient justification to instrument properly.

The operational cost of continuous re-fitting. The Bootstrap Protocol and Drift Triggers together imply a cadence of re-measurement: quarterly, or whenever a Drift Trigger fires. The friction of each re-fit is easy to understate. A team’s first full Measurement Recipe run — a CO-free, open-loop load generator with production-representative traffic distribution, six-point curve fitting, sanity check — takes two to three days, not counting the traffic distribution work described above. A quarterly re-fit by a team with a maintained harness takes four to eight hours. A quarterly re-fit by a team whose harness has rotted since last quarter reverts toward the two-to-three-day estimate. The difference between these cases is invisible before the Drift Trigger fires; it becomes visible the moment the on-call engineer opens the measurement runbook and discovers the last working invocation was a year ago.

The $T_{\text{drift}}$ field in the governance tax vector is designed to surface this. Record the actual re-fit time in engineer-hours on the first run and update it after the first quarterly re-fit. The gap between estimated and actual re-fit cost is a measurement in itself: if the estimated $T_{\text{drift}}$ is 2 hours and the actual is 8 hours, the governance overhead in the birth certificate is understated by $4\times$ . That discrepancy is the shadow friction of continuous USL fitting — the cost that accumulates silently until it surfaces under the pressure of a capacity event.

---

Degraded Measurement Guide — Estimating $\kappa$ from Legacy Observability

The Measurement Recipe above requires a CO-free, open-loop load generator with high-resolution latency histogram output. Many production systems run on observability stacks with none of these: APM dashboards reporting fixed-percentile summaries, database monitoring reporting lock wait averages, and network dashboards reporting packet counters. This guide derives actionable approximations of the total coherency cost $\kappa_{\text{total}}$ and its components — $\kappa_{\text{phys}}$ (hardware floor) and $\beta$ (protocol overhead), as previewed in Definition 11 — from those legacy signals. The approximations are bounds, not measurements — sufficient for the decision “does our current $N_{\text{current}}$ warrant investigation?” and insufficient for a precise USL curve fit. Use this guide as a triage tool, not a substitute for the Measurement Recipe.

Proxy Coherency Coefficient $\hat{\kappa}$ . Hardware-level coherency manifests in three observable legacy signals. Compute each and take the maximum as $\hat{\kappa}$ .

Signal 1 — Connection pool saturation rate. Connection pool exhaustion events per minute per node indicate when per-node coordination overhead saturates available handles. A saturation rate above 0.5 events/minute/node means $\kappa \in [0.001, 0.01]$ for the current $N$ :

$$\hat{\kappa}_{\text{pool}} \approx 0.001 \times \left\lceil \frac{\text{pool\_saturation\_events\_per\_min}}{N_{\text{current}} \times 0.5} \right\rceil$$

Signal 2 — NIC drop rate. Network interface drop rate above 0.01% of total packets signals that physical coordination bandwidth is saturated. At $N = 10$ nodes with 1 Gbps NICs and 64-byte coordination messages, more than 200K coordination messages/sec/node-pair exceeds NIC interrupt processing capacity. Drop rate above 0.01% gives a conservative lower bound: $\hat{\kappa}_{\text{nic}} > 0.001$ .

Combine the two cross-node signals:

$$\hat{\kappa} = \max\!\left(\hat{\kappa}_{\text{pool}},\; \hat{\kappa}_{\text{nic}}\right)$$

Estimate $\alpha_{\text{proxy}}$ from two complementary signals. From APM CPU utilization: $\alpha_{\text{proxy}} \approx (\text{CPU}_{\text{peak}} - \text{CPU}_{\text{baseline}}) / (N_{\text{current}} \times \text{CPU}_{\text{baseline}})$ . From lock wait monitoring: $\alpha_{\text{proxy}} \approx W_{P50} / T_{P50}$ , where $W_{P50}$ is the median lock wait duration and $T_{P50}$ is the median end-to-end request latency — the fraction of total request time spent acquiring local serialization primitives approximates the Amdahl serial fraction. When $W_{P95}/W_{P50} > 4$ , contention is non-uniform and $\alpha$ is likely elevated regardless of node count; this is a local serialization signal and belongs in $\alpha$ , not $\kappa$ . Lock wait variance measures local mutex contention — the in-process serialization that Amdahl’s term captures — not the cross-node state exchange that drives $\kappa$ . Mapping it to $\kappa$ via $N(N-1)$ would predict retrograde collapse at node counts where the network is completely healthy. Take the higher of the two $\alpha_{\text{proxy}}$ estimates. Then compute $\hat{N}_{\max} = \sqrt{(1 - \alpha_{\text{proxy}}) / \hat{\kappa}}$ .

Synthetic Isolation Protocol. To decompose the total coherency cost $\kappa_{\text{total}} = \kappa_{\text{phys}} + \beta$ from Definition 11 into its hardware and protocol components, construct a synthetic probe that bypasses application logic and exercises only the consensus coordination layer. The probe measures the throughput ceiling of the coordination protocol in isolation.

The probe has two configurations:

Single-consistency sweep (measuring $\kappa_{\text{total}}$ ). Issue minimal state-mutating operations — the smallest write the coordination layer accepts, with no application payload — at increasing offered rates against cluster sizes $N \in \{1, 3, 5\}$ (or whatever replica counts your topology supports). At each $N$ , ramp offered load until throughput saturates; record that saturation throughput as $X_{\text{sat}}(N)$ . Fit the USL formula to the $(N, X_{\text{sat}})$ pairs. The fitted coherency coefficient is $\kappa_{\text{total}}$ — the combined hardware-plus-protocol floor for your coordination layer.

Dual-consistency sweep (decomposing $\kappa_{\text{phys}}$ from $\beta$ ). For systems that expose a tunable consistency level, run the same probe twice at identical offered load: once with single-node acknowledgment (exercises only hardware-layer state synchronization, approximating $\kappa_{\text{phys}}$ ) and once with quorum acknowledgment (exercises hardware plus protocol agreement, approximating $\kappa_{\text{total}}$ ). The throughput ratio at the same offered load isolates the protocol-overhead component: $\beta \approx \kappa_{\text{total,quorum}} - \kappa_{\text{phys,local}}$ . For systems that do not expose a consistency knob, treat the single-consistency sweep result as $\kappa_{\text{total}}$ and accept that hardware and protocol components remain bundled.

The probe bypasses authentication middleware, application-level caches, and business logic — it measures coordination cost at the state-machine replication layer. The result is $\kappa_{\text{total}}$ for the consensus component in isolation; the Measurement Recipe on the full service stack captures the full production-path overhead.

Measurement Sufficiency Threshold. The proxy approximation is actionable when the uncertainty budget does not obscure the capacity decision — the margin to $\hat{N}_{\max}$ exceeds twice the 15% measurement uncertainty:

$$\frac{|N_{\text{current}} - \hat{N}_{\max}|}{N_{\text{current}}} > 2\varepsilon$$

where $\varepsilon = 0.15$ (15% measurement uncertainty budget). When this condition holds, the system is either clearly in the interior (wide margin to $\hat{N}_{\max}$ ) or clearly in the retrograde regime — in both cases the proxy estimate justifies the next action without a full recipe run. When $|N_{\text{current}} - \hat{N}_{\max}| / N_{\text{current}} \leq 0.30$ — the proxy estimate places $N_{\text{current}}$ within 30% of $\hat{N}_{\max}$ — run the full Measurement Recipe before making capacity decisions. The proxy approximation is a triage signal, not a substitute for a CO-free USL fit.

The Perf Lab Axiom. All frontier geometry is characterized in a controlled, isolated environment — deliberately pushing the system past $N_{\max}$ to map the exact shape of retrograde collapse, under sterile conditions with no co-tenant interference. Production telemetry serves one function: detecting whether real-world operating coordinates deviate from the lab-characterized model. Discovery happens in the lab. Anomaly detection happens in production. A production anomaly triggers a lab re-run; it does not constitute a measurement.

Perf lab track — 1 day. The only valid measurement environment for frontier geometry. Hardware counters replace APM time-series; coordinated-omission-free load maps the full N-to-throughput curve including the retrograde descent past $N_{\max}$ ; direct USL fits deliver parameters with 90% confidence intervals.

Phase	Action	Output
Morning (0–3h)	NUMA memory access statistics + hardware performance counter monitoring for cache-miss rates (LLC and L3) at idle and under synthetic write load. NIC drop rate monitoring sampled at 1-second intervals. Derive $\hat{\kappa}_{\text{bare}}$ from hardware counters directly.	$\hat{\kappa}_{\text{bare}}$ from hardware; NUMA topology ($\kappa_{\text{intra}}$ vs $\kappa_{\text{inter}}$ ); NIC baseline
Midday (3–6h)	Synthetic Isolation Protocol at $N \in \{1, 3, 5\}$ on dedicated nodes. Open-loop load at fixed Poisson arrival rate via a CO-free load generator; 5-min windows per N point; high-resolution latency histograms. Deliberately push past saturation to observe retrograde descent — the wind tunnel must reach collapse.	$\kappa_{\text{total}}$ and $\beta$ from probe USL fit; retrograde onset confirmed
Afternoon (6–10h)	Full Measurement Recipe at $N \in \{1, 2, 4, 8, 16\}$ including points beyond $\hat{N}_{\max}$ . Synthetic workload with production-representative key distribution; stall filtering (P99/P50 > 8 criterion); NLS fit with 10 windows per N point.	$\alpha$ , $\kappa_{\text{phys}}$ , $\beta$ with 90% CI; full frontier shape including retrograde slope
End of day	NLS fit, bootstrap CI on 1,000 resamples. Validate: probe $\kappa_{\text{total}}$ vs. full-recipe $\kappa$ within 20%. Set autoscaler ceiling to CI lower bound minus two nodes. Record NUMA footprint in birth certificate.	Frontier geometry parameters; autoscaler ceiling; birth certificate entry

Production anomaly detection. Once the lab has characterized the frontier geometry, production monitoring checks whether current operating coordinates match the lab model. Production does not measure — it compares. If actual throughput at $N_{\text{current}}$ deviates more than 15% from the USL prediction at the lab-fitted parameters, or if $\kappa_{\text{eff}}$ estimated from current P50/P99 latency ratios exceeds the lab-measured $\hat{\kappa}_{\max}$ , the system’s operating point has drifted outside the lab-characterized model. That deviation is the trigger for a lab re-run — not a measurement in its own right.

Signal	Lab reference	Production anomaly condition	Action
Throughput at $N_{\text{current}}$	Lab-fitted $X(N_{\text{current}})$	Actual throughput < 85% of lab prediction	Schedule lab re-run within 5 business days
Lock wait fraction $\hat{\alpha}_{\text{lock}}$	Lab-measured $\alpha$	$W_{P50}/T_{P50} > 1.3 \times \hat{\alpha}_{\text{lab}}$	Investigate serialization source; schedule lab aging run
Pool saturation events/min	Lab-predicted: zero at current $N$	Any sustained pool saturation > 0.5 events/min/node	$\kappa$ is above lab-measured value; re-run Synthetic Isolation Protocol
NIC drop rate	Lab-characterized: below 0.01%	Drop rate above 0.01% sustained 10 min	Lab-measured NIC headroom is being consumed; schedule re-run

The lab track reads hardware counters directly and removes cloud jitter from the fit — the resulting $\kappa_{\text{phys}}$ is cleaner by a full digit of precision compared to any APM extraction approach.

Frontier Measurement Protocol — Extracting $\alpha$ and $\kappa$ from Messy Data

The six-step recipe above works when your test generator is closed-loop and your service has no GC pauses, no kernel stalls, and no hot shards. Production systems have all three. Running a closed-loop test on a JVM service at 20 nodes and fitting USL to the results does not measure the frontier — it measures the frontier as experienced by a synthetic client that politely waits for each response before sending the next. That client does not exist in production. Your users are an open-loop arrival process. The frontier you compute from closed-loop data is consistently optimistic in the wrong direction: $\alpha$ is underestimated, $\kappa$ is underestimated, and $N_{\max}$ is overestimated. Your confidence in that frontier should be zero.

Coordinated omission. A closed-loop generator issues one outstanding request at a time: send, wait, record, repeat. When the service slows — a GC pause, a network blip, an election timeout — the generator backs off. Offered load drops at precisely the moment you need to observe the system under pressure. The throughput curve appears to continue climbing past the true $N_{\max}$ because the generator never applied the pressure needed to expose the coherency ceiling. Worse, the P99 you measure is the P99 experienced by a patient client. The P99 your users experience is the P99 of a request that arrived during the stall and queued — an event the closed-loop generator never generated.

The fix is open-loop testing with a Poisson arrival process: schedule request arrivals independently of prior responses, at a fixed rate $R$ ops/sec. Outstanding requests accumulate if the service cannot keep up — exactly the behavior under production load. At saturation, the queue grows without bound; measure the point where throughput plateaus regardless of additional offered load. That plateau is $X(N)$ for USL fitting. Use a CO-free, open-loop load generator that schedules arrivals at a fixed rate independently of response state. Do not use a closed-loop load generator — any tool whose default behavior pauses request issuance when a prior response is pending suppresses coordinated omission silently and produces systematically optimistic frontier estimates.

HDR Histograms and the stall boundary. A fixed-bucket histogram compresses GC pauses and structural tail latency into the same bucket — they are indistinguishable. An HDR Histogram captures values across six orders of magnitude at sub-percent relative precision, so a 300ms GC pause at P99.97 is visible and distinct from the 15ms structural tail at P99.9. Record full HDR histograms — not summary percentiles — for every measurement window. Summary percentiles cannot be re-analyzed; histograms can be merged, sliced, and inspected after the fact.

Stall noise is the population of requests whose latency reflects an external interruption rather than the system’s architectural behavior — GC collection, kernel preemption, NIC interrupt coalescing. Including stall noise in USL fitting inflates $\alpha$ : the optimizer interprets stall-induced throughput dips as additional serialization, producing a model that overstates contention and underestimates $N_{\max}$ . Detection criterion: at each $N$ , compute $P99 / P50$ from the HDR histogram. When $P99 / P50 > 8$ , the tail contains a stall population distinct from the body distribution. For JVM services, the stall boundary is typically visible as a sharp upward kink in the log-slope of the percentile curve above P99.5.

Filtering procedure: collect ten independent 5-minute open-loop windows at each $N$ . For each window, identify the stall boundary $\tau_{\text{stall}}$ as the latency at which the HDR log-slope increases by more than a factor of three — the kink where GC pauses begin to dominate. Exclude request slots above $\tau_{\text{stall}}$ from the throughput count for that window. Recompute $X(N)$ as the median of the ten filtered per-window throughput values. Discard any window where more than 2% of requests fell above $\tau_{\text{stall}}$ — that window experienced a structural stall event (full GC collection, VM live-migration) and should be re-run rather than filtered. The resulting $X(N)$ values represent architectural throughput without stall contamination.

Non-linear least squares fitting. The two-point closed-form estimate in Step 4 above is a starting guess, not a fit. It uses two measurements to solve two unknowns; measurement error in either point propagates directly to $\hat{\alpha}$ and $\hat{\kappa}$ with no smoothing. With stall-filtered throughput at $N \in \{1, 2, 4, 8, 12, 16, 20, 24, 32\}$ , fit the USL model via weighted non-linear least squares:

$$\hat{\alpha},\, \hat{\kappa},\, \hat{\gamma} \;=\; \arg\min_{\alpha,\,\kappa,\,\gamma} \sum_{k} \frac{1}{s_k^2} \left( X(N_k) - \frac{\gamma N_k}{1 + \alpha(N_k - 1) + \kappa N_k(N_k - 1)} \right)^{\!2}$$

where $s_k$ is the standard deviation of $X(N_k)$ across the ten filtered windows at $N_k$ . Weighting by $1/s_k^2$ is heteroskedastic regression: noisier measurements at high $N$ receive less weight than the stable single-node baseline. Use the two-point closed-form estimates as initial values $(\alpha_0, \kappa_0, \gamma_0)$ for the optimizer. Constrain $\alpha \geq 0$ , $\kappa \geq 0$ — negative values are not architecture, they are measurement error or violated equipartition, and should halt the fitting process rather than produce a nominal result.

Confidence intervals for the frontier. The NLS optimizer returns fitted parameters and a covariance matrix $\Sigma(\hat{\alpha}, \hat{\kappa})$ . Propagate parameter uncertainty to $N_{\max}$ via the delta method:

$$\text{Var}(\hat{N}_{\max}) \;\approx\; \left(\frac{\partial N_{\max}}{\partial \alpha}\right)^{\!2} \text{Var}(\hat{\alpha}) \;+\; \left(\frac{\partial N_{\max}}{\partial \kappa}\right)^{\!2} \text{Var}(\hat{\kappa}) \;+\; 2\,\frac{\partial N_{\max}}{\partial \alpha}\,\frac{\partial N_{\max}}{\partial \kappa}\,\text{Cov}(\hat{\alpha}, \hat{\kappa})$$

with $\partial N_{\max}/\partial \alpha = -1/(2\hat{\kappa}\hat{N}_{\max})$ and $\partial N_{\max}/\partial \kappa = -\hat{N}_{\max}/(2\hat{\kappa})$ . A 90% CI is $\hat{N}_{\max} \pm 1.65\sqrt{\text{Var}(\hat{N}_{\max})}$ . For the full frontier curve, bootstrap: resample the $(N_k, X(N_k))$ pairs with replacement 1,000 times, refit USL each time, and take the 5th and 95th percentile of $X_{\text{USL}}(N)$ at each $N$ . That 90% band — not the single fitted curve — is what the data supports. Plot the band.

Operational consequence. Set the autoscaler ceiling to the lower bound of the $N_{\max}$ confidence interval minus two nodes. If $\hat{N}_{\max} = 18$ with CI = [14, 22], the ceiling is 12. Operating at the point estimate risks sitting in the retrograde region whenever the true $\kappa$ is at the high end of its uncertainty range — which it will be, periodically, as load patterns and data distributions shift. The CI margin is not conservatism. It is the honest answer to the question “where is my frontier, given what my data actually supports?”

The Tax Return: Node ROI

Adding a node is an architectural commitment that must produce a positive return. Node efficiency — the throughput gained per unit of baseline capacity added — is the Tax Return:

$$\text{Efficiency}(N) = \frac{X(N)}{N \cdot \gamma}$$

Efficiency = 1.0 means perfect linear scaling. Efficiency at $N_{\max}$ is the minimum before the system crosses into regression. When efficiency is falling and $N > N_{\max}$ , each additional node is negative ROI: you are paying for hardware to reduce throughput.

Worked example. USL parameters fitted from a load test: $\gamma = 1{,}000$ ops/sec, $\alpha = 0.02$ , $\kappa = 0.003$ , giving $N_{\max} = \sqrt{0.98/0.003} \approx 18$ nodes.

$N$	Throughput (ops/sec)	Efficiency	Decision
4	3,650	0.91	Interior — add freely
8	6,116	0.76	Interior — returns declining
16	7,921	0.50	Approaching $N_{\max}$ — justify each node
18	7,972	0.44	At $N_{\max}$ — next node returns nearly nothing
20	7,937	0.40	Past $N_{\max}$ — throughput declining
32	6,962	0.22	14 extra nodes cost 12% throughput

The team at 32 nodes has 14 nodes with negative ROI. Throughput at 32 nodes (6,962 ops/sec) is lower than at 16 nodes (7,921 ops/sec). Every node above 18 is measurable, paid-for waste.

Adding a node past $N_{\max}$ without first reducing $\kappa$ is not cautious — it is provably counterproductive. The Tax Return forces the question: which structural change moves $N_{\max}$ outward far enough to make the next node efficiency-positive? Sharding (adds independent coherency domains), protocol relaxation (conflict-free merge for non-serializable operations), or consistency weakening (causal instead of linearizable) are the levers. Without the calculation, adding capacity past $N_{\max}$ is provably counterproductive — the numbers are measurable and the loss is documented.

Interior waste diagnostic. Two causes produce an efficiency curve that falls faster than the USL predicts from fitted parameters. First: $\alpha$ dominance — if efficiency drops below 0.6 at $N = 4$ , the serial bottleneck is binding before coherency. Adding nodes helps nothing until the serial path is eliminated. Second: hot-shard skew — if the fitted $\kappa$ comes from a uniform load test but production has one shard absorbing 80% of traffic, the real ceiling is the hot shard’s local $\alpha_{\text{hot}}$ , not the cluster-wide $\kappa$ . This yields a shadow constraint: $X_{\text{shadow}} = \gamma / (\alpha_{\text{hot}} \cdot f)$ , always below the USL-predicted maximum. The USL model will predict a higher $N_{\max}$ than observed in production. The gap between predicted and observed peak is the signal that the equipartition assumption is violated.

The Equipartition Trap — Shadow Constraint and Detection

The USL model fitted from a uniform load test is a model of a system that does not exist in production. The load generator distributes traffic evenly; every node sees the same offered load; the fitted $\alpha$ and $\kappa$ reflect averages. That model’s $N_{\max}$ is the theoretical peak of a hypothetical uniform cluster. The production cluster has hot shards — partitions receiving disproportionate traffic because key space is not uniform: popular user IDs, high-churn product categories, monotonically increasing write keys that funnel into a single partition. That shard runs hotter than the model, and its local serialization coefficient $\alpha_{\text{hot}}$ exceeds $\alpha_{\text{avg}}$ by a factor that no uniform load test measures. The result is a shadow constraint: the achievable region contracts inward and the theoretical maximum becomes unreachable.

Formally: let $f \in (0,1)$ be the fraction of total cluster writes routed to the hottest shard, $\alpha_{\text{hot}}$ the serialization coefficient measured on that shard under production key distribution, and $r$ its replication factor. The hot shard’s coherency domain has $N_{\text{hot}} = r$ nodes — typically 3. At that scale the quadratic penalty $\kappa \cdot N_{\text{hot}}(N_{\text{hot}}-1)$ is negligible; the binding constraint is the linear Amdahl term. The quadratic blowup does not materialize because $N_{\text{hot}}$ is too small; what crushes the shard is serialization: $\alpha_{\text{hot}} = 0.30$ with $r = 3$ caps the shard at $\gamma / 0.30 \approx 3.3\gamma$ regardless of how many nodes the full cluster has. This localized rescue applies only to strictly intra-shard operations. The moment the hot shard coordinates a cross-shard transaction — as it does in any system with distributed transactions (Spanner, CockroachDB) — it acquires locks across all participating shards and becomes the transaction coordinator. Its coherency domain instantly balloons to encompass every node holding an intersecting lock, re-invoking the full quadratic $\kappa N^2$ penalty across that expanded domain. The small-$N_{\text{hot}}$ rescue collapses; the hot shard now imposes both its Amdahl serialization ceiling and a coherency blowup proportional to the cross-shard fan-out under contention. The Constraint Sequence Framework later in this section identifies this condition: “throughput falls faster than USL predicts” with “cross-shard operation rate” as the discriminating signal for exactly this failure mode. The cluster-wide throughput cannot exceed that shard ceiling divided by the traffic fraction, regardless of total node count:

$$X_{\text{shadow}} = \min\!\left(X_{\text{USL}}(N_{\max}),\; \frac{X_{\text{hot\_max}}}{f}\right) = \min\!\left(X_{\text{USL}}(N_{\max}),\; \frac{\gamma}{\alpha_{\text{hot}} \cdot f}\right)$$

When $X_{\text{shadow}} < X_{\text{USL}}(N_{\max})$ , the Pareto frontier has contracted inward: there exists a throughput value that the USL model says is achievable but the hot shard prevents reaching. Every node added past the shadow constraint contributes zero throughput and positive coordination cost. The load test showed headroom; the headroom was in the wrong place.

Numbers. A shard receiving 80% of writes ($f = 0.8$ ) with local $\alpha_{\text{hot}} = 0.30$ caps the cluster at $\gamma / (0.30 \times 0.8) \approx 4.2\gamma$ . A 100-node cluster with USL fit of $\alpha = 0.02$ , $\kappa = 0.001$ predicts $X_{\text{USL}}(N_{\max}) \approx 12\gamma$ . The shadow constraint cuts the achievable maximum to 35% of the model’s promise. The model is correct for the uniform case; the production case is a different problem.

    
    %%{init: {'theme': 'neutral'}}%%
flowchart LR
    UNIFORM["Uniform load test
alpha=0.02, kappa=0.001
N_max=32, X_max=12x gamma"]:::ok
    PROD["Production key distribution
Hot shard: f=0.80 of writes
alpha_hot=0.30"]:::warn
    SHADOW_CEIL["Shadow ceiling
gamma / (alpha_hot x f) = 4.2x gamma"]:::warn
    GAP["Frontier contraction
12x predicted, 4.2x achievable
7.8x gamma unreachable"]:::warn
    NODES["Adding nodes past shadow ceiling
Zero throughput gain
Coordination cost still rises"]:::warn

    UNIFORM -->|"diverges in production"| PROD
    PROD -->|"Amdahl ceiling on hot shard"| SHADOW_CEIL
    SHADOW_CEIL -->|"shadow constraint"| GAP
    GAP -->|"consequence"| NODES

    classDef ok fill:none,stroke:#22c55e,stroke-width:2px;
    classDef warn fill:none,stroke:#b71c1c,stroke-width:2px,stroke-dasharray: 4 4;

Constraint Sequence Framework (CSF) — detecting when the model stops matching the telemetry. The USL fit from a load test is a hypothesis. When the production system diverges from that hypothesis, a structured detection sequence identifies which assumption broke and where.

Step 1 — Observe the divergence. Compare the USL -predicted throughput curve to production peak throughput at each node count. A gap of more than 15% between model and measurement at any $N$ value flags a violated assumption. Plot efficiency ($X(N) / (N \cdot \gamma)$ ) against node count: if the production curve bends downward earlier than the model curve, the production $\alpha_{\text{eff}}$ exceeds the modeled $\alpha$ . If it plateaus rather than peaks, the shadow constraint is capping throughput before the coherency regime.

Step 2 — Classify the constraint type. Four signatures appear in telemetry, each pointing at a different root cause:

Signal	Root cause	Next step
P50 and P99 latency scale linearly with disk iowait	WAL fsync saturation (storage physical limit)	Review group commit batch sizing and storage IOPS quota; run a single-threaded synchronous write benchmark (one write at a time, each flushed to durable storage before the next begins) to confirm the raw per-thread fsync ceiling
P99 diverges, P50 is flat	Hot shard read amplification	Per-shard latency histograms
Throughput plateaus below $N_{\max}$	Hot shard write serialization ($\alpha_{\text{hot}}$ )	Per-shard CPU and queue depth
Throughput falls faster than USL predicts	Cross-shard coordination restoring $\kappa$	Cross-shard operation rate

Step 3 — Isolate and measure $\alpha_{\text{hot}}$ . Run the hot shard at 100% of its production traffic fraction in a staging environment, in isolation from other shards. Fit the USL on the isolated shard. The resulting $\alpha_{\text{hot}}$ is the binding serialization coefficient. Compute $X_{\text{shadow}} = \gamma / (\alpha_{\text{hot}} \cdot f)$ and compare to the cluster-wide $X_{\text{USL}}(N_{\max})$ . If $X_{\text{shadow}} / X_{\text{USL}}(N_{\max}) < 0.8$ , the shadow constraint is the dominant ceiling; horizontal scaling is structurally blocked until the hot shard bottleneck is resolved.

Step 4 — Apply the structural fix, then re-fit. Shadow constraints have three remedies: key re-partitioning (distribute hot keys across shards — reduces $f$ ), shard splitting (dedicate additional shards to the hot key range — reduces $\alpha_{\text{hot}}$ by reducing serialization domain size), or read replicas (offload hot shard reads to replicas — separates read $\alpha$ from write $\alpha$ ). After each structural change, re-run the measurement recipe from the section above with production key distribution, not uniform traffic. The USL fit under production keys is the only model that describes the production system.

Named failure mode: Shadow constraint blindness. A load test at uniform traffic fits a healthy $\alpha = 0.02$ , $\kappa = 0.001$ , and predicts $N_{\max} = 32$ . The team scales to 40 nodes under production load. Peak throughput is lower than at 16 nodes. The team attributes it to noisy neighbors, adds 20 more nodes, and observes further degradation. The on-call log for 18 months says “scaling has stopped working.” The USL model was correct under uniform load; the production key distribution activated a shadow constraint at $N = 8$ — the hot shard’s Amdahl ceiling divided by 0.8 — which the load test never exercised. Fix: run all USL fits under production key distributions. Treat any load test with synthetic uniform keys as a lower bound on contention, not a measurement of actual production behavior.

Named failure mode: bimodal tenant spike — the pulsating shadow constraint. In multi-tenant systems, the shadow constraint is not a static ceiling discovered during capacity planning. It pulsates with the tenant activity mix. A USL fit from average-day production traffic models the background cohort well: many small tenants generating low-variance, approximately uniform load. That fit produces a stable $\alpha$ and a correspondingly large $N_{\max}$ . It does not model the spike cohort — the small number of large tenants whose batch exports, fan-out events, or product launches concentrate writes onto a narrow hot key range for minutes at a time.

Let $f_{\text{spike}}(t)$ be the fraction of cluster writes routed to the hot shard at time $t$ (driven by spike-cohort activity) and $\alpha_{\text{hot}}$ be the serialization coefficient measured on that shard. The shadow ceiling is:

$$X_{\text{shadow}}(t) = \frac{\gamma}{\alpha_{\text{hot}} \cdot f_{\text{spike}}(t)}$$

When $f_{\text{spike}}(t)$ rises from a background 0.05 to a spike 0.30 — a single large tenant initiating a batch export — with $\alpha_{\text{hot}} = 0.60$ , the shadow ceiling drops from $\gamma/(0.60 \times 0.05) = 33\gamma$ to $\gamma/(0.60 \times 0.30) = 5.6\gamma$ . A cluster provisioned at its USL $N_{\max}$ — where the model promises 9$\gamma$ — crosses the shadow constraint the moment the spike fires. The system was not over-provisioned; it was provisioned for a traffic distribution that did not exist at the moment of failure.

Operational consequence: a static USL fit produces a capacity ceiling for the traffic distribution the test measured — not a ceiling for all possible distributions the system will serve. In multi-tenant systems, the correct artifact is a $N_{\max}$ distribution across the observed tenant-mix percentiles: $N_{\max}$ at P50 traffic (background), P90 (moderate spike), and P99 (severe spike).

Provisioning at the P50 $N_{\max}$ is a bet that the P99 scenario never coincides with peak offered load. For quota-critical systems — rate limiters, billing counters, shared capacity pools — that bet fails on a weekly basis. The USL gives you a number; the number is only valid when the distribution matches.

When the cluster-wide $N_{\max}$ becomes vacuous. Under extreme skew — Zipf exponents above 1.5, Pareto 80/20 distributions concentrating over 60% of writes onto a single partition — the shadow ceiling is reached at such small $N$ that the cluster-wide $N_{\max}$ calculation offers no practical guidance. The binding condition is:

$$X_{\text{shadow}} < X_{\text{USL}}(N) \quad \text{for all } N \geq N_{\text{shadow\_entry}}$$

where $N_{\text{shadow\_entry}}$ is the node count at which the shadow ceiling first binds. When $N_{\text{shadow\_entry}} \ll N_{\max}$ , the cluster is in the retrograde regime for the hot shard before the aggregate throughput model predicts any degradation. The cluster-wide $N_{\max}$ is not wrong — it correctly describes a cluster that the hot shard prevents from ever existing in production.

The correct modeling unit in this regime is the localized coherency domain: a partition of the cluster into independent sub-clusters, each described by its own $(\alpha_i, \kappa_i)$ pair fitted under its actual traffic share. The cluster-level throughput ceiling is the minimum shadow ceiling across all domains:

$$X_{\text{cluster}} = \min_i \frac{\gamma}{\alpha_i \cdot f_i}$$

where $f_i$ is the traffic fraction of domain $i$ . Under a Zipf distribution with exponent $s$ , the hot domain’s traffic share $f_1 \approx 1/H_{N,s}$ (where $H_{N,s}$ is the generalized harmonic number). At $s = 1.8$ and $N = 100$ keys, $f_1 \approx 0.54$ : over half of all writes route to a single partition regardless of cluster size. At $s = 2.0$ , $f_1 \approx 0.61$ . In either regime, horizontal scaling of the cluster does not move $X_{\text{cluster}}$ — only sharding the hot key range (splitting domain $i$ to reduce $f_i$ ) or reducing $\alpha_i$ (restructuring the hot shard’s serialization path) changes the ceiling.

Proposition 7a -- Coherency Domain Decomposition: under extreme skew, replace the cluster USL with per-domain fits; the cluster ceiling is the minimum domain shadow ceiling

Axiom: Proposition 7a: Coherency Domain Decomposition

Formal Constraint: For a cluster partitioned into coherency domains $\{D_1, \ldots, D_k\}$ with per-domain serialization coefficients $\{\alpha_1, \ldots, \alpha_k\}$ and traffic fractions $\{f_1, \ldots, f_k\}$ summing to 1, the achievable cluster throughput is bounded by:

$$X_{\text{cluster}} \leq \min_{1 \leq i \leq k} \frac{\gamma}{\alpha_i \cdot f_i}$$

The cluster-wide $N_{\max} = \sqrt{(1-\alpha_{\text{avg}})/\kappa_{\text{avg}}}$ is a valid ceiling only when all $f_i$ are approximately equal. When $\max_i f_i > 2/k$ (the hot domain receives more than twice the equipartition share), the cluster-wide model overestimates the achievable ceiling; the domain decomposition ceiling is the tighter, binding constraint.

Engineering Translation: Fit USL per coherency domain, not per cluster. A domain is any partition of state that is serialized independently — a shard, a keyspace range, a tenant bucket. For each domain, measure $\alpha_i$ under its actual production traffic fraction $f_i$ , not under the cluster-wide average. The minimum $\gamma/(\alpha_i \cdot f_i)$ is the ceiling the architecture must be designed around. If that ceiling is below the required throughput, shard the hot domain — no amount of horizontal scaling at the cluster level moves it.

Watch out for: the Zipf exponent itself shifts with product maturity. A new product launch has near-uniform key access (low exponent); as power users emerge, access concentrates (rising exponent); viral events can temporarily push the exponent past 2.0 for minutes. A USL fit from early-phase uniform traffic is not a model for late-phase concentrated traffic. Measure the skew exponent from production key-access histograms, re-fit per-domain $\alpha_i$ when the P99/P50 per-shard traffic ratio exceeds 5, and re-compute the domain decomposition ceiling before each capacity planning cycle.

The Skeptic’s Audit — Three Forces the Model Ignores

The USL formula is deterministic: given $\alpha$ and $\kappa$ , it returns the exact throughput at every $N$ . That determinism is the model’s strength and its lie. Real clusters are stochastic systems where three forces introduce variance that the formula cannot capture. Each force widens the gap between the model’s promise and what the telemetry actually delivers.

Force 1 — Hardware heterogeneity. Nominally identical instances are not identical. Within-SKU variation in CPU silicon revision, thermal throttling state, NUMA topology, and memory timing produces single-node throughput variance of 10–20% in cloud environments [4] . The USL treats $\gamma$ as a constant; the real $\gamma$ is a distribution across nodes. The binding constraint in any serialization path is the slowest node — the one that holds the next lock or commits the next write to the quorum. When the worst-performing node in a 5-node quorum runs at $0.85\gamma$ , the effective $\alpha$ for every operation routed through it rises because its service time is longer than the model assumed. The USL fit captures mean $\gamma$ ; the bottleneck experiences worst-case $\gamma$ . Diagnostic: compute the ratio of P99 to P50 single-node throughput across your cluster under identical offered load. A ratio above 1.2 means hardware variance is contributing to $\alpha$ above the load-test estimate and the real $N_{\max}$ is lower than the model predicts.

Force 2 — Network congestion is stochastic. The USL assumes round-trip time is a constant folded into $\kappa$ . Production round-trip time is a heavy-tailed stochastic process. A single TCP retransmit event under CUBIC congestion control reduces the sender’s congestion window to approximately 70% (Linux TCP CUBIC’s multiplicative decrease factor is 0.7, not 0.5 as in TCP Reno), reducing effective throughput on the affected connection by 30–60% for 200ms–2s while the window recovers [5] . In a cluster where one retransmit event occurs per 10,000 messages at 2,000 msg/sec total throughput, that event fires every 5 seconds — and temporarily raises effective $\kappa$ by a factor of 3–5 for the affected node pairs during recovery. The USL curve describes expected throughput; the production system realizes a throughput distribution whose variance grows with $N$ and with offered load, because more nodes and higher load mean more retransmit opportunities. Diagnostic: measure TCP retransmit rate under sustained production traffic — not a load test. Every 0.1% retransmit rate adds approximately 0.003 to effective $\alpha$ at 10Gbps, eroding the model’s $N_{\max}$ estimate.

Force 3 — Temporal $\kappa$ drift. The USL fit from commissioning does not describe the cluster six months later. State accumulates: conflict-free merge tombstone sets grow, compaction debt defers GC, write-ahead logs lengthen recovery windows, Bloom filter false-positive rates rise as fill ratios increase. Each of these raises the per-operation coherency overhead that contributes to $\kappa$ . A cluster tuned to $N_{\max} = 32$ at deployment may have $N_{\max} = 22$ at six months — coherency overhead per node pair has grown while $\alpha$ remained stable. The model’s promise at commissioning is not the model’s promise in production. Diagnostic: run the Interior Diagnostics procedure at each deploy and on a weekly schedule — not once at commissioning. A declining $N_{\max}$ with stable $\alpha$ is temporal $\kappa$ drift; the remedies are compaction, tombstone pruning, or Bloom filter reconstruction — not more nodes. $\kappa$ derivation must use CO -free P99 measurements from a CO-free, open-loop load generator with high-resolution histogram output; a CO -afflicted $\kappa$ estimate systematically underestimates drift because the overload events where hardware jitter and kernel scheduling variance spike $\kappa_{\text{eff}}$ are the same events the tool omits.

Frontier Drift — $N_{\max}$ as a Time-Varying Signal. The three forces above share a structural property: they fluctuate. Hardware variance changes with thermal state and hypervisor scheduling. Network stochasticity spikes during congestion. State accumulation raises $\kappa$ continuously between compaction events. In multi-tenant systems, a fourth force operates independently of all three: the key-access distribution $D(t)$ shifts with tenant activity mix, driving the effective serialization coefficient $\alpha_{\text{eff}}(D(t))$ up and down throughout the day. The consequence is that $N_{\max}$ is not a fixed design parameter — it is a real-time signal with two independent drivers:

$$N_{\max}(t) = \sqrt{\frac{1 - \alpha_{\text{eff}}(D(t))}{\kappa_{\text{eff}}(t)}}$$

Both parameters fluctuate continuously and independently. $\kappa_{\text{eff}}(t)$ drifts on the timescale of state accumulation (days to weeks) and spikes on the timescale of infrastructure events (seconds to minutes). $\alpha_{\text{eff}}(D(t))$ shifts on the timescale of tenant behavior — batch job schedules, event fan-outs, product launches — and can change by an order of magnitude in under a minute. A static USL fit captures neither driver.

Kernel jitter. Linux CFS scheduling introduces epoll wake-up latency variance of 1–5ms for processes blocked on network I/O. A consensus follower awaiting a heartbeat sees 1–5ms added to the round-trip floor during a CFS scheduling contention event. At intra-DC baseline RTT = 0.3ms, a 2ms wake-up jitter raises the effective round-trip to 2.3ms — a 7.7x multiplier on the coordination cost contribution to $\kappa_{\text{eff}}$ . During CFS contention, $\kappa_{\text{eff}}$ spikes by 3–8x for 10–200ms windows. A counter-shard cluster tuned to $N_{\max} = 10$ at commissioning finds $N_{\max}(t) \approx 4$ during a CFS contention event — six nodes transition from the scaling regime to the retrograde regime without any configuration change.

Noisy-neighbor vCPU steal. Hypervisor scheduling interference introduces steal time — periods when the physical CPU services another tenant’s vCPU. At 15–20% steal (common in public cloud during peak neighbor activity), effective $\gamma$ drops proportionally while consensus round-trip variance increases. The combined effect: $N_{\max}(t)$ can contract by 30–40% during a 90-second steal event — from 10 to 6 nodes in the counter-shard cluster — while per-node CPU utilization looks normal. Steal time is not CPU utilization; standard dashboards do not surface it.

The compounding failure. The load spike that triggers a noisy-neighbor event is the same spike that increases quota-increment write rate to the counter shards. The frontier contracts precisely when the cluster is under maximum stress. A cluster operating at $N = 24$ with commissioning $N_{\max} = 10$ already has 14 nodes in the retrograde regime at idle. During a steal event that drops $N_{\max}(t)$ to 6, every one of those 24 nodes is past the summit — adding coordination cost to a system already degrading under load. Detection: live efficiency $X(N)/(N \cdot \gamma)$ falling faster than the commissioning model predicts flags $\kappa_{\text{eff}}(t) > \kappa_0$ and confirms the frontier has drifted inward.

The Shadow Frontier. The three forces above share a common consequence: each raises the production $\kappa_{\text{eff}}$ above the commissioning-time $\kappa_0$ . The throughput ceiling the model promises — $X(N_{\max}(\kappa_0))$ — is not the ceiling the production system can reach. The Shadow Frontier is the area between the theoretical USL curve and the production-reality curve: throughput that is achievable on the map but unavailable on the terrain.

$$\Delta X_{\text{shadow}} = X\!\left(N_{\max}(\kappa_0)\right) - X\!\left(N_{\max}(\kappa_{\text{eff}})\right)$$

Rate limiter specimen. Counter-shard cluster at six months without a Drift Trigger: ungoverned temporal state accumulation (Bloom filter fill ratio growth, Raft log compaction debt) drives $\kappa_{\text{eff}} = 0.00114$ from the commissioning $\kappa_0 = 0.0005$ . New summit: $N_{\max}(\kappa_{\text{eff}}) = \sqrt{0.98/0.00114} \approx 29$ nodes. Production ceiling: $X(29,\; \kappa_{\text{eff}}) = 29{,}000/(1 + 0.02 \times 28 + 0.00114 \times 29 \times 28) \approx 11{,}670$ ops/sec. Shadow Frontier gap: $\Delta X_{\text{shadow}} = 15{,}681 - 11{,}670 = 4{,}011$ ops/sec — 26% of the commissioning ceiling — not recoverable by adding nodes, only by compaction and tombstone pruning that drive $\kappa_{\text{eff}}$ back toward $\kappa_0$ .

This gap is always non-negative in practice. The commissioning load test measures $\kappa$ under ideal, synthetic, CO -free conditions; production adds hardware jitter, noisy-neighbor interference, kernel scheduling variance, and accumulated state overhead. The gap widens between re-measurements. A team that treats $\kappa_0$ as the current operating parameter has invisibly ceded $\Delta X_{\text{shadow}}$ of throughput headroom.

The CO -free constraint is non-negotiable here. A Shadow Frontier measurement derived from a CO -afflicted tool confirms the model’s optimistic assumptions rather than measuring the actual gap. The overload events where hardware jitter and kernel scheduling variance spike $\kappa_{\text{eff}}$ are precisely the periods a CO-afflicted benchmark omits. If the measurement tool cannot see the overload, it cannot see the Shadow Frontier.

The synthesis. Linear scalability — efficiency = 1.0 at every $N$ — requires $\gamma$ constant (no hardware heterogeneity), RTT constant (no network variance), and $\kappa$ constant (no temporal state accumulation). None of these conditions hold in production. The USL curve is the idealized envelope; production throughput is a stochastic process centered below it with variance that grows with $N$ . The achievable region is not a deterministic set — it is a probabilistic region, the set of operating points the system reaches with high probability under production conditions, which is strictly smaller than the set the model declares theoretically reachable. Named failure mode: linear scalability commitments. A team that presents the USL model’s $N_{\max}$ to stakeholders as a guaranteed scaling ceiling has made a commitment the hardware cannot keep. Treat $N_{\max}$ as the P50 of the achievable scaling range; the hot-shard shadow constraint is the P10 floor; hardware variance and network stochasticity eat the remaining margin; and temporal $\kappa$ drift widens the Shadow Frontier gap between model and telemetry. In multi-tenant systems add a fifth erosion: the P99 shadow constraint is not a fixed floor but an intraday oscillation driven by tenant activity mix — $N_{\max}$ at peak-spike traffic can sit 30–50% below its background-traffic value, and both states are real operating conditions. The model is a map, not a contract.

Interior Diagnostics — Deriving $\kappa$ from Production

The Measurement Recipe earlier in this section derives $\kappa$ from a controlled load test. Interior Diagnostics operationalizes that derivation as a continuous process — running under production key distributions, with CO -free tooling, on a schedule, not once at commissioning.

The procedure follows Measurement Recipe steps 1–6 with four mandatory constraints:

1. Coordinated-omission-free throughput measurements only. Run a CO-free, open-loop load generator with high-resolution histogram output at each $N$ value. A closed-loop tool pauses issuing requests when the previous one is slow — it stops generating load during the overload events that define saturation, the exact moments you need to measure. Throughput values from such tools underestimate saturation throughput and therefore underestimate $\kappa$ . A CO -afflicted $\kappa$ is an optimistic lower bound, not a measurement.

2. Production key distributions, not uniform synthetic traffic. Hot-shard skew concentrates load on a subset of nodes. A uniform load test measures the fleet-average $\kappa$ ; the hot shard’s effective $\kappa_{\text{eff, hot}}$ may be an order of magnitude higher. Interior Diagnostics run under production key distributions report the binding $\kappa$ , not the ideal-case $\kappa$ . The simplest implementation: replay a recent production traffic sample through the CO-free load generator against a staging cluster at $N = 1, 2, 4$ nodes.

3. Track $\kappa$ over time. Derive $\kappa$ at each deploy and on a weekly schedule. Store the time series. A monotonically rising $\kappa$ with stable $\alpha$ is temporal drift from state accumulation or protocol overhead growth — the Shadow Frontier is widening between measurements. A step-change in $\kappa$ at a deploy boundary is a regression introduced by that deploy, identifiable before the system reaches production saturation.

4. Emit the Shadow Frontier gap as a monitoring metric. After each Interior Diagnostics run, compute $\Delta X_{\text{shadow}} = X(N_{\max}(\kappa_0)) - X(N_{\max}(\kappa_{\text{current}}))$ where $\kappa_0$ is the commissioning-time baseline. When $\Delta X_{\text{shadow}}$ exceeds 15% of the commissioning-time ceiling, investigate $\kappa$ sources before authorizing the next horizontal scale-out. Adding nodes when the Shadow Frontier is already wide does not recover the gap — it adds coordination cost to a system that is already paying above its model’s assumptions.

Cognitive Map — The Coordination Tax. The USL places a quantitative ceiling on throughput growth, set by the Contention Tax $\alpha$ and the Coherency Tax $\kappa$ — both inevitable when nodes share state. Past $N_{\max}$ , every added node is a net loss: decreasing throughput at increasing cost. Hot-shard skew and $\kappa$ drift introduce a Shadow Frontier gap — the throughput the model promises that production physics cannot deliver. The USL fit is a hypothesis; CO-free Interior Diagnostics is how you measure the gap.

The throughput ceiling is one constraint on the achievable region; the irreducible tail-latency floor is the other — and it has an entirely different origin.

---

The Geometric Tax — Tail Latency and Fan-Out

Average latency is a lie. It is the metric your monitoring dashboard shows by default, and it is precisely the metric that hides the failure mode your users actually experience. In a distributed system, the response your user receives is not the average across your fleet — it is the maximum across every server their request touched. Fan-out does not average the tails away; it collects them.

The coordination tax limits throughput. The latency tax compounds it: every request that fans out to multiple nodes amplifies the worst-case latency geometrically. The achievable region contracts not only from above (throughput ceiling) but from the right (latency floor moving upward with fan-out). These are independent taxes on the same operating point, and they are invisible to anyone watching P50.

Proposition 8 -- Fan-Out Tail Amplification: composite P99 grows geometrically with fan-out because the slowest of N servers determines the response time

Axiom: Proposition 8: Tail Latency Fan-Out Amplification — Dean & Barroso 2013

Formal Constraint: For a request fanning out to $N$ independent servers, each with probability $p$ of completing within time $T$ :

$$P(\max \text{ latency} \leq T) = p^N$$

For $p = 0.99$ and $N = 100$ : $P(\text{all complete within } T) = 0.99^{100} \approx 0.366$ . The effective P99 of the composite request is the single-server latency at the $0.99^{1/N}$ -th percentile — growing without bound as $N$ increases. [2]

Engineering Translation: To deliver composite P99 at fan-out $N = 100$ , each server must hit P99.99. On a JVM service with P99 = 10ms and P99.99 = 500ms (50x heavy-tail multiplier), the 100ms composite SLA is structurally impossible. Reduce fan-out depth before tuning individual server P99 — each tree depth level charges one decade of required precision regardless of fan-out width.

Proof sketch -- Fan-out tail amplification (Dean & Barroso 2013): why the composite response time is the maximum across all servers and why averaging P99s understates the tail by orders of magnitude

Axiom: Fan-Out Tail Amplification — Dean & Barroso 2013

Formal Constraint: By independence, the joint CDF of the maximum is the product of individual CDFs: $F_{\max}(T) = p^N$ . To meet composite P99 at time $T^*$ : $p(T^*)^N = 0.99$ , so $p(T^*) = 0.99^{1/N}$ — the single-server CDF value at which all N servers simultaneously meet the composite P99. As $N$ grows, $0.99^{1/N} \to 1$ , pushing the required single-server response time toward progressively deeper percentiles approaching P100.

Engineering Translation: At fan-out $N = 10$ , each server must hit P99.9; at $N = 100$ , P99.99. Heavy-tailed distributions (JVM GC-dominated services) amplify this nonlinearly — observed amplification routinely exceeds the exponential prediction. The table below uses a bimodal model representative of production; for exponential service times, amplification is smaller but the direction is identical.

Fan-out $N$	Single-server P99	Composite P99	Amplification
1	10ms	10ms	$1.0\times$
10	10ms	~16ms	$1.6\times$
50	10ms	~55ms	$5.5\times$
100	10ms	~105ms	$10.5\times$
1,000	10ms	~700ms	$70\times$

The diagram below shows how a single client request fans out to $N$ servers, each with an independent P99 of 10ms. The aggregator cannot respond until the slowest server completes — so the composite P99 is determined by the maximum, not the average, across all $N$ servers.

    
    %%{init: {'theme': 'neutral'}}%%
flowchart TD
    REQ["Client request"]:::entry
    FAN["Fan-out to N servers"]:::work
    S1["Server 1 -- P99=10ms"]:::work
    S2["Server 2 -- P99=10ms"]:::work
    SN["Server k (one of N) -- P99=10ms"]:::work
    AGG["Aggregator: waits for ALL"]:::decide
    RESP["Response -- P99 = 10ms x amplification"]:::work

    REQ --> FAN
    FAN --> S1
    FAN --> S2
    FAN --> SN
    S1 --> AGG
    S2 --> AGG
    SN --> AGG
    AGG --> RESP

    classDef entry fill:none,stroke:#333,stroke-width:2px;
    classDef decide fill:none,stroke:#ca8a04,stroke-width:2px;
    classDef work fill:none,stroke:#333,stroke-width:1px;

Every server added to the fan-out widens the composite P99 gap — the amplification factor grows geometrically with $N$ , as the table above shows.

Escaping the geometric tax. The primary architectural escape from fan-out tail amplification is tied requests (also called hedged requests) [2] : issue the same request to two servers in parallel and cancel the slower the moment the first responds. At 100-way fan-out this reduces the effective required percentile from P99.99 down toward P99.9, recovering roughly 40% of P99.9 at the cost of 5–10% additional aggregate throughput — an explicit point on the latency-load frontier, not a free lunch. Two limits bound the strategy: the throughput overhead scales with cancellation rate (higher fan-out means more cancelled duplicates), and tied requests offer no relief when the slow event is correlated across servers sharing a switch or hypervisor, because both copies hit the same slow path simultaneously. Tied requests are the correct first response to geometric tail amplification when reducing fan-out depth is not feasible; reducing fan-out depth is the correct first response when it is.

Independence assumption. Proposition 8 assumes component latencies are independent — that each server’s slow event is uncorrelated with every other server’s. In practice, shared infrastructure (top-of-rack switches, hypervisors, shared DNS resolvers) creates positive correlation: when a switch micro-bursts or a hypervisor stalls, every server behind that component sees elevated latency simultaneously. Under positive correlation, the true composite P99 exceeds the $p^N$ prediction — the formula is an optimistic lower bound, not an exact value. The direction of the error is consistent: the formula always understates composite tail latency when correlation is present.

Why P99s Don’t Average

A persistent misconception is that the composite P99 of a scatter-gather request approximates the average of the per-server P99s. This is wrong by orders of magnitude at non-trivial fan-out, and the error direction is always optimistic. P99 is a quantile, not an expectation. Quantiles of the maximum follow different arithmetic than expectations — and the difference is not a rounding error.

The precise statement. Let $X_1, X_2, \ldots, X_N$ be the response times of $N$ servers, each with CDF $F$ , and let $M_N = \max(X_1, \ldots, X_N)$ be the composite response time. The composite P99 is the value $T^*$ solving $P(M_N \leq T^*) = 0.99$ . By independence:

$$P(M_N \leq T) = F(T)^N$$

Solving for $T^*$ : $F(T^*)^N = 0.99 \implies F(T^*) = 0.99^{1/N}$ . So the composite P99 is $F^{-1}(0.99^{1/N})$ — not the 99th percentile of a single server, but the $(100 \times 0.99^{1/N})$ -th percentile.

Fan-out $N$	$0.99^{1/N}$	Required single-server percentile	JVM P99 = 10ms; required single-server latency
1	0.990	P99.0	10ms
10	0.999	P99.9	~30ms (3x heavy-tail multiplier)
100	0.9999	P99.99	~100–500ms (10–50x)
1,000	0.99999	P99.999	~1–5s (100–500x)

Why averaging is wrong. The average of N identical P99 values is P99 — this is the arithmetic of expectations. But $P99(M_N) \neq \text{avg}(P99(X_i))$ ; the P99 of the maximum is not the average of the per-server P99s. The correct relation is $P99(M_N) = F^{-1}(0.99^{1/N})$ , which is the $F^{-1}$ evaluated at a percentile approaching 1 as $N$ grows — pushing the required single-server value into the extreme tail where latency distributions are heavy. For any service with heavy tails — JVM GC, shared storage, network micro-bursts — the extreme tail grows much faster than linear in the percentile. A JVM service whose P99 = 10ms may have P99.99 = 500ms (50x, not 1.01x). Fan-out of 100 requires each server to hit P99.99; the system with composite SLA = 100ms cannot meet it. The averaging intuition fails because it confuses the aggregation law for expectations (linear) with the aggregation law for quantiles of the maximum (non-linear). There is no correct way to “average” P99s across a scatter-gather request; the only correct operation is to compute the required per-server percentile via $0.99^{1/N}$ and look it up on the empirical tail distribution.

The Relay Race — Case Study: Tail Latency Geometry in Scatter-Gather Quota Checks. The rate limiter’s enforcement path is a scatter-gather relay: before admitting a request, the coordinator reads all $K$ counter shards simultaneously and waits for every shard to confirm the global quota is below 1,000 req/min. With $K = 10$ shards and per-shard P99 = 5ms on a JVM runtime: composite quota-check P99 requires all 10 shards to respond within $T$ , meaning each shard must hit its $0.99^{1/10}$ -th percentile — P99.9. On a JVM shard with G1GC, P99 = 5ms but $P99.9 \approx 15$ ms (3x heavy-tail multiplier from stop-the-world pause distribution). Composite quota-check $P99 \approx 15$ ms. Within the 50ms SLA .

Add kernel jitter. Each shard process runs on a shared hypervisor. G1GC fires one 50ms pause per 30 seconds per shard. With $K = 10$ , the expected gap between any shard’s GC pause is $30/10 = 3$ seconds. At 300 quota checks per second, the expected rate of checks landing during a GC window is $300 \times (0.05/3) = 5$ per second. These 5 checks per second see a composite P99 of 65ms — a persistent SLA breach, not a spike. P50 reports 5ms. Average latency reports 5.2ms. An alarm on P99 over a 1-minute window averages the 5 affected checks against 295 clean ones and does not fire. Only an HDR histogram at sub-second resolution reveals the 65ms population. The benchmark said the relay was fast. Kernel jitter is why one runner is always late.

A composite request is a relay where every shard must finish before the aggregator can respond. The composite P99 is not the speed of the fastest shard — it is the time until the last shard crosses the line. Reducing fan-out removes runners; hedged requests take the faster of two parallel sends. But when kernel jitter is correlated across shards sharing a hypervisor — a GC storm elevating latency on multiple shards simultaneously — both the primary and the hedge see the same slow starting gun, and no hedge saves the race.

Mental Model: The Microservices Latency Loan. A single fan-out is one interest payment. A microservices tree — service A calls B and C, each of which calls D and E — is a compounding loan against your latency budget, charging a new interest installment at every level of depth. The required per-service percentile to deliver composite P99 at the root is $0.99^{1/N^d}$ , where $N$ is the fan-out width at each level and $d$ is the tree depth. At $N = 10$ and $d = 1$ : leaf services must hit P99.9 — one decade of precision borrowed. At $d = 2$ : P99.99 — two decades. At $d = 3$ : P99.999 — three decades. Each level of depth charges one additional decade of precision as interest, regardless of the fan-out width at that level. Most JVM services cannot repay a two-level loan: a service with P99 = 10ms typically has P99.99 between 200ms and 1,000ms (20–100x, not 1x) due to stop-the-world GC. A two-level, ten-wide microservices tree with JVM leaf services and a 100ms composite SLA is structurally insolvent — the loan terms exceed the balance before a single request is issued. The prescription: minimize depth $d$ before optimizing width $N$ . Reducing a three-level tree to two levels saves one full decade of required precision regardless of fan-out. Halving $N$ at any single level saves only half a decade. This is the budget arithmetic that fan-out diagrams hide: the tree structure, not the width, determines the interest rate.

The SLA Budget: fan-out inverts the latency target. The table above reads in one direction: given a single-server P99, what composite P99 does fan-out produce? The operationally critical direction is the inverse: given a composite P99 SLA of $T_{\text{target}}$ , what per-server performance is required?

From the proof sketch, to achieve composite P99 $= T_{\text{target}}$ at fan-out $N$ , each server must respond at its $q$ -th percentile within $T_{\text{target}}$ , where $q = 0.99^{1/N}$ :

Fan-out $N$	Required per-server percentile	Typical heavy-tail multiplier at that percentile	Effective SLA budget consumed
1	P99.0	1.0x	none
10	P99.90	1.5–2x	budget shrinks 1.5–2x
100	P99.990	5–10x	budget shrinks 5–10x
1,000	P99.999	50–100x	budget shrinks 50–100x

The “heavy-tail multiplier” is the ratio of per-server latency at the required percentile to P99 — empirically 1.5–2x for tuned C++ services and 5–10x for JVM services with stop-the-world GC. If your composite SLA is 100ms and fan-out is 100, you need each server at P99.990 within 100ms. If your server’s P99.99 is 500ms (a 5x multiplier over P99 = 100ms), the achievable composite P99 is 500ms — the SLA is unachievable at this fan-out.

Achievable Region shrinkage. In the (latency, fan-out) design space, a fixed SLA draws a curve of feasible (N, per-server P99) pairs. That curve descends steeply — the allowed per-server latency budget collapses as $N$ grows. Reducing the heavy-tail multiplier (eliminating GC pauses, adopting hedged requests, removing shared infrastructure components) expands the achievable region on the latency axis: operating points that were excluded (large fan-out + tight SLA ) become reachable. This is frontier expansion. Adding servers to a fan-out without first reducing the tail multiplier moves deeper into the interior — it does not change the curve’s shape.

Physical translation. At 100-way fan-out your 10ms P99 becomes 105ms; at 1,000-way, 700ms. Adding replicas reduces MTTF but each replica added to a scatter-gather operation raises composite tail — availability improvement costs latency, a genuine frontier trade-off. Hedged requests [2] — duplicate to a second server after P90 latency, take the first response — recover roughly 40% of P99.9 at 5% extra load: a documented operating point on the latency-load frontier.

Watch out for — correlated infrastructure and the independence trap. The $p^N$ formula is not just a lower bound on composite P99 — it is an optimistic approximation that assumes the worst-case events at each server are independent. In modern cloud environments, they are not. Top-of-rack switches, hypervisors, and storage controllers are shared across many tenant VMs. When a switch enters a micro-burst (incast congestion) or a hypervisor stalls for GC or live migration, every server behind that shared component sees elevated latency simultaneously. The latency spike is correlated across all servers in the fan-out, not independent.

Model the fan-out latency under shared infrastructure as a mixture of two regimes. In the normal regime (probability $1 - q$ ), servers respond independently with per-server P99 equal to $p$ . In the correlated regime (probability $q$ ), a shared component is degraded and all $N$ servers see elevated latency simultaneously — effectively reducing the composite response to a single slow draw. The composite P99 then satisfies:

$$P(\max \leq T) \approx (1 - q)\,p^N + q\,p_{\text{burst}}$$

where $p_{\text{burst}} \approx 0$ at the normal-operation P99 threshold $T$ . The consequence: even if each server’s P99 improves to $p = 0.9999$ , the composite P99 is bounded above by $1 - q$ — the probability of not hitting a correlated event. For AWS micro-burst rates of $q \approx 0.005$ –$0.02$ , no per-server optimization can push composite P99 below this floor. Named failure mode: “independence blindness” — a team that improves individual server P99 from 10ms to 5ms and observes no improvement in composite P99 under fan-out is hitting a correlated-event floor, not independent tail amplification. Hedged requests do not help: if the shared switch is bursting, both the primary and the hedge are equally slow. Rack-aware and zone-aware placement reduce $q$ by moving servers to different shared components — but cannot eliminate it at the hypervisor or availability-zone network layer.

Coordinated Omission — Why Your Benchmark Is Lying

The tail latency numbers above assume you can measure them accurately. Most benchmarking tools cannot.

A benchmark that pauses during system overload never records the latency of the queued requests that real users experience — it measures the idle distribution, not the load distribution.

Definition 13 -- Coordinated Omission: a benchmark error where request issuance pauses during system overload, systematically underestimating tail latency by omitting the queued high-latency samples

Axiom: Definition 13: Coordinated Omission

Formal Constraint: A latency benchmark exhibits coordinated omission when it does not issue requests at the intended arrival rate during periods of system overload — it pauses while the system is slow. The benchmark synchronizes with the system’s slowness, omitting the latency samples that would occur during backlog clearance. The result: tail latency is systematically underestimated relative to what users experience under steady load. [3]

Engineering Translation: The omitted samples are the highest-latency ones — the requests that queued during the overload transient and experienced the full backlog drain time. These define the P99.9 tail. A benchmark that omits them reports a P99 drawn from the idle distribution; the true P99 is drawn from the load distribution and can be 30–40× higher at utilization ρ = 0.9. If your load generator is not using open-loop scheduling, all P99 numbers above 90% utilization are systematically understated by a factor proportional to 1/(1-ρ).

Proposition 9 (Coordinated Omission Bias). A benchmark issuing requests at rate $\lambda$ with response time distribution $F(t)$ that pauses during overload (effective rate $\lambda' < \lambda$ during overload periods) underestimates true P99 by a factor of at least $1/(1 - \rho)$ where $\rho$ is the load factor at saturation — a first-order lower bound. In practice, heavy-tailed response time distributions produce gaps substantially larger than this bound: at $\rho = 0.9$ , M/M/1 analysis yields $30$ –$40\times$ underestimation in practice, not the $10\times$ suggested by the bound alone.

Proof sketch -- Coordinated omission bias (Tene 2015): why benchmarks that pause under load omit the worst-case latency samples and systematically understate tail latency

Axiom: Proposition 9: Coordinated Omission Bias (Tene 2015)

Formal Constraint: During an overload period of duration $D$ , a correct benchmark issues $\lambda \cdot D$ requests, all queuing behind the backlog and experiencing latencies proportional to queue position. An omitting benchmark issues at most one request per $D$ — it waits for the previous to complete before sending the next — missing the entire backlog. The missing samples are the high-latency ones; the bias is structural, not random.

Engineering Translation: At $\rho = 0.9$ , a coordinated-omission benchmark reports P99 = 10–15ms while the true P99 is ~460ms — a 30–40x underestimate. Three signals confirm CO is active: P99 flat until saturation then vertical spike; benchmark error rate 0% while load-balancer timeout counters show errors; P99 identical at 80% and 100% load. Use a CO-free, open-loop load generator with high-resolution latency histogram output; any closed-loop generator — one that pauses issuance while waiting for a prior response — structurally omits the tail by never recording the queued requests that users actually experience during overload.

Worked example. Nominal service time: 10ms. Target load: $\lambda = 1{,}000$ req/sec. During a 100ms overload transient, a correct benchmark records approximately 100 requests with latencies between 10ms and 100ms (queue drains as requests are processed). The omitting benchmark records one request at ~100ms. Over a 1,000-second run with ten such transients, the correct distribution has 1,000,000 samples, of which ~1,000 are in the overload tail (0.1%). The CO benchmark has ~999,010 samples, of which ~10 are slow. The CO benchmark’s P99 is the 99th percentile of a sample set where slow requests represent 0.001% instead of 0.1% — off by $100\times$ . The $1/(1-\rho)$ bound is a lower bound on mean queue length, not the P99 gap directly. The M/M/1 P99 at utilization $\rho$ is $-\ln(0.01)/(\mu(1-\rho))$ — at $\rho = 0.9$ and $\mu^{-1} = 10\text{ms}$ : true P99 $\approx 460$ ms. CO benchmark P99: 10–15ms. The actual gap is $30$ –$40\times$ — larger than the lower bound because the bound measures mean queue length while P99 is in the tail of an exponential distribution.

Three signals in your existing metrics confirm coordinated omission is active. First: P99 is flat under increasing load until saturation, then spikes vertically — CO distributions show a cliff rather than gradual degradation because the benchmark never records the queue buildup. Second: benchmark error rate is 0% while your load balancer or client-side timeout counters show errors — the benchmark is not sending requests during the periods users experience failures. Third: P99 under 80% load equals P99 under 100% load — a CO benchmark reports the same tail regardless of offered load because it samples only the idle distribution.

Coordinated omission is why your benchmark says P99 = 5ms and your users report timeouts. The benchmark stopped issuing requests when the system was slow. Real users do not stop — the backlog accumulated during the slowdown is invisible in the CO benchmark’s sample set. The correct approach — described by Gil Tene [3] — is a CO-free, open-loop load generator with high-resolution latency histogram output: the minimum-viable benchmark for any service with a latency SLA . Open-loop scheduling issues each request at its scheduled wall-clock arrival time regardless of whether previous responses have arrived, so outstanding requests accumulate when the service slows — exactly what a real user population does. A closed-loop generator waits for a response before sending the next request, backing off precisely during overload and never generating the queued requests that users actually experience during a stall. The backlogged high-latency samples that define the true tail are the ones a closed-loop generator omits. If your load generator does not use open-loop scheduling, all P99 numbers above 90% utilization are fiction.

Watch out for: back-pressure is a latency-load trade-off on the frontier, and the optimal policy is continuous, not binary. Classic back-pressure treats throttling as a binary decision: a token bucket either issues a request or drops it. The optimal back-pressure policy applies graduated delay proportional to queue depth, equivalent to minimizing total work-in-progress inventory subject to SLA constraints — Little’s Law: $Q = \lambda \cdot W$ ($Q$ = mean queue length, $W$ = mean wait time), minimize $W$ under an SLA bound on $Q$ . Named failure mode: “discrete back-pressure collapse” — systems with binary back-pressure (send or drop) oscillate: they permit requests until the queue hits the limit, drop everything, drain, and repeat. The oscillation frequency is inversely proportional to the token bucket size. Users experience periodic bursts of 100% error rate interspersed with normal operation. Continuous back-pressure — graduated delay, not binary gating — eliminates the oscillation. This is movement toward the frontier: binary back-pressure is an interior point; continuous back-pressure is closer to optimal.

---

Synthesis — The Two Physics Taxes on the Achievable Region

Every result in this post is a contraction of the achievable region established in The Impossibility Tax. The Coherency Tax ($\kappa$ ) contracts the throughput axis — past $N_{\max}$ , horizontal scaling reverses. The Geometric Tax (fan-out amplification) contracts the latency axis — composite P99 grows geometrically with scatter-gather depth.

The following diagram shows how the two physics taxes apply independently to different axes of the achievable region, arriving at the actual contracted boundary.

    
    %%{init: {'theme': 'neutral'}}%%
flowchart TD
    COORD["Coherency Tax
kappa clips throughput at N_max
scaling reverses above ceiling"]:::warn
    LAT["Geometric Tax
fan-out raises P99 floor
composite tail grows with depth"]:::warn
    REGION["Actual achievable region
both physics taxes applied
interior points are genuine gains"]:::ok

    COORD -->|"clips throughput at N_max"| REGION
    LAT -->|"raises latency floor"| REGION

    classDef ok fill:none,stroke:#22c55e,stroke-width:2px;
    classDef warn fill:none,stroke:#b71c1c,stroke-width:2px,stroke-dasharray: 4 4;

The three movement types from The Impossibility Tax apply directly.

Movement toward the frontier. Reducing $\kappa$ without violating correctness — sharding state to reduce cross-shard coordination, replacing synchronous replication with conflict-free merge where consistency requirements permit, applying continuous back-pressure instead of binary. These are movements from the interior toward the Pareto boundary: lower coordination cost with no correctness regression.

Movement along the frontier. Trading throughput for durability — increasing the replication factor to survive more failures at the cost of higher $\kappa$ . Trading latency for fan-out coverage — reducing scatter-gather width at the cost of availability. These are genuine trade-offs where every gain has a measurable cost.

Expansion of the frontier. Sharding to reduce cross-shard coordination, adopting continuous back-pressure to replace binary throttling — these expand the achievable boundary and make previously impossible operating points reachable.

The deepest implication of both taxes is one that scaling narratives hide: adding nodes past $N_{\max}$ — the frontier’s throughput peak under the measured $\kappa$ — does not move a system toward the Pareto frontier; it moves it away. Every node past the coherency peak adds coordination cost to the denominator without adding throughput to the numerator; efficiency falls, the operating point drifts further from optimal, and the team has paid hardware cost to make their system measurably worse on the throughput axis. Meanwhile, every level of depth added to a scatter-gather tree compounds the latency loan geometrically, pushing the required per-service percentile deeper into the tail where JVM pauses and network jitter make the SLA physically unachievable. Scaling out is not inherently improvement. Without a fitted $\kappa$ , a known $N_{\max}$ , and a coordinated-omission-free P99 at each fan-out depth, horizontal scaling is as likely to move a system further from its frontier as toward it.

The operating principle is categorical: you cannot scale your way out of a coherency penalty. A higher $\kappa$ does not become manageable through additional nodes — every node past N_max compounds it. The only structural paths are to reduce $\kappa$ by weakening or redesigning coordination, to shard state to contract coherency domains, or to accept a throughput ceiling that reflects the coordination protocol in place. Treating horizontal scaling as a performance remedy without a fitted $\kappa$ and a known N_max is not capacity planning — it is spending money to traverse the retrograde region.

The taxes in this post are irreducible. They are not penalties for bad engineering — they are the physics of distribution, the floor that every system pays before any protocol choice is made. No protocol eliminates the $\kappa N(N-1)$ coherency term when nodes share state; no scatter-gather architecture escapes $F^{-1}(0.99^{1/N})$ as the composite P99 ceiling. The structure is fixed; the specific $\kappa$ a system pays in production is not — it varies with load distribution, co-tenant pressure, and infrastructure generation, making the floor’s exact height a measurement target rather than a specification constant.

The logical taxes layer on top of this floor: they determine which specific $\kappa$ a system pays — Raft at $\kappa \approx 0.002$ –$0.005$ ($N_{\max} \approx 14$ –$22$ ), EPaxos fast-path at $\kappa \approx 0.0005$ ($N_{\max} \approx 44$ ), conflict-free merge at $\kappa \approx 0$ on writes with coordination relocated to the read path. But every one of those $\kappa$ values lives inside the denominator of $X(N)$ from Proposition 7 . The protocol determines the coefficient; the USL determines what that coefficient costs. Choose the protocol; measure the physics in this post — and measure them again after the protocol change, because the floor shifts when the architecture does.

Flexible quorums — a Pareto movement within the consistency guarantee. One additional degree of freedom the table above omits: quorum sizing within a fixed acceptor set. Heidi Howard’s Flexible Paxos [6] demonstrates that the standard majority-quorum requirement is not necessary for safety: any two quorums that intersect suffice. This allows trading read quorum size against write quorum size while maintaining the same consistency guarantee. A system with seven nodes can use write quorum of five and read quorum of three ($5 + 3 > 7$ ), or write quorum of four and read quorum of four (standard majority), or write quorum of six and read quorum of two. Reducing the read quorum decreases read latency — each read touches fewer nodes — at the cost of a larger write quorum and higher write latency. This is a direct along-frontier Pareto movement within the strict-serial region: write latency and read latency trade against each other without changing the consistency guarantee or the USL $\kappa$ ceiling in aggregate. The total coherency work is the same; its distribution between the read and write paths shifts. This flexibility is not captured in the standard Raft vs. EPaxos comparison: it applies within any quorum-based protocol and is independent of leader vs. leaderless architecture. Systems that are read-heavy should size quorums to minimize read cost; write-heavy systems should do the opposite. Most production deployments use majority quorums by default, leaving this degree of freedom unexploited. Matchmaker Paxos [7] extends the result: where Flexible Paxos addresses quorum sizing within a fixed acceptor set, Matchmaker Paxos enables reconfiguration — replacing failed acceptors with new nodes — with little to no impact on ongoing consensus latency or throughput, a degree of freedom that Flexible Paxos leaves open.

Map update. The map now has two dimensions filled in: the impossibility boundary from The Impossibility Tax defines the excluded corners; the physics taxes from this post define the contraction of the interior. Measure $\kappa$ . Measure P99 with a coordinated-omission-free tool. These are not academic exercises — they are the receipts for the operating point you already occupy.

---

The Pareto Ledger

A team claiming to have “optimized” a distributed system must prove two things: that they were interior before the change, and that they moved toward the frontier rather than along it. Without measurement, “optimization” is unfalsifiable. The Pareto Ledger is the minimum set of numbers required to make the claim. Its two primary coordinates — $\kappa$ (Coherency Tax, Definition 11 ) and $N_{\max}$ (scalability bound, Definition 12 ) — are the same variables that The Impossibility Tax identifies as the operational coordinates of the achievable region.

Ledger Update — $\mathbf{T}_{\text{phys}}$ . The impossibility taxes from The Impossibility Tax define the shape of $\Omega$ by removing excluded corners. This post adds the first measurable component of the cumulative tax vector $\mathbf{T}$ : $\mathbf{T}_{\text{phys}} = (\alpha, \kappa, \text{fan-out depth})$ . The Contention Tax $\alpha$ sets the Amdahl ceiling; the Coherency Tax $\kappa$ sets $N_{\max}$ ; the Geometric Tax (fan-out depth) sets the composite P99 floor. All three are extractable from a single load test. Until they are measured, $\mathbf{T}_{\text{phys}}$ is unknown — and a system with unknown taxes cannot know whether it is on $\mathcal{F}$ or how far inside $\Omega$ it sits.

The Pareto Ledger here is not a formula — it is the floor of your P99. Let L denote the P99 inter-node round-trip time. The Geometric Tax makes L load-bearing: for a request that fans out to N servers, the composite P99 floor is $F^{-1}(0.99^{1/N})$ — a number that grows with fan-out depth and tracks L as the per-server latency distribution shifts. The fitted $\kappa$ and $N_{\max}$ tell you where the throughput ceiling is; the composite P99 at your actual fan-out depth tells you where the latency floor is. Both must appear on the same ledger entry. A system can sit well inside $N_{\max}$ and still be pinned to an unacceptable latency floor by its scatter-gather topology — throughput headroom and latency debt are independent coordinates, not substitutes.

Measuring Your Position

Step 1: Measure $\alpha$ and $\kappa$ . From the Measurement Recipe in Section 1, run your service at $N = 1, 2, 4$ nodes, record stable-state throughput at each point. Solve the two-equation system jointly:

$$\kappa = \frac{4\,X(1)/X(4) - 6\,X(1)/X(2) + 2}{6} \qquad \alpha = 2\cdot\frac{X(1)}{X(2)} - 1 - 2\kappa$$

Note: the closed-form expression $2 \cdot X(1)/X(2) - 1$ yields $\alpha + 2\kappa$ , not $\alpha$ alone. Solving $\kappa$ first and substituting gives the correct result. For systems where $\kappa \ll \alpha$ ($\kappa < 0.001$ ), the shortcut approximation $\alpha \approx 2 \cdot X(1)/X(2) - 1$ introduces less than 0.2% error. For Raft-class systems ($\kappa \sim 0.003\text{--}0.010$ ), use the full two-equation form.

$$N_{\max} = \sqrt{\frac{1 - \alpha}{\kappa}}$$

Result	Healthy range	Warning	Critical
$\alpha$	$< 0.05$	$0.05\text{--}0.15$	$> 0.15$ — serial bottleneck limits frontier
$\kappa$	$< 0.001$	$0.001\text{--}0.005$ (Raft range)	$> 0.01$ — throughput collapses at small $N$
Operating $N$ vs $N_{\max}$	$N < 0.7 N_{\max}$	$0.7\text{--}1.0 \cdot N_{\max}$	$N > N_{\max}$ — every new node reduces throughput

Step 2: Calculate effective $\kappa$ under sharding. If your deployment has $K$ shards and fraction $f$ of writes cross shard boundaries (requiring 2PC ):

$$\kappa_{\text{eff}} = (1 - f) \cdot \frac{\kappa}{K} + f \cdot \kappa$$

At $f = 0$ (all traffic intra-shard), $\kappa_{\text{eff}} = \kappa / K$ — sharding scales the coherency cost perfectly. At $f = 1$ (all traffic cross-shard), $\kappa_{\text{eff}} = \kappa$ — sharding gives zero benefit. The formula makes the cross-shard fraction the primary design variable: reducing $f$ by co-locating related data is often more valuable than adding shards.

The Auditor’s Questions

Eight questions that reveal interior waste. If a team cannot answer them from measurements — not estimates, not architecture diagrams — the system is interior by definition.

1. “What are your measured $\alpha$ and $\kappa$ from the last load test at $N = 1, 2, 4$ ?” If unavailable: the team is operating without a frontier map. They cannot know their position relative to it.

2. “What is your $N_{\max}$ , and how many nodes are you currently running?” If $N > N_{\max}$ : every node above $N_{\max}$ is reducing throughput while consuming resources. This is measurable, paid-for interior waste.

3. “What fraction of writes cross shard boundaries (your $f$ )?” If $f > 0.20$ with a high $\kappa$ : the sharding topology is not aligned with the access pattern. $\kappa_{\text{eff}}$ is much larger than the advertised per-shard $\kappa$ , and the throughput ceiling is correspondingly lower.

4. “At 80% target load, is your observed P99 within $2\times$ of your P99 at 20% load?” If yes: coordinated omission is active in the benchmark — true P99 at saturation is commonly 10–100$\times$ higher than reported, with 30–40$\times$ typical for systems with periodic GC stalls. The team believes they are on the frontier; their P99 measurement is fiction.

5. “Which consistency level does each API endpoint actually require vs. what the database provides?” If strict serializability for operations that only need read-your-writes: the team is paying 1–5ms per write intra-DC and accepting leader-bound throughput for a guarantee no one is using. Interior waste on the consistency axis.

6. “What is your leader’s CPU utilization relative to your followers at peak load?” If leader CPU > 70% and follower CPU < 30%: $\alpha$ dominates. The system is paying the full distribution tax — replication overhead, leader election, heartbeats — but the throughput ceiling is the single leader pipeline, not coherency. This is $\alpha$ -dominated interior waste. $N_{\max}$ may be large ($\kappa$ is low), but that ceiling is unreachable while reads serialize through the leader. The fix is not more replicas: it is distributing reads to followers (sequential consistency via stale reads, or linearizable via batched ReadIndex), sharding leadership across multiple consensus groups, or vertical leader scaling for write-dominated workloads.

7. “What is your hottest shard’s throughput as a multiple of your average shard’s throughput?” If max shard throughput > 3x average shard throughput: the system is operating on the local achievable region of the hot shard, not the global achievable region of the fleet. The theoretical $N_{\max}$ from fleet-wide $\kappa$ is unreachable — the hot shard saturates first. Adding shards to the fleet does not redistribute the hot shard’s traffic without routing changes or data rebalancing. This is the local-frontier trap: the team measures global $N_{\max}$ and sees capacity headroom; the hot shard’s effective $\kappa_{\text{eff, hot}} = K \cdot \kappa$ means its local $N_{\max}$ is 1 for the overloaded key range.

8. “Is your baseline throughput $\gamma$ measured with synchronous or asynchronous WAL flushes?” If asynchronous (fsync disabled or backgrounded): the throughput figure is a vanity metric. The team has silently traded strict durability for speed, moving off the strict-serializable frontier — if the process crashes, acknowledged writes are lost. Acknowledging the true physical cost of durability typically cuts claimed $\gamma$ by an order of magnitude: a system reporting $\gamma = 50{,}000$ ops/sec with fsync disabled may deliver $\gamma = 2{,}000\text{--}5{,}000$ ops/sec at production durability settings on cloud-attached block storage. Every USL figure derived from the async baseline — $N_{\max}$ , the coherency trade-off table, the frontier position — inherits this fiction.

Two $\kappa_{\text{eff}}$ formulas serve different purposes and must not be mixed.

Per-shard hotness model — $\kappa_{\text{eff}} = K \cdot \kappa$ : use this to model a single overloaded shard where $K$ is its load multiple. It gives the local $N_{\max}$ ceiling for the bottleneck shard.

Fleet-wide traffic-weighted sum — $\kappa_{\text{eff}} = f_{\text{cross}} \cdot \kappa_{\text{cross}} + (1 - f_{\text{cross}}) \cdot \kappa_{\text{intra}}$ : use this to estimate fleet-wide coherency overhead when traffic splits across coordination domains. It gives a fleet-average fit for USL regression across the full cluster — as in the Cassandra case study below.

Named failure mode: Hot Shard Contraction. A platform shards user data by user ID across 100 shards. One shard handles 40% of total write volume — a viral account. Fleet-wide $\kappa = 0.003$ gives $N_{\max}^{\text{global}} \approx 18$ nodes per shard. The hot shard carries 40x its intended load: applying the per-shard hotness model, $\kappa_{\text{eff, hot}} = 40 \times 0.003 = 0.12$ , giving $N_{\max}^{\text{hot}} = \sqrt{(1-\alpha)/0.12} \approx 2.8$ — effectively a single-node system for that key range. The team observes global P99 rising and adds capacity; P99 continues rising. They are measuring the global frontier; they are bounded by the local frontier. Detection: if global P99 co-moves with one shard’s P99 while all other shards are below 50% utilization, hot shard contraction is the cause. The fix is key redistribution (split the hot account’s data across multiple logical shards), load-aware routing (rate-limit writes beyond a shard’s local $N_{\max}$ ), or application-level fan-out mitigation.

Sample Ledger Entry: Raft Cluster with Leader Bottleneck

A 5-node Raft cluster handles 90% reads and 10% writes. All reads route to the leader via ReadIndex — the team chose “linearizability everywhere” as the safe default. The team reports low latency and high availability. The Pareto Ledger tests that claim.

Load test at $N = 1, 2, 4$ nodes yields $\alpha = 0.45$ , $\kappa = 0.003$ , $\gamma = 10{,}000$ ops/sec. The 45% serial fraction is the leader pipeline: every ReadIndex serializes through it. $N_{\max} = \sqrt{(1 - 0.45) / 0.003} \approx 13.5$ — coherency headroom is ample. The constraint is not coherency; it is serialization.

Metric	Current (all reads via ReadIndex)	After (90% reads to followers, batched ReadIndex)	Net
$\alpha$	0.45	0.05 (writes + rare linearizable reads only)	-0.40
$N_{\max}$	~13.5	~18	+1.3x ceiling
$X(5)$ at $\gamma = 10{,}000$	~17,500 ops/sec	~40,000 ops/sec	+2.3x
Consistency	Linearizable (all traffic)	Linearizable (10% writes + critical reads); sequential (90% reads)	Relaxed for non-critical reads
Hardware change	—	None	—

Ledger verdict. The system delivers 17,500 ops/sec when the same hardware, with reads routed to followers, supports ~40,000 ops/sec — a 2.3x gap recoverable by a routing change, without hardware addition. This is the defining characteristic of interior waste from The Impossibility Tax: the gap is between two configurations of the same hardware, free improvement is available, and no genuine trade-off is required to claim it.

The auditor’s question surfaces the decision that created the waste: “linearizability everywhere” applied uniformly to idempotent catalog reads that no client code depends on for ordering guarantees. For those reads, sequential consistency is indistinguishable from linearizability in practice. Moving them to followers changes no external contract; it recovers 2.3x throughput.

Sample Ledger Entry: RDBMS to Sharded NoSQL

A platform migrates from a single PostgreSQL instance to a 10-shard Cassandra cluster with a Saga coordinator for cross-shard writes. The team claims “$3\times$ performance improvement.” The ledger tests that claim.

Metric	Before (PostgreSQL, $N=1$ )	After (Cassandra, $K=10$ , $f=0.12$ )	Net
$\alpha$	~0 (single node)	0.02 (Saga coordinator serializes cross-shard)	increased slightly
$\kappa_{\text{intra}}$	0	0.0001 (intra-shard Raft, 3-way)	increased
$\kappa_{\text{eff}}$	0	$0.88 \times 0.00001 + 0.12 \times 0.0001 \approx 0.000021$	increased from 0
$N_{\max}$	1 node (cannot scale)	$\sqrt{0.98 / 0.000021} \approx 216$ nodes	increased from 1 to 216
Consistency $c$	6 (strict serializable — everything)	3 intra-shard (snapshot ISO); 1 cross-shard (saga eventual)	decreased for cross-shard; level 3 intra, level 1 cross
P99 intra-shard write	5ms	2ms	decreased (improved)
P99 cross-shard write	5ms	15ms ( 2PC phases + Saga)	$+3\times$ regression
Availability $a$	0.999 (single node)	0.9999 (no SPOF, replicated)	increased
Throughput ceiling $t$	~5K ops/sec (single node)	~1M ops/sec ($N_{\max} \times 5K$ )	$+200\times$ frontier expansion

Note on consistency levels. The $c$ values in the table use the six-level partial order from Definition 5 in The Impossibility Tax: 6 = strict serializable (strongest), 5 = linearizable, 4 = sequential, 3 = snapshot isolation, 2 = causal, 1 = eventual consistency (weakest). The Cassandra migration accepts level 3 for intra-shard operations (snapshot isolation via intra-shard Raft) and level 1 for cross-shard operations (saga eventual with bounded convergence lag). Level 3 means reads may observe stale state within the Saga’s convergence window for cross-shard items; the write path is still coordinated within each shard.

Note on ledger $\kappa$ values. The $\kappa_{\text{intra}} = 0.0001$ and $\kappa_{\text{eff}} \approx 0.000021$ values in the table above are illustrative estimates, not measurements from a load test. They are derived from the stated traffic fractions and representative coordination-overhead ratios for Cassandra intra-shard Raft and Saga cross-shard coordination. Do not use them as reference starting points for your own system — $\kappa$ varies by an order of magnitude or more depending on replication protocol, network topology, and key distribution. Your actual $\kappa$ must come from the Measurement Recipe run against your service. The $\kappa_{\text{eff}}$ formula ($0.88 \times 0.00001 + 0.12 \times 0.0001$ ) treats intra- and cross-shard coherency costs as linearly additive with traffic-fraction weights. This is a practical approximation for capacity planning, not a result derivable from USL theory: the USL $\kappa$ is a per-node-pair synchronization overhead that does not distinguish coordination topology. The 12% cross-shard traffic fraction ($f$ ) is a traffic-mix parameter, not a USL parameter — multiplying structurally different per-pair costs ($\kappa_{\text{intra}} \neq \kappa_{\text{cross-shard}}$ ) by traffic fractions is a heuristic that may be off by a factor of 2–5 depending on the actual coordination topology. Measure $\kappa_{\text{eff}}$ directly from a load test that reproduces your real cross-shard traffic fraction.

Ledger verdict. The throughput ceiling expanded $200\times$ : this is genuine frontier expansion — new operating points became reachable that were impossible before. The availability improved. Intra-shard latency dropped. These are real gains.

The cross-shard P99 regression from 5ms to 15ms is not interior waste — it is the cost of 2PC , a movement along the frontier. The team should document it explicitly: “12% of writes now cost 15ms for strict cross-shard coordination. This is the consistency tax for cross-shard atomicity.”

The auditor’s question then surfaces actual interior waste: of the 12% cross-shard writes, how many truly require strict atomicity? If post-deployment analysis shows only 4% involve financial transactions requiring cross-shard strict isolation, the other 8% are using Saga + 2PC unnecessarily. Those 8% are paying 15ms when 2ms would suffice — interior waste identifiable only because the ledger forced the measurement.

The claim “$3\times$ performance improvement” is partially correct, partially incomplete. Throughput ceiling improved $200\times$ ; intra-shard latency improved $2.5\times$ ; cross-shard latency worsened $3\times$ ; consistency weakened for cross-shard operations. The Pareto Ledger converts a marketing claim into a navigational statement: here is where the system was, here is where it moved, and here is the interior waste that remains.

The Ledger as defined here is a point-in-time measurement. At the scale of production systems, the operating point does not stay fixed — $\kappa$ drifts as traffic patterns change, N_max shifts as hardware degrades, and RTT floors move as network topology evolves. The logical extension is a continuous Pareto Ledger: the same measurements, running as monitoring panels, feeding automated triggers rather than one-time audits. That extension — how to build the Ledger into a production governor — is the subject of the final post in this series.

Pareto Ledger — Physics Taxes

Tax Type	Metric / Notation	Price Paid — Rate Limiter Case Study	Drift Trigger
Physics — Throughput Baseline	$\gamma$ (single-node peak throughput, measured at $N = 1$ with CO-free load)	$\gamma = 1{,}000$ ops/sec (EPaxos fast path at N = 1, bare node)	$\gamma$ drop > 20% vs. birth certificate — storage or CPU contention review. Note: $\gamma$ is the throughput baseline that $\alpha$ and $\kappa$ erode — it does not appear in the tax vector $\mathbf{T}_{\text{phys}}$ because it is the value being taxed, not a component of the tax itself. A drop in $\gamma$ means the hardware’s raw capacity has degraded; $N_{\max} = \sqrt{(1-\alpha)/\kappa}$ is unchanged, but the throughput at every node count is uniformly lower.
Physics — Contention	$\alpha$ (Amdahl serial fraction)	$\alpha = 0.02$ (2% residual cross-region serialization from periodic global quota sync; reduced from the initial 4% by replacing per-write leader serialization with EPaxos fast-path consensus on non-conflicting keys)	$\alpha > 0.10$ sustained 30 days — shard topology review
Physics — Coherency	$\kappa \to N_{\max} = \sqrt{(1-\alpha)/\kappa}$	$\kappa = 0.0005$ ( EPaxos fast path), $N_{\max} = 44$ nodes	$\kappa > 0.0006$ sustained 7 days — USL re-fit within 7 business days
Physics — Tail Latency	$F^{-1}(0.99^{1/N})$ , irreducible P99 floor	$L_{\text{intra}} = 1\,\text{ms}$ P99 floor at $N = 3$ (regional Raft group)	Intra-region P99 > 5ms — Raft group health check

---

Measuring the Physics Tax Under Production Noise

The Pareto Ledger demands two numbers: $\alpha$ and $\kappa$ . Extracting them from a load test in an isolated lab environment is a matter of algebra. Extracting them from a system running inside a multi-tenant hypervisor, sharing ToR switches with 60,000 other containers, is a statistical estimation problem under adversarial noise conditions. Every measurement instrument you point at $X(N)$ is reading a superposition of signal and three structurally distinct noise sources: hypervisor steal-time injections, network micro-burst latency spikes, and CPU cache-line ping-pong from cross-tenant NUMA pressure. The naive approach — fit the USL formula to mean observed throughput at $N = 1, 2, 4$ — absorbs all three noise sources into the $\kappa$ coefficient. The result is a $\kappa$ that overstates coherency cost, a $N_{\max}$ that understates scaling headroom, and a frontier map that is wrong in the direction that most damages engineering decisions: it shows the system as closer to the frontier than it is.

Decomposing $\alpha$ : Logical Contention vs. Physical Noise

The contention coefficient $\alpha$ from Definition 10 aggregates two mechanisms that have different causes and require different remediation. Logical contention — $\alpha_{\text{logic}}$ — is protocol-determined: lock acquisition queues, write-coordinator serialization, leader election wait time. It responds to protocol redesign: replacing single-leader Raft with EPaxos, partitioning the write coordinator, batching lock acquisitions. Physical contention — $\alpha_{\text{phys}}$ — is hardware-determined: hypervisor CPU steal cycles, L3 cache-line invalidation from cross-tenant NUMA pressure, NIC interrupt coalescing that serializes bursts of small packets through a single core. It does not respond to protocol redesign; it yields only to hardware topology changes, CPU affinity pinning, or load reduction below the steal threshold.

$$\alpha_{\text{observed}} = \alpha_{\text{logic}} + \alpha_{\text{phys}}$$

The USL fit gives you $\alpha_{\text{observed}}$ . To separate the components: instrument a benchmark run with Linux Performance Monitoring Counters (PMC) simultaneously. $\alpha_{\text{phys}}$ manifests as elevated LLC (Last-Level Cache) miss rates and NIC interrupt queue depth that correlates with throughput degradation — it is present even when no protocol-level locking is occurring. $\alpha_{\text{logic}}$ manifests as mutex contention depth, leader CPU utilization asymmetry (leader CPU > 70%, follower CPU < 30%), and scheduler wait time on lock-owning threads. Profile both simultaneously under the same load levels used for USL fitting. Report $\alpha_{\text{logic}}$ and $\alpha_{\text{phys}}$ separately. A team that collapses them into a single number is conflating a protocol problem with a topology problem — the wrong intervention for each wastes engineering cycles.

Contention source	Detection instrument	Remediation path
Lock queue depth / write serialization	Lock contention profiler (runtime or OS-level)	Protocol redesign: leaderless, sharding, batching
L3 cache-line ping-pong	Hardware performance counter monitoring for last-level cache miss rates	CPU affinity pinning, NUMA-aware memory allocation
NIC interrupt coalescing	NIC interrupt coalescing configuration inspection; RX interrupt rate vs. throughput	Interrupt affinity, RX queue steering, larger coalesce intervals
Hypervisor steal time	Host CPU steal time accounting (available from cloud provider instance metrics or OS-level CPU utilization telemetry)	Steal-time gate, container placement policy, dedicated tenancy

Steal-Time Gating

Hypervisor steal time is the fraction of CPU cycles the guest OS requested but the hypervisor gave to another VM. At 60,000 containers on shared hosts, steal rates of 2–8% are background conditions, not anomalies. A throughput measurement taken during a 6% steal window reports $X_{\text{obs}}(N) \approx X_{\text{true}}(N) \times (1 - s)$ where $s$ is the steal fraction — an artificial depression of observed throughput that the USL formula then attributes to increased coherency cost, inflating $\hat{\kappa}$ .

The gate is mechanical: before accepting a throughput measurement at any $N$ , verify that $\Delta \text{steal}/\Delta t < s_{\text{max}}$ throughout the measurement window. Read the steal fraction from the host’s CPU utilization telemetry (cloud provider instance metrics or OS-level CPU accounting) at the start and end of each measurement interval; reject and repeat if the delta exceeds the threshold. A practical value is $s_{\text{max}} = 0.02$ (2%). At fleet scale, quiet windows are infrequent but not rare — a 15-minute measurement window run against a canary cluster during off-peak hours will find them within hours. Do not relax the gate to accelerate measurement; a $\kappa$ derived from noisy data is not a conservative estimate — it is a biased estimate with no predictable direction of error.

Quantile Regression for Noise-Robust Fitting

Ordinary least squares (OLS) fits the USL formula by minimizing the sum of squared residuals. In the presence of micro-burst latency spikes, noisy-neighbor events, and thermal throttle episodes, OLS is the wrong estimator: it is pulled toward outliers, and the outliers in a production throughput series are uniformly downward — temporary throughput collapses during steal events or incast bursts. The correct estimator for a floor boundary is quantile regression at the $\tau = 0.10$ quantile: fit the USL curve to the 10th-percentile envelope of observed throughput at each $N$ , not the mean.

$$(\hat{\alpha},\, \hat{\kappa}) = \arg\min_{\alpha,\kappa} \sum_{i} \rho_{0.10}\!\left(X_i - \frac{\gamma N_i}{1 + \alpha(N_i - 1) + \kappa N_i(N_i - 1)}\right)$$

where $\rho_\tau(u) = u(\tau - \mathbf{1}_{u < 0})$ is the quantile regression check function [8] . The P10 envelope captures the throughput the system sustains under production noise conditions — the floor of the achievable region, not its occasional peaks. Pair this with a bootstrap confidence interval: resample the $(N_i, X_i)$ pairs with replacement 1,000 times, refit on each resample, and report the P5–P95 interval on $\hat{\kappa}$ . A 90% CI of $[0.003, 0.004]$ is actionable; a CI of $[0.001, 0.009]$ indicates the measurement is noise-dominated and more data collection under gated windows is required before the $\kappa$ estimate can be trusted. The CI width is the honesty check that mean-based fitting omits.

The Five-Point Minimum and the Retrograde Confirmation

Fitting $\alpha$ and $\kappa$ from three data points ($N = 1, 2, 4$ ) is algebraically overdetermined only if the USL formula is exact. Under measurement noise it is underdetermined for CI construction: three points yield a point estimate with no confidence interval. Use a minimum of five measurement points at $N = 1, 2, 4, 8, 16$ for quantile regression to have degrees of freedom for variance estimation.

More importantly: include at least one measurement point in the retrograde region — at $N > N_{\max}$ . The coherency term $\kappa N(N-1)$ in the denominator of $X(N)$ is what separates the USL from Amdahl’s Law; it is only visible as decreasing throughput at large $N$ . A fit derived entirely from the scaling regime ($N < N_{\max}$ ) cannot distinguish a low-$\kappa$ curve from a $\kappa = 0$ curve in the observed data range — both fit well where throughput is still rising. A confirmed retrograde point is the empirical evidence that the quadratic denominator term is real, not a model artifact. If your current deployment operates at $N = 8$ and fitted $N_{\max} = 14$ , run one measurement at $N = 20$ to confirm the retrograde region exists at the predicted location. If throughput at $N = 20$ is still rising, $\hat{\kappa}$ is wrong.

Measuring $\kappa$ from the scaling region alone is like measuring a parabola’s peak by fitting only the left branch. The coefficient exists in the data; it becomes visible only when you look far enough right to see the curve turn over. Teams that never deploy a retrograde measurement point are fitting a model without testing its central prediction.

---

References

1. N. Gunther. “Guerrilla Capacity Planning.” Springer, 2007.

2. J. Dean, L. Barroso. “The Tail at Scale.” Communications of the ACM, 56(2):74-80, 2013.

3. G. Tene. “How NOT to Measure Latency.” Strange Loop, 2015.

4. P. Leitner, J. Cito. “Patterns in the Chaos — A Study of Performance Variation and Predictability in Public IaaS Clouds.” ACM Transactions on Internet Technology, 16(3), 2016.

5. V. Vasudevan, A. Phanishayee, H. Shah, E. Krevat, D. Andersen, G. Ganger, G. Gibson, B. Mueller. “Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication.” SIGCOMM, 2009.

6. H. Howard, D. Malkhi, A. Spiegelman. “Flexible Paxos: Quorum Intersection Revisited.” arXiv:1608.06696, 2016.

7. M. Whittaker, N. Giridharan, A. Szekeres, J. M. Hellerstein, H. Howard, F. Nawab, I. Stoica. “Matchmaker Paxos: A Reconfigurable Consensus Protocol.” Journal of Systems Research (JSys), 2021.

8. R. Koenker, G. Bassett Jr. “Regression Quantiles.” Econometrica, 46(1):33–50, 1978.

---

Back to top