The Barrier the Scaler Cannot Hold

25 October 2026 • Yuriy Polyulya •

#distributed-systems #control-theory #systems-thinking #queueing-theory

Watch the first ninety seconds of an overload from inside the service. Demand steps up tenfold. The autoscaler notices within a sync, and its recommendation climbs. Pods are ordered. For the next ninety seconds, while those pods are scheduled, pulled and started, the service runs on exactly the capacity it had before the step. In the specimen of the previous post, good work fell to zero within the first thirty seconds. By the time the first new pod served a request, the queue had already decided how the next hour would go.

Every result in this series has lived inside that interval. The saturated metric of the first post probed one round at a time because each round cost a dead time. The backlog loop of the second needed a standing delay of about one dead time to rest. The retry debt of the third was built during the dead time, when nothing the autoscaler ordered could reach the queue. Each post priced a different consequence of the same fact. During the provisioning delay, the scaler has authority over nothing.

The previous posts let the queue absorb whatever the dead time produced. The queue held all of it, and the controller’s only job was to catch up. This post asks where the excess should go instead, who controls that choice, and what each choice costs. It ends where the series must: with what an engineer can compute before the next overload arrives.

Notation used across the series

where:

• $\tau$ - the dead time, from ordering a pod to its first served request; 90 seconds in the specimen
• $k$ - the size of an overload, as a multiple of the load before it; $K$ when it is drawn at random
• $u^\ast$ and $\rho$ - the utilization target and the utilization a service actually runs at
• $\ell$ and $r$ - a pod’s CPU limit and CPU request
• $g = \ell/(r\,u^\ast)$ - the growth factor: the most the controller can multiply its ready pods by in one round
• $W$ and $w = W/\tau$ - the downscale stabilization window, and the same window measured in dead times
• $T$ , $c$ and $D = T/c$ - a backlog threshold in messages per pod, the rate of one pod, and the standing delay every message waits at rest
• $\Phi(w)$ - the standing delay, in dead times, a backlog loop with window $w$ needs to rest
• $A$ and $A_{\text{fail}}$ - the retry amplification: units of work per first attempt once every attempt fails, and the same quantity whatever sets it, retry counts, budgets or limiters
• $\lambda$ , $C$ , $\tau_{\text{out}}$ and $\Delta$ - demand, capacity, the client timeout and the length of a spike
• $M$ and $b$ - a replica ceiling and an admission bound in seconds of capacity
• L1 · Bound, L2 · Fit and L3 · Estimate - the tag on each claim: a proof or a cited fact, a judgment or approximation the text argues for, and a worked example computed from the model; each part’s ledger names the layer of every law it records

The Belief: a Limit Removes a Risk

L1 · Bound Limits are how operations teams make systems safe, and the tools ship with them already set. KEDA’s specification gives every scaled object a replica ceiling unless told otherwise: maxReplicaCount: 100, “Optional. Default: 100”[1] . The Kubernetes autoscaler holds every recommendation under the maximum its owner configures. Retry libraries cap attempts, and queues have sizes.

L1 · Bound This blog has recommended limits too, for good reasons. In The Governance Tax an autoscaler ran past the peak of a service’s scalability curve, where each added pod lowers throughput. The fix that post gave was to set “the maximum pod count to 80% of the worst-case” scalability peak, “as a hard ceiling”. The ceiling there removes a real risk, a controller that would otherwise climb into the retrograde region and stay.

L1 · Bound Load shedding is the limit applied at the door. Yanacek describes the practice at a large cloud provider. “In addition to bounding the size of queues, we’ve found it’s extremely important to place an upper bound on the amount of time that an incoming request sits on a queue, and we throw it out if it’s too old”[2] .

Each of these limits does remove a risk. What the belief leaves out is where the risk goes. A limit on one quantity holds that quantity in place and lets the excess flow into the next quantity that has no limit. Whether the limit made the system safer depends on what it pushed the excess into, and on whether anyone is watching that quantity. This post follows the excess through three limits, a replica ceiling, a retry policy and an admission bound, and prices where each one sends it.

Where Each Limit Sends the Excess

Every limit in this series held one quantity in place, and every one of them pushed the excess somewhere else. Followed through the four parts, the destinations are specific.

• The stabilization window of The Delay a Window Demands limited how fast pods could leave. The excess went into held pods after each peak, and from there into a backlog drained below rest, which the window turned into a hunt.
• The replica ceiling limits how many pods the autoscaler may run. When it sits below the amplified need, the excess stays in the queue as retry debt, and the service waits there indefinitely.
• The retry cap limits how many times a client tries. The excess the cap stops is a request that fails for the user instead of costing the service another attempt.
• The admission bound limits how much work the queue holds. The excess becomes refusals at the door, visible as errors to the clients that receive them.

L2 · Fit Read as a list, the destinations differ in one property that matters more than the size of the excess: who can see it. Held pods appear on the cost report a month later. Retry debt appears as CPU at 100% on pods that are serving nobody, which most dashboards read as a busy service. User failures appear in the success rate and in support tickets. Refusals appear as an error rate the service itself reports at the moment it happens. The admission bound sends the excess to the most visible destination available, and that visibility is part of what makes it the right default.

Authority and Time Scale

The first question is who can act during the dead time at all. The answer turns on a quantity this series has used without naming.

Definition 10 -- Authority: the change an actuator can make to the queue's rate of change within a given horizon

Definition 10 (Authority). An actuator’s authority over a queue within a horizon h is the largest change it can make to the queue’s rate of change, $dL/dt$ , by actions that take effect within h. An actuator whose actions all take effect after h has zero authority within h.

Proposition 9 -- Zero Authority in the Dead Time: within one dead time the autoscaler cannot change the queue; only faster actuators can

Proposition 9 (Zero Authority in the Dead Time). Let a pod ordered at time t first serve at $t + \tau$ . Within any horizon shorter than $\tau$ , the autoscaler has zero authority over the queue. The queue then evolves as

$$\frac{dL}{dt} = \lambda_{\text{adm}}(t) - C(t)$$

where:

• $L$ - all accepted work not yet served, including attempts whose clients have gone
• $\lambda_{\text{adm}}(t)$ - the work admitted per second, first attempts and retries alike
• $C(t)$ - the capacity serving at time t, which within the horizon is set entirely by orders placed before it began

and only actuators that change $\lambda_{\text{adm}}(t)$ within the horizon, or capacity committed before it began, can change the outcome.

Proof. No pod ordered inside the horizon serves inside it, so within the horizon C(t) depends only on orders placed before it began. The queue’s rate of change is admitted work minus served work, and only the first term remains open to control. $\square$

L2 · Fit The proposition is almost a definition, and its consequence is not small. Three levers act faster than a pod can start. Clients can send less, by retrying less, and the service can admit less, by refusing work at the door. Both change $\lambda_{\text{adm}}(t)$ . Or the service can need less, by having held spare capacity before the overload began, which raises C(t) for the whole horizon and keeps $\lambda_{\text{adm}}(t) - C(t)$ small. Everything the autoscaler does arrives one dead time later, and by then the outcome may already be fixed. Proposition 8 of Demand Is Not Debt put a number on that. With a 90-second dead time and a ten-second timeout, any sustained overload above about 11% triggers the trap before the first new pod serves.

L2 · Fit The design question therefore has two time scales, and they belong to different mechanisms. On the scale of the dead time, minutes, only admission, retry policy and spare capacity act. On the scale of many dead times, the autoscaler acts, and it is the right tool for matching capacity to demand that has changed for good. A design that asks the autoscaler to handle the first time scale has asked it for authority it does not have.

L2 · Fit The dead time itself can be shortened, and that is the one change that helps every result in this series at once. Its parts are well known: scheduling, pulling an image, starting a process, passing a readiness check. Pre-pulled images, smaller start-up paths and a pool of warm instances each remove part of it. None removes it entirely, and each has a price of its own, usually in idle capacity or operational complexity. Shortening the dead time narrows the interval in which only the fast actuators act. It does not close it, which is why the levers of this post remain necessary however fast the provisioning path becomes.

Three Levers, Three Bills

Three levers act within the dead time. Each one can keep a service out of the trap of the previous post, and each sends its bill to a different place.

• Run colder. A lower utilization target keeps the amplified utilization below one and leaves spare capacity for the first minutes of an overload. The bill is steady-state pods, paid every hour of every day.
• Retry less. A retry budget, a lower cap or a client-side concurrency limiter reduces the amplification. A limiter counts only if no layer above it retries what it rejects, the condition Demand Is Not Debt gave. The bill is paid by the clients, who must honour the policy, and by users whose requests fail rather than being retried.
• Refuse at the door. An admission bound caps how much work the queue holds. The bill is rejected requests during the overload, and it is the only lever the service operator controls without trusting anyone else.

Under the same sustained tenfold step, with a ten-second timeout, two retries and an autoscaler on first attempts, the three levers compare like this:

Lever	Steady pods per pod of demand	Who must cooperate	First-attempt work lost
Target lowered to 25%	4	nobody	338 s
Retry budget of 20%, target 50%	2	every client	249 s
Admission bound of 5 s, target 50%	2	nobody	87 s

Model: 90-second dead time, fluid queue, abandoned work served; lost work in seconds of the stepped demand. Without any lever, the same configuration stays failed.

Definition 11 -- Lever Achievable Region: steady pods held against work lost, one point per lever setting, with its frontier

Definition 11 (Lever Achievable Region). For a given overload, each setting of the three levers maps to the point

$$\Big( 1/u^\ast,\; L_{\text{first}} \Big)$$

whose first coordinate is the steady pods held per pod of demand and whose second is the first-attempt work lost. A setting is on the frontier if no other has both fewer steady pods and less loss.

L3 · Estimate In this region the admission bound dominates both alternatives for this overload. It holds the same two pods per pod of demand as the retry budget and loses about a third as much work. It loses about a quarter as much as the colder target while holding half the pods. The bound is also the only one of the three that the service team can deploy alone. A retry budget is a promise every client must keep, including clients the service team does not own. A colder target is paid for in every hour that is not an overload.

The table compares three levers on one signal. Put every signal of Demand Is Not Debt and every lever of this post on the same axes, and the comparison sharpens. Each run is the same sustained tenfold step with a ten-second timeout. The horizontal axis is the pods held over the hour after the step, per pod of stepped demand at full rate, so that a 50% target sits near two and a 25% target near four. The vertical axis is the first-attempt work lost.

L3 · Estimate Only two points reach the frontier, and both sit behind an admission bound: the first-attempt scaler at 1.98 pods and 87 seconds lost, and the CPU scaler at 1.88 pods and 180 seconds. Every other configuration holds more pods, loses more work, or both. The colder targets buy their protection with capacity, 3.30 and 3.96 pods for 878 and 338 seconds lost. The retry budgets hold the same pods as the bound and lose about three and four and a half times as much. The CPU scaler without a bound holds 4.70 pods and still loses 736 seconds, paying the overshoot of One Bit per Dead Time and the debt of the previous post at once.

L2 · Fit The levers are not exclusive, and the comparison above uses each one alone to show its bill. A service can run a moderate target, ask its clients for a retry budget, and still bound its queue. The point of the comparison is the dependency. The bound keeps working when a client ignores its budget or a new caller arrives with an aggressive retry library. That is the case the other two levers cannot cover.

L3 · Estimate Each lever also has a range, and the ranges explain why the bound wins this comparison. A colder target protects in proportion to how cold it runs. At a 30% first-attempt target the same step recovers after 922 seconds, and at 25% after 384, because each step down moves the amplified utilization further from one. The cost is fixed and permanent: a 25% target holds twice the pods of a 50% target in every hour, overload or not. A retry budget protects in proportion to how small it is: at a 50% target, a 20% budget recovers after 274 seconds and a 50% budget after 422. Its cost falls on users whose failed requests are not retried, and its protection lasts only while every client honours it. The bound protects in proportion to how short it is relative to the timeout, and its cost appears only during an overload, as refusals.

L2 · Fit The colder target deserves one more word, because it is the lever most teams already pull. Spare capacity is capacity held in advance, and it is the only one of the three that acts before the overload begins. The first post of this series, One Bit per Dead Time, priced it as an information rate. The third, Demand Is Not Debt, priced it as distance from the trap. Here it shows up a third time, as the lever that shrinks the excess the dead time produces. Gandhi and three coauthors made the same case for servers with long setup times. In their words, “to successfully meet response time SLAs, it suffices to simply manage existing capacity carefully and not give away spare capacity recklessly”[3] . It is a sound lever with a constant bill. The question this post raises is whether that bill is being paid for protection the admission bound would provide during the few minutes that need it.

The Fast Actuator

An admission bound acts on the term Proposition 9 leaves open. It needs a precise definition, because one parameter of it decides whether it works.

Definition 12 -- Admission Bound: the queue holds at most b seconds of its current capacity; work beyond that is refused at the door and costs no service time

Definition 12 (Admission Bound). A service with an admission bound of b seconds admits a new attempt only if its queue, after admitting it, would hold at most $b \cdot C(t)$ work, where $C(t)$ is the capacity serving at that moment. Refused attempts consume no service time. Their clients may retry, under the same retry policy as for a timeout.

Proposition 10 -- Bounded Debt: an admission bound shorter than the client timeout leaves no retry debt, so the trap cannot hold

Proposition 10 (Bounded Debt). In the fluid model of first-in-first-out service, let the admission bound satisfy $b < \tau_{\text{out}}$ . Then every admitted attempt is served before its client times out, the service performs no abandoned work, and the failed state of Proposition 6 of Demand Is Not Debt cannot be sustained. All excess becomes refused attempts.

Proof. An admitted attempt finds at most $b\,C(t)$ work ahead of it. Capacity does not fall while it waits, so it waits at most b seconds, which is less than $\tau_{\text{out}}$ . No admitted attempt is abandoned, so the retry debt of Definition 7 of Demand Is Not Debt is zero. The trap requires abandoned work to be served at rate $A\lambda$ , which cannot happen. $\square$

L2 · Fit The proposition puts the bound in the unit that matters. A queue limit set in requests, or in seconds unrelated to anything the clients do, protects the service only by accident. A limit set below the client timeout guarantees that every unit of work the service accepts is work someone is still waiting for. Yanacek’s practice bounds the time a request may sit in a queue. The condition here says what that time must be compared against.

Physical translation. A bank branch that lets everyone into the hall, however long the wait, ends the afternoon serving customers who gave up an hour ago and left their tickets on the floor. A branch that shuts the door once the hall holds more than the tellers can clear before anyone runs out of patience turns some people away at the entrance and serves everyone it lets in. The people turned away were going to be failed either way. The door decides only whether the tellers spend the afternoon on them.

L1 · Bound Other mechanisms reach the same goal by different routes. CoDel, specified by Nichols and Jacobson for network queues, drops packets when the minimum queueing delay over an interval stays above a target[4] . Apache HBase applies the idea to remote calls in its AdaptiveLifoCoDelCallQueue, which credits Facebook’s “Fail at Scale” paper in its source[5] . The queue switches to last-in-first-out order once it passes a fill threshold, serving the newest calls, whose clients are most likely still waiting. While the interval’s minimum delay exceeds the target, it drops calls that have waited more than twice the target. Yanacek also describes moving to last-in-first-out service “if the protocol supports it”. Each of these spends capacity on requests whose clients are still waiting, which is what Proposition 10 guarantees for first-in-first-out service with a short enough bound.

L2 · Fit The bound in this post is static: a fixed number of seconds of capacity, checked at the door, with no feedback loop of its own. That matters for a debt this blog has already named. The post Multi-Resource Capacity and the Price of Anarchy noted that an admission policy which refuses work under pressure and resumes it later “is still a control loop with delay in it”. It left that loop’s stability margin unproven. A static bound does not pay that debt, and it does not incur it either. An adaptive shedder, one that adjusts its threshold from measured load, is such a loop and owes that analysis.

What the Bound Buys

L3 · Estimate Put a five-second bound in front of the specimen of the previous post, below its ten-second timeout, and run the same sustained tenfold step with two retries. Without the bound, the CPU-driven autoscaler peaked at twelve and a half times the pods needed and lost 736 seconds of demand. With it, the same autoscaler reaches capacity in the same 345 seconds with no overshoot at all and loses 180 seconds. The first-attempt autoscaler, which without the bound never recovered, reaches capacity in 120 seconds, never overshoots, and loses 87 seconds, the least of any configuration in the series.

L2 · Fit The bound reverses the previous post’s ranking of signals. Without it, the first-attempt signal was the worst choice, because it could not see the debt. With it, there is no debt to see, and the signal’s virtues return: it reacts at once, it does not saturate, and it provisions the need to the pod. The previous post, Demand Is Not Debt, stated the reversal condition as a service that drops abandoned work. An admission bound below the timeout is a service that never accepts work it would later abandon.

L1 · Bound The pairing has a known failure on the CPU side, and Yanacek names it. “Consider the following example: a service is configured for CPU-based reactive scaling and also has load shedding configured to reject requests at a similar CPU target. In this case, the load shedding system will reduce the number of requests to keep the CPU load low, and reactive scaling will never receive or get a delayed signal to launch new instances”. A shedder that keeps the scaler’s own metric in range hides the overload from the scaler.

L2 · Fit The bound in this post sheds by time in queue, so CPU stays saturated behind it and the CPU scaler still scales, slowly. The first-attempt signal escapes Yanacek’s failure by construction. It counts demand before the door, so whatever the door refuses still appears in the signal. The cleanest signal and the fastest actuator need each other: the actuator removes the debt the signal cannot see, and the signal sees the demand the actuator turns away.

L2 · Fit The bound also explains why the CPU scaler’s twelvefold overshoot disappears. In the previous post that overshoot came from the debt. Abandoned attempts kept every pod busy long after first-attempt demand was covered, the metric stayed saturated, and the controller kept multiplying. Behind a bound shorter than the timeout there is no debt. The queue holds at most five seconds of work. Once capacity meets demand the queue empties within seconds, the metric falls below saturation, and the controller stops at the need. The staircase of the first post still sets the time to capacity, at 345 seconds. What the bound removes is everything the staircase did after capacity arrived.

The Ceiling and the Amplified Need

The replica ceiling is the limit the belief trusts most, and the previous post’s trap gives it a floor.

Proposition 11 -- Ceiling Trap: a ceiling below the amplified load keeps a failing service failed; above it, recovery falls as one over the margin

Proposition 11 (Ceiling Trap). Let a service with retry amplification A, demand $\lambda$ and per-pod rate c be limited to M replicas. After a trigger, it stays failed if

$$M \le M^\ast = \frac{A\lambda}{c}$$

whatever its scaling signal. Above that ceiling, the debt drains at $c\,(M - M^\ast)$ , so recovery time falls roughly as $1/(M - M^\ast)$ .

Proof. At the ceiling, capacity is $cM$ and utilization is $\lambda / (cM)$ . Proposition 6 of Demand Is Not Debt traps the service when $A\lambda/(cM) \ge 1$ , which is the stated condition. Above it, the failed state’s backlog falls at the capacity minus the amplified arrival rate, $cM - A\lambda = c(M - M^\ast)$ , and the time to clear a given debt is that debt divided by this rate. $\square$

L3 · Estimate The runs show both halves. With two retries after a tenfold spike, the amplified load needs fifteen pods. Ceilings of fourteen and fifteen never recover. A ceiling of sixteen recovers after 40,293 seconds, more than eleven hours. Twenty takes 7,921 seconds, twenty-five takes 3,925 and forty takes 1,530. Recovery time multiplied by the margin above fifteen stays between 38,250 and 40,293 across those four ceilings, as the proposition predicts. The same law appears at the size of the cluster kit, where the amplified need is six pods. Ceilings of five and six never recover, and seven, eight and twelve recover after 6,974, 3,401 and 1,085 seconds.

Corollary 6 -- Feasible Ceiling Band: a ceiling must exceed the amplified need and stay below the scalability limit; when the amplified need exceeds the limit, no ceiling is safe

Corollary 6 (Feasible Ceiling Band). Combine Proposition 11 with the ceiling rule of The Governance Tax, which places the ceiling at 80% of the worst-case scalability peak $N_{\max}$ . A safe ceiling then satisfies

$$\frac{A\lambda}{c} < M \le 0.8\, N_{\max}$$

If $A\lambda/c \ge 0.8\,N_{\max}$ , the band is empty: every ceiling either traps the service after a trigger or lets the autoscaler climb past the scalability peak.

Proof. The lower bound is Proposition 11. The upper bound is the earlier post’s rule. $\square$

L2 · Fit The corollary reconciles two pieces of advice that look opposed. The Governance Tax was right that a ceiling protects against the retrograde region. Proposition 11 is right that a ceiling below the amplified need creates a second trap. Both hold, and together they say a ceiling is safe only inside a band whose width is set by the amplification. When retries across a stack multiply A until the band closes, no ceiling setting rescues the service. The levers that remain are the ones that lower A or bound the debt, and the admission bound is the one the operator controls.

L3 · Estimate The earlier post’s own example makes the band concrete. Its service had a scalability peak of about 18 pods, so its ceiling sat near 14. With two retries the amplified need is three times the pods that demand alone requires. A service in that example that needs five pods of capacity at full use has an amplified need of fifteen. That is above its ceiling of fourteen, so the band is empty before any setting is chosen. Retries behind a gateway that retries again, an amplification of nine, would close the band for a service needing just two pods. The ceiling rule was correct for the risk it addressed. It was written without the amplification in view, and the amplification decides whether any ceiling can satisfy both risks.

Heavy-Tailed Demand

Overloads in practice are not tenfold or thirtyfold by design. Their sizes vary, and the large ones are rarer than they are large. The Newsvendor Problem Under a Heavy Tail showed what a heavy tail does to a capacity target. The question here is what it does to the four results of this series.

L3 · Estimate Time to capacity stays light-tailed when demand is heavy-tailed. Draw 3,000 overloads from a Pareto distribution with tail index 1.5 and a smallest overload of 1.5 times the base load, capped at 2,000 times, so that one overload in a hundred exceeds thirty times the base load. From the median to the 99th percentile the overload grows about fourteenfold, from 2.3 to 31.3 times the base load. The time until capacity meets it grows less than fivefold, from 105 seconds to 480, because the rounds of One Bit per Dead Time grow with the logarithm of the overload. Behind an admission bound, on the first-attempt signal, the 99th percentile is 135 seconds.

L3 · Estimate A lighter tail changes the numbers and not the pattern. With a tail index of 3, the 99th-percentile overload is 6.9 times the base load. The time until capacity meets it is 240 seconds on CPU and 120 on the first-attempt signal behind a bound. Half of these overloads stay below twice the base load, which a service at a 50% target meets with the capacity it already has. The tail of demand sets the size of what arrives. Because the rounds grow only with the logarithm, the time to meet it stays short under either tail.

L3 · Estimate Size behaves differently: the peak backlog grows in proportion to the overload times its round count. Its local growth exponent falls from 2.84 between 3x and 5x to 1.27 between 100x and 300x. Divided by the overload times the rounds, the backlog settles near 94 seconds of base load, from 35 at 3x through 61 at 10x and 88 at 30x. Since the round count grows like the logarithm of the overload, the backlog grows like the overload times its logarithm.

Proposition 12 -- Tail Index Preserved: a backlog that grows like K log K has the same tail index as the overload K

Proposition 12 (Tail Index Preserved). Let the overload K have a regularly varying tail, $\Pr[K > x] = x^{-\alpha}s(x)$ with s slowly varying, and let the peak backlog satisfy $B \sim \gamma\,K\log K$ for a constant $\gamma > 0$ for large K. Then B has a regularly varying tail with the same index $\alpha$ .

Proof sketch. The function $f(x) = x \log x$ is increasing for large x with inverse $f^{-1}(y) \sim y/\log y$ . So $\Pr[B > y] \sim \Pr[K > y/(\gamma\log y)] = (y/\log y)^{-\alpha}\,s_1(y)$ , where $s_1$ absorbs the constant and is slowly varying, and the factor $(\log y)^{\alpha}s_1(y)$ is slowly varying too. The index of regular variation is $\alpha$ . $\square$

L3 · Estimate The logarithm still matters in practice. A tail-index estimate computed over the overloads that actually occur sees the local growth exponent, not the asymptotic one. Over the practical range that exponent is between about 1.3 and 2.8, so the backlog’s apparent index comes out near the overload’s index divided by it. The Hill estimate on the backlog of the 3,000 overloads is 0.89 against 1.57 for the overloads themselves, roughly 1.5 divided by an exponent of 1.7. The backlog looks heavier-tailed than demand over every range anyone will observe, and it is not heavier in the limit. Both statements are true, and the first is the one an operator lives with.

L3 · Estimate Those two estimates carry uncertainty of their own, and it is worth seeing. Resampled 400 times, with the top tenth of the sample treated as the tail, the overload’s estimate has a 90% interval of 1.46 to 1.72 and the backlog’s an interval of 0.83 to 0.97. The intervals do not overlap, so the gap is not sampling noise. The choice of tail moves both estimates more than resampling does. Treating only the top 2% as the tail raises the backlog’s estimate to 1.23 and the overload’s to 1.79, and treating the top 30% lowers them to 0.65 and 1.51. The backlog’s estimate climbs toward the overload’s as the tail considered gets deeper, which is the logarithm of Proposition 12 fading as the overloads grow.

The Tail Does Not Leave

An admission bound caps the backlog by construction. The question under heavy-tailed demand is what happens to the part of the tail the backlog used to hold.

Definition 13 -- Tail Achievable Region: backlog against refused work in the tail, one point per admission bound, with its frontier

Definition 13 (Tail Achievable Region). For a distribution of overloads, each admission bound b maps to the point

$$\Big( B_{99}(b),\; L_{99}(b) \Big)$$

whose coordinates are the 99th percentiles of peak backlog and of refused or lost work. A bound is on the frontier if no other bound has both a smaller backlog quantile and a smaller loss quantile.

L3 · Estimate The sweep over bound sizes, with the CPU signal, a ten-second timeout and two retries, divides cleanly at the timeout. With no bound, the 99th percentile of peak backlog is 50,563 seconds of base load and the 99th percentile of lost work is 30,905. Most of that backlog is debt: work the service performs for nobody. Bounds longer than the timeout shrink the backlog but leave most of the loss in place, because work still waits past the timeout and the debt still forms: at 12 seconds the loss is still 22,064. Bounds shorter than the timeout collapse the backlog to a few hundred seconds of base load or less and halve the loss: 10,979 at 8 seconds. The change happens between the two bounds that straddle the timeout, not gradually across the sweep. The figure labels every bound.

L3 · Estimate The last number is the one to keep. The same 3,000 overloads, run with no timeouts and no retries at all, leave a 99th-percentile backlog of 11,170 seconds of base load. That is the excess the overload itself brings, with nothing added and nothing refused. A bound below the timeout refuses 11,056 at the same percentile. Its tail index is 0.88, against 0.89 for that unretried backlog, and their bootstrap intervals, 0.82 to 0.96 and 0.83 to 0.97, nearly coincide. The bound removed the debt, which at the 99th percentile accounted for nearly two-thirds of the loss. What remained it moved, almost unit for unit and with the same tail, from the queue to the door.

L3 · Estimate The lighter tail shows the same exchange. With a tail index of 3, the queue without timeouts or retries holds 871 seconds of base load at the 99th percentile. A five-second bound with two retries refuses 845 at the same percentile. The refused volume tracks the unretried backlog in both distributions. The bound changes where the excess sits, and the distribution of demand decides how much of it there is.

L3 · Estimate On Definition 13’s region the sweep has a cliff. The five-second bound has a smaller backlog quantile and a smaller loss quantile than no bound or a bound of sixty, twenty or twelve seconds. Every bound longer than the timeout is therefore dominated by one shorter than it. The frontier lies entirely below the timeout. Among the bounds below it the trade is small and real: two seconds holds about a sixth of the backlog that eight seconds holds and refuses 1.4% more. The choice that matters is which side of the timeout the bound sits on, and that choice is not a matter of degree.

L2 · Fit This is Safe in Probability, Not in Size measured in a different system. That post’s Definition 3 bounded the probability of leaving a safe region of queue occupancy. Its Definition 3a priced a fixed budget split between a probability margin and a magnitude margin. An admission bound is a hard magnitude barrier on the same quantity: the queue can never hold more than b seconds of capacity. It achieves that bound exactly, and the magnitude it refuses to hold reappears outside the region, in refused requests, with the tail it had inside. The barrier makes the queue safe in size. It makes the users safe in neither.

L2 · Fit Time and size answer to different levers, and they are linked. Time can be bought. A shorter dead time makes every round cheaper, a larger growth factor makes rounds fewer, a signal that does not saturate cuts them to one, and the time until capacity meets an overload stays light-tailed under any demand. Size cannot be limited away. The excess an overload brings before capacity arrives exists whatever the door does, and a limit chooses only where it lands: in the queue as latency, at the door as refusals, or in idle pods held for the day it comes. What shrinks the excess is time. Behind the same five-second bound, the first-attempt signal reaches capacity at the 99th percentile in 135 seconds instead of 480, and it refuses 3,535 seconds of base load at that percentile instead of 11,056. Removing the debt and buying time are the two reductions on offer. The first requires a bound shorter than the patience of the clients. The second stops at one dead time, which no signal removes.

What the Bound Does Not Fix

L3 · Estimate The bound does not buy time. Across every bound size in the heavy-tailed sweep, from none to two seconds, the 99th percentile of time to capacity stays at 480 seconds on the CPU signal. That time is set by the rounds of the first post, by the dead time and the growth factor, and the door has no authority over either. What the bound changes is what happens to the work that arrives while those rounds run.

L2 · Fit The bound also depends on knowing the timeout it must stay under. In a chain of services, each caller has its own timeout, and the one that matters is the shortest among the clients whose abandoned work the service would otherwise perform. A bound chosen against a generous outer timeout can still exceed a tight inner one, and then the debt forms from the inner caller’s retries. The bound is a promise about the patience of the clients, and it is only as good as the team’s knowledge of that patience.

L2 · Fit Finally, refusals are real failures. In the heavy-tailed runs, a five-second bound refused 11,056 seconds of base load at the 99th percentile. That is as many requests as the service normally receives in about three hours, turned away during a single overload. The comparison that makes this acceptable is with the alternative: without the bound, the same percentile lost 30,905 seconds of base load to timeouts while serving work nobody wanted. The bound does not create the loss. It removes the part of the loss the service caused itself, and makes the rest visible and immediate.

What to Compute

The four posts of this series reduce to five numbers an engineer can compute before an overload, each from quantities a service already knows.

• The growth factor and its rounds. Take the limit over the request over the target to get g. If g is at most 1 plus the tolerance, the controller will not scale on a saturated metric at all. Otherwise a k-fold overload needs at least $\lceil \log_g (k u^\ast r/\ell) \rceil$ rounds of one dead time plus one metric window each.
• The standing delay. For a backlog-scaled consumer, divide the threshold by the per-pod rate. It must be at least the effective dead time times $\Phi$ of the window over the dead time, and in the idealized loop a standing delay of e dead times is enough after an upward step, whatever the window.
• The threshold of the trap. Multiply retry factors across every layer to get A. A service that completes abandoned work is vulnerable whenever its utilization is at least 1/A, and a scaler on a debt-excluding signal must keep its target below that.
• The ceiling band. A replica ceiling must exceed A times the demand divided by the per-pod rate, and stay below the scalability limit. If it cannot do both, the ceiling is not the lever.
• The bound. An admission bound protects against debt only if it is shorter than the client timeout. Set in seconds of capacity, below the timeout, it turns the trap into refusals and leaves the excess the overload brings.

L2 · Fit None of these numbers requires a simulation, a load test or a tuning session. Each needs a value the team already has: a target, a limit, a dead time, a window, a retry count, a timeout. What they require is the habit of computing them before choosing the settings, rather than after the incident that the settings permitted.

The Binding Constraint, in Sequence

L2 · Fit The Constraint Sequence Framework orders constraints by which binds first, and for the whole series the order is now visible. The dead time binds first, because it sets the interval in which only fast actuators can act and the price of every round after it. The amplification binds second, because it sets the threshold every other setting is measured against and the floor under every ceiling. The admission bound binds third, because it is the only fast actuator the operator owns, and it works only when it is shorter than the timeout. The signal binds fourth. It decides whether the metric ceiling of One Bit per Dead Time binds at all, and with the debt removed the cleanest signal is the best one. The target and the window bind last, because once the signal is chosen they only tune what it does, and they are where most tuning effort goes. This order merges the sequences of the earlier parts without reversing any of them. The dead time led the first two, and it stands ahead of the amplification that led the third because, by Proposition 8 of Demand Is Not Debt, the dead time is what turns a sustained step into a trigger.

L2 · Fit Tuning the autoscaler first is natural, because its settings are the ones in front of the operator. The sequence says it is the last place to look. Consider a team that shortens its dead time, knows its amplification, bounds its queue below its clients’ patience and only then chooses a signal. It has done the work that settings cannot do for it.

Falsification Criteria

The cluster kit for this series supports an admission bound, timeouts and retries. The predictions below come from the fluid model at its size: four pods at a 50% target, a ten-second timeout and two retries.

F1 (the bound removes the overshoot and the loss).

• Condition: a sustained fivefold step on the CPU signal with a replica ceiling of 50, with and without a five-second admission bound.
• Prediction: without the bound, pods climb to the ceiling of 50 and 10.3% of demand is lost; with it, pods peak at 21 and 1.2% is lost.
• Refuted if: the bounded run still overshoots to the ceiling, or loses as much as the unbounded one.

F2 (the ceiling floor).

• Condition: a fivefold spike for 300 seconds, then back, on the CPU signal, with replica ceilings of 6, 7 and 12; the amplified load after the spike needs six pods.
• Prediction: a ceiling of 6 never recovers; 7 recovers after about 6,974 seconds and 12 after about 1,085.
• Refuted if: the ceiling of 6 recovers on its own, or recovery time does not fall roughly as one over the margin above six.

F3 (the bound must be shorter than the timeout).

• Condition: the F1 step with an admission bound of 20 seconds, twice the client timeout, and again with 5 seconds.
• Prediction: the 20-second run still serves abandoned attempts, about 6,262 seconds of base load, climbs to the ceiling of 50 and loses 7.5% of demand; the 5-second run serves none, peaks at 21 pods and loses 1.2%.
• Refuted if: the 20-second bound removes the abandoned work as completely as the 5-second one.

The Ledger of What the Loop Cannot Know

The series kept one artifact across its four parts. Here it is complete.

Size. Belief: the metric measures load. Law: at most one factor of g per round; no scale-up at all inside the tolerance band. Bill: fifteen rounds for a thirtyfold overload at the API’s default target.

Time. Belief: a stabilization window makes the loop stable. Law: rest only if the standing delay is at least $\tau\,\Phi(W/\tau)$ , never more than e dead times, and e is exact. Bill: about ninety seconds of waiting on every message at the default window.

State. Belief: a cleaner signal is a safer one. Law: trapped after a trigger if and only if $A\rho \ge 1$ ; a scaler blind to the debt is trapped at any target of 1/A or more. Bill: with two retries, a target of a third or more turns a tenfold step into a permanent failure at twice the capacity demand needs.

Where the excess goes.

Belief: a limit removes a risk.

Mechanism: during the dead time the autoscaler has no authority; whatever a limit holds back flows into the next quantity without one.

Law: Proposition 10, an admission bound shorter than the client timeout removes the debt. Proposition 11, a replica ceiling at or below $A\lambda/c$ traps the service. Proposition 12, the backlog keeps the overload’s tail index. Layer 1 for the propositions; Layer 3 for the sweeps.

Bill: refused requests carrying the full tail of the overload’s excess, about 11,000 seconds of base load at the 99th percentile on the CPU signal in the heavy-tailed runs and about a third of that on the first-attempt signal, in exchange for a queue that never holds more than five seconds of work.

The Dashboard, Read Again

One Bit per Dead Time opened on a dashboard. Every pod read 100% CPU, the replica count said 20, then 40, then 80, two minutes apart, and the engineer on call could not tell how much load was arriving. The four parts give that screen a reading.

L3 · Estimate The stairs are rounds. Each costs one dead time plus one metric window, and at a 50% target each can at most double the ready pods. A tenfold step needs three of them and meets its load at 345 seconds. The count that kept climbing to 317 pods was the backlog holding the metric at its ceiling after the need had passed. Give the clients behind that service a ten-second timeout and two retries, and the same step leaves debt behind. The count climbs to twelve and a half times the need, and the service is healthy only after 771 seconds. Put a five-second admission bound at the door and the stairs do not change, because the door has no authority over the dead time. The count stops at the need, and the service loses 180 seconds of demand instead of 736.

None of this needed a new controller. It needed knowing which quantity each number on the screen measures, and which quantity it cannot.

What the Four Beliefs Share

L2 · Fit Each belief this series examined is a sound piece of local reasoning. A busier pod does mean more work. A window does stop a noisy metric from discarding warm pods. A demand signal is closer to the cause than CPU. A limit does hold the quantity it limits. In each case the engineer reasoned correctly about the mechanism in front of them. The loop it sat in then moved the problem somewhere the reasoning did not reach.

L2 · Fit What the four have in common is relocation. The saturated metric moved the size of the overload out of the signal and into the number of rounds. The window moved the instability out of the replica count and into the threshold. The clean signal moved the debt out of view without moving it out of the queue. The limit moved the excess from the quantity it held to the next one without a limit. None of these mechanisms removed what it appeared to remove. Each changed where it lived, and in a loop with a dead time in it, where it lives decides who sees it and when.

L2 · Fit The method that found each relocation was the same. Read what the mechanism actually computes, from its source rather than its summary. Write down the quantity it holds fixed. Then follow the excess to the first quantity nobody holds, and price it there. That is slower than tuning a setting, and it gives answers a setting cannot: a number of rounds, a standing delay, a threshold, a band, a bound. The settings follow from those numbers. They do not replace them.

What This Post Did Not Claim

Where Each Claim Holds

Claim. During the dead time the autoscaler has zero authority over the queue.

• Assumption: new pods serve only after the provisioning delay, and no capacity is held warm in advance.
• Failure mode: a warm pool or pre-started pods shorten the effective dead time, which narrows the interval without removing it.

Claim. An admission bound shorter than the client timeout removes the retry debt.

• Assumption: first-in-first-out service at a capacity that does not fall while a request waits, and a bound measured against current capacity.
• Failure mode: capacity that drops suddenly, for example when pods crash under load, can make an admitted request wait longer than the bound.

Claim. A replica ceiling at or below $A\lambda/c$ traps a failing service.

• Assumption: retries and abandoned work as in the previous post.
• Failure mode: client-side circuit breakers or deadline propagation lower the effective amplification and move the floor down.

Claim. The refused volume under a bound below the timeout carries the tail the backlog had.

• Assumption: Pareto overloads with index 1.5, from 1.5 times the base load and capped at 2,000 times, applied as sustained steps, 3,000 trials, with a tail-index estimate on the top tenth.
• Failure mode: overloads that are short spikes rather than sustained steps produce less excess per event, and correlated overloads would change the sample entirely.

Reversal condition. The advice of this post is to bound the queue below the client timeout and to treat ceilings as a band rather than a safety margin. It reverses when refusing work is more expensive than delaying it, for example for writes that cannot be retried or work that must not be lost. There the queue is the right place for the excess, the debt has to be removed by other means, and the only remaining lever is capacity held in advance.

What Is Not Claimed

The propositions of this post are fluid-model statements, and the sweeps that illustrate them use one specimen, sustained steps and Pareto draws. The cliff at the timeout, the ceiling floor and the tail exchange are shown at the sizes tested. Correlated overloads, short spikes and service times with heavy tails could move the numbers, and none of them is in the model.

It has not been claimed that admission control is new. Yanacek, CoDel and the HBase call queue all pursue the same goal. This post adds the condition that the bound must be shorter than the client timeout, and the measurement of what such a bound does to the tail. Nor has it been claimed that a static bound is the best one. An adaptive shedder can do better, and it owes the delay-margin analysis this post does not supply.

The series has also left open which point on a frontier a service should choose. Each achievable region in it sets pods held against what users lose, whether rounds spent short, delay or work refused, and each is drawn without a price for a miss. Choosing a point takes that price, and only the service can set it. The regions also count the users’ side as volume: seconds of demand lost or seconds of delay. A service that manages against the share of requests inside a latency target, or against an error budget, would read the same runs through its own target.

Two open questions are named on purpose. A bound measured against current capacity assumes that capacity does not fall while a request waits, and pods that crash under load break that assumption. And the tail exchange is shown for overloads that arrive as steps; how it behaves for overloads that arrive as bursts is not shown.

One correction belongs here as well. The ceiling rule of The Governance Tax is extended rather than overturned. Its upper bound remains right for the risk it addressed, and this post adds the lower bound that retries impose, which can leave no room between the two.

What the Series Leaves Open

The series has not proved that any loop in it is stable in the sense a control theorist would accept. The laws are derived in fluid models and checked against simulations of the deployed controller, and each part says which steps are proofs and which are numerical. No stochastic barrier theorem with delay compensation is claimed. The delay-margin analysis that an adaptive admission loop would need, named in an earlier series, is still unpaid. Last-in-first-out service, deadline propagation, heavy-tailed service times and several services in a chain are all outside the models. And no claim here has yet met a real cluster. The falsification criteria in each part are written so that it can, and so that a refutation would show.

An autoscaler adds capacity for the load it can measure, and an overload is the load it cannot. The four parts of this series measured what that sentence costs. The price is a probe of one bit per dead time and a standing delay of one dead time per message. It also includes a debt that a clean signal cannot see, and an excess that a limit can only relocate. The capacity always arrives in arrears. What a design chooses is where the arrears are held while it does.

Read the Cognitive Map as plain text

Authority

1. No pod ordered within a dead time serves within it, so the queue follows admitted work minus the capacity already serving. Only clients, admission and capacity held in advance can act.
2. A 25% target holds 4 pods per pod of demand and loses 338 s; a 20% retry budget needs every client and loses 249 s; a 5 s admission bound holds 2 pods and loses 87 s.
3. An admission bound shorter than the client timeout means every admitted attempt is served in time, so no debt forms and the trap cannot hold; CoDel and adaptive LIFO pursue the same goal.

Limits

4. Behind a 5 s bound the CPU scaler loses its twelvefold overshoot and the first-attempt scaler, trapped without it, loses only 87 s: with no debt, the clean signal is best.
5. A replica ceiling at or below $A\lambda/c$ traps a failing service; above it recovery falls as one over the margin, 40,293 s at a margin of one pod and 1,530 s at twenty-five.
6. A ceiling must exceed the amplified need and stay below 80% of the scalability peak; when amplification closes that band, no ceiling is safe.

The Tail

7. Across Pareto overloads with index 1.5, the 99th-percentile overload is 31.3 times base, yet time to capacity is 480 s on CPU and 135 s on the first-attempt signal behind a bound: time grows with the logarithm of size.
8. Peak backlog grows like $K \log K$ , so its tail index equals the overload’s, while looking heavier over any observed range: an estimate of 0.89 against 1.57.
9. On the CPU signal, a bound below the timeout removes the debt and then refuses 11,056 s of base load at the 99th percentile, almost exactly the 11,170 a queue without retries would hold, with the same tail.

Compute it. Write down your dead time, your target and limit, your downscale window, your retry counts across every layer, your client timeout and your replica ceiling. From them compute g and the rounds for your largest overload, the standing delay your queue consumers need, 1/A, the ceiling band, and whether your admission bound is shorter than your timeout. If any of the five fails, the next overload has already been decided.

---

^[1] KEDA Authors (2026). KEDA documentation v2.19: ScaledObject specification. keda.sh (kedacore/keda-docs).

^[2] Yanacek, D. (2019). Using load shedding to avoid overload. Amazon Builders’ Library.

^[3] Gandhi, A., Harchol-Balter, M., Raghunathan, R. & Kozuch, M.A. (2012). AutoScale: Dynamic, Robust Capacity Management for Multi-Tier Data Centers. ACM Transactions on Computer Systems 30(4), Article 14.

^[4] Nichols, K. & Jacobson, V. (2018). Controlled Delay Active Queue Management. RFC 8289, IETF.

^[5] Apache HBase Authors (2026). AdaptiveLifoCoDelCallQueue. hbase-server, Apache HBase source (apache/hbase, master branch).

---

Back to top