![\"\"](\"https:\/\/www.infoq.com\/articles\/engineering-speed-scale\/articles\/engineering-speed-scale\/en\/resources\/170figure-1-1767008190654.jpg\")
<\/p>\n\t\t\t\t\t\t\t\t\tKey Takeaways
\n\t\t\t\t\t\t\t\t\t
\n\tTreat latency as a first-class product concern \u2014 designed with the same discipline as security and reliability.
\n\tUse a latency budget to turn “sub-100ms” into enforceable constraints across every hop in the request path.
\n\tExpect speed to regress unless you actively guard it as the system, traffic, and dependencies evolve.
\n\tKeep performance ownership broad by baking it into reviews, dashboards, and release practices \u2014 not a single “performance team.”
\n\tLet architecture create the fast path, and let culture (measurement + accountability) keep it fast over time.
\n\t\u00a0
<\/p>\n
\t\t\t\t\t\t\t\tThe Cost of a Millisecond: Why Latency Shapes Experience<\/p>\n
When we talk about API performance, it\u2019s tempting to think in neat technical terms – response times, CPU cycles, connection pools, and the occasional flame graph. But in real-world systems, especially global commerce and payments platforms, latency has a very human cost. A delay of just 50 or 100 milliseconds rarely registers in isolation, but at scale it can nudge a customer away from completing a purchase, disrupt a payment flow, or simply chip away at the trust users place in your product.<\/p>\n
Speed shapes perception long before it shapes metrics. Users don\u2019t measure latency with stopwatches – they feel it. The difference between a 120 ms checkout step and an 80 ms one is invisible to the naked eye, yet emotionally it becomes the difference between “smooth” and “slightly annoying”. On a small scale, that\u2019s forgettable. Across millions of sessions, it becomes the friction that compacts into lower conversion rates, abandoned carts, and reduced revenue. And the irony? The engineering effort needed to recover from that friction – new features, experiments, retention strategies – often dwarfs the work needed to prevent it in the first place.<\/p>\n
<\/p>\n
In high-throughput platforms, latency amplifies. If a service adds 30 ms in normal conditions, it might add 60 ms during peak load, then 120 ms when a downstream dependency wobbles. Latency doesn\u2019t degrade gracefully; it compounds. And once your tail latency (p95, p99) drifts, it silently “taxes” every upstream service that depends on you. Each service adds its own jitter, serialization overhead, and network hop. What starts as a tiny bump in one API becomes a cascading slowdown across dozens of interconnected services.<\/p>\n
This is why high-performing architecture teams treat speed as a product feature, not a pleasant side effect. They design for latency the same way they design for security and reliability: intentionally, with clear budgets, well-defined expectations, and patterns that protect the user experience under stress.<\/p>\n
A helpful way to see this is through a “latency budget”. Instead of thinking about performance as a single number – say, “API must respond in under 100 ms” – modern teams break it down across the entire request path:<\/p>\n
\n\t10 ms at the edge
\n\t5 ms for routing
\n\t30 ms for application logic
\n\t40 ms for data access
\n\t10\u201315 ms for network hops and jitter
<\/p>\n
Each layer is allocated a slice of the total budget. This transforms latency from an abstract target into a concrete architectural constraint. Suddenly, trade-offs become clearer: “If we add feature X in the service layer, what do we remove or optimize so we don\u2019t blow the budget?” These conversations – technical, cultural, and organizational – are where fast systems are born.<\/p>\n
The heart of this article is simple: low latency isn\u2019t an optimization – it\u2019s a design outcome. It emerges from the choices we make about data locality, async vs. sync flows, cache boundaries, error isolation, and observability. Achieving sub-100 ms is possible for many systems, but sustaining it under load takes alignment across engineering, product, and operations.<\/p>\n
In the sections that follow, we\u2019ll break down how real systems are structured, how engineering teams make trade-offs when milliseconds matter, and how organizations sustain performance long after the first release ships. Fast systems don\u2019t happen accidentally – they\u2019re engineered with intent.<\/p>\n
Inside the Fast Lane: How Low-Latency Systems Are Structured<\/p>\n
Before we talk about optimizing performance, we need to zoom out and understand what a low-latency system actually looks like. Sub-100 ms responses don\u2019t come from a single clever trick; they emerge from a carefully orchestrated pipeline of components that work together with minimal friction. Think of it less as “making one thing fast” and more as “removing unnecessary steps from an entire journey”.<\/p>\n
Most modern systems – especially in commerce and payments – follow a layered architecture that looks deceptively simple from the outside: a client makes a request, it hits an API gateway, flows through a service layer, talks to a database, and returns. But behind that simple flow is an intricate chain where every hop, every serialization, every cache hit or miss shapes the user\u2019s experience.<\/p>\n
Let\u2019s walk through the anatomy of a fast system and where milliseconds typically hide.<\/p>\n
The Request Journey: Where Latency Sneaks In<\/p>\n
A typical sub-100 ms request flow might look like this:<\/p>\n
\n\tClient \u2192 CDN or Edge Network
\n\tThe closest node absorbs the request and routes it smartly.<
\n\tLatency target: 5\u201315 ms
\n\tEdge \u2192 API Gateway
\n\tAuthentication, routing, throttling.
\n\tLatency target: 5 ms
\n\tGateway \u2192 Service Layer
\n\tBusiness logic, orchestration, fan-out.
\n\tLatency target: 20\u201330 ms
\n\tService Layer \u2192 Data Layer
\n\tReads from databases, caches, or search systems.
\n\tLatency target: 25\u201340 ms
\n\tService \u2192 Gateway \u2192 Client
\n\tSerialization and network hop back.
\n\tLatency target: 5\u201310 ms
<\/p>\n
When done right, the entire hop chain stays predictable – even during peak load. But if any single hop drifts, the whole chain inherits the slowdown. This is why fast systems start with understanding the complete journey, not just the part you own.<\/p>\n
![\"\"](\"https:\/\/www.infoq.com\/articles\/engineering-speed-scale\/articles\/engineering-speed-scale\/en\/resources\/131figure-2-1767008190654.jpg\")
<\/p>\n
Where Latency Really Comes From (Not Where You Expect)<\/p>\n
Latency is rarely caused by “slow code”. In production systems, it usually comes from:<\/p>\n
1. Network Hops<\/p>\n
Each hop adds cost:<\/p>\n
\n\tTLS handshakes
\n\tConnection pool waits
\n\tDNS lookups
\n\tRegion-to-region travel
<\/p>\n
Shaving one hop often saves more latency than rewriting 100 lines of Java.<\/p>\n
2. Serialization & Payload Size<\/p>\n
JSON serialization\/deserialization is more expensive than people realize. Every unnecessary field is extra work. Binary formats (e.g., Protobuf) can help, but they add operational overhead.<\/p>\n
3. Cold Caches<\/p>\n
A cache miss at the wrong time can double or triple service latency. This is why “warming strategies” matter when deploying new versions.<\/p>\n
4. Database Query Shape<\/p>\n
Database latency is often an access-pattern problem: query shape, indexes, and cardinality matter.A poorly indexed read can turn a 10 ms query into a 120 ms spike.Multiply that across thousands of requests per second, and tail latency explodes.<\/p>\n
5. Dependent Services<\/p>\n
This is where latency becomes unpredictable: If your service calls three downstreams, your response time is often gated by the slowest one.<\/p>\n
This is why async fan-out, caching, and circuit breakers become critical. (We\u2019ll go deeper soon)<\/p>\n
The Latency Budget: Your Most Important Architectural Tool<\/p>\n
High-performing engineering teams don\u2019t just “measure latency”; they budget it. A latency budget is like a financial budget: Everyone gets a fixed allowance, and nobody is allowed to overspend.<\/p>\n
A typical 100 ms budget might look like:<\/p>\n
\n\t
\n\t\t
\n\t\t\tLayer
\n\t\t\tBudget (ms)
\n\t\t
\n\t\t
\n\t\t\tEdge\/CDN
\n\t\t\t10
\n\t\t
\n\t\t
\n\t\t\tGateway
\n\t\t\t5
\n\t\t
\n\t\t
\n\t\t\tService Logic
\n\t\t\t30
\n\t\t
\n\t\t
\n\t\t\tDatabase \/ Cache
\n\t\t\t40
\n\t\t
\n\t\t
\n\t\t\tNetwork Jitter
\n\t\t\t10
\n\t\t
\n\t
<\/p>\n
Budgets make performance manageable and negotiable. Engineers can now ask:<\/p>\n
<\/p>\n
“If we add feature X, which layer gives up its milliseconds?”<\/p>\n
<\/p>\n
Without budgets, performance conversations become vague and subjective.<\/p>\n
Why Understanding the System Structure Matters<\/p>\n
Everything we cover in later sections – async fan-out, caching hierarchies, circuit breakers, fallback strategies – only makes sense when you understand the system anatomy. Optimizing a single service without understanding the ecosystem is like upgrading a car engine but ignoring the wheels, brakes, and fuel system.<\/p>\n
Fast systems share these traits:<\/p>\n
\n\tFewer hops
\n\tAggressive local caching
\n\tPredictable data-access paths
\n\tParallelism over serial execution
\n\tIsolation of slow components
\n\tStable tail latency under load
<\/p>\n
With the system anatomy clear, we can now move into the engineering playbook – how to actually make these systems fly.<\/p>\n
The Engineering Playbook: Trade-Offs That Keep APIs Lightning-Fast<\/p>\n
Engineering for low latency is really engineering for predictability. Fast systems aren\u2019t built through micro-optimizations – they\u2019re built through a series of deliberate, layered decisions that minimize uncertainty and keep tail latency under control. This section breaks down the actual patterns, trade-offs, and guardrails used in high-throughput systems.<\/p>\n
Async Fan-Out: Parallelism Without Pain<\/p>\n
Slow APIs often boil down to one root cause: serial dependencies.<\/p>\n
If your system performs three downstream calls at 40 ms each, you\u2019ve already lost 120 ms without doing any real business work.<\/p>\n
Fan out in parallel<\/p>\n
Java\u2019s CompletableFuture is a natural fit, especially when paired with a custom executor tuned for downstream concurrency:<\/p>\n
\nExecutorService pool = new ThreadPoolExecutor(
\n\u00a0 \u00a0 \u00a0 \u00a0 20, 40, 60, TimeUnit.SECONDS,
\n\u00a0 \u00a0 \u00a0 \u00a0 new LinkedBlockingQueue<>(500),
\n\u00a0 \u00a0 \u00a0 \u00a0 new ThreadPoolExecutor.CallerRunsPolicy()
\n);
\n
\nCompletableFuture profileFuture =
\n\u00a0 \u00a0 \u00a0 \u00a0 CompletableFuture.supplyAsync(() -> profileClient.getProfile(userId), pool);
\n
\nCompletableFuture> recsFuture =
\n\u00a0 \u00a0 \u00a0 \u00a0 CompletableFuture.supplyAsync(() -> recClient.getRecs(userId), pool);
\n
\nCompletableFuture orderFuture =
\n\u00a0 \u00a0 \u00a0 \u00a0 CompletableFuture.supplyAsync(() -> orderClient.getOrders(userId), pool);
\n
\nreturn CompletableFuture.allOf(profileFuture, recsFuture, orderFuture)
\n\u00a0 \u00a0 \u00a0 \u00a0 .thenApply(v -> new HomeResponse(
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 profileFuture.join(),
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 recsFuture.join(),
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 orderFuture.join()
\n\u00a0 \u00a0 \u00a0 \u00a0 ));<\/p>\n
But here\u2019s the caution most articles never mention:<\/p>\n
Async code doesn\u2019t eliminate blocking – it just hides it inside a thread pool.<\/p>\n
If your executor is misconfigured, you can trigger:<\/p>\n
\n\tCPU thrashing
\n\tThread contention
\n\tQueue buildup
\n\tOut-of-memory errors
\n\tCascading slowdowns across the entire fleet
<\/p>\n
Thread Pool Rule of Thumb:<\/p>\n
For downstream IO-bound calls, size your pool to:
\n2\u00d7CPU cores \u00d7 expected parallel downstream calls per request
\n(adjust using p95\/p99 load testing)<\/p>\n
![\"\"](\"https:\/\/www.infoq.com\/articles\/engineering-speed-scale\/articles\/engineering-speed-scale\/en\/resources\/111figure-3-1767010410236.jpg\")
<\/p>\n
Multi-Level Caching: The Art of Fast Paths<\/p>\n
Fast systems don\u2019t eliminate work – they avoid doing the same expensive work repeatedly.<\/p>\n
A typical hierarchy:<\/p>\n
\n\tLocal cache (Caffeine) – sub-millisecond
\n\tRedis cache – 3\u20135 ms
\n\tDatabase – 20\u201360+ ms
<\/p>\n
Use the dual-level caching pattern. In this example, Redis uses a 10-minute TTL, and the local in-memory cache should also be time-bounded (usually shorter), otherwise it can quietly become a “forever cache” and serve stale data across instances.<\/p>\n
\npublic ProductService(RedisClient redis, ProductDb db) {
\n\u00a0 \u00a0 this.redis = redis;
\n\u00a0 \u00a0 this.db = db;
\n\u00a0 \u00a0 this.localCache = Caffeine.newBuilder()
\n\u00a0 \u00a0 \u00a0 \u00a0 .maximumSize(50_000)
\n\u00a0 \u00a0 \u00a0 \u00a0 .expireAfterWrite(Duration.ofMinutes(1)) \/\/ shorter than Redis
\n\u00a0 \u00a0 \u00a0 \u00a0 .build();
\n\u00a0 }
\n
\npublic ProductInfo getProductInfo(String productId) {
\n\u00a0 \u00a0 ProductInfo local = localCache.getIfPresent(productId);
\n\u00a0 \u00a0 if (local != null) return local;
\n
\n\u00a0 \u00a0 ProductInfo redisValue = redis.get(productId);
\n\u00a0 \u00a0 if (redisValue != null) {
\n\u00a0 \u00a0 \u00a0 \u00a0 localCache.put(productId, redisValue);
\n\u00a0 \u00a0 \u00a0 \u00a0 return redisValue;
\n\u00a0 \u00a0 }
\n
\n\u00a0 \u00a0 ProductInfo dbValue = db.fetch(productId);
\n
\n\u00a0 \u00a0 redis.set(productId, dbValue, Duration.ofMinutes(10));
\n\u00a0 \u00a0 \/\/ localCache is configured with expireAfterWrite(1, MINUTES)
\n\u00a0 \u00a0 localCache.put(productId, dbValue);
\n\u00a0 \u00a0 return dbValue;
\n}
\n
<\/p>\n
This drives most requests into the fast path and reserves slow work for the cold path.<\/p>\n
![\"\"](\"https:\/\/www.infoq.com\/articles\/engineering-speed-scale\/articles\/engineering-speed-scale\/en\/resources\/76figure-4-1767010410236.jpg\")
<\/p>\n
Cache Invalidation: The Hardest Problem in Computer Science (Still True)<\/p>\n
Low-latency systems are heavily cache-driven, but caching without a clear invalidation strategy is a time bomb.<\/p>\n
There are three kinds of invalidation:<\/p>\n
\n\t
\n\t\t
\n\t\t\tCache invalidation style
\n\t\t\tPros
\n\t\t\tCons
\n\t\t
\n\t\t
\n\t\t\t1. Time-based (TTL)
\n\t\t\tSimple, safe, widely used
\n\t\t\tstale data risk increases with longer TTLs
\n\t\t
\n\t\t
\n\t\t\t2. Event-based
\n\t\t\tProducer sends “invalidate” events to downstream caches whenever data changes
\n\t\t\tRequires strong data ownership
\n\t\t
\n\t\t
\n\t\t\t3. Version-based
\n\t\t\tCache keys include a version: product:v2:12345
\n\t\t\tWhen you bump the version, old data becomes unreachable
\n\t\t
\n\t
<\/p>\n
There is no universally “best” invalidation strategy. The right choice depends on how often data changes and how costly staleness is – which is exactly why classification matters.<\/p>\n
Data Classification: Not Everything Belongs in Cache<\/p>\n
This is the part almost every caching article ignores, but real systems cannot.<\/p>\n
You cannot treat all data as equal. Before caching anything, classify the data:<\/p>\n
\n\t
\n\t\t
\n\t\t\tClassification
\n\t\t\tWhat it means
\n\t\t\tCaching guidance
\n\t\t\tExamples
\n\t\t
\n\t\t
\n\t\t\tPublic
\n\t\t\tSafe to cache anywhere (CDN, Redis, local memory).
\n\t\t\tCache freely. TTL based is usually fine.
\n\t\t\tproduct titles, images, metadata
\n\t\t
\n\t\t
\n\t\t\tInternal
\n\t\t\tCacheable with restrictions.
\n\t\t\tCache with guardrails (scope, TTL, access control).
\n\t\t\tinternal IDs, flags
\n\t\t
\n\t\t
\n\t\t\tConfidential (PII)
\n\t\t\tSensitive user data.
\n\t\t\tCan only be cached if encrypted and with strict TTL.
\n\t\t\temail, phone, full user info
\n\t\t
\n\t\t
\n\t\t\tRestricted (PCI)
\n\t\t\tHighly regulated payment data.
\n\t\t\tNever cached.
\n\t\t\traw card numbers, CVV, unmasked PAN
\n\t\t
\n\t
<\/p>\n
When to be strict vs. loose?<\/p>\n
The caching strategy depends on the type of data\u2026 For example:<\/p>\n
\n\tProduct catalog \u2192 loose TTL is fine (staleness OK)
\n\tPricing, offers \u2192 tighter TTL or event-based
\n\tPayments, balances \u2192 never cached, or only tokenized\/aggregated versions
<\/p>\n
A simple classification check can protect engineering teams from accidental compliance violations.<\/p>\n
\nif (data.isRestricted()) {
\n\u00a0 \u00a0 throw new UnsupportedOperationException(“Cannot cache PCI\/PII data”);
\n}<\/p>\n
Circuit Breakers: Don\u2019t Let Slow Dependencies Infect Your Tail Latency Downstream<\/p>\n
Slowness is one of the biggest drivers of p99 spikes. A dependency doesn\u2019t need to be fully down to cause trouble – sustained latency is enough. If every request waits on a degrading downstream call, you start consuming threads, building queues, and turning a local slowdown into a broader tail-latency problem.<\/p>\n
A circuit breaker helps by acting as a boundary between your service and an unstable dependency. When errors or timeouts cross a threshold, the breaker opens and temporarily stops sending traffic there. That shifts the system from “wait and accumulate” to a predictable outcome: fail fast and fall back, keeping your own API responsive.<\/p>\n
Resilience4j gives lightweight protection:<\/p>\n
\nCircuitBreakerConfig config = CircuitBreakerConfig.custom()
\n\u00a0 \u00a0 \u00a0 \u00a0 .failureRateThreshold(50)
\n\u00a0 \u00a0 \u00a0 \u00a0 .slidingWindowSize(20)
\n\u00a0 \u00a0 \u00a0 \u00a0 .waitDurationInOpenState(Duration.ofSeconds(5))
\n\u00a0 \u00a0 \u00a0 \u00a0 .build();
\n
\nCircuitBreaker cb = CircuitBreaker.of(“recs”, config);
\n
\nSupplier> supplier =
\n\u00a0 \u00a0 \u00a0 \u00a0 CircuitBreaker.decorateSupplier(cb, () -> recClient.getRecs(userId));
\n
\ntry {
\n\u00a0 \u00a0 return supplier.get();
\n} catch (Exception ex) {
\n\u00a0 \u00a0 return Collections.emptyList(); \u00a0\/\/ fast fallback
\n}
\n
<\/p>\n
When the breaker opens:<\/p>\n
\n\tCalls fail fast (<1 ms)
\n\tNo threads are blocked
\n\tYour API stays stable
<\/p>\n
Fallbacks: When “Fast and Partial” Beats “Slow and Perfect”<\/p>\n
Fallbacks keep your fast path intact when a dependency is slow or unavailable. The point isn\u2019t to pretend nothing happened – it\u2019s to stop downstream slowness from consuming your latency budget. In many user flows, a slightly degraded response delivered quickly is better than a perfect response delivered late.<\/p>\n
Fallbacks should:<\/p>\n
\n\tProvide something useful
\n\tBe predictably fast
\n\tNot cause additional load
\n\tBe easy to reason about
<\/p>\n
Timeouts are part of the design. If a downstream timeout is “a few seconds”, it can quietly destroy a sub-100ms target. Timeouts need to align with the latency budget you set earlier and the dependency\u2019s p95\/p99 behavior – especially in fan-out paths where one slow call can dominate tail latency.<\/p>\n
Here\u2019s an example that returns a cached snapshot if the full page can\u2019t be assembled quickly. This only works because it builds on the caching approach discussed earlier – another reminder that low latency is holistic (budgets, caching, timeouts, and resilience patterns working together):<\/p>\n
\npublic ProductPageResponse getPage(String productId) {
\n\u00a0 \u00a0 try {
\n\u00a0 \u00a0 \u00a0 \u00a0 return fetchFullPage(productId);
\n\u00a0 \u00a0 } catch (TimeoutException e) {
\n\u00a0 \u00a0 \u00a0 \u00a0 return fetchCachedSnapshot(productId); \u00a0\/\/ warm, minimal, safe
\n\u00a0 \u00a0 }
\n}<\/p>\n
Fallbacks don\u2019t eliminate failures – they bound the user impact when things get slow.<\/p>\n
Data Partitioning: Reducing Hotspots and Tail Spikes<\/p>\n
Partitioning reduces lock contention, narrows index scans, and improves locality.<\/p>\n
Here is a simple example where data is partitioned by region:<\/p>\n
\nCREATE TABLE orders_us PARTITION OF orders FOR VALUES IN (‘US’);
\nCREATE TABLE orders_eu PARTITION OF orders FOR VALUES IN (‘EU’);<\/p>\n
The application layer needs corresponding updates to use the partition effectively:<\/p>\n
\nString table = region.equals(“US”) ? “orders_us” : “orders_eu”;
\nreturn jdbc.query(“SELECT * FROM ” + table + ” WHERE user_id=?”, userId);<\/p>\n
Partitioning is essential for read-heavy API systems.<\/p>\n
Observability: Making Speed Measurable<\/p>\n
Fast systems are not just the result of good architecture – they\u2019re the result of relentless observability. Latency budgets, circuit breakers, caching layers, thread pools\u2026 none of them matter if you don\u2019t know when and where your system drifts under real traffic.<\/p>\n
The biggest myth about low latency is that once you achieve it, you\u2019re done. The truth is the opposite:<\/p>\n
Speed decays unless you actively guard it.<\/p>\n
This is why high-performing engineering teams treat observability as a first-class citizen – not a debugging tool, but a continuous performance governance mechanism.<\/p>\n
Measure What Matters: p50, p95, p99, and Beyond<\/p>\n
Most dashboards proudly show average latency, which is almost useless in distributed systems. What users actually feel is tail latency:<\/p>\n
\n\tp50 \u2192 “typical user”
\n\tp95 \u2192 “slightly unlucky user”
\n\tp99 \u2192 “customer who will abandon your product if this happens too often”
<\/p>\n
If your p50 is 45 ms but your p99 is 320 ms, your system isn\u2019t fast – it\u2019s merely nice sometimes.<\/p>\n
Fast systems tune for predictability, not just averages.<\/p>\n
Instrumentation with Micrometer<\/p>\n