Blog on Chris Kennelly

Performance Tip of the Week #99: Illuminating the processor core with llvm-mca

Tue, 07 Oct 2025 00:00:00 +0000

The RISC versus CISC debate ended in a draw: Modern processors decompose instructions into micro-ops handled by backend execution units. Understanding how instructions are executed by these units can give us insights on optimizing key functions that are backend bound. In this episode, we walk through using llvm-mca to analyze functions and identify performance insights from its simulation.

Preliminaries: Varint optimization

llvm-mca, short for Machine Code Analyzer, is a tool within LLVM. It uses the same datasets that the compiler uses for making instruction scheduling decisions. This ensures that improvements made to compiler optimizations automatically flow towards keeping llvm-mca representative. The flip side is that the tool is only as good as LLVM’s internal modeling of processor designs, so certain quirks of individual microarchitecture generations might be omitted. It also models the processor behavior statically, so cache misses, branch mispredictions, and other dynamic properties aren’t considered.

Performance Tip of the Week #98: Measurement has an ROI

Fri, 03 Oct 2025 00:00:00 +0000

Effectively measuring optimization projects is an important part of the lifecycle of an optimization. Overlooking a large positive (or negative) externality can cause us to make the wrong decisions for choosing our next steps and future projects. Nevertheless, this quest for accuracy needs to be balanced against the ROI from a better measurement. In this episode, we discuss strategies for deciding when to invest more time in measuring a project and when to move on.

Performance Tip of the Week #97: Virtuous ecosystem cycles

Thu, 27 Nov 2025 00:00:00 +0000

Software ecosystems aim to maximize qualities like efficiency, correctness, and reliability while minimizing the costs of achieving these properties. Improving a single service through customization can help build an optimization more expediently, but it has an inherent limited scope to its upside and increases technical debt. A point solution fails to provide the full benefits from applying features horizontally. In this episode, we discuss how lessons learned from partnerships to improve individual applications can improve efficiency for everyone.

Performance Tip of the Week #95: Spooky action at a distance

Mon, 14 Jul 2025 00:00:00 +0000

Shared resources can cause surprising performance impacts on seemingly unchanged software. In this episode, we discuss how to anticipate these effects and externalities.

Normalization techniques

Workload changes can confound longitudinal analysis: If you optimize a library like protocol buffers, does spending more time in that code mean your optimization didn’t work or that the application now serves more load?

A/B tests can control for independent variables. Nevertheless, load balancing can throw a wrench into this. A client-side load balancing algorithm (like Weighted Round Robin) might observe the better performance of some tasks and send more requests to them.

Performance Tip of the Week #94: Decision making in a data-imperfect world

Fri, 27 Jun 2025 00:00:00 +0000

Profiling tools are vital to narrowing the search space of all possible changes we could make to highlight the best uses of our time. In this episode, we discuss strategies for identifying when we might want to seek out more data and when we are in the midst of diminishing returns.

Focusing on outcomes

The ultimate outcomes that we want from our tools are insights that let us improve performance.

Performance Tip of the Week #93: Robots never sleep

Tue, 03 Jun 2025 00:00:00 +0000

Techniques like presubmits are essential tools for effective software engineering. For performance optimization, changing observable but unspecified behaviors can be a large source of opportunities. In this episode, we discuss strategies for leaning on automation and additional tools to make it easier to evolve software.

Prevent problems via technical means

Use technical means, rather than (eventually fallible) humans, to prevent problems. Humans can be a valuable line of defense for unknown unknowns, but they cannot be the only line of defense for everything. While human-executed checklists are great, they also have to be short and that makes them unsuitable for encoding a bunch of knowledge about a problem domain. It can be tempting to add to the checklist after every incident and never subtract from it, but this leads to inevitable toil or skipped steps. Robots never get tired.

Performance Tip of the Week #90: How to estimate

Thu, 06 Feb 2025 00:00:00 +0000

Estimating savings can be useful for improving our decision making. In this episode, we discuss how to make and use estimates in the optimization lifecycle.

Why estimate?

While looking for and developing optimizations, we use performance estimates frequently to decide how to approach problems and spend time:

A sense of how large a problem domain is might tell us where to look for opportunities in the first place.
Given finite time and multiple projects to work on, we will prioritize those with the highest return on investment. Estimation lets us fill in a guess for “return,” before we have it in hand. Our goal is to make better decisions, not perfect-in-hindsight ones.
Within a specific optimization project, an estimate of the benefit might inform us how much complexity (fiddly edge cases, technical debt, etc.) we might be willing to tolerate.

The outcome of better estimates is better decisions, which informs us how much precision we might need. If we’re considering project A that will make things 5% faster and project B that will make things 0.1% faster, we don’t need to worry about additional significant figures for project A’s estimate. A more precise estimate for project A of 5.134% won’t change our prioritization all things being equal (effort, complexity, etc.). In that situation, we should instead prefer to focus on information that could change our decision rather than gathering unneeded precision that won’t.

Performance Tip of the Week #88: Measurement methodology: Avoid the jelly beans trap

Mon, 18 Nov 2024 00:00:00 +0000

Measurement of performance optimizations often requires in-depth analysis due to the inherent stochasticity of workloads. We need to gather data to calculate the metrics we’ve selected. In this episode, we discuss the importance of defining your measurement methodology ahead of time rather than fishing for possible sources of significance after the data has been collected.

Choose a methodology, without peeking

Choose and publish a methodology before looking at the data. Ideally, this process is completed before any changes land. It is otherwise easy to pick the methodology that tells the best story (the “biggest number”) unwittingly after the fact. Preregistration of experiment methodology also helps avoid false positives in statistical analysis.

Performance Tip of the Week #87: Two-way doors

Fri, 08 Nov 2024 00:00:00 +0000

Jeff Bezos divides decisions between “one-way doors”–ones that are hard to reverse–and “two-way doors”–those that are easy to reverse. Different optimizations fall on each side of this divide. In this episode, we discuss patterns common to two-way doors to reduce risk without exhaustively analyzing the situation. Good decisions endure, while missteps can be corrected along the way.

Assessing reversibility

As we explore a new optimization idea, we want to prioritize blockers to landing and ignore (for now) less important details. This is easier said than done, since we need to figure out which subproblems are actually on our critical path and those which can be ignored.

Performance Tip of the Week #83: Reducing memory indirections

Wed, 04 Sep 2024 00:00:00 +0000

Memory indirections are a frequent cause for latency and memory bandwidth bottlenecks. By forcing the processor to follow pointers to get to the useful data it needs, our programs incur slowdowns and require more memory bandwidth than they might need from a more efficient layout. In this episode, we discuss tools for identifying inefficient data structures and improving them.

Latency and throughput

In Google’s fleet, our processors spend 40-50% of their time waiting for data coming from their caches or main memory. Memory access latency and throughput are inextricably linked:

Performance Tip of the Week #79: Make at most one tradeoff at a time

Wed, 04 Sep 2024 00:00:00 +0000

Developing and enabling optimizations can often involve tradeoffs: using more RAM and less CPU, choosing which problems to solve right away and which to defer, and so on. In this episode, we discuss examples and strategies for breaking down projects into smaller steps to increase velocity and maximize area under the curve.

Step-by-step migrations: SwissTable

Hash tables have many different implicit and explicit properties that affect their contracts and behaviors. In designing SwissTables and planning for their associated migrations, we made careful choices to defer, avoid, or consciously embrace a great number of tradeoffs when modifying implementation contracts.

Performance Tip of the Week #74: Avoid sweeping street lights under rugs

Fri, 10 Nov 2023 00:00:00 +0000

While their issues go by multiple names (i.e. the streetlight effect, the McNamara fallacy), proxy metrics present a seductive danger. Simply put, people often focus on improving the things they can measure and neglecting the things they cannot even if those other things are important. In this episode, we explore multiple stories of how this can go wrong to help ground ourselves in the real world failure modes that proxy metrics present.

Most folks are familiar with the streetlight effect’s titular story:

Performance Tip of the Week #75: How to microbenchmark

Fri, 10 Nov 2023 00:00:00 +0000

Imagine having access to two Oracles. One gives 90% accurate predictions and responds in 10 minutes and one that gives 99% accurate predictions and responds in 1 month. With access to these secret seers, which will let us be best able to make good decisions?

“Production is ultimately what matters” might obviate the need for microbenchmarks altogether, but they are a useful tool in the performance optimization toolbox. While reduced fidelity doesn’t sound desirable at face value, it comes with an important tradeoff: speed.

Performance Tip of the Week #72: Optimizing optimization

Wed, 04 Sep 2024 00:00:00 +0000

Finding optimizations is increasingly crucial as Moore’s Law ends and we can no longer expect a free lunch from continued hardware innovation. In this episode, we discuss the process of finding and developing optimizations to work on.

Planning

“Plans are worthless, but planning is everything”

Just as efficiency projects are trying to maximize the productivity of our hardware, the projects themselves can be optimized by effective planning. Project selection makes a huge difference in outcomes and impact.

Performance Tip of the Week #70: Defining and measuring optimization success

Fri, 20 Oct 2023 00:00:00 +0000

Engineers optimizing performance are ultimately trying to maximize the things Google does (serve search queries, videos on YouTube, etc.) and minimize the things Google buys (CPUs, RAM, disks, etc.). In this episode, we discuss how to choose metrics that help us influence the optimizations we work on, make effective decisions, and measure the outcome of projects.

Economic value

Things like search queries and video playbacks on YouTube represent economic value. Useful work is happening to deliver an experience to an end-user which then translates into Google revenue.

Performance Tip of the Week #64: More Moore with better API design

Tue, 10 Oct 2023 00:00:00 +0000

Optimizing library implementations only carries us so far in making software more efficient. In this episode, we discuss the importance of good APIs and the right abstractions for finding optimization opportunities. As we can make the hardware–especially with the end of Moore’s Law–and software run only so fast, the right abstractions give us continued optimization opportunities.

Correctness is paramount

We can simplify an implementation down to return 42; regardless of the input to see blazing fast results, but an API that doesn’t work correctly isn’t doing its job.

Performance Tip of the Week #62: Identifying and reducing memory bandwidth needs

Wed, 04 Sep 2024 00:00:00 +0000

To accomplish useful work with processors, programs need to access main memory. The rate that programs can transfer data to/from main memory has not grown as fast as processors getting greater per-core performance and more cores. The Memory Wall, long predicted, is here. Memory bandwidth is increasingly the bottleneck for how much useful work modern data centers can accomplish with its CPUs. Optimizing for memory bandwidth productivity can make code faster–by reducing memory stalls–and RAM footprints smaller. In this episode, we discuss techniques for identifying hotspots and reducing their impact.

Performance Tip of the Week #60: In-process profiling: lessons learned

Sun, 15 Oct 2023 00:00:00 +0000

Google-Wide Profiling collects data not just from our hardware performance counters, but also from in-process profilers. These have been covered in previous episodes covering hashtables.

In-process profilers can give deeper insights about the state of the program that are hard to observe from the outside, such as lock contention, where memory was allocated, and the distribution of collisions on a hashtable. In this tip we discuss how to determine that a new profiler is necessary, and the best practices for producing one.

Performance Tip of the Week #39: Beware microbenchmarks bearing gifts

Thu, 02 Mar 2023 00:00:00 +0000

Benchmarks are only a tool for debugging efficiency: Production is ultimately what matters. Benchmarks analyze the performance of code under the specific circumstances created and maintained by the benchmark. They cannot perfectly predict the performance of code in the real world. In this episode, we discuss some of the pitfalls of microbenchmarks and mitigation strategies.

For example, we can use the following series of benchmarks for evaluating changes to search query performance:

Performance Tip of the Week #26: Fixing things with hashtable profiling

Wed, 16 Jul 2025 00:00:00 +0000

As discussed by Matt Kulukundis in his 2019 CppCon talk, identifying a “slow” hashtable from a pure CPU profile can be challenging. Abseil’s C++ hashtables have a built-in profiler. In this episode we describe what insights about the hash function quality and hash collisions it can provide, making them discernible at scale. We also look at a couple of case studies where this information was used to improve Google’s production fleet efficiency.

Performance Tip of the Week #9: Optimizations past their prime

Thu, 02 Mar 2023 00:00:00 +0000

Overview

Optimizations don’t always age gracefully. Faster yesterday might mean slower today.

Benchmarks citing performance on Intel Pentium 3’s or AMD Opterons may have been meaningful several years ago, but optimization equilibria, originally chosen for long-unplugged platforms, may have changed since. Let’s look at a couple examples where well-intended optimizations ultimately hurt performance in the long run.

Popcount

In 2008, Intel introduced the popcnt instruction to determine the number of set bits in a 32- or 64-bit integer. This is of interest for computing hamming distances and a bunch of other things. Without the instruction, we can use a slightly more complex, longer sequence of shifts, bitwise ands, and adds to achieve the same.