CPU Profiling

CPU profiling answers one question: where is my program spending its CPU time? This is the first tool you reach for when a service is CPU-bound or when you simply don’t know what’s slow.

Sampling vs. Instrumentation Profiling

Sampling is preferred for production profiling because the low overhead means you can profile under real load without significantly distorting results.

Flame Graphs

Flame graphs, created by Brendan Gregg, are the single most effective visualization for CPU profiling data.

How to read a flame graph:

  ┌─────────────────────────────────────────┐
  │          main()                         │  ← root (100% of CPU time)
  │  ┌──────────────┐  ┌─────────────────┐  │
  │  │ process()    │  │ handleIO()     │  │  ← top-level callees
  │  │ ┌──────────┐  │  │                 │  │
  │  │ │ parse()  │  │  │                 │  │  ← deeper calls
  │  │ │ ┌──────┐ │  │  │                 │  │
  │  │ │ │token │ │  │  │                 │  │  ← leaf (CPU is burning here)
  │  │ │ └──────┘ │  │  │                 │  │
  │  │ └──────────┘  │  │                 │  │
  │  └──────────────┘  └─────────────────┘  │
  └─────────────────────────────────────────┘

  Width  = proportion of CPU time in that function
  Height = call stack depth
  Color  = usually random (no semantic meaning); some tools color by module

Reading rules:

• Look for the widest towers — these consume the most CPU.
• The top of each tower is where CPU is actually being spent (leaf functions).
• If a function spans the full width of the graph, it’s on every call path — optimizing it helps everything.

Linux perf

perf is the standard Linux profiling tool. It uses hardware performance counters (PMU) for near-zero overhead sampling.

perf stat — High-Level Summary

$ perf stat ./my_server

 Performance counter stats for './my_server':

      3,241.52 msec  task-clock           #    0.998 CPUs utilized
            12      context-switches     #    3.701 /sec
             1      cpu-migrations       #    0.308 /sec
         4,521      page-faults          #    1.395 K/sec
 9,876,543,210      cycles               #    3.046 GHz
 7,654,321,098      instructions         #    0.775  insn per cycle  ← IPC! Low = CPU-bound, stalls
   123,456,789      cache-references     #   38.099 M/sec
    12,345,678      cache-misses         #   10.008 % of all cache refs

    3.247812345 seconds time elapsed

Key metric: Instructions Per Cycle (IPC). Low IPC (< 1.0) often means cache misses or branch mispredictions. High IPC (> 2.0) means the CPU is executing efficiently.

perf record + perf report — Detailed Profiles

# Record CPU profile for a specific PID
$ perf record -p <PID> -g -- sleep 30

# Record with call graph (dwarf unwinding for accurate stacks)
$ perf record -g --call-graph dwarf -p <PID>

# View the report (interactive TUI)
$ perf report

# Generate a flame graph (using Brendan Gregg's scripts)
$ perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg

Language-Specific Tools

Java

# async-profiler: generate flame graph in 30 seconds
$ ./profiler.sh -d 30 -f flame.svg <pid>

# JFR: start recording
$ jcmd <pid> JFR.start name=profiling duration=60s filename=recording.jfr

Python

# py-spy: top-like live view
$ py-spy top --pid <pid>

# py-spy: generate flame graph
$ py-spy record -o flame.svg --pid <pid> --duration 30

# cProfile
$ python -m cProfile -s cumtime my_script.py

Go

// In your server:
import _ "net/http/pprof"
// Then: go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

// Generate flame graph:
// go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile

Rust

# Build with debug info
$ cargo build --release

# Profile with perf
$ perf record -g ./target/release/my_app
$ perf report

# Or use samply for nicer flame graphs
$ cargo install samply
$ samply record ./target/release/my_app

Common CPU Bottlenecks

Interview Questions

1. What’s the difference between sampling and instrumentation profiling? When would you choose each?
2. How do you read a flame graph? What does the width of a stack frame represent?
3. You run perf stat and see IPC of 0.5. What does that tell you? What would you investigate next?
4. How would you profile a Java application running in a container with limited CPU?
5. What is async-profiler and why is it preferred over jstack for production Java profiling?
6. How does pprof in Go work under the hood?
7. A service has 80% CPU but low throughput. Walk me through your diagnosis.
8. How would you detect and diagnose lock contention as a CPU bottleneck?