Memory-Corruption Exploitation and the Mitigation Arms Race

Every mitigation in a modern toolchain exists because a specific exploitation technique defeated the previous one. Read them as a 30-year argument between attackers who need one memory write and defenders who must make every write harmless:

 overflow writes return address
   |  [1998] stack canary        -> smash detected at function return
   |  [2003] NX / W^X           -> injected shellcode pages not executable
   |  [1997] ret2libc            -> "code" already in executable libc
   |  [2007] ROP                 -> libc itself is a gadget concatenation machine
   |  [2005] ASLR + PIE          -> gadget/libc addresses randomized per run
   |  [2010] info leaks          -> read a pointer first, defeat randomization
   |  [2013+] CFI / CET / PAC    -> even a hijacked branch can only land on valid targets
   v  [2020s] memory-safe languages -> remove the write primitive itself

The exploitation ladder, mechanism by mechanism

Stack smashing to code injection. A strcpy(buf, input) past the end of a stack frame overwrites the saved return address with a pointer to attacker bytes on the same stack. Defeated by NX (no-execute page permissions), so attacker-supplied bytes can no longer be executed.

ret2libc. Point the return address at system() with a crafted argument already on the stack – no injected code, only control flow. NX is irrelevant; only address randomization helps.

ROP (Return-Oriented Programming, Shacham CCS’07). Chain many short existing code fragments (“gadgets”) that each end in ret; the stack itself becomes the program counter feed. Given a large binary or libc, gadget catalogs show a handful of pop rdi; ret-style gadgets suffice to call arbitrary syscalls – ROP is Turing-complete over conventional ISAs. Forward-edge variants (JOP/COP) use jmp/call dispatch tables with a dedicated “dispatcher gadget”, defeating CFI schemes that only police ret.

Heap exploitation. glibc malloc keeps freed chunks in bins: per-thread tcache (64 bins, up to 7 chunks each, single next pointer), fastbins (LIFO, small sizes, no coalescing), then small/large/unsorted. A use-after-free or double-free lets you edit a freed chunk’s next pointer; the next two malloc()s of that size return an attacker- chosen address (tcache poisoning) – typically the __free_hook/__malloc_hook era (and post-hook-removal, exit handlers or stack targets). Allocator hardening followed the same one-step-at-a-time pattern: glibc 2.29 added a tcache double-free key; glibc 2.32 added safe-linking, storing next as (chunk_addr >> 12) ^ next so a leak-free heap cannot forge the pointer. See the glibc malloc internals wiki for the authoritative bin layout.

The mitigation stack, and what each actually stops

Deployment reality: Intel CET shipped in Tiger Lake (2020) and mainline Linux gained user-space shadow-stack support in 6.6; arm64 phones have shipped PAC since 2018 and MTE hardware since the Pixel 8 (2023); kernel CFI (kCFI) merged in Linux 6.2, with FineIBT (IBT + hash checks combined) as its successor design. None of these are full solutions – they raise per-exploit cost from “one write” to “write + leak + oracle”.

ASLR is probability, not prevention

Randomization only raises expected attacker cost. Two exact computations every security engineer should be able to do (the second explains why canaries alone were never enough):

"""Brute-force economics of ASLR entropy and stack-canary attacks.
Exact arithmetic, no RNG."""

print("ASLR: expected tries and wall-clock to hit a 1-in-2^bits guess")
print("  (assumes a crashing service that restarts at 1000 attempts/second)")
for bits, name in ((16, "32-bit-era partial"), (24, "weak mmap"), (28, "x86-64 mmap/PIE default"),
                   (32, "full 32-bit")):
    exp_tries = 2.0 ** bits
    secs = exp_tries / 1000.0
    print("  %2d bits %-24s expected %12.0f tries  = %.1f hours" %
          (bits, "(%s)" % name, exp_tries, secs / 3600.0))

print()
print("P(at least one hit within K tries), 28-bit ASLR:")
for k in (10**3, 10**6, 10**9):
    p = 1.0 - (1.0 - 2.0 ** -28) ** k
    print("  K=1e%-2d -> %.4f" % (len(str(k)) - 1, p))

print()
print("Stack canary brute force (32-bit SSP canary, LSB forced 0x00):")
whole = 2.0 ** 32
print("  whole-value guessing:      2^32 = %d tries (and one crash each)" % whole)
print("  byte-at-a-time w/ restart: max %d, expected %.1f tries"
      % (3 * 256, 3 * (256 + 1) / 2.0))
print("  speedup factor:            %.1e x" % (whole / (3 * (256 + 1) / 2.0)))
print("  64-bit canary (7 unknown bytes): max %d tries" % (7 * 256))

Real output (Python 3.12):

ASLR: expected tries and wall-clock to hit a 1-in-2^bits guess
  (assumes a crashing service that restarts at 1000 attempts/second)
  16 bits (32-bit-era partial)     expected        65536 tries  = 0.0 hours
  24 bits (weak mmap)              expected     16777216 tries  = 4.7 hours
  28 bits (x86-64 mmap/PIE default) expected    268435456 tries  = 74.6 hours
  32 bits (full 32-bit)            expected   4294967296 tries  = 1193.0 hours

P(at least one hit within K tries), 28-bit ASLR:
  K=1e3  -> 0.0000
  K=1e6  -> 0.0037
  K=1e9  -> 0.9759

Stack canary brute force (32-bit SSP canary, LSB forced 0x00):
  whole-value guessing:      2^32 = 4294967296 tries (and one crash each)
  byte-at-a-time w/ restart: max 768, expected 385.5 tries
  speedup factor:            1.1e+07 x
  64-bit canary (7 unknown bytes): max 1792 tries

Read the rows as a defender. Sixteen-bit randomization falls in under a minute; the x86-64 default 28 bits survives a crash-looping attacker only because restart cost is in the denominator – a service that reconnects at 1M attempts/s (in-process retries, no crash needed, e.g. attack via a leaked pointer side channel) closes 74.6 hours to under five minutes. The canary rows show the classic forking-server attack: because a crashed child gets a fresh canary but the byte you already guessed stays guessed, the 2^32 space collapses to at most 768 probes. That asymmetry – attacker learns across restarts, defender entropy does not grow – is the entire game; deeper background on randomization itself lives in the kernel ASLR deep-dive.

Why the industry changed strategy

By 2019 the data was unambiguous: Microsoft’s MSRC attributed roughly 70% of tracked CVEs to memory-safety violations, and Chromium’s security team reports the same proportion for its most severe bugs. Mitigation stacks were not reversing the trend; exploit kits simply priced in canaries, ROP, and one info leak. The response moved from containing corruption to eliminating the write primitive: Android’s memory-safety bugs fell from 76% of all Android CVEs in 2019 to 24% in 2024 (Google’s security blog, “Safer with Google”) – overwhelmingly correlated with new code being Rust, not with better sanitizers. The Linux kernel merged initial Rust support in 6.1 (drivers first) for the same reason, and Android/Chromium ship hardened allocators (scudo, PartitionAlloc) with quarantine and deterministic double-free aborts so that UAF becomes a crash, not an exploit primitive.

Practical defaults (what a reviewer checks)

• Build: -fstack-protector-strong, -D_FORTIFY_SOURCE=2, -fPIE -pie, -Wl,-z,relro,-z,now, -fcf-protection=full; sanitizer builds in CI (-fsanitize=address,undefined) to catch the bug class before mitigations matter.
• Allocator: know your runtime’s behavior on double-free/UAF (glibc aborts on tcache key mismatch; hardened allocators quarantine instead of reusing).
• Operations: entropy is per-boot; a fleet that leaks one address per process gives an attacker fleet-wide de-randomization – gate on vm.mmap_rnd_bits, keep 32 where supported.
• See system hardening for sysctl-level hardening, side-channel-resistant crypto for the adjacent constant-time discipline, and microarchitectural attacks for why speculative execution re-opened even correct sandboxing.

References

• Chromium: Memory Safety (share of serious security bugs)
• Android: Test Memory Safety (allocation hygiene, MTE, hardened allocators)
• Linux kernel x86 user shadow-stack (CET) documentation
• glibc Malloc Internals (bins, tcache, fastbin layout)
• Check Point Research: Safe-Linking – eliminating a 20-year-old malloc exploit primitive