Stack Traces

Overview

A stack trace (also called a backtrace or call stack) shows the chain of function calls that led to a particular point in the kernel’s execution. Stack traces are one of the most important debugging tools in the Linux kernel—they appear in kernel oops messages, warning dumps, lockdep reports, and can be captured programmatically.

The kernel provides several mechanisms for producing stack traces:

dump_stack() — print a stack trace to the kernel log
save_stack_trace() — capture stack trace into a buffer
WARN_ON() / BUG() — automatically include stack traces
/proc/<pid>/stack — per-task stack traces
/sys/kernel/debug/stack_trace — saved stack traces
ftrace — function graph tracing with stack traces

Unwinding Mechanisms

The kernel uses different “unwinders” to walk the call stack. Each has different trade-offs in terms of accuracy, speed, and space requirements.

ORC Unwinder

ORC (Oops Rewind Capability) is the default unwinder on x86_64 since Linux 4.14. It was developed as a replacement for the DWARF unwinder for in-kernel stack unwinding.

How ORC Works

ORC uses a compact lookup table that maps instruction pointers to unwind information:

struct orc_entry {
    s16 sp_off;    /* SP offset from the previous frame */
    s16 bp_off;    /* BP offset (for frame pointer chains) */
    u32 type:2;    /* UNWIND_HINT_TYPE_* */
    u32 end:1;     /* End of stack marker */
    u32 sp_reg:4;  /* Which register holds SP */
    u32 bp_reg:4;  /* Which register holds BP */
    u32 signal:1;  /* Interrupt/signal frame */
};

The ORC table is generated at compile time by the objtool tool, which analyzes the compiled object files and generates unwind information.

ORC vs DWARF

Property	ORC	DWARF
Table size	~2–4% of code size	~10–15% of code size
Runtime speed	Fast (binary search)	Slower (complex state machine)
Generation	objtool (post-compile)	Compiler (inline)
Accuracy	Good	Excellent
Out-of-line data	Yes (separate section)	Yes (`.debug_frame` or `.eh_frame`)

ORC was created because the DWARF unwinder was too slow and its tables too large for in-kernel use, especially for livepatch and BPF.

ORC Table Location

# View the ORC table size
objdump -h vmlinux | grep orc

# ORC sections:
# .orc_unwind_ip — instruction pointer table
# .orc_unwind    — ORC entry table

Enabling ORC

# Kernel config
CONFIG_UNWINDER_ORC=y    # Default on x86_64
# vs
CONFIG_UNWINDER_FRAME_POINTER=y  # Alternative
# vs
CONFIG_UNWINDER_GUESS=y  # Last resort (no unwinder)

Frame Pointer Unwinder

The frame pointer unwinder follows the chain of frame pointers (BP/RBP on x86_64, FP on ARM64).

How It Works

Each function prologue saves the caller’s frame pointer:

; x86_64 function prologue
push rbp          ; Save caller's frame pointer
mov rbp, rsp      ; Set up new frame pointer

To unwind, start at the current RBP, read the saved return address at RBP+8, then follow the saved RBP to the previous frame:

Frame 3: RBP3 → [saved RBP2 | ret addr3]
Frame 2: RBP2 → [saved RBP1 | ret addr2]
Frame 1: RBP1 → [saved RBP0 | ret addr1]
Frame 0: (current)

Requirements

Frame pointer unwinding requires that all functions in the call chain use frame pointers. This means:

# Compile with frame pointers
CONFIG_FRAME_POINTER=y

# For userspace:
gcc -fno-omit-frame-pointer ...

If any function omits the frame pointer (common with optimized code), the chain breaks and the unwinder fails.

Advantages and Disadvantages

Fast: just pointer chasing
Small overhead: one push/mov per function
Accurate when enabled: works if all functions cooperate
Breaks with optimization: -O2 often omits frame pointers by default
Architecture-dependent: each arch has its own frame pointer convention

DWARF Unwinder

DWARF unwinding uses .eh_frame (or .debug_frame) tables generated by the compiler.

How It Works

Each function has a DWARF Call Frame Information (CFI) entry describing how to unwind:

FDE: function "foo"
  CFA: rsp + 16    ; Canonical Frame Address = RSP + 16
  rbp: cfa - 16    ; RBP saved at CFA - 16
  rip: cfa - 8     ; Return address at CFA - 8

The unwinder evaluates these DWARF expressions to recover the caller’s registers.

In-Kernel Use

The DWARF unwinder was used in-kernel on x86_64 before ORC:

CONFIG_UNWINDER_ORC=n
CONFIG_UNWINDER_FRAME_POINTER=n
# Falls back to DWARF if available

DWARF tables are large (.eh_frame can be 5–15% of code size), making them unsuitable for in-kernel use. ORC was specifically designed to replace them.

Userspace DWARF

Userspace stack unwinding typically uses DWARF via libunwind:

#include <libunwind.h>

unw_cursor_t cursor;
unw_context_t context;

unw_getcontext(&context);
unw_init_local(&cursor, &context);

while (unw_step(&cursor) > 0) {
    char name[256];
    unw_word_t offset;
    unw_get_proc_name(&cursor, name, sizeof(name), &offset);
    printf("%s+0x%lx\n", name, offset);
}

Guess Unwinder

The guess unwinder is a last resort when no proper unwind information is available:

CONFIG_UNWINDER_GUESS=y

It scans the stack looking for values that look like kernel text addresses (within _stext to _etext). This is unreliable—it produces false positives and misses frames—but works when no unwind data exists.

Not recommended for production. Use only for debugging when other unwind mechanisms are broken.

Kernel API

dump_stack()

#include <linux/kernel.h>

/* Print a stack trace to the kernel log */
void dump_stack(void);

dump_stack() is the simplest way to get a stack trace. It prints the current call chain to dmesg:

[  123.456789] Call Trace:
[  123.456789]  dump_stack+0x64/0x8c
[  123.456789]  my_function+0x42/0x100
[  123.456789]  caller_function+0xab/0x200
[  123.456789]  entry_point+0x12/0x50

Each line shows: function_name+offset/size

save_stack_trace()

#include <linux/stacktrace.h>

struct stack_trace {
    unsigned int nr_entries;
    unsigned int max_entries;
    unsigned long *entries;
    int skip;           /* Number of entries to skip */
};

/* Capture a stack trace */
void save_stack_trace(struct stack_trace *trace);

/* Free the trace (if needed) */
void save_stack_trace_tsk(struct task_struct *tsk,
                          struct stack_trace *trace);

Use this when you need to capture a stack trace for later analysis:

unsigned long entries[16];
struct stack_trace trace = {
    .entries = entries,
    .max_entries = ARRAY_SIZE(entries),
    .nr_entries = 0,
    .skip = 0,
};

save_stack_trace(&trace);

for (int i = 0; i < trace.nr_entries; i++)
    printk("  [<%p>] %pS\n", (void *)entries[i], (void *)entries[i]);

New Stack Trace API (stack_trace_save)

The newer API (4.x+) is simpler:

#include <linux/stacktrace.h>

/* Returns number of entries stored */
unsigned int stack_trace_save(unsigned long *store,
                              unsigned int size,
                              unsigned int skipnr);

/* For a specific task */
unsigned int stack_trace_save_tsk(struct task_struct *tsk,
                                  unsigned long *store,
                                  unsigned int size,
                                  unsigned int skipnr);

stack_trace_print()

/* Print a saved stack trace */
void stack_trace_print(const unsigned long *trace,
                       unsigned int nr_entries,
                       int spaces);

WARN_ON, BUG_ON, and Stack Traces

WARN_ON

/* Produces a stack trace + warning message */
WARN_ON(condition);

/* With custom message */
WARN(condition, "Something went wrong: %d\n", value);

WARN_ON() prints a full stack trace prefixed with WARNING: CPU: X PID: Y at file:line:

[  123.456789] ------------[ cut here ]------------
[  123.456789] WARNING: CPU: 2 PID: 1234 at drivers/foo/bar.c:42 do_something+0x42/0x100
[  123.456789] Modules linked in: ...
[  123.456789] CPU: 2 UID: 0 PID: 1234 Comm: test Not tainted 6.1.0 #1
[  123.456789] Call Trace:
[  123.456789]  <TASK>
[  123.456789]  do_something+0x42/0x100
[  123.456789]  caller+0xab/0x200
[  123.456789]  </TASK>

BUG_ON

/* Terminates the kernel (or current process) with a stack trace */
BUG_ON(condition);

/* With message */
BUG(condition, "Fatal: %s\n", reason);

BUG_ON() is for unrecoverable errors. It prints a “BUG” message with a stack trace and then either panics or kills the current process, depending on CONFIG_BUG_ON_DATA_CORRUPTION and related settings.

WARN_ON_ONCE / BUG_ON_ONCE

/* Only warn/bug once, then suppress */
WARN_ON_ONCE(condition);
BUG_ON_ONCE(condition);

Useful for conditions that might fire many times but you only want one report.

/proc and /sys Interfaces

`/proc/<pid>/stack`

# View current stack trace of a process
cat /proc/1234/stack

# Output:
# [<0>] futex_wait_queue+0x64/0x120
# [<0>] futex_wait+0x58/0x100
# [<0>] do_futex+0x140/0x800
# [<0>] __x64_sys_futex+0x40/0xc0
# [<0>] do_syscall_64+0x5c/0x90
# [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

This is useful for checking what a stuck process is doing.

`/proc/<pid>/syscall`

# Show current system call
cat /proc/1234/syscall

/sys/kernel/debug/stack_trace/

The stack trace collection framework can be enabled to save stack traces for various events:

# Enable stack trace collection
echo 1 > /proc/sys/kernel/stack_tracer_enabled

# View stack traces
cat /sys/kernel/debug/stack_trace/stack_max_size
cat /sys/kernel/debug/stack_trace/stack_trace

ftrace Stack Traces

Function Graph Tracer with Stack

# Enable function graph tracing
echo function_graph > /sys/kernel/debug/tracing/current_tracer

# Enable stack traces after each function
echo 1 > /sys/kernel/debug/tracing/options/stacktrace

# View output
cat /sys/kernel/debug/tracing/trace

Trace Events with Stack

# Enable an event with stack trace
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/stacktrace

# View
cat /sys/kernel/debug/tracing/trace

Stack Histogram

# Enable stack trace collection for allocation profiling
echo stacktrace > /sys/kernel/debug/tracing/set_event
echo 1 > /sys/kernel/debug/tracing/options/stacktrace

# Or use the dedicated stack tracer
echo 1 > /proc/sys/kernel/stack_tracer_enabled
sleep 10
cat /sys/kernel/debug/stack_trace/stack_trace

Architecture-Specific Notes

x86_64

Default unwinder: ORC (since 4.14)
Fallback: frame pointer, DWARF, or guess
objtool generates ORC tables at build time
NMI stacks: NMI handlers have their own stack; unwinding through NMI boundaries requires special handling

ARM64

Default: frame pointer unwinder
DWARF: available via .eh_frame if compiled with -fasynchronous-unwind-tables
FP register: x29 (frame pointer)
LR register: x30 (link register, holds return address)

RISC-V

Frame pointer: s0 (x8)
ORC: not yet supported
DWARF: available

s390x

Backchain: native backchain support (similar to frame pointers)
DWARF: available
ORC: not applicable

Display Format

Standard Format

[<address>] function_name+offset/size

address: kernel text address (may be 00000000 if CONFIG_KALLSYMS=n)
function_name: symbol name from kallsyms
offset: offset within the function (hex)
size: total function size (hex)

Task Tag

Modern kernels tag the stack trace with the process context:

Call Trace:
 <TASK>
 function_a+0x10/0x20
 function_b+0x30/0x40
 </TASK>

Call Trace:
 <NMI>
 nmi_handler+0x5/0x10
 </NMI>
 <IRQ>
 irq_handler+0x20/0x30
 </IRQ>

Register Dump

A full oops includes registers before the stack trace:

RIP: 0010:bad_function+0x42/0x100
RSP: 0018:ffffc90000abcd00 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88810abcd000
RCX: 0000000000000001 RDX: 0000000000000002
...
Call Trace:
 <TASK>
 ...

Practical Debugging

Analyzing a Kernel Oops

Identify the crash point: look at RIP: or the first WARNING/BUG line
Read the stack trace bottom-up: the topmost frame is the crash site, read downward for the call chain
Use addr2line: addr2line -e vmlinux <address> to get exact source line
Use objdump: objdump -d vmlinux | grep -A20 <function> to see the disassembly
Check gdb: gdb vmlinux and use list *(address) for source context

Getting Stack Traces from Crashed Kernels

If you have a crash dump (via kdump):

# Using crash utility
crash vmlinux /var/crash/vmcore

# Inside crash:
crash> bt          # Current task backtrace
crash> bt -a       # All tasks
crash> bt <pid>    # Specific task
crash> log         # Kernel log (dmesg)

Saving Stack Traces Programmatically

#include <linux/stacktrace.h>
#include <linux/slab.h>

void capture_and_save_trace(void)
{
    unsigned long *entries;
    unsigned int nr_entries;

    entries = kmalloc(64 * sizeof(unsigned long), GFP_KERNEL);
    if (!entries)
        return;

    nr_entries = stack_trace_save(entries, 64, 0);

    /* Process the trace */
    stack_trace_print(entries, nr_entries, 2);

    kfree(entries);
}

Custom Stack Trace in a Module

#include <linux/module.h>
#include <linux/stacktrace.h>
#include <linux/printk.h>

static void my_debug_function(void)
{
    unsigned long entries[16];
    unsigned int nr;

    nr = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
    pr_err("Stack trace with %u entries:\n", nr);
    stack_trace_print(entries, nr, 4);
}

Source Files

arch/x86/kernel/dumpstack.c — x86 stack dumping
arch/x86/kernel/unwind_orc.c — ORC unwinder
arch/x86/kernel/unwind_frame.c — frame pointer unwinder
arch/x86/kernel/unwind_guess.c — guess unwinder
arch/arm64/kernel/stacktrace.c — ARM64 stack tracing
lib/stacktrace.c — generic stack trace infrastructure
kernel/stacktrace.c — stack trace collection framework
tools/objtool/ — ORC table generator
include/linux/stacktrace.h — API declarations

Keyboard shortcuts

Placement Preparation Knowledge Base