Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

CI/CD for the Linux Kernel

Introduction

Continuous Integration and Continuous Delivery (CI/CD) for the Linux kernel is a massive undertaking. The kernel is one of the largest and most complex open-source projects, with over 40 million lines of code (Linux 6.14, 2025), thousands of configuration options, and hundreds of target architectures. Testing it requires specialized infrastructure that can build, boot, and stress-test kernels across diverse hardware.

This chapter covers the major CI/CD systems used by the kernel community, from Intel’s 0-Day bot to the community-driven KernelCI, and practical approaches for CI/CD in kernel module development.

The Kernel CI Landscape

graph TB
    subgraph "Automated Testing"
        ZDAY["0-Day Bot<br>Intel"] -->|builds &amp; tests| REPORT1[Email reports to LKML]
        KCI["KernelCI<br>Community"] -->|multi-arch builds| REPORT2[Dashboard]
        LAVA["LAVA<br>Linaro"] -->|hardware testing| REPORT3[Test results]
        SYZKALLER["syzkaller<br>Google"] -->|fuzzing| BUGS[Bug reports]
    end
    
    subgraph "Manual Testing"
        NEXT[linux-next tree] -->|developer testing| FEEDBACK[Feedback]
        KTEST[ktest] -->|automated boot tests| RESULTS[Boot logs]
    end
    
    subgraph "CI Systems"
        GH[GitHub Actions] -->|module CI| BUILD[Build artifacts]
        GL[GitLab CI] -->|custom pipelines| DEPLOY[Deploy]
        JENKINS[Jenkins] -->|enterprise| PACKAGE[Package builds]
    end
    
    style ZDAY fill:#f96,stroke:#333,stroke-width:2px
    style KCI fill:#bbf,stroke:#333,stroke-width:2px

Intel’s 0-Day Bot

Overview

The 0-Day bot is Intel’s automated kernel testing system. It’s one of the most prolific contributors to the kernel—not by writing code, but by finding bugs.

0-Day Bot Statistics (2024)
───────────────────────────
Patches tested per day:     ~200-400
Builds per day:             ~1,000+
Configurations tested:      ~200+
Architectures:              x86_64, ARM64, RISC-V
Bug reports per year:       ~5,000+
Performance tests:          Continuous
Maintained by:              Intel OSS Technology Center

How 0-Day Works

sequenceDiagram
    participant Dev as Developer
    participant LKML as Mailing List
    participant Zero as 0-Day Bot
    participant Build as Build Farm
    participant Test as Test Farm
    
    Dev->>LKML: git send-email (PATCH)
    LKML->>Zero: Monitor for new patches
    Zero->>Zero: Parse patch, identify subsystem
    Zero->>Build: Build with relevant configs
    Build->>Zero: Build success/failure
    
    alt Build Success
        Zero->>Test: Boot test (QEMU)
        Zero->>Test: Static analysis (smatch, sparse)
        Zero->>Test: Performance benchmarks
        Test->>Zero: Test results
    end
    
    Zero->>LKML: Reply with report
    Note over Zero: Reported-by, Reviewed-by tags

0-Day Report Example

From: kernel test robot <lkp@intel.com>
Subject: [net] e1000e: fix regression - build error

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
head:   abc123def456
commit: 7890abcdef12 ("e1000e: add new device support")
date:   3 days ago
config: x86_64-randconfig-001-20250721
compiler: gcc-13 (GCC 13.2.0)

Error (expand):
   drivers/net/ethernet/intel/e1000e/netdev.c: In function 'e1000e_setup_rx_resources':
   drivers/net/ethernet/intel/e1000e/netdev.c:1234:25: error: implicit declaration of function 'new_api_func'

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202507210000.ABCDEF@lkp/

Static Analysis Tools Used by 0-Day

# Sparse — Semantic parser for C
$ make C=1 CHECK=sparse -j$(nproc)
# Or for specific files:
$ make C=2 drivers/net/ethernet/intel/e1000e/

# Smatch — Static analysis for kernel code
$ ./smatch_scripts/test_kernel.sh

# Coccinelle — Semantic patching
$ make coccicheck
# Or for specific semantic patches:
$ sp --sp-file scripts/coccinelle/free/kfree.cocci fs/ext4/

# GCC static analyzer
$ make KCFLAGS="-fanalyzer" -j$(nproc)

# Clang static analyzer
$ make CC=clang-analyzer -j$(nproc)

KernelCI

Overview

KernelCI is a community-driven project that tests the Linux kernel across multiple architectures and hardware platforms.

KernelCI Facts
──────────────
URL: https://kernelci.org/
Goal: Test every kernel commit on real hardware
Architectures: x86_64, ARM, ARM64, RISC-V, MIPS
Hardware: 100+ physical devices in labs
Test types: Boot, kselftest, LTP, benchmarks
Data: kernelci.org API and dashboard
Funding: Linux Foundation, Collabora, Google, ARM

KernelCI Architecture

graph LR
    SRC[Kernel Source] --> BUILD[Build Service]
    BUILD --> ARTIFACTS["Build Artifacts<br>kernel + rootfs + DTBs"]
    ARTIFACTS --> LAVA[LAVA Lab]
    ARTIFACTS --> OTHER[Other Labs]
    
    LAVA --> HW["Physical Hardware<br>ARM, RISC-V, x86"]
    HW --> RESULTS[Test Results]
    RESULTS --> API[KernelCI API]
    API --> DASHBOARD[Dashboard]
    API --> NOTIFY[Notifications]
    
    style BUILD fill:#f96,stroke:#333
    style LAVA fill:#bbf,stroke:#333
    style API fill:#9f9,stroke:#333

Using KernelCI API

# Query recent build results
$ curl -s "https://api.kernelci.org/build?kernel_branch=mainline&limit=10" | \
    jq '.data[] | {kernel: .kernel.version, arch: .arch, status: .status}'

# Get test results for a specific kernel version
$ curl -s "https://api.kernelci.org/test?kernel_version=6.12&limit=10" | \
    jq '.data[] | {test: .test_suite, device: .device, status: .status}'

# List available devices
$ curl -s "https://api.kernelci.org/device" | \
    jq '.data[] | {name: .name, arch: .arch}'

LAVA (LAb for Validation Architecture)

Overview

LAVA is a test automation framework developed by Linaro for deploying and testing software on real hardware.

# Install LAVA (Debian/Ubuntu)
$ sudo apt-get install lava-server lava-dispatcher

# Or use the LAVA Docker setup
$ git clone https://git.lavasoftware.org/lava/lava-docker
$ cd lava-docker
$ docker-compose up

LAVA Job Definition

# Example LAVA job definition for ARM64 boot test
device_type: qemu-arm64
job_name: kernel-boot-test-arm64

timeouts:
  job:
    minutes: 30
  action:
    minutes: 10

priority: medium
visibility: public

actions:
- deploy:
    to: tftp
    kernel:
      url: https://storage.kernelci.org/mainline/v6.12/arm64/gcc-13/Image
    ramdisk:
      url: https://storage.kernelci.org/rootfs/buildroot/arm64/rootfs.cpio.gz
      compression: gz
    os: oe

- boot:
    method: qemu
    media: tmpfs
    prompts:
    - "root@lava:~#"

- test:
    timeout:
      minutes: 10
    definitions:
    - repository: https://github.com/kernelci/test-definitions.git
      from: git
      path: automated/linux/boot
      name: boot-test

LAVA REST API

# Submit a job
$ curl -X POST \
    -H "Authorization: Token $LAVA_TOKEN" \
    -H "Content-Type: application/yaml" \
    -d @job.yaml \
    https://lava.example.net/api/v0.2/jobs/

# Check job status
$ curl -H "Authorization: Token $LAVA_TOKEN" \
    https://lava.example.net/api/v0.2/jobs/12345/

# Get job logs
$ curl -H "Authorization: Token $LAVA_TOKEN" \
    https://lava.example.net/api/v0.2/jobs/12345/logs/

syzkaller: Kernel Fuzzing

Overview

syzkaller is a coverage-guided kernel fuzzer developed by Google. It has found thousands of kernel bugs.

# Install syzkaller
$ go install github.com/google/syzkaller/syz-manager@latest

# Create configuration
cat > syz-manager.cfg << 'EOF'
{
    "target": "linux/amd64",
    "http": "localhost:56741",
    "workdir": "/syzkaller/workdir",
    "kernel_obj": "/path/to/kernel/build",
    "image": "/path/to/image.qcow2",
    "sshkey": "/path/to/ssh/key",
    "syzkaller": "/syzkaller",
    "procs": 4,
    "type": "qemu",
    "vm": {
        "count": 4,
        "kernel": "/path/to/kernel/build/arch/x86/boot/bzImage",
        "cpu": 2,
        "mem": 2048
    }
}
EOF

# Run syzkaller
$ syz-manager -config=syz-manager.cfg

# Dashboard: https://syzkaller.appspot.com/

syzkaller Bug Report Example

Title: KASAN: use-after-free in ext4_write_inline_data

syzkaller found a use-after-free bug in ext4:

BUG: KASAN: use-after-free in ext4_write_inline_data+0x234/0x300 fs/ext4/inline.c:234
Read of size 8 at addr ffff888123456789 by task syz-executor.0/1234

CPU: 0 PID: 1234 Comm: syz-executor.0 Not tainted 6.12.0-rc1 #1
Hardware name: QEMU Standard PC
Call Trace:
 <TASK>
 dump_stack_lvl+0x91/0xf0 lib/dump_stack.c:107
 print_report+0x17a/0x4b0 mm/kasan/report.c:399
 kasan_report+0xc4/0x100 mm/kasan/report.c:496
 ext4_write_inline_data+0x234/0x300 fs/ext4/inline.c:234
 ...

GitHub Actions for Kernel Modules

Basic CI Pipeline

# .github/workflows/kernel-module-ci.yml
name: Kernel Module CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        kernel_version: ['6.1', '6.6', '6.12']
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Install dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y build-essential bc bison flex \
            libelf-dev libssl-dev linux-headers-$(uname -r)
    
    - name: Build kernel module
      run: |
        make KVER=${{ matrix.kernel_version }}.0-generic
    
    - name: Check module
      run: |
        modinfo *.ko
    
    - name: Upload artifact
      uses: actions/upload-artifact@v4
      with:
        name: module-${{ matrix.kernel_version }}
        path: '*.ko'

Advanced CI with Cross-Compilation

# .github/workflows/cross-build.yml
name: Cross-Platform Build

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include:
          - arch: arm64
            cross_compile: aarch64-linux-gnu-
            defconfig: defconfig
          - arch: arm
            cross_compile: arm-linux-gnueabihf-
            defconfig: multi_v7_defconfig
          - arch: riscv
            cross_compile: riscv64-linux-gnu-
            defconfig: defconfig
          - arch: x86_64
            cross_compile: ""
            defconfig: defconfig

    steps:
    - uses: actions/checkout@v4
    
    - name: Install cross-compiler
      run: |
        sudo apt-get update
        if [ "${{ matrix.cross_compile }}" = "aarch64-linux-gnu-" ]; then
          sudo apt-get install -y gcc-aarch64-linux-gnu
        elif [ "${{ matrix.cross_compile }}" = "arm-linux-gnueabihf-" ]; then
          sudo apt-get install -y gcc-arm-linux-gnueabihf
        elif [ "${{ matrix.cross_compile }}" = "riscv64-linux-gnu-" ]; then
          sudo apt-get install -y gcc-riscv64-linux-gnu
        fi
        sudo apt-get install -y build-essential bc bison flex \
            libelf-dev libssl-dev

    - name: Get kernel source
      run: |
        git clone --depth=1 --branch v6.12 \
          https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    
    - name: Configure
      run: |
        cd linux
        make ARCH=${{ matrix.arch }} \
             CROSS_COMPILE=${{ matrix.cross_compile }} \
             ${{ matrix.defconfig }}
    
    - name: Build
      run: |
        cd linux
        make ARCH=${{ matrix.arch }} \
             CROSS_COMPILE=${{ matrix.cross_compile }} \
             -j$(nproc)

CI with QEMU Boot Test

# .github/workflows/boot-test.yml
name: Kernel Boot Test

on: [push]

jobs:
  boot-test:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Install QEMU
      run: |
        sudo apt-get update
        sudo apt-get install -y qemu-system-x86
    
    - name: Build kernel
      run: |
        make defconfig
        make -j$(nproc)
    
    - name: Create minimal initramfs
      run: |
        mkdir -p rootfs/{bin,sbin,etc,proc,sys,dev}
        # Install busybox
        wget https://busybox.net/downloads/busybox-1.36.1.tar.bz2
        tar xf busybox-1.36.1.tar.bz2
        cd busybox-1.36.1
        make defconfig
        sed -i 's/# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
        make -j$(nproc)
        make CONFIG_PREFIX=../rootfs install
        cd ..
        
        # Create init
        cat > rootfs/init << 'INITEOF'
        #!/bin/sh
        mount -t proc proc /proc
        mount -t sysfs sysfs /sys
        echo "Boot test passed!"
        poweroff -f
        INITEOF
        chmod +x rootfs/init
        
        cd rootfs && find . | cpio -o -H newc | gzip > ../initramfs.cpio.gz
    
    - name: Boot test with QEMU
      run: |
        timeout 60 qemu-system-x86_64 \
          -kernel arch/x86/boot/bzImage \
          -initrd initramfs.cpio.gz \
          -append "console=ttyS0 rdinit=/init" \
          -nographic \
          -no-reboot \
          2>&1 | tee boot.log
        
        grep -q "Boot test passed!" boot.log

Performance Testing in CI

Kernel Performance Benchmarks

# .github/workflows/perf-test.yml
name: Performance Regression Test

on:
  pull_request:
    branches: [main]

jobs:
  perf:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v4
    
    - name: Build kernel with perf
      run: |
        make defconfig
        scripts/config --enable CONFIG_PERF_EVENTS
        make -j$(nproc)
        make -C tools/perf
    
    - name: Run benchmark
      run: |
        # Boot test VM and run benchmarks
        # Example: network throughput test
        tools/perf/bench/sched/messaging -g 4 -l 1000 > perf_results.txt
        
        # Compare against baseline
        if [ -f baseline_results.txt ]; then
          python3 scripts/ci/compare_perf.py baseline_results.txt perf_results.txt
        fi
    
    - name: Upload results
      uses: actions/upload-artifact@v4
      with:
        name: perf-results
        path: perf_results.txt

ktest: Automated Kernel Testing

ktest is a kernel testing framework by Steven Rostedt:

# ktest configuration file
cat > ktest.conf << 'EOF'
[MACHINE]
MACHINE = localhost
SSH_PORT = 2222

[BUILD]
BUILD_TYPE = make
OUTPUT_DIR = /tmp/ktest/output
MAKE_CMD = make -j$(nproc)

[TEST]
TEST_TYPE = boot
POWER_CYCLE = qemu
QEMU_CMD = qemu-system-x86_64 -kernel ${KERNEL} -append "console=ttyS0" -nographic

[DEFAULTS]
TIMEOUT = 300
EOF

# Run ktest
$ ktest.pl ktest.conf

CI Pipeline Architecture

flowchart TD
    PATCH[Developer Patch] --> GATEWAY[CI Gateway]
    
    GATEWAY --> BUILD{Build Matrix}
    BUILD --> X86[x86_64 Build]
    BUILD --> ARM64[ARM64 Cross Build]
    BUILD --> RISCV[RISC-V Cross Build]
    
    X86 --> STATIC[Static Analysis]
    ARM64 --> STATIC
    RISCV --> STATIC
    
    STATIC --> BOOT[Boot Tests]
    BOOT --> QEMU[QEMU Boot]
    BOOT --> HW[Hardware Boot]
    
    QEMU --> KSELFTEST[kselftest Suite]
    HW --> LTP[LTP Tests]
    
    KSELFTEST --> REPORT[CI Report]
    LTP --> REPORT
    STATIC --> REPORT
    
    REPORT --> LKML[Email to Developer]
    REPORT --> DASHBOARD[CI Dashboard]
    
    style GATEWAY fill:#f96,stroke:#333,stroke-width:2px
    style REPORT fill:#9f9,stroke:#333,stroke-width:2px

Setting Up Your Own Kernel CI

Minimal CI with Git + Make + QEMU

#!/bin/bash
# Simple kernel CI script

set -e

KERNEL_REPO="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git"
BRANCH="master"
BUILD_DIR="/tmp/kernel-ci"

# Clone/update
if [ -d "$BUILD_DIR" ]; then
    cd "$BUILD_DIR"
    git pull
else
    git clone --depth=1 --branch "$BRANCH" "$KERNEL_REPO" "$BUILD_DIR"
    cd "$BUILD_DIR"
fi

# Configure
make defconfig
scripts/config --enable CONFIG_DEBUG_INFO
scripts/config --enable CONFIG_KASAN
scripts/config --disable CONFIG_DEBUG_INFO_BTF  # Faster builds

# Build
make -j$(nproc) 2>&1 | tee build.log

# Static analysis (if sparse available)
if command -v sparse &> /dev/null; then
    make C=1 CHECK=sparse -j$(nproc) 2>&1 | tee sparse.log
fi

# Create initramfs and boot test
if command -v qemu-system-x86_64 &> /dev/null; then
    # Create minimal initramfs (see boot-test.yml above)
    make_initramfs
    
    # Boot test
    timeout 60 qemu-system-x86_64 \
        -kernel arch/x86/boot/bzImage \
        -initrd initramfs.cpio.gz \
        -append "console=ttyS0 rdinit=/init" \
        -nographic -no-reboot 2>&1 | tee boot.log
    
    if grep -q "Boot test passed!" boot.log; then
        echo "BOOT TEST: PASS"
    else
        echo "BOOT TEST: FAIL"
        exit 1
    fi
fi

echo "CI completed successfully"

References and Further Reading

  • The Linux Kernel Documentation

  • GNU Project Documentation

  • GNU Manuals

  • Free Software Directory

  • Planet GNU

  • Free Software Books

  • Intel 0-Day: https://01.org/lkp/documentation/0-day

  • KernelCI: https://kernelci.org/

  • LAVA: https://lavasoftware.org/

  • syzkaller: https://github.com/google/syzkaller

  • ktest: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/ktest.git

  • kernelci-core: https://github.com/kernelci/kernelci-core

  • Linux Kernel Selftests: https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html

  • LWN.net Kernel CI coverage: https://lwn.net/Kernel/

  • “Kernel Testing and CI” — Linux Plumbers Conference talks

  • GitHub Actions documentation: https://docs.github.com/en/actions

  • kernelci API: https://api.kernelci.org/