Building DRIFTBENCH: Turning Kernel Git History Into a Reproducible Evaluation Corpus

Evaluating kernel migration tools requires real cases, not synthetic ones. You cannot assess whether a system handles the complexity of kernel API evolution by testing it on contrived examples. The edge cases that matter — the ones that break real drivers in production — emerge from the actual history of kernel development, with all its messiness, implicit conventions, and cross-subsystem dependencies.

DRIFTBENCH is our answer to this problem. It is a deterministic, reproducible evaluation corpus mined from the Linux kernel's git history. Each case represents a real API migration: a kernel change that modified an internal API, paired with the corresponding driver adaptations that maintainers made in response. The dataset includes the before state, the kernel change, the golden-reference after state, and a validation harness that verifies any candidate patch through KDRIFT's four-stage pipeline.

This post describes how we built it, the design decisions that make it reproducible, and why existing benchmarks were insufficient for our needs.

The Problem with Existing Benchmarks

SWE-bench and its variants have become the standard evaluation for code generation and automated program repair. They test a model's ability to resolve GitHub issues across popular Python repositories. This is valuable work, but it does not transfer to kernel code for several fundamental reasons:

• Build system complexity: Kernel code requires Kconfig, Kbuild, cross-compilation toolchains, and architecture-specific configuration. A "compile" check is not just running gcc — it is configuring a kernel, resolving hundreds of Kconfig dependencies, and building within the kernel's make infrastructure.
• Hardware dependencies: Many driver changes are only testable with specific hardware or accurate hardware emulation. DRIFTBENCH cases are selected to be testable in QEMU with appropriate machine types.
• Cross-subsystem interactions: A single API change can require coordinated modifications across multiple drivers in different subsystems. SWE-bench cases are typically self-contained within a single repository.
• Strict coding style: The kernel has checkpatch.pl and a rigid coding style that rejects patches with whitespace errors, incorrect comment formatting, or non-standard macro usage. Generated patches must conform to these standards.
• Validation depth: In Python projects, "does the test pass?" is a reasonable proxy for correctness. In kernel code, "does it compile?" is the minimum bar, and real validation requires static analysis, boot testing, and runtime verification.

We needed a benchmark that was native to the kernel development workflow, not adapted from a different domain. DRIFTBENCH is designed from the ground up to reflect how kernel migration actually works.

Mining Kernel Commits

The Linux kernel's git history is the richest source of API migration examples in existence. Every time a kernel developer changes an internal API, they are expected to update all in-tree users of that API in the same commit or patch series. This convention — "if you break it, you fix it" — means that the kernel's git history contains thousands of paired examples: an API change and its corresponding driver adaptations.

We identify migration-relevant commits through a multi-step filter:

• Path filter: The commit must touch both header files in include/linux/ (or include/net/, include/sound/, etc.) and driver source files in drivers/. This identifies commits where an API definition changed and driver call sites were updated.
• Diff structure: The changes to include/ files must represent API modifications (function signature changes, struct reorganization, macro redefinition), not just documentation or comment updates.
• Driver scope: The commit must update at least 2 drivers. Single-driver changes are often driver-specific bug fixes rather than API migrations.
• Complexity filter: The driver-side changes must involve more than simple find-and-replace. We filter out trivial renames and argument reorderings that can be handled by Coccinelle scripts, focusing on cases that require semantic understanding.

Applying these filters to kernel versions v5.10 through v6.10 yielded approximately 2,400 candidate commits. Manual review reduced this to 340 high-quality migration examples. From these, we selected 71 cases for DRIFTBENCH based on diversity of change type, subsystem coverage, and testability in QEMU.

Case Extraction Pipeline

For each selected commit, we extract a self-contained migration case. The extraction pipeline produces:

• Before state: The driver source code at the commit immediately preceding the migration. This is the "input" to the migration tool — the code that needs to be updated.
• API change specification: The diff to the header files, extracted from the kernel commit. This defines what changed in the API.
• Golden patch: The diff applied to the driver in the original commit. This is the reference solution — the adaptation that the kernel maintainer actually made.
• After state: The driver source code after applying the golden patch. Used for AST similarity comparison.
• Change classification: Manual annotation of the change type according to the 9-class taxonomy (signature change, return type change, deprecation, struct reshape, lifecycle ordering, concurrency, error handling, security, cross-subsystem).
• Kernel configuration: A minimal defconfig that enables the target driver and its dependencies, suitable for QEMU boot testing.

The extraction is deterministic: given a kernel commit hash, the pipeline always produces the same case. We pin exact kernel versions using git tags, not branch heads, ensuring that DRIFTBENCH results are reproducible across time.

# Extract a case from a kernel commit
driftbench extract --commit abc123def \
  --driver drivers/gpu/drm/example/example_drv.c \
  --kernel-version v6.4 \
  --output cases/case_047/

# Produces:
# cases/case_047/before/       — driver source at v6.3
# cases/case_047/api_change/   — header diffs
# cases/case_047/golden/       — reference patch
# cases/case_047/config         — kernel defconfig
# cases/case_047/metadata.json — classification, subsystem, etc.

Stratified Splits

The 71 cases are divided into 48 API migration cases (where a kernel API change requires driver adaptation) and 23 regression cases (where a kernel change introduces a driver bug that must be detected and fixed). The split reflects the two primary use cases for KDRIFT: proactive migration and reactive regression repair.

Within each category, we create train/dev/test splits that are stratified along three dimensions:

• Change type: Each split contains a representative mix of all 9 change classes. No split is dominated by a single class.
• Kernel subsystem: Cases span 14 subsystems (DRM, networking, USB, PCI, I2C, SPI, sound, platform, clocksource, watchdog, input, hwmon, media, GPIO). Each split covers at least 8 subsystems.
• Difficulty: Annotated on a 1-5 scale based on number of files changed, lines modified, and whether the change is syntactic or semantic. Each split has similar difficulty distributions.

Critically, the splits enforce strict temporal ordering. All test cases come from kernel versions strictly newer than all training cases. This prevents temporal leakage — a model cannot achieve high test performance by memorizing patterns from the same kernel version. In practice, training cases are drawn from v5.10-v6.2, dev cases from v6.3-v6.5, and test cases from v6.6-v6.10.

Validation Harness

Each DRIFTBENCH case includes everything needed to validate a candidate patch through KDRIFT's four-stage pipeline. The validation harness is packaged as a self-contained environment:

• Dockerfile: Builds a container with the exact kernel source, toolchain, Sparse, Smatch, and QEMU version needed for the case. Pinned to specific package versions for reproducibility.
• Kernel config: Minimal defconfig that enables the target driver. We use allnoconfig as a base and add only the Kconfig symbols needed, reducing build time from minutes to seconds.
• Validation script: Orchestrates the four stages: apply patch, compile, run Sparse+Smatch, boot in QEMU, run lockdep+KASAN tests. Each stage produces a structured pass/fail result with diagnostic output.

Running validation is a single command:

# Validate a candidate patch against case 047
driftbench run --case cases/case_047/ \
  --patch candidate.patch \
  --stages all

# Output:
# Stage 1 (compile):     PASS  [2.3s]
# Stage 2 (sparse):      PASS  [1.1s]
# Stage 3 (qemu-boot):   PASS  [18.4s]
# Stage 4 (runtime):     FAIL  [34.2s]
#   lockdep: possible deadlock in example_ioctl
#
# AST similarity to golden: 0.84
# Overall: FAIL (3/4 stages passed)

The harness also computes AST similarity between the candidate patch and the golden patch. This is not a pass/fail metric — it is a diagnostic that helps understand how close a failed patch is to the correct solution. A patch that fails Stage 4 with 0.90 AST similarity is much closer to correct than one that fails Stage 2 with 0.30 similarity.

The Numbers

The median case complexity is 3 files and 47 lines changed. This is deliberately in the range where the task is non-trivial (not a single-line rename) but tractable (not a full subsystem rewrite). The distribution has a long tail — the most complex case touches 11 files and modifies 340 lines across a cross-subsystem DMA API change.

The 48/23 split between API migration cases and regression cases means DRIFTBENCH evaluates both proactive and reactive capabilities. Migration cases test whether a tool can adapt a driver to a known API change. Regression cases test whether a tool can identify that a kernel change broke a driver and generate the appropriate fix — a harder task because the tool must first diagnose the problem.

Comparison with Prior Work

To our knowledge, DRIFTBENCH is the first benchmark specifically designed for kernel code migration. The closest prior work includes:

• SWE-bench (Jimenez et al., 2024): General-purpose software engineering benchmark. Python-centric, no kernel-specific validation.
• Defects4J (Just et al., 2014): Java bug benchmark. Focuses on bug detection and repair, not API migration.
• CodeXGLUE (Lu et al., 2021): Code understanding/generation benchmark. Includes code repair tasks but not system-level migration.
• Coccinelle test suites: Evaluation cases for the Coccinelle semantic patching tool. Covers syntactic transformations but not the semantic changes that DRIFTBENCH emphasizes.

None of these include kernel-specific validation (Sparse, QEMU boot, lockdep/KASAN), none enforce temporal splits to prevent leakage, and none cover the full spectrum of kernel API change types.

Availability

DRIFTBENCH is released on HuggingFace under the Apache 2.0 license. The dataset includes all 71 cases with metadata, golden patches, validation scripts, and Dockerfiles. The driftbench CLI tool handles case extraction, validation, and result aggregation.

We welcome contributions: new cases from kernel versions beyond v6.10, additional subsystem coverage, and improvements to the validation harness. The extraction pipeline is documented and designed to be extended by the community.

If you evaluate a kernel migration tool and want to report results on DRIFTBENCH, please use the official test split and report all four validation stages separately. Reporting only compile-pass rates without Sparse, QEMU, and runtime results is not meaningful for this domain.