Cross-Kernel Testing

Use this file for testing whether an eBPF object loads and attaches across multiple kernel versions.

When to Use This File

• validating kernel compatibility in CI
• checking whether a compiled object loads on older and newer kernels
• reproducing "works on my kernel only" issues
• testing feature fallback decisions against a kernel matrix

Recommended Approach

For CI-based cross-kernel loading tests, use lightweight VMs rather than trying to swap kernels in-place on the CI runner.

The most practical pattern here is:

• use cilium/little-vm-helper (LVH) to boot a lightweight VM per kernel under test
• choose a prebuilt LVH image variant such as kind
• mount the repository or build artifacts into the guest
• run bpftool prog load or the userspace loader inside that guest
• treat load success or verifier failure as the compatibility signal

This is a good fit for GitHub Actions.

Why This Pattern Works

The host runner kernel does not change during CI, so testing against multiple kernels needs virtualization. LVH gives a lightweight way to boot kernel-specific guests and run your load/attach checks inside them.

Use it when the question is:

• "does this object load on kernel X, Y, and Z?"
• "did my new helper/map/hook choice narrow compatibility?"
• "which kernel is the first one where this object stops loading?"

Key Building Blocks

little-vm-helper

little-vm-helper is a VM management tool used for kernel-dependent development and testing, including BPF-focused workflows.

What matters for this skill:

• it can boot lightweight VMs for development and testing
• it can download or build kernels
• it supports pulling images prepared by little-vm-helper-images

little-vm-helper Images

LVH images are published separately and come in variants such as:

• base
• kind
• complexity-test

Choose based on how much userspace you need inside the guest. If unsure, kind is a practical default because it is relatively complete.

Practical CI Pattern

The basic GitHub Actions approach is:

1. Build or download the .bpf.o artifact on the runner.
2. Create a matrix of kernel image tags.
3. Launch one LVH VM per matrix entry.
4. Mount the repository or artifact directory into the guest.
5. Inside the guest, print uname -a, then try bpftool prog load.
6. Fail the job if the load fails.

Minimal pattern:

name: Cross Kernel Load

on:
  workflow_dispatch:

jobs:
  load-vm:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        kernel:
          - "6.6-20250616.013250-amd64"
          - "5.15-20250616.013250-amd64"
          - "5.10-20250610.043823-amd64"

    steps:
      - uses: actions/checkout@v4

      - name: Build object
        run: |
          clang -O2 -g -target bpf -D__TARGET_ARCH_x86 -I. -c program.bpf.c -o program.bpf.o

      - name: Provision LVH VM
        uses: cilium/little-vm-helper@v0.0.23
        with:
          test-name: load-test
          image: "kind"
          image-version: ${{ matrix.kernel }}
          host-mount: .
          install-dependencies: "true"
          cmd: |
            uname -a
            cd /host
            sudo bpftool prog load ./program.bpf.o /sys/fs/bpf/test

What To Assert

Start with the smallest useful assertion:

• object loads successfully

Then expand only if needed:

• attach succeeds
• maps are created as expected
• the userspace loader starts correctly
• a smoke event is observed

Recommended progression:

• first gate on bpftool prog load
• then add attach tests
• then add behavior checks only if the project needs them

Good Kernel Matrix Strategy

Include:

• your oldest supported kernel
• one or two common production kernels
• your newest target kernel

If the project uses newer features, make the expected cutoff explicit. Example:

• RINGBUF implies 5.8+
• USER_RINGBUF implies 6.2+
• TCX implies 6.6+

This makes CI failures informative instead of surprising.

Failure Interpretation

If one kernel fails and another passes, classify the failure:

• unsupported helper/map/program type
• verifier rejection caused by older kernel behavior
• missing runtime config or kernel option
• image/userspace mismatch rather than BPF logic

First follow-ups:

uname -a
bpftool feature
bpftool prog load ./program.bpf.o /sys/fs/bpf/test

If needed, escalate to:

• verifier log inspection
• checking guest userspace packages
• comparing image variants such as base vs kind

Practical Recommendations

• keep the first cross-kernel test focused on loadability, not full end-to-end behavior
• prefer a matrix that reflects real support policy, not every kernel you can find
• keep fail-fast: false so you see the whole compatibility picture
• print the guest kernel version in every run
• mount the host workspace into the guest so artifacts stay easy to inspect

When To Mention This Workflow

Bring this file into the answer when the user asks about:

• testing on multiple kernels
• CI compatibility gates
• validating the oldest supported kernel
• loading eBPF in GitHub Actions with a kernel matrix

References

• Blog inspiration: https://h0x0er.github.io/blog/2025/06/21/ebpf-loading-in-different-kernels/
• LVH project: https://github.com/cilium/little-vm-helper
• LVH image tags and variants: https://github.com/cilium/little-vm-helper-images and https://quay.io/organization/lvh-images