Introduction
This blog post uses Fedora 40 and the 6.10 kernel. The focus is to get the technology working on a 'real' system via a series of experiments. Serial experiments help build-up the knowledge, experience, tooling etc. to handle real-world use-cases.
Kernel livepatching is a nice technology that doesn't see heavy usage outside of some specific (read, expensive) use cases. The technology is simple enough to understand: A function is marked for replacement and the kernel attempts to replace all references to it. The kernel can do that since it schedules all tasks, and in the pauses between tasks (while context switching) the kernel can modify the task environment while the task is effectively "frozen".
I hope to explore in this blog series the more unusual ways the technology can be used to:
-
test changes
-
debug problems
-
address security issues (specifically, unusual ways to "fix" security issues, like disabling syscalls)
Preparation
Install the following generic dependencies:
dnf -y install git make gcc wget tar
Install Linux kernel headers:
dnf -y install kernel-devel
Sanity test
Before creating a new livepatch module it is a good idea that the setup can compile a known-good livepatching sample.
It is highly recommended to create a separate user to run the kernel Makefiles, as they have had bugs (a long time ago)
that would delete the root filesystem (i.e. a classic rm -rf / hanging around in a makefile). I run these commands
in a VM, so the precaution isn't as relevant.
To create a new user and switch to it, run:
adduser build
su - build
gpasswd -a build wheel
To enable passwordless sudo on the user, run visudo and uncomment the following line:
## Same thing without a password
%wheel ALL=(ALL) NOPASSWD: ALL
Create a directory for the sanity test:
mkdir -p sanity-test
cd sanity-test
wget https://raw.githubusercontent.com/torvalds/linux/master/samples/livepatch/livepatch-sample.c
Create a Makefile:
obj-m = livepatch-sample.o
Compile the livepatch sample:
make -C/lib/modules/$(uname -r)/build M=$(pwd) modules
The compilation will output something similar to the following:
make -C/lib/modules/$(uname -r)/build M=$(pwd) modules
make: Entering directory '/usr/src/kernels/6.10.4-200.fc40.x86_64'
CC [M] /home/build/sanity-test/livepatch-sample.o
MODPOST /home/build/sanity-test/Module.symvers
CC [M] /home/build/sanity-test/livepatch-sample.mod.o
LD [M] /home/build/sanity-test/livepatch-sample.ko
BTF [M] /home/build/sanity-test/livepatch-sample.ko
Skipping BTF generation for /home/build/sanity-test/livepatch-sample.ko due to unavailability of vmlinux
make: Leaving directory '/usr/src/kernels/6.10.4-200.fc40.x86_64'
There will be a number of files generated, we're only interested in the livepatch-sample.ko file:
[build@fedora sanity-test]$ ls
livepatch-sample.c livepatch-sample.ko
Insert the module into the running kernel with:
sudo insmod livepatch-sample.ko
On Fedora 40 this will result in the following message in the dmesg -w output:
[ 1693.302006] livepatch_sample: loading out-of-tree module taints kernel.
[ 1693.302985] livepatch_sample: tainting kernel with TAINT_LIVEPATCH
[ 1693.303560] livepatch_sample: module verification failed: signature and/or required key missing - tainting kernel
[ 1693.305566] livepatch: enabling patch 'livepatch_sample'
[ 1693.311902] livepatch: 'livepatch_sample': starting patching transition
[ 1694.920570] livepatch: 'livepatch_sample': patching complete
Configuring kernel module signatures is a separate topic. The important thing is the last line: "patching complete" which indicates that all references to the function have been replaced. We can check that the livepatch has been applied by running:
cat /proc/cmdline
this has been live patched
To disable the patch, we can run:
echo "0" > /sys/kernel/livepatch/livepatch_sample/enabled
The output of dmesg -w can again be checked to see the progres of the unpatching operation:
[ 1693.311902] livepatch: 'livepatch_sample': starting patching transition
[ 1694.920570] livepatch: 'livepatch_sample': patching complete
[ 1999.451428] livepatch: 'livepatch_sample': starting unpatching transition
[ 2000.945583] livepatch: 'livepatch_sample': unpatching complete
Now, remove the module from the kernel:
rmmod livepatch_sample
The cmdline can again be checked to see that it is back to the original value:
cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.10.4-200.fc40.x86_64 root=UUID=356cb170-4518-42db-b02a-ad6530cf8956 ro console=tty0 rd_NO_PLYMOUTH console=ttyS0,115200
First livepatch
Let's see if we can patch something a bit more exciting, like the output of cat /proc/cpuinfo.
For that, a copy of the original function is required. The easiest way to obtain a copy of the kernel .src.rpm package is to
navigate to https://koji.fedoraproject.org and search for the "kernel" package, or to use dnf directly:
dnf download --source kernel
Depending on how unreliable the local mirrror is, it might be better to use koji. For the system I am using the correct kernel .src.rpm package is:
wget https://kojipkgs.fedoraproject.org//packages/kernel/6.10.4/200.fc40/src/kernel-6.10.4-200.fc40.src.rpm
Check the output of rpm -qa | grep kernel to find the correct srpm version to download.
Unpack the package with:
rpm -Uvh kernel-6.10.4-200.fc40.src.rpm
Untar the kernel source in rpmbuild/SOURCES:
tar -xvf linux-6.10.4.tar.xz
The file we are looking for is for the x86 architecture:
/home/build/rpmbuild/SOURCES/linux-6.10.4/arch/x86/kernel/cpu/proc.c
Let's replace a function that seems to be easy to replace on x86_64: show_cpuinfo.
Dealing with references is a chore, so the function has been modified to remove some
features (displaying CPU bugs, displaying power management).
The whole file would look similar to this:
// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>
#include <linux/livepatch.h>
#include <linux/seq_file.h>
// See linux-6.10.4/arch/x86/kernel/cpu/proc.c for detailed licensing information
static void show_cpuinfo_core(struct seq_file *m, struct cpuinfo_x86 *c,
unsigned int cpu)
{
seq_printf(m, "physical id\t: %d\n", c->topo.pkg_id);
seq_printf(m, "siblings\t: %d\n",
cpumask_weight(topology_core_cpumask(cpu)));
seq_printf(m, "core id\t\t: %d\n", c->topo.core_id);
seq_printf(m, "cpu cores\t: %d\n", c->booted_cores);
seq_printf(m, "apicid\t\t: %d\n", c->topo.apicid);
seq_printf(m, "initial apicid\t: %d\n", c->topo.initial_apicid);
}
static void show_cpuinfo_misc(struct seq_file *m, struct cpuinfo_x86 *c)
{
seq_printf(m,
"fpu\t\t: yes\n"
"fpu_exception\t: yes\n"
"cpuid level\t: %d\n"
"wp\t\t: yes\n",
c->cpuid_level);
}
static int patched_show_cpuinfo(struct seq_file *m, void *v)
{
struct cpuinfo_x86 *c = v;
unsigned int cpu;
cpu = c->cpu_index;
seq_printf(m, "processor\t: %u\n"
"vendor_id\t: %s\n"
"cpu family\t: %s\n"
"model\t\t: %s\n"
"model name\t: %s\n",
cpu,
"Anderson Robotics",
"Sentient Rocks",
"REDACTED",
"Standard Lightning-infused rock MK2");
if (c->x86_stepping || c->cpuid_level >= 0)
seq_printf(m, "stepping\t: %d\n", c->x86_stepping);
else
seq_puts(m, "stepping\t: unknown\n");
if (c->microcode)
seq_printf(m, "microcode\t: 0x%x\n", c->microcode);
/* Cache size */
if (c->x86_cache_size)
seq_printf(m, "cache size\t: %u KB\n", c->x86_cache_size);
show_cpuinfo_core(m, c, cpu);
show_cpuinfo_misc(m, c);
seq_printf(m, "\nbogomips\t: %lu.%02lu\n",
c->loops_per_jiffy/(500000/HZ),
(c->loops_per_jiffy/(5000/HZ)) % 100);
if (c->x86_tlbsize > 0)
seq_printf(m, "TLB size\t: %d 4K pages\n", c->x86_tlbsize);
seq_printf(m, "clflush size\t: %u\n", c->x86_clflush_size);
seq_printf(m, "cache_alignment\t: %d\n", c->x86_cache_alignment);
seq_printf(m, "address sizes\t: %u bits physical, %u bits virtual\n",
c->x86_phys_bits, c->x86_virt_bits);
seq_puts(m, "\n\n");
return 0;
}
static struct klp_func lfuncs[] = {
{
.old_name = "show_cpuinfo",
.new_func = patched_show_cpuinfo,
}, { }
};
static struct klp_object lobjs[] = {
{
.funcs = lfuncs,
}, { }
};
// The .replace = true indicates that loading this patch replaces
// the previously loaded patch
// See https://docs.kernel.org/livepatch/cumulative-patches.html
// Hence, the highlander references
static struct klp_patch lpatch = {
.mod = THIS_MODULE,
.objs = lobjs,
.replace = true,
};
static int linit(void)
{
pr_info("Adding highlander livepatch...");
return klp_enable_patch(&lpatch);
}
static void lexit(void)
{
pr_info("Removing highlander livepatch...");
}
module_init(linit);
module_exit(lexit);
MODULE_LICENSE("GPL");
MODULE_INFO(livepatch, "Y");
Save the above file as highlander.c
Modify Makefile to contain:
obj-m=highlander.o
And compile the module with:
make -C/lib/modules/$(uname -r)/build M=$(pwd) modules
Load the module with:
sudo insmod highlander.ko
You should now be able to see the following when running cat /proc/cpuinfo:
cat /proc/cpuinfo | head -n 25
processor : 0
vendor_id : Anderson Robotics
cpu family : Sentient Rocks
model : REDACTED
model name : Standard Lightning-infused rock MK2
stepping : 1
microcode : 0xa404102
cache size : 512 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
bogomips : 5789.09
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
Conclusion of part 1
There's more that could be explored here but I am time-limited today.
The two important take-aways are:
1) Resolving references to functions is non-trivial. You might have to copy-paste a lot of code which is non-ideal.
2) Not all functions can be patched. Check /proc/kallsyms, if the function is there, it can be patched. If the function you wish to replace is called by a function from kallsyms, (i.e. it is likely inlined) replace the kallsyms function and modify the function so that it calls the function you couldn't patch otherwise. For example:
[ 5011.921889] livepatch: symbol 'show_cpuinfo_misc' not found in symbol table
While it's not possible to directly patch show_cpuinfo_misc, show_cpuinfo can
be patched and the call to the function show_cpuinfo_misc replaced with a different
function.
References
https://docs.kernel.org/livepatch/livepatch.html
LFD420 from the Linux Foundation or Jerry Cooperstein's books are a good reference.