Introduction
This blog post uses Fedora 40 and the 6.10 kernel. The focus is to get the technology working on a 'real' system via a series of experiments. Serial experiments help build-up the knowledge, experience, tooling etc. to handle real-world use-cases.
In this part we'll try using the Fedora-native solution (kpatch). This is a vanilla solution for people working in reasonable environments, however, it is a good idea to cover the normal solution so that people don't get mislead about the most appropriate solution for the problem they are usually facing.
I rarely encounter the "normal" cases. Part 1 covers one of these less ordinary situations which are useful preparation for dealing with situations where the kernel sources or headers are unavailable for one reason or another. If livepatching is enabled, it's still possible to patch the kernel, although a bit more wrangling is needed.
The article is a work in progress. Sadly, I am having difficulties with the tool on Fedora 40. I will retry with CentOS7 CentOS8 and then work forward.
Preparation
I would recommend following part 1 of the blog series since passwordless sudo is setup there. For the sake of brevity I have ommited the preparation steps from that article and only reproduced the kpatch-specific steps.
Follow the install instructions, reproduced briefly below Clone the kpatch repository:
git clone https://github.com/dynup/kpatch
Install the dependencies and install the kpatch binary:
make dependencies
make
sudo make install
Sanity test
The example from the Quick Start can be used. I can confirm it is not working on Fedora 40 with a 6.10 kernel. However, if one regenerates the patch one can get something similar to the following:
--- src.orig/fs/proc/meminfo.c 2024-08-24 17:47:37.879473974 -0400
+++ src/fs/proc/meminfo.c 2024-08-24 17:48:19.342729445 -0400
@@ -129,7 +129,7 @@
seq_printf(m, "VmallocTotal: %8lu kB\n",
(unsigned long)VMALLOC_TOTAL >> 10);
show_val_kb(m, "VmallocUsed: ", vmalloc_nr_pages());
- show_val_kb(m, "VmallocChunk: ", 0ul);
+ show_val_kb(m, "VMALLOCCHUNK: ", 0ul);
show_val_kb(m, "Percpu: ", pcpu_nr_pages());
memtest_report_meminfo(m);
Note that the above indentation is actually tabs. You will likely have to remove all the whitespace and hit tab
to get the patch to apply.
This will work if you run kpatch-build against it:
kpatch-build meminfo.patch
I would highly recommend running the above in a tmux session. The above command will take a while to complete.
Note that kpatch will rebuild the linux kernel which will take a long time. The required resources for a kernel compilation are usually:
8+ CPU cores
1 GB of RAM per CPU core
~25GB of space
I ended up giving up running the demo on my laptop (12 cores) and switched to a proper (64 core EPYC Milan) system.
You might need to run yum --enablerepo updates-debuginfo install kernel-debuginfo before kpatch-build will be able to start.
Also make sure that your runtime kernel (visible with uname -r) matches your latest installed kernel (visible with rpm -qa | grep kernel)
otherwise you are likely to see errors due to the missing kernel-debuginfo package.
My thoughts on kpatch
It compiles the entire kernel to be able to apply a 10 line patch. It seems to be a developer tool, and not an ops tool. For comparison, in part1, the compiler was barely involved at all.
It seems decent enough for a developer tool. How much of a pain it is to use long-term is a different question. I might have to revisit the tool once I have more experience writing my own. It is nice that it is open source and props to the different teams from Oracle, SUSE, RedHat for collaborating and making a unified kernel solution (even though the userspace tools are different).
However if you are a kernel hacker, or want to replace random functions in the kernel with Rust, it's not the right tool. A lot of problems occur "in the wild" where it's necessary to MacGyver a solution. It might become necessary to reimplement livepatching using just ftrace, or deal with missing symbols, kernel headers, or the like.
Well-maintained tools that can work with ELF binaries directly (or binaries in general) are hard to find. Better tools that deal with object code (i.e. the stuff of linkers) and a reliable decompiler-compiler combination would do wonders for embedded, airgapped, and other difficult systems (EOL systems).
In an ideal world, it should be possible to just edit the kernel on the fly (or any process on the system) through a TUI without having to install a huge number of tools. It seems very strange not to be able to edit object code (and patch it).