linux/lib
Gary Guo 3a2486cc1d kbuild: rust: provide an option to inline C helpers into Rust
A new experimental Kconfig option, `RUST_INLINE_HELPERS` is added to
allow C helpers (which were created to allow Rust to call into
inline/macro C functions without having to re-implement the logic in
Rust) to be inlined into Rust crates without performing global LTO.

If the option is enabled, the following is performed:
* For helpers, instead of compiling them to an object file to be linked
  into vmlinux, they're compiled to LLVM IR bitcode. Two versions are
  generated: one for built-in code (`helpers.bc`) and one for modules
  (`helpers_module.bc`, with -DMODULE defined). This ensures that C
  macros/inlines that behave differently for modules (e.g. static calls)
  function correctly when inlined.
* When a Rust crate or object is compiled, instead of generating an
  object file, LLVM bitcode is generated.
* llvm-link is invoked with --internalize to combine the helper bitcode
  with the crate bitcode. This step is similar to LTO, but this is much
  faster since it only needs to inline the helpers.
* clang is invoked to turn the combined bitcode into a final object file.
* Since clang may produce LLVM bitcode when LTO is enabled, and objtool
  requires ELF input, $(cmd_ld_single) is invoked to ensure the object
  is converted to ELF before objtool runs.

The --internalize flag tells llvm-link to treat all symbols in
helpers.bc using `internal` linkage [1]. This matches the behavior of
`clang` on `static inline` functions, and avoids exporting the symbol
from the object file.

To ensure that RUST_INLINE_HELPERS is not incompatible with BTF, we pass
the -g0 flag when building helpers. See commit 5daa0c35a1 ("rust:
Disallow BTF generation with Rust + LTO") for details.

We have an intended triple mismatch of `aarch64-unknown-none` vs
`aarch64-unknown-linux-gnu`, so we pass --suppress-warnings to llvm-link
to suppress it.

I considered adding some sort of check that KBUILD_MODNAME is not
present in helpers_module.bc, but this is actually not so easy to carry
out because .bc files store strings in a weird binary format, so you
cannot just grep it for a string to check whether it ended up using
KBUILD_MODNAME anywhere.

[ Andreas writes:

    For the rnull driver, enabling helper inlining with this patch
    gives an average speedup of 2% over the set of 120 workloads that
    we publish on [2].

    Link: https://rust-for-linux.com/null-block-driver [2]

  This series also uncovered a pre-existing UB instance thanks to an
  `objtool` warning which I noticed while testing the series (details
  in the mailing list).

      - Miguel ]

Link: https://github.com/llvm/llvm-project/pull/170397 [1]
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Co-developed-by: Matthew Maurer <mmaurer@google.com>
Signed-off-by: Matthew Maurer <mmaurer@google.com>
Signed-off-by: Gary Guo <gary@garyguo.net>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Tested-by: Andreas Hindborg <a.hindborg@kernel.org>
Reviewed-by: Andreas Hindborg <a.hindborg@kernel.org>
Link: https://patch.msgid.link/20260203-inline-helpers-v2-3-beb8547a03c9@google.com
[ Some changes, apart from the rebase:

  - Added "(EXPERIMENTAL)" to Kconfig as the commit mentions.

  - Added `depends on ARM64 || X86_64` and `!UML` for now, since this is
    experimental, other architectures may require other changes (e.g.
    the issues I mentioned in the mailing list for ARM and UML) and they
    are not really tested so far. So let arch maintainers pick this up
    if they think it is worth it.

  - Gated the `cmd_ld_single` step also into the new mode, which also
    means that any possible future `objcopy` step is done after the
    translation, as expected.

  - Added `.gitignore` for `.bc` with exception for existing script.

  - Added `part-of-*` for helpers bitcode files as discussed, and
    dropped `$(if $(filter %_module.bc,$@),-DMODULE)` since `-DMODULE`
    is already there (would be duplicated otherwise).

  - Moved `LLVM_LINK` to keep binutils list alphabetized.

  - Fixed typo in title.

  - Dropped second `cmd_ld_single` commit message paragraph.

      - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2026-03-30 02:03:52 +02:00
..
842
crc
crypto lib/crypto: tests: Add a .kunitconfig file 2026-03-02 15:35:05 -08:00
dim Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
fonts
kunit kunit: reduce stack usage in kunit_run_tests() 2026-03-02 10:11:06 -07:00
lz4
lzo
math
pldmfw Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
raid6 First set of RISC-V updates for v6.19-rc1 2025-12-05 16:26:57 -08:00
reed_solomon Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_fortify fortify: Cleanup temp file also on non-successful exit 2026-01-14 19:49:55 -08:00
tests Convert more 'alloc_obj' cases to default GFP_KERNEL arguments 2026-02-21 20:03:00 -08:00
vdso vdso/gettimeofday: Force inlining of __cvdso_clock_getres_common() 2026-01-26 22:27:07 +01:00
xz Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
zlib_deflate
zlib_dfltcc
zlib_inflate Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
zstd
.gitignore
Kconfig lib/glob: convert selftest to KUnit 2026-01-26 19:07:13 -08:00
Kconfig.debug kbuild: rust: provide an option to inline C helpers into Rust 2026-03-30 02:03:52 +02:00
Kconfig.kasan
Kconfig.kcsan
Kconfig.kfence
Kconfig.kgdb
Kconfig.kmsan
Kconfig.ubsan Kconfig.ubsan: Remove CONFIG_UBSAN_REPORT_FULL from documentation 2026-01-07 12:16:03 -08:00
Makefile mm.git review status for linus..mm-nonmm-stable 2026-02-12 12:13:01 -08:00
alloc_tag.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
argv_split.c
ashldi3.c
ashrdi3.c
asn1_decoder.c
asn1_encoder.c
assoc_array.c Convert more 'alloc_obj' cases to default GFP_KERNEL arguments 2026-02-21 20:03:00 -08:00
atomic64.c
atomic64_test.c
audit.c
base64.c
bcd.c
bch.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
bitmap-str.c
bitmap.c
bitrev.c
bootconfig-data.S
bootconfig.c bootconfig: Terminate value search if it hits a newline 2026-02-05 22:21:07 +09:00
bsearch.c
btree.c
bucket_locks.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
bug.c bug: Hush suggest-attribute=format for __warn_printf() 2025-12-12 10:26:26 +01:00
build_OID_registry oid_registry: allow arbitrary size OIDs 2026-01-20 19:44:15 -08:00
buildid.c procfs: avoid fetching build ID while holding VMA lock 2026-02-05 14:10:00 -08:00
bust_spinlocks.c
cache_maint.c
check_signature.c
checksum.c
closure.c
clz_ctz.c
clz_tab.c
cmdline.c
cmpdi2.c
cmpxchg-emu.c
codetag.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
compat_audit.c
cpu_rmap.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
cpumask.c
ctype.c
debug_info.c
debug_locks.c
debugobjects.c debugobject: Make it work with deferred page initialization - again 2026-02-10 13:26:19 +01:00
dec_and_lock.c compiler-context-analysis: Remove __cond_lock() function-like helper 2026-01-05 16:43:33 +01:00
decompress.c
decompress_bunzip2.c
decompress_inflate.c
decompress_unlz4.c
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c xz: fix arm fdt compile error for kmalloc replacement 2026-02-22 12:05:31 -08:00
decompress_unzstd.c
devmem_is_allowed.c
devres.c
dhry.h
dhry_1.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
dhry_2.c
dhry_run.c
digsig.c
dump_stack.c
dynamic_debug.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
dynamic_queue_limits.c
earlycpio.c
errname.c
error-inject.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
errseq.c
extable.c
fault-inject-usercopy.c
fault-inject.c
fdt.c
fdt_addresses.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
find_bit.c lib/find_bit: fix uninitialized variable use in FIND_NTH_BIT 2026-02-08 18:47:29 -05:00
find_bit_benchmark.c
find_bit_benchmark_rust.rs
flex_proportions.c flex_proportions: make fprop_new_period() hardirq safe 2026-01-26 19:03:46 -08:00
fw_table.c
genalloc.c
generic-radix-tree.c
glob.c
group_cpus.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
hexdump.c kernel.h: drop hex.h and update all hex.h users 2026-01-20 19:44:19 -08:00
hweight.c
idr.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
inflate.c
interval_tree.c
interval_tree_test.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
iomap.c
iomap_copy.c
iomem_copy.c
iommu-helper.c
iov_iter.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
irq_poll.c
irq_regs.c
is_single_threaded.c
kasprintf.c
kfifo.c kfifo: fix kmalloc_array_node() argument order 2026-01-26 19:07:09 -08:00
klist.c
kobject.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
kobject_uevent.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
kstrtox.c lib/kstrtox: fix kstrtobool() docstring to mention enabled/disabled 2026-01-26 19:07:10 -08:00
kstrtox.h
linear_ranges.c
list_debug.c
list_sort.c
llist.c
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-rtmutex.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c
lockref.c compiler-context-analysis: Remove __cond_lock() function-like helper 2026-01-05 16:43:33 +01:00
logic_iomem.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
logic_pio.c
lru_cache.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
lshrdi3.c
lwq.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
maple_tree.c
memcat_p.c
memory-notifier-error-inject.c
memregion.c
memweight.c
min_heap.c
muldi3.c
net_utils.c
netdev-notifier-error-inject.c
nlattr.c
nmi_backtrace.c
notifier-error-inject.c
notifier-error-inject.h
objagg.c Convert remaining multi-line kmalloc_obj/flex GFP_KERNEL uses 2026-02-22 08:26:33 -08:00
objpool.c objpool: fix the overestimation of object pooling metadata size 2026-02-12 15:45:57 -08:00
of-reconfig-notifier-error-inject.c
oid_registry.c
once.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
packing.c
packing_test.c
parman.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
parser.c
percpu-refcount.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
percpu_counter.c
percpu_test.c
plist.c treewide: Update email address 2026-01-11 06:09:11 -10:00
pm-notifier-error-inject.c
polynomial.c
radix-tree.c
radix-tree.h
random32.c
ratelimit.c
rbtree.c
rbtree_test.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
rcuref.c
ref_tracker.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
refcount.c
rhashtable.c rhashtable: Enable context analysis 2026-01-05 16:43:35 +01:00
sbitmap.c
scatterlist.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
seq_buf.c
sg_pool.c
sg_split.c treewide: Replace kmalloc with kmalloc_obj for non-scalar types 2026-02-21 01:02:28 -08:00
siphash.c
smp_processor_id.c
sort.c
stackdepot.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
stmp_device.c
string.c
string_helpers.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
strncpy_from_user.c
strnlen_user.c
sys_info.c
syscall.c
test-kstrtox.c
test_bitmap.c
test_bitops.c
test_bpf.c Convert 'alloc_flex' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_context-analysis.c compiler-context-analysis: Remove __assume_ctx_lock from initializers 2026-01-28 20:45:25 +01:00
test_debug_virtual.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_dynamic_debug.c
test_firmware.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_fpu.h
test_fpu_glue.c
test_fpu_impl.c
test_free_pages.c
test_hexdump.c
test_hmm.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_hmm_uapi.h
test_ida.c
test_kho.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_kmod.c
test_lockup.c
test_maple_tree.c
test_memcat_p.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_meminit.c
test_module.c
test_objagg.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_objpool.c
test_parman.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_ref_tracker.c
test_rhashtable.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_static_key_base.c
test_static_keys.c
test_sysctl.c
test_ubsan.c
test_vmalloc.c Convert 'alloc_obj' family to use the new default GFP_KERNEL argument 2026-02-21 17:09:51 -08:00
test_xarray.c
textsearch.c
timerqueue.c
trace_readwrite.c
ts_bm.c
ts_fsm.c
ts_kmp.c
ubsan.c
ubsan.h
ucmpdi2.c
ucs2_string.c
union_find.c
usercopy.c
uuid.c kernel.h: drop hex.h and update all hex.h users 2026-01-20 19:44:19 -08:00
vsprintf.c kernel.h: drop hex.h and update all hex.h users 2026-01-20 19:44:19 -08:00
win_minmax.c
xarray.c
xxhash.c