The makefile logic to detect if rust is enabled is not working
the way it was expected: instead of using the current setup
for CONFIG_RUST, it uses a cached version from a previous build.
The root cause is that the current logic inside docs/Makefile
uses a cached version of CONFIG_RUST, from the last time a non
documentation target was executed. That's perfectly fine for
Sphinx build, as it doesn't need to read or depend on any
CONFIG_*.
So, instead of relying at the cache, move the logic to the
wrapper script and let it check the current content of .config,
to verify if CONFIG_RUST was selected.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <c06b1834ef02099735c13ee1109fa2a2b9e47795.1763722971.git.mchehab+huawei@kernel.org>
Now that we have tools/lib/python for our Python modules, turn them into
proper packages with a single namespace so that everything can just use
tools/lib/python in sys.path. No functional change.
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-3-corbet@lwn.net>
"scripts/lib" was always a bit of an awkward place for Python modules. We
already have tools/lib; create a tools/lib/python, move the libraries
there, and update the users accordingly.
While at it, move the contents of tools/docs/lib. Rather than make another
directory, just put these documentation-oriented modules under "kdoc".
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-2-corbet@lwn.net>
The option list to sphinx-build via SPHINXOPTS should have higher
priority than those the wrapper comes up with.
sphinx-build will choose the latest one if there are duplicates.
To restore the behavior of Makefile era, when the documentation builds
at https://www.kernel.org/doc/html/next/ had been depending on it,
reorder the flag list.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Closes: https://lore.kernel.org/20251007-awesome-guan-of-greatness-e6ec75@lemur/
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/c35e690f-0579-49cb-850c-07af98e5253a@gmail.com/
Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <eaf4bfd8-fb80-45d0-b3ec-4047692ebbed@gmail.com>
As reported by Konstantin, sphinx-build -q is a boolean, not an integer.
Fix the code.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Closes: https://lore.kernel.org/all/871pnepxfy.fsf@trenco.lwn.net/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <cafa10cddce3e5342a66c73f3f51a17fb6c7f5d3.1759851791.git.mchehab+huawei@kernel.org>
Fix the logic which checks for pdflatex. Currently, it complains
for both pdfdocs and latexdocs, but for the latter, all it is
needed is Sphinx.
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/linux-doc/cover.1758881658.git.mchehab+huawei@kernel.org/T/#ma81ff2e11b8579e5edc23e4381e464081ae668b7
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <313df7b4aab653e8fc40c30120c0dbebf8a0bcb1.1759328070.git.mchehab+huawei@kernel.org>
Most targets have two steps:
- step 1: run sphinx-build;
- step 2: run a post-build logic.
The second step can be as simple as copying static files like CSS,
but may may also envolve running make. allowing to skip the first
step helps debugging what's broken, and also allows using make
command line arguments like --ignore-errors.
Add an option to skip step 1.
Requested-by: Akira Yokosawa <akiyks@gmail.com>
Link: https://lore.kernel.org/linux-doc/5031e0c4-f17e-41b8-8955-959989e797f2@gmail.com/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <30dae295606ae1735f3bbeef4ca041a76dcd4540.1758539031.git.mchehab+huawei@kernel.org>
On recent versions of openSUSE Tumbleweed, sphinx-buildis is no longer
a Python script, but something else. Such change is due to how
it now handles alternatives:
https://en.opensuse.org/openSUSE:Migrating_to_libalternatives_with_alts
The most common approach that distros use for alternatives is via
symlinks:
lrwxrwxrwx 1 root root 22 out 31 2024 /usr/bin/java -> /etc/alternatives/java
lrwxrwxrwx 1 root root 37 mar 5 2025 /etc/alternatives/java -> /usr/lib/jvm/java-21-openjdk/bin/java
With such approach, one can sun the script with either:
<sphinx>
python3 <script>
However, openSUSE's implementation uses an ELF binary (/usr/bin/alts),
which breaks the latter format.
It is needed to allow users to specify the Python version to be
used while building docs, as some distros like Leap 15.x are
shipped with:
- older, unsupported python3/python3-sphinx packages;
- more modern python3xx/python3xx-sphinx packages that work
properly.
On such distros, building docs require running make with:
make PYTHON3=python3.11 htmldocs
Heh, even on more moderen distros where python3-sphinx
is supported, one may still want to use a newer package,
for instance, due to performance issues, as:
- with Python < 3.11, kernel-doc is 3 times slower;
- while building htmldocs with Python 3.13/Sphinx 8.x
takes about 3 minutes on a modern machine, using
Sphinx < 8.0 can take up to 16 minutes to build docs
(7.x are the worse ones and require lots of RAM).
So, even with not too old distros, one still may want to use
for instance PYTHON3=python3.11.
To acommodate using PYTHON3 without breaking on Tumbleweed,
add a workaround that will only use:
$(PYTHON3) sphinx-build
if PYTHON3 env var is not default.
While here, drop the special check for venv, as, with venv,
we can just call sphinx-build directly without any extra
checks.
Reported-by: Randy Dunlap <rdunlap@infradead.org>
Link: https://lore.kernel.org/all/883df949-0281-4a39-8745-bcdcce3a5594@infradead.org/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <8917f862e0b8484c68408c274129c9f37a7aefb4.1758539031.git.mchehab+huawei@kernel.org>
The code here was meant to handle 3 functions:
1. allow having a separate conf.py file, per subdir;
2. generate a list of latex documents.
3. set "subproject" tag if SPHINXDIRS points to a subdir.
We don't have (1) anymore, and (3) is now properly handled
entirely inside conf.py.
So, only (3) is still needed, and this is a single-line change
at conf.py.
So, drop it, moving the remaining code to conf.py.
While here, drop a duplicated $(RUSTDOC) command-line argument.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <ec998f9f268a401ca6aa36e3221d39c97efeccaa.1758361087.git.mchehab+huawei@kernel.org>
If sphinx-build returns an error, exit the script.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <b7e152291fadd91694cbb6b086caefa4b6470fdd.1758361087.git.mchehab+huawei@kernel.org>
Sometimes, it is desired to run Sphinx from a virtual environment.
Add a command line parameter to automatically build Sphinx from
such environment.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <e34fa63a61e75a0ec86b37c9b5fafa6677f44c6c.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Before this patch, building htmldocs on opensuseLEAP works
fine:
# make htmldocs
Available Python versions:
/usr/bin/python3.11
Python 3.6.15 not supported. Changing to /usr/bin/python3.11
Python 3.6.15 not supported. Changing to /usr/bin/python3.11
Using alabaster theme
Using Python kernel-doc
...
As the logic detects that Python 3.6 is too old and recommends
intalling python311-Sphinx. If installed, documentation builds
work like a charm.
Yet, some develpers complained that running python3.11 instead
of python3 should not happen. So, let's break the build to make
them happier:
$ make htmldocs
Python 3.6.15 not supported. Bailing out
You could run, instead:
/usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs \
--sphinxdirs=. --conf=conf.py --builddir=Documentation/output --theme= --css= \
--paper=
Python 3.6.15 not supported. Bailing out
make[2]: *** [Documentation/Makefile:76: htmldocs] Error 1
make[1]: *** [Makefile:1806: htmldocs] Error 2
make: *** [Makefile:248: __sub-make] Error 2
It should be noticed that:
1. after this change, sphinx-pre-install needs to be called
by hand:
$ /usr/bin/python3.11 tools/docs/sphinx-pre-install
Detected OS: openSUSE Leap 15.6.
Sphinx version: 7.2.6
All optional dependencies are met.
Needed package dependencies are met.
2. sphinx-build-wrapper will auto-detect python3.11 and
suggest a way to build the docs using the parameters passed
via make variables. In this specific example:
/usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs --sphinxdirs=. --conf=conf.py --theme= --css= --paper=
3. As this needs to be executed outside docs Makefile, it won't run
the validation check scripts nor build Rust documentation if
enabled, as the extra scripts are part of the docs Makefile.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <0635c311295300e9fb48c0ea607e2408910036e3.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Simplify even further the docs Makefile by moving rust build logic
to the wrapper.
After this change, running make on an environment with rust enabled
works as expected.
With CONFIG_RUST:
$ make O=/tmp/foo LLVM=1 SPHINXDIRS=peci htmldocs
make[1]: Entrando no diretório '/tmp/foo'
Using alabaster theme
Using Python kernel-doc
GEN Makefile
DESCEND objtool
CC arch/x86/kernel/asm-offsets.s
INSTALL libsubcmd_headers
CALL /new_devel/docs/scripts/checksyscalls.sh
RUSTC L rust/core.o
BINDGEN rust/bindings/bindings_generated.rs
BINDGEN rust/bindings/bindings_helpers_generated.rs
...
Without it:
$ make SPHINXDIRS=peci htmldocs
Using alabaster theme
Using Python kernel-doc
Both work as it is it is supposed to do.
After the change, it is also possible to build directly with the script
by passing "--rustodoc".
if CONFIG_RUST, this works fine:
$ ./tools/docs/sphinx-build-wrapper --sphinxdirs peci --rustdoc -- htmldocs
Using alabaster theme
Using Python kernel-doc
SYNC include/config/auto.conf
...
RUSTC L rust/core.o
...
If not, it will produce a warning that RUST may be disabled:
$ ./tools/docs/sphinx-build-wrapper --sphinxdirs peci --rustdoc -- htmldocs
Using alabaster theme
Using Python kernel-doc
***
*** Configuration file ".config" not found!
***
*** Please run some configurator (e.g. "make oldconfig" or
*** "make menuconfig" or "make xconfig").
***
make[1]: *** [/new_devel/docs/Makefile:829: .config] Error 1
make: *** [Makefile:248: __sub-make] Error 2
Ignored errors when building rustdoc: Command '['make', 'LLVM=1', 'rustdoc']' returned non-zero exit status 2.. Is RUST enabled?
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <fa1235ccf859f6ebfeef7ffba0ebde2015a75042.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Generating man files currently requires running a separate
script. The target also doesn't appear at the docs Makefile.
Add support for mandocs at the Makefile, adding the build
logic inside sphinx-build-wrapper, updating documentation
and dropping the ancillary script.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <3d248d724e7f3154f6e3a227e5923d7360201de9.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
When SPHINXDIRS is used, basename may be identical for different
files. If this happens, the summary and error detection won't be
accurate.
Fix it by using relative names from builddir.
While here, don't duplicate names. Report, instead:
- SUCCESS
output PDF file was built
- FAILED
latexmk/xelatex didn't build any PDF output
- FAILED: no .tex files were generated
Sphinx didn't build any tex file for SPHINXDIRS directories
- FAILED ({python exception})
When a concurrent.futures is catched. Usually indicates an
internal error at the build logic.
With that, building multiple dirs with the same name is reported
properly:
$ make V=1 SPHINXDIRS="admin-guide/media driver-api/media userspace-api/media" pdfdocs
Summary
=======
admin-guide/media/pdf/media.pdf : SUCCESS
driver-api/media/pdf/media.pdf : SUCCESS
userspace-api/media/pdf/media.pdf: SUCCESS
And if at least one of them fails, return code will be 1.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <d4a4f16f6c0c423ad38531a490888be3bf01e574.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
On a properly set system, LANG and LC_ALL is always defined.
However, some distros like Debian, Gentoo and their variants
start with those undefioned.
When Sphinx tries to set a locale with:
locale.setlocale(locale.LC_ALL, '')
It raises an exception, making Sphinx fail. This is more likely
to happen with test containers.
Add a logic to detect and workaround such issue by setting
locale to C.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <1d0afad8fe3d83182be3a08eb00dd71322e23e69.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Use POSIX jobserver when available or -j<number> to run PDF
builds in parallel, restoring pdf build performance. Yet,
running it when debugging troubles is a bad idea, so, when
calling directly via command line, except if "-j" is splicitly
requested, it will serialize the build.
With such change, a PDF doc builds now takes around 5 minutes
on a Ryzen 9 machine with 32 cpu threads:
# Explicitly paralelize both Sphinx and LaTeX pdf builds
$ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs -j 33
real 5m17.901s
user 15m1.499s
sys 2m31.482s
# Use POSIX jobserver to paralelize both sphinx-build and LaTeX
$ make cleandocs; time make pdfdocs
real 5m22.369s
user 15m9.076s
sys 2m31.419s
# Serializes PDF build, while keeping Sphinx parallelized.
# it is equivalent of passing -jauto via command line
$ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs
real 11m20.901s
user 13m2.910s
sys 1m44.553s
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <42eef319f9af6f9feb12bcd74ca6392c8119929d.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
By default, we use LaTeX batch mode to build docs. This way, when
an error happens, the build fails. This is good for normal builds,
but when debugging problems with pdf generation, the best is to
use interactive mode.
We already support it via LATEXOPTS, but having a command line
argument makes it easier:
Interactive mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i
...
Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Default batch mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v
...
Running 'xelatex --no-pdf -no-pdf -interaction=batchmode -no-shell-escape -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <9e5b9a8becc981b47ca3bf3ddce034f273400738.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
There are too much magic inside docs Makefile to properly run
sphinx-build. Create an ancillary script that contains all
kernel-related sphinx-build call logic currently at Makefile.
Such script is designed to work both as an standalone command
and as part of a Makefile. As such, it properly handles POSIX
jobserver used by GNU make.
On a side note, there was a line number increase due to the
conversion (ignoring comments) is:
Documentation/Makefile | 131 +++----------
tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
2 files changed, 323 insertions(+), 101 deletions(-)
Comments and descriptions adds:
tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++-
So, about half of the script are comments/descriptions.
This is because some things are more verbosed on Python and because
it requires reading env vars from Makefile. Besides it, this script
has some extra features that don't exist at the Makefile:
- It can be called directly from command line;
- It properly return PDF build errors.
When running the script alone, it will only take handle sphinx-build
targets. On other words, it won't runn make rustdoc after building
htmlfiles, nor it will run the extra check scripts.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <80ae57b01fcfb1d338d93b8f8e26e57b69b5f16b.1758196090.git.mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Mauro Carvalho Chehab
Jonathan Corbet
Akira Yokosawa