Now that we have tools/lib/python for our Python modules, turn them into
proper packages with a single namespace so that everything can just use
tools/lib/python in sys.path. No functional change.
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-3-corbet@lwn.net>
"scripts/lib" was always a bit of an awkward place for Python modules. We
already have tools/lib; create a tools/lib/python, move the libraries
there, and update the users accordingly.
While at it, move the contents of tools/docs/lib. Rather than make another
directory, just put these documentation-oriented modules under "kdoc".
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-2-corbet@lwn.net>
Improve the suggestions algorithm by using get_close_matches() if
no suggestions with the same name are found. As we're now building
a dict, when the name is identical, but on a different domain,
the search is O(1), making it a lot faster.
The get_close_matches is also fast, as there is just one loop,
instead of 3.
This can be useful to detect typos on references, with could
be the base of a futuere extension that will handle ref unmatches
for the entire build, allowing someone to find typos and fix them.
As difflib and get_close_matches are there since the early
Python 3.x days, we don't need to handle any extra dependencies
to use it.
We're keeping the default values for the search, e.g. n=3, cutoff=0.6.
With that, we now have things like:
$ make SPHINXDIRS="userspace-api/media" htmldocs
...
include/uapi/linux/videodev2.h:199: WARNING: Invalid xref: c:type:`v4l2_memory`. Possible alternatives:
c:type:`v4l2_meta_format` (from v4l/dev-meta)
c:type:`v4l2_rect` (from v4l/dev-overlay)
c:type:`v4l2_area` (from v4l/ext-ctrls-image-source) [ref.missing]
...
include/uapi/linux/videodev2.h:1985: WARNING: Invalid xref: c:type:`V4L.v4l2_queryctrl`. Possible alternatives:
std🏷️`v4l2-queryctrl` (from v4l/vidioc-queryctrl)
std🏷️`v4l2-query-ext-ctrl` (from v4l/vidioc-queryctrl)
At the first example, it was not a typo, but a symbol that doesn't
seem to be properly documented. The second example points to
v4l2-queryctrl, which is a close match for the symbol.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <7365feb74cbdd6b982c87baf5863360ab98cf727.1759329363.git.mchehab+huawei@kernel.org>
Add an extra parameter to parse_file to make it handle exceptions
internally, cleaning up the API.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <8575bbc94ff706aa7e7cc3a188399ca17a3169e6.1759329363.git.mchehab+huawei@kernel.org>
Specially when using c::namespace, it is not hard to break
a reference by forgetting to add a domain. Also, different
cases and using "-"/"_" the wrong way are typical cases that
people often gets wrong.
We might use a more complex logic here to also check for typos,
but let's keep it plain, simple.
This is enough to get thos exeptions from media controller:
.../include/uapi/linux/media.h:26: WARNING: Invalid xref: c:type:`media_device_info`. Possible alternatives:
c:type:`MC.media_device_info` (from mediactl/media-ioc-device-info)
.../include/uapi/linux/media.h:149: WARNING: Invalid xref: c:type:`media_entity_desc`. Possible alternatives:
c:type:`MC.media_entity_desc` (from mediactl/media-ioc-enum-entities)
.../include/uapi/linux/media.h:228: WARNING: Invalid xref: c:type:`media_link_desc`. Possible alternatives:
c:type:`MC.media_link_desc` (from mediactl/media-ioc-enum-links)
.../include/uapi/linux/media.h:235: WARNING: Invalid xref: c:type:`media_links_enum`. Possible alternatives:
c:type:`MC.media_links_enum` (from mediactl/media-ioc-enum-links)
.../include/uapi/linux/media.h:212: WARNING: Invalid xref: c:type:`media_pad_desc`. Possible alternatives:
c:type:`MC.media_pad_desc` (from mediactl/media-ioc-enum-links)
.../include/uapi/linux/media.h:298: WARNING: Invalid xref: c:type:`media_v2_entity`. Possible alternatives:
c:type:`MC.media_v2_entity` (from mediactl/media-ioc-g-topology)
.../include/uapi/linux/media.h:312: WARNING: Invalid xref: c:type:`media_v2_interface`. Possible alternatives:
c:type:`MC.media_v2_interface` (from mediactl/media-ioc-g-topology)
.../include/uapi/linux/media.h:307: WARNING: Invalid xref: c:type:`media_v2_intf_devnode`. Possible alternatives:
c:type:`MC.media_v2_intf_devnode` (from mediactl/media-ioc-g-topology)
.../include/uapi/linux/media.h:341: WARNING: Invalid xref: c:type:`media_v2_link`. Possible alternatives:
c:type:`MC.media_v2_link` (from mediactl/media-ioc-g-topology)
.../include/uapi/linux/media.h:333: WARNING: Invalid xref: c:type:`media_v2_pad`. Possible alternatives:
c:type:`MC.media_v2_pad` (from mediactl/media-ioc-g-topology)
.../include/uapi/linux/media.h:349: WARNING: Invalid xref: c:type:`media_v2_topology`. Possible alternatives:
c:type:`MC.media_v2_topology` (from mediactl/media-ioc-g-topology)
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <4c75d277e950e619ea00ba2dea336853a4aac976.1759329363.git.mchehab+huawei@kernel.org>
On TOC output, we need to embeed line numbers with ViewList.
Change the parse class to produce a line-number parsed result,
and adjust the output accordingly.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <74eed96e32f79eaaef7a99ffe7c3224fed369c27.1759329363.git.mchehab+huawei@kernel.org>
The ErrorString() and SafeString() docutils functions were helpers meant to
ease the handling of encodings during the Python 3 transition. There is no
real need for them after Python 3.6, and docutils 0.22 removes them,
breaking the docs build
Handle this by just injecting our own one-liner version of ErrorString(),
and removing the sole SafeString() call entirely.
Reported-by: Zhixu Liu <zhixu.liu@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <87ldmnv2pi.fsf@trenco.lwn.net>
As reported by Stephen, building docs with O= is now
broken. Fix it by ensuring that it will seek files under
Kernel source tree.
The original logic was defined to accept including files
under Documentation/output. The new logic doesn't need it
anymore for media, but it might still be useful to preserve
the previous behavior. So, I ended preserving it.
Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
Closes: https://lore.kernel.org/all/20250901142639.4de35a11@canb.auug.org.au/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/da91980ce42f31730dc982920167b2757b9d2769.1756732363.git.mchehab+huawei@kernel.org
While the original code came from the Sphinx Include class,
such class is monolithic: it has only one function that does
everything, and 3 variables that are used:
- required_arguments
- optional_arguments
- option_spec
So, basically those are the only members that remain from
the original class, but hey! Those are the same vars that every
other Sphinx directive extension has to define!
In summary, keeping inheritance here doesn't make much sense.
Worse than that, kernel-include doesn't support the current set
of options that the original Include class has, but it also
has its own set of options.
So, let's fill in the argument vars with what it does
support, dropping the rest.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/a9f2eebf11c6b0c3a2e3bf42e71392cdfd2835d1.1755872208.git.mchehab+huawei@kernel.org
It is best to point to the original line of code that generated
an error than to point to the beginning of a directive.
Add support for it. It should be noticed that this won't work
for literal or code blocks, as Sphinx will ignore it, pointing
to the beginning of the directive. Yet, when the output is known
to be in ReST format, like on TOC, this makes the error a lot
more easier to be handled.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/a0953af8b71e64aaf2e0ba4593ad39e19587d50a.1755872208.git.mchehab+huawei@kernel.org
When generate-cross-refs is used, instead of just implementing
the default of generating a literal block, we can also
generate a ReST file as a TOC.
The advantage is that, by being a ReST file, missing references
will point to the place inside the header file that has the
broken link.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/c0d32cd1ef94017e05984b0a38bd2516f7db21e2.1755872208.git.mchehab+huawei@kernel.org
Originally, parse-readers were generating an output where
the first two lines were setting a literal block.
The script now gets only the actual parsed data without that,
so it is now safe to allow start-line and end-line parameters
to be handled.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/5dff693860a6a3faade15c24abdc380f09db468d.1755872208.git.mchehab+huawei@kernel.org
In the past, Sphinx used to warn about broken references. That's
basically the rationale for adding media uAPI files: to get
warnings about missed symbols.
This is not true anymore. So, we need to explicitly check them
after doctree-resolved event.
While here, move setup() to the end, to make it closer to
what we do on other extensions.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/73be9a198746421687e2eee916ccf8bf67980b7d.1755872208.git.mchehab+huawei@kernel.org
kernel_include extension was originally designed to be used by the
media comprehensive uAPI documentation, where, instead of simpler
kernel-doc markups, the uAPI documentation is enriched with a larger
text, with images, complex tables, graphs, etc.
There, we wanted to include the much simpler yet documented .h
file.
This extension is needed to include files from other parts of the
Kernel tree outside Documentation, because the original Sphinx
include tag doesn't allow going outside of the directory passed
via sphinx-build command line.
Yet, the cross-references themselves to the full documentation
were using a perl script to create cross-references against the
comprehensive documentation.
As the perl script is now converted to Phython and there is a
Python class producing an include-compatible output with cross
references, add two optional arguments to kernel_include.py:
1. :generate-cross-refs:
If present, instead of reading the file, it calls ParseDataStructs()
class, which converts C data structures into cross-references to
be linked to ReST files containing a more comprehensive documentation;
Don't use it together with :start-line: and/or :end-line:, as
filtering input file line range is currently not supported.
2. :exception-file:
Used together with :generate-cross-refs:. Points to a file containing
rules to ignore C data structs or to use a different reference name,
optionally using a different reference type.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/efc39c8e54a2056ae2fdb94d5006fcb19e227198.1755872208.git.mchehab+huawei@kernel.org
The setup() function of a Sphinx-extension can return a dictionary. This
is treated by Sphinx as metadata of the extension [1].
With metadata "parallel_read_safe = True" a extension is marked as
save for "parallel reading of source". This is needed if you want
build in parallel with N processes. E.g.:
make SPHINXOPTS=-j4 htmldocs
will no longer log warnings like:
WARNING: the foobar extension does not declare if it is safe for
parallel reading, assuming it isn't - please ask the extension author
to check and make it explicit.
Add metadata to extensions:
* kernel-doc
* flat-table
* kernel-include
[1] http://www.sphinx-doc.org/en/stable/extdev/#extension-metadata
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Tested-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
The kernel-include directive is needed to include the auto generated rst
content from a build (pre-) process. E.g. the linux_tv Makefile
generates intermediate reST-files from header files. Since there is a O=
option:
make O=dir [targets] Locate all output files in "dir"
We need to include intermediate reST files from arbitrary (O=/tmp/foo)
locations:
The 'kernel-include' reST-directive is a replacement for the 'include'
directive. The 'kernel-include' directive expand environment variables
in the path name and allows to include files from arbitrary locations.
.. hint::
Including files from arbitrary locations (e.g. from '/etc') is a
security risk for builders. This is why the 'include' directive from
docutils *prohibit* pathnames pointing to locations *above* the
filesystem tree where the reST document with the include directive is
placed.
Substrings of the form $name or ${name} are replaced by the value of
environment variable name. Malformed variable names and references to
non-existing variables are left unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarIT.de>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Jonathan Corbet
Mauro Carvalho Chehab
Akira Yokosawa
Mauro Carvalho Chehab
Markus Heiser