When printing --help, we'd like the name of the files
from __doc__ to match the displayed positional arguments at
both usage and argument description lines.
Use a custom formatter class to convert ``foo`` into ANSI SGR
code to bold the argument, if is TTY, and adjust the help
text to match the argument names.
Here on Plasma, that makes it display it colored, wich is
really cool. Yet, I opted for SGR, as the best is to follow
the terminal color schema for bold.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/2c1e61d1fb1b2a2838b443beee89c1528831997f.1755872208.git.mchehab+huawei@kernel.org
When the Kernel started to use Sphinx, we had to come up with
a solution to parse media headers. On that time, we didn't have
much experience with Sphinx extensions. So, we came up with our
own script-based solution that were basically implementing a
set of rules we used to have at the Makefile.
Convert it to Python, keeping it bug-compatible with the
original script.
While here, try to better document it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/ae5cfa8dff37e280cc9493fc95a51cd0cc0ba127.1755872208.git.mchehab+huawei@kernel.org
As reported by Donald, this code:
rst_parser = RSTParser()
rst_parser.parse('\n'.join(result), document)
breaks line parsing. As an alternative, I tested a variant of it:
rst_parser.parse(result, document)
but still line number was not preserved. As Donald noted,
standard Parser classes don't have a direct mechanism to preserve
line numbers from ViewList().
So, instead, let's use a mechanism similar to what we do already at
kerneldoc.py: call the statemachine mechanism directly there.
I double-checked when states and statemachine were introduced:
both were back in 2002. I also tested doc build with docutils 0.16
and 0.21.2. It worked with both, so it seems to be stable enough
for our needs.
Reported-by: Donald Hunter <donald.hunter@gmail.com>
Closes: https://lore.kernel.org/linux-doc/m24ivk78ng.fsf@gmail.com/T/#u
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Instead of printing line numbers from the temp converted ReST
file, get them from the original source.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Add a simple sphinx.Parser to handle yaml files and add the
the code to handle Netlink specs. All other yaml files are
ignored.
The code was written in a way that parsing yaml for different
subsystems and even for different parts of Netlink are easy.
All it takes to have a different parser is to add an
import line similar to:
from doc_generator import YnlDocGenerator
adding the corresponding parser somewhere at the extension:
netlink_parser = YnlDocGenerator()
And then add a logic inside parse() to handle different
doc outputs, depending on the file location, similar to:
if "/netlink/specs/" in fname:
msg = self.netlink_parser.parse_yaml_file(fname)
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
The logic there which adds a dependency note to Sphinx cache
is not taking into account that the build dir may not be
the source dir. This causes a performance regression:
$ time make O=/tmp/foo SPHINXDIRS=admin-guide htmldocs
[OUTDATED]
Added: set()
Changed: {'abi-obsolete', 'abi-removed', 'abi-stable-files', 'abi-obsolete-files', 'abi-stable', 'abi', 'abi-removed-files', 'abi-testing-files', 'abi-testing', 'gpio/index', 'gpio/obsolete'}
Removed: set()
All docs count: 385
Found docs count: 385
real 0m11,324s
user 0m15,783s
sys 0m1,164s
To get the root cause of the problem (ABI files reported as changed),
I used this changeset:
diff --git a/Documentation/conf.py b/Documentation/conf.py
index e8766e689c1b..ab486623bd8b 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -571,3 +571,16 @@ def setup(app):
"""Patterns need to be updated at init time on older Sphinx versions"""
app.connect('config-inited', update_patterns)
+ app.connect('env-get-outdated', on_outdated)
+
+def on_outdated(app, env, added, changed, removed):
+ """Track cache outdated due to added/changed/removed files"""
+ print("\n[OUTDATED]")
+ print(f"Added: {added}")
+ print(f"Changed: {changed}")
+ print(f"Removed: {removed}")
+ print(f"All docs count: {len(env.all_docs)}")
+ print(f"Found docs count: {len(env.found_docs)}")
+
+ # Just return what we have
+ return added | changed | removed
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/linux-doc/c174f7c5-ec21-4eae-b1c3-f643cca90d9d@gmail.com/
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Tested-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/e25673d87357457bc54ee863e97ff8f75956580d.1752752211.git.mchehab+huawei@kernel.org
Docutils emits a deprecation warning when the set_class() element method is
used; that warning disappears into the ether, but it also causes a crash
with docutils 0.19.
Avoid the deprecated function and just append directly to the "classes"
attribute like the documentation says instead.
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Tested-by: Akira Yokosawa <akiyks@gmail.com>
Closes: https://lore.kernel.org/de7bae91-3200-481f-9db2-c0dc382c91dd@gmail.com/
Fixes: d6d1df92c2 ("docs: automarkup: Mark up undocumented entities too")
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
The automarkup code generates markup and a cross-reference link for
functions, structs, etc. for which it finds kerneldoc documentation.
Undocumented entities are left untouched; that creates an inconsistent
reading experience and has caused some writers to go to extra measures to
cause the markup to happen.
Mark up detected C entities regardless of whether they are documented.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Remove a few declarations that are no longer doing anything now that we
have left Sphinx 2 behind.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Several of the markup functions contain the same code, calling into
sphinx's pending_xref and resolve_xref functions to add and resolve a
cross-reference, with only a few of the parameters changed (domain,
reference type, markup content). Move this logic to its own function and
reuse it in the markup functions.
No functional change.
Signed-off-by: Nícolas F. R. A. Prado <nfraprado@collabora.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/20250408-automarkup-resolve-xref-helper-v2-1-e0a9b8fc7fdd@collabora.com
Documentation/driver-api/infiniband.rst has a kernel-doc tag
with "\" characters at the end:
.. kernel-doc:: drivers/infiniband/ulp/iser/iscsi_iser.c
:functions: iscsi_iser_pdu_alloc iser_initialize_task_headers \
iscsi_iser_task_init iscsi_iser_mtask_xmit iscsi_iser_task_xmit \
iscsi_iser_cleanup_task iscsi_iser_check_protection \
iscsi_iser_conn_create iscsi_iser_conn_bind \
iscsi_iser_conn_start iscsi_iser_conn_stop \
iscsi_iser_session_destroy iscsi_iser_session_create \
iscsi_iser_set_param iscsi_iser_ep_connect iscsi_iser_ep_poll \
iscsi_iser_ep_disconnect
This is not handled well, as the "\" strings will be just stored inside
Sphinx options.
While the actual problem deserves being fixed, better to relax the
keneldoc.py extension to silently strip "\" from the end of strings,
as otherwise this may cause troubles when preparing arguments to
be executed by kernel-doc.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/4c652d6c57b20500c135b95294e554d9e9a97f42.1744106242.git.mchehab+huawei@kernel.org
Sphinx logging system doesn't like warnings during module load,
as it understands that such logs are produced at the wrong time:
WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/physical_package_id is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:27; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/ppin is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:89; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
So, use a function to allocate/process ABI files and use it to
be called at kernel_abi.py, as automarkup also needs it to
produce the right cross-references.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/b0e79dc60d556e3b39fa6774d3b7bf734b73f352.1739182025.git.mchehab+huawei@kernel.org
Now that all ABI files are handled together, we can add a feature
at automarkup for it to generate cross-references for ABI symbols.
The cross-reference logic can produce references for all existing
files, except for README (as this is not parsed).
For symbols, they need to be an exact match of what it is
described at the docs, which is not always true due to wildcards.
If symbols at /sys /proc and /config are identical, a cross-reference
will be used.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/0b97a51b68b1c20127ad4a6a55658557fe0848d0.1739182025.git.mchehab+huawei@kernel.org
In zh_CN part of translations.pdf, there are several ASCII-art
diagrams whose vertical lines look sometimes jagged.
This is due to the interference between default settings of xeCJK
and fancyvrb (employed in sphinxVerbatim env), where extra space
is inserted between a latin char and a non-latin char when they
are next to each other (i.e., no explicit white space).
This issue can be suppressed by invoking \CJKsetecglue{} at the
beginning of every sphinxVerbatim enviornment.
\AtBeginEnvironment, provided by the etoolbox package, is useful in
this case.
Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/20240905050941.31439-1-akiyks@gmail.com
As discussed (see Links), there is some inertia to move to the recent
Sphinx versions for the doc build environment.
As first step, drop the version constraints and the related comments. As
sphinx depends on jinja2, jinja2 is pulled in automatically. So drop that.
Then, the sphinx-pre-install script will fail though with:
Can't get default sphinx version from ./Documentation/sphinx/requirements.txt at ./scripts/sphinx-pre-install line 305.
The script simply expects to parse a version constraint with Sphinx in the
requirements.txt. That version is used in the script for suggesting the
virtualenv directory name.
To suggest a virtualenv directory name, when there is no version given in
the requirements.txt, one could try to guess the version that would be
downloaded with 'pip install -r Documentation/sphinx/requirements.txt'.
However, there seems no simple way to get that version without actually
setting up the venv and running pip. So, instead, name the directory with
the fixed name 'sphinx_latest'.
Finally update the Sphinx build documentation to reflect this directory
name change.
Link: https://lore.kernel.org/linux-doc/874jf4m384.fsf@meer.lwn.net/
Link: https://lore.kernel.org/linux-doc/20240226093854.47830-1-lukas.bulwahn@gmail.com/
Reviewed-by: Akira Yokosawa <akiyks@gmail.com>
Tested-by: Vegard Nossum <vegard.nossum@oracle.com>
Signed-off-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20240301141800.30218-1-lukas.bulwahn@gmail.com>
Now that Sphinx 2.4.4 or better is required, get rid of
\providecommand{}'s for compatibility with Sphinx 1.7.9.
While at it, reword the comment on \sphinxtableofcontentshook
for better description of why it needs to be emptied.
Signed-off-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <ed1ec6f2-0050-46f6-807d-8679f26427e9@gmail.com>
If the directory passed to the '.. kernel-feat::' directive does not
exist or the get_feat.pl script does not find any files to extract
features from, Sphinx will report the following error:
Sphinx parallel build error:
UnboundLocalError: local variable 'fname' referenced before assignment
make[2]: *** [Documentation/Makefile:102: htmldocs] Error 2
This is due to how I changed the script in c48a7c44a1 ("docs:
kernel_feat.py: fix potential command injection"). Before that, the
filename passed along to self.nestedParse() in this case was weirdly
just the whole get_feat.pl invocation.
We can fix it by doing what kernel_abi.py does -- just pass
self.arguments[0] as 'fname'.
Fixes: c48a7c44a1 ("docs: kernel_feat.py: fix potential command injection")
Cc: Justin Forbes <jforbes@fedoraproject.org>
Cc: Salvatore Bonaccorso <carnil@debian.org>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
Link: https://lore.kernel.org/r/20240205175133.774271-2-vegard.nossum@oracle.com
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Akira Yokosawa reported [1] that the "translations" extension we added in
commit 7418ec5b15 ("docs: translations: add translations links when they
exist") broke the build on Sphinx versions v6.1.3 through 7.1.2 (possibly
others) with the following error:
Exception occurred:
File "/usr/lib/python3.12/site-packages/sphinx/util/nodes.py", line 624, in _copy_except__document
newnode = self.__class__(rawsource=self.rawsource, **self.attributes)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LanguagesNode.__init__() missing 1 required positional argument: 'current_language'
The full traceback has been saved in /tmp/sphinx-err-7xmwytuu.log, if you want to report the issue to the developers.
Solve this problem by making 'current_language' a true element attribute
of the LanguagesNode element, which is probably the more correct way to do
it anyway.
Tested on Sphinx 2.x, 3.x, 6.x, and 7.x.
[1]: https://lore.kernel.org/all/54a56c2e-a27c-45a0-b712-02a7bc7d2673@gmail.com/
Fixes: 7418ec5b15 ("docs: translations: add translations links when they exist")
Reported-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
Closes: https://lore.kernel.org/all/54a56c2e-a27c-45a0-b712-02a7bc7d2673@gmail.com/
Tested-by: Akira Yokosawa <akiyks@gmail.com> # Sphinx 4.3.2, 5.3.0 and 6.2.1
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Link: https://lore.kernel.org/r/20240215064109.1193556-1-vegard.nossum@oracle.com
If the directory passed to the '.. kernel-feat::' directive does not
exist or the get_feat.pl script does not find any files to extract
features from, Sphinx will report the following error:
Sphinx parallel build error:
UnboundLocalError: local variable 'fname' referenced before assignment
make[2]: *** [Documentation/Makefile:102: htmldocs] Error 2
This is due to how I changed the script in c48a7c44a1 ("docs:
kernel_feat.py: fix potential command injection"). Before that, the
filename passed along to self.nestedParse() in this case was weirdly
just the whole get_feat.pl invocation.
We can fix it by doing what kernel_abi.py does -- just pass
self.arguments[0] as 'fname'.
Fixes: c48a7c44a1 ("docs: kernel_feat.py: fix potential command injection")
Cc: Justin Forbes <jforbes@fedoraproject.org>
Cc: Salvatore Bonaccorso <carnil@debian.org>
Cc: Jani Nikula <jani.nikula@intel.com>
Cc: Mauro Carvalho Chehab <mchehab@kernel.org>
Cc: stable@vger.kernel.org
Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
Link: https://lore.kernel.org/r/20240205175133.774271-2-vegard.nossum@oracle.com
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
