Most of the rules inside CTransforms are of the type CMatch.
Don't re-parse the source code every time.
Doing this doesn't change the output, but makes kdoc almost
as fast as before the tokenizer patches:
# Before tokenizer patches
$ time ./scripts/kernel-doc . -man >original 2>&1
real 0m42.933s
user 0m36.523s
sys 0m1.145s
# After tokenizer patches
$ time ./scripts/kernel-doc . -man >before 2>&1
real 1m29.853s
user 1m23.974s
sys 0m1.237s
# After this patch
$ time ./scripts/kernel-doc . -man >after 2>&1
real 0m48.579s
user 0m45.938s
sys 0m0.988s
$ diff -s before after
Files before and after are identical
Manually checked the differences between original and after
with:
$ diff -U0 -prBw original after|grep -v Warning|grep -v "@@"|less
They're due:
- whitespace fixes;
- struct_group are now better handled;
- several badly-generated man pages from broken inline kernel-doc
markups are now fixed.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <1cc2a4286ebf7d4b2d03fcaf42a1ba9fa09004b9.1773770483.git.mchehab+huawei@kernel.org>
Changeset 2b957decdb6c ("docs: kdoc: don't add broken comments inside prototypes")
revealed a hidden bug at split_struct_proto(): some comments there may break
its capability of properly identifying a struct.
Fixing it is as simple as stripping comments before calling it.
Fixes: 2b957decdb6c ("docs: kdoc: don't add broken comments inside prototypes")
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <dcff37b6da5329aea415de31f543b6a1c2cbbbce.1773770483.git.mchehab+huawei@kernel.org>
Better handle comments inside structs. After those changes,
all unittests now pass:
test_private:
TestPublicPrivate:
test balanced_inner_private: OK
test balanced_non_greddy_private: OK
test balanced_private: OK
test no private: OK
test unbalanced_inner_private: OK
test unbalanced_private: OK
test unbalanced_struct_group_tagged_with_private: OK
test unbalanced_two_struct_group_tagged_first_with_private: OK
test unbalanced_without_end_of_line: OK
Ran 9 tests
This also solves a bug when handling STRUCT_GROUP() with a private
comment on it:
@@ -397134,7 +397134,7 @@ basic V4L2 device-level support.
unsigned int max_len;
unsigned int offset;
struct page_pool_params_slow slow;
- STRUCT_GROUP( struct net_device *netdev;
+ struct net_device *netdev;
unsigned int queue_idx;
unsigned int flags;
};
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <f83ee9e8c38407eaab6ad10d4ccf155fb36683cc.1773074166.git.mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <054763260f7b5459ad0738ed906d7c358d640692.1773770483.git.mchehab+huawei@kernel.org>
Parsing a file like drivers/scsi/isci/host.h, which contains
broken kernel-doc markups makes it create a prototype that contains
unmatched end comments.
That causes, for instance, struct sci_power_control to be shown this
this prototype:
struct sci_power_control {
* it is not. */ bool timer_started;
*/ struct sci_timer timer;
* requesters field. */ u8 phys_waiting;
*/ u8 phys_granted_power;
* mapped into requesters via struct sci_phy.phy_index */ struct isci_phy *requesters[SCI_MAX_PHYS];
};
as comments won't start with "/*" anymore.
Fix the logic to detect such cases, and keep adding the comments
inside it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Message-ID: <18e577dbbd538dcc22945ff139fe3638344e14f0.1773074166.git.mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <12ac4a97e2bd5a19d6537122c10098690c38d2c7.1773770483.git.mchehab+huawei@kernel.org>
Drop context analysis and lock (tracking) attributes to avoid
kernel-doc warnings.
There are now lots of warnings like these:
Documentation/core-api/kref:328: ../include/linux/kref.h:72: WARNING: Invalid C declaration: Expected end of definition. [error at 96]
int kref_put_mutex (struct kref *kref, void (*release)(struct kref *kref), struct mutex *mutex) __cond_acquires(true# mutex)
------------------------------------------------------------------------------------------------^
Documentation/core-api/kref:328: ../include/linux/kref.h:94: WARNING: Invalid C declaration: Expected end of definition. [error at 92]
int kref_put_lock (struct kref *kref, void (*release)(struct kref *kref), spinlock_t *lock) __cond_acquires(true# lock)
--------------------------------------------------------------------------------------------^
The regex is suggested by Mauro; mine was too greedy. Thanks.
Updated context analysis and lock macros list provided by PeterZ. Thanks.
[mchehab: modified to be applied after xforms_lists split]
Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
Closes: https://lore.kernel.org/all/20260107161548.45530e1c@canb.auug.org.au/
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <3c7fdfc364a8920f92530b47bdbf4bb29a40371f.1772469446.git.mchehab+huawei@kernel.org>
Over the time, most of the changes at kernel-doc are related
to maintaining a list of transforms to convert macros into pure
C code.
Place such transforms on a separate module, to cleanup the
parser module.
There is an advantage on that: QEMU also uses our own kernel-doc,
but the xforms list there is different. By placing it on a
separate module, we can minimize the differences and make it
easier to keep QEMU in sync with Kernel upstream.
No functional changes.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <ccd74b7589e1fff340a74bf8ed16a974532cb54f.1772469446.git.mchehab+huawei@kernel.org>
Mangling with #defines is not nice, as we may end removing
the macro names, preventing several macros from being properly
documented.
Also, on defines, we have something like:
#define foo(a1, a2, a3, ...) \
/* some real implementation */
The prototype part (first line on this example) won't contain
any macros, so no need to apply any regexes on it.
With that, move the apply_transforms() logic to ensure that
it will be called only on functions.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Aleksandr Loktionov <aleksandr.loktionov@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <8f9854c8ca1c794b6a3fe418f7adbc32aa68b432.1772469446.git.mchehab+huawei@kernel.org>
Handle functions that are marked with __exit to prevent warnings:
Documentation/networking/iucv:35: ../net/iucv/iucv.c:1918: WARNING: Error in declarator or parameters
Invalid C declaration: Expecting "(" in parameters. [error at 12]
void __exit iucv_exit (void)
------------^
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Mauro Carvalho Chehab <mchehab@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20260206065440.2412185-1-rdunlap@infradead.org>
Mauro's work to include documentation from our Python modules. His cover
letter follows:
This is an extended version of:
https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/
It basically adds everything we currently have inside libs/tool/python
to "tools" book inside documentation.
This version should be independent of the other series yet to be merged,
(including the jobserver one).
The vast amount of changes here are docstring cleanups and additions.
They mainly consists on:
- ensuring that every phrase will end with a period, making it uniform
along all files;
- cleaning ups to better uniform docstrings;
- variable descriptions now use "#:" markup, as it allows autodoc to
add them inside the documentation;
- added some missing docstrings;
- some new blank lines at comments to make ReST syntax parser happy;
- add a couple of sphinx markups (mainly, code blocks).
Most of those are minor changes, affecting only comments.
It also has one patch per libarary type, adding them to docs.
For kernel-doc, I did the cleanups first, as there is one code block
inside tools/lib/python/kdoc/latex_fonts.py that would cause a Sphinx
crash without such markups.
The series actually starts with 3 fixes:
- avoid "*" markups on indexes with deep> 3 to override text
- a variable rename to stop abusing doctree name
- don't rely on cwd to get Documentation/ location
patch 4 adds support to document scripts either at:
- tools/
- scripts/
patch 5 contains a CSS to better display autodoc html output.
For those who want to play with documentation, documenting a python
file is very simple. All it takes is to use:
.. automodule:: lib.python.<dir+name>
Usually, we add a couple of control members to it to adjust
the desired documentation scope (add/remove members, showing class
inheritance, showing members that currently don't have
docstrings, etc). That's why we're using:
.. automodule:: lib.python.kdoc.enrich_formatter
:members:
:show-inheritance:
:undoc-members:
(and similar) inside tools/kdoc*.rst.
autodoc allows filtering in/out members, file docstrings, etc.
It also allows documenting just some members or functions with
directives like:
..autofunction:
..automember:
Sphinx also has a helper script to generate .rst files with
documentation:
$ sphinx-apidoc -o foobar tools/lib/python/
which can be helpful to discover what should be documented,
although changes are needed to use what it produces.
Changeset 469c1c9eb6 ("kernel-doc: Issue warnings that were silently discarded")
didn't properly addressed the missing messages behavior, as
it was calling directly python logger low-level function,
instead of using the expected method to emit warnings.
Basically, there are two methods to log messages:
- self.config.log.warning() - This is the raw level to emit a
warning. It just writes the a message at stderr, via python
logging, as it is initialized as:
self.config.log = logging.getLogger("kernel-doc")
- self.config.warning() - This is where we actually consider a
message as a warning, properly incrementing error count.
Due to that, several parsing error messages are internally considered
as success, causing -Werror to not work on such messages.
While here, ensure that the last ignored entry will also be handled
by adding an extra check at the end of the parse handler.
Fixes: 469c1c9eb6 ("kernel-doc: Issue warnings that were silently discarded")
Closes: https://lore.kernel.org/linux-doc/20260112091053.00cee29a@foz.lan/
Cc: stable@vger.kernel.org
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <95109a6585171da4d6900049deaa2634b41ee743.1768823489.git.mchehab+huawei@kernel.org>
Inline kernel-doc blocks failed to parse tags containing dots (e.g.
creator.process_name in panfrost_gem.h) because the @name regex only
matched word characters. Modify the single-line pattern to match
doc_inline_sect so it includes \. and parses the same as a multi-line
comment.
Signed-off-by: Steven Price <steven.price@arm.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org>
Tested-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251211104851.45330-1-steven.price@arm.com>
kdoc is looking for "@value" here, so use that kind of string in the
warning message. The "%value" can be confusing.
This changes:
Warning: drivers/net/wireless/mediatek/mt76/testmode.h:92 Excess enum value '%MT76_TM_ATTR_TX_PENDING' description in 'mt76_testmode_attr'
to this:
Warning: drivers/net/wireless/mediatek/mt76/testmode.h:92 Excess enum value '@MT76_TM_ATTR_TX_PENDING' description in 'mt76_testmode_attr'
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251126061752.3497106-1-rdunlap@infradead.org>
Recognize and ignore __rcu (in struct members), __private (in struct
members), and __always_unused (in function parameters) to prevent
kernel-doc warnings:
Warning: include/linux/rethook.h:38 struct member 'void (__rcu *handler' not described in 'rethook'
Warning: include/linux/hrtimer_types.h:47 Invalid param: enum hrtimer_restart (*__private function)(struct hrtimer *)
Warning: security/ipe/hooks.c:81 function parameter '__always_unused' not described in 'ipe_mmap_file'
Warning: security/ipe/hooks.c:109 function parameter '__always_unused' not described in 'ipe_file_mprotect'
There are more of these (in compiler_types.h, compiler_attributes.h)
that can be added as needed.
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251127063117.150384-1-rdunlap@infradead.org>
Now that we have tools/lib/python for our Python modules, turn them into
proper packages with a single namespace so that everything can just use
tools/lib/python in sys.path. No functional change.
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-3-corbet@lwn.net>
"scripts/lib" was always a bit of an awkward place for Python modules. We
already have tools/lib; create a tools/lib/python, move the libraries
there, and update the users accordingly.
While at it, move the contents of tools/docs/lib. Rather than make another
directory, just put these documentation-oriented modules under "kdoc".
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Message-ID: <20251110220430.726665-2-corbet@lwn.net>
