On ARM64, when running with --configs '36*SRCU-P', I noticed that only 1 instance
instead of 36 for starting.
Fix it by checking for Image files, instead of bzImage which ARM does
not seem to have. With this I see all 36 instances running at the same
time in the batch.
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Back in the day, rcutorture was about the only thing that tested off-stack
CPU masks, but now any arm64 system with more than 256 CPUs tests it
full time. In fact, it is necessary to hack the kernel to prevent such
a system from testing off-stack CPU masks. This means that there is
no longer much point in rcutorture going out of its way to test this.
And given the differences in how CPUMASK_OFFSTACK is enabled in x86 and
arm64, rcutorture would need to go out of its way.
This commit therefore removes CONFIG_CPUMASK_OFFSTACK=y (and the
CONFIG_MAXSMP=y required to enable it on x86) from TREE01.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
The TREE01.boot nr_cpus kernel boot parameter has been set to 43 for
more than seven years, but it can cause RCU CPU stall warnings on arm64,
most of the time involving the stop-machine subsystem. This should
not be too surprising, given that this causes 43 vCPUs to spin with
interrupts disabled when there are only eight physical CPUs.
The point of this CPU overcommit is to test the ability of expedited RCU
grace period initialization to handle races with incoming CPUs that have
never previously been online. But limiting to 17 CPUs instead of 43
allows time for this code to be exercised, and eliminates (or at least
greatly reduces) the incidence of RCU CPU stall warnings on arm64.
So this commit therefore sets nr_cpus=17 in TREE01.boot.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
This commit adds a --do-rcu-rust parameter to torture.sh, which invokes
a rust_doctests_kernel kunit run. Note that kunit wants a clean source
tree, so this runs "make mrproper", which might come as a surprise to
some users. Should there be a --mrproper parameter to torture.sh to make
the user explicitly ask for it?
Co-developed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Right now, torture.sh runs normal runs unconditionally, which can be slow
and thus annoying when you only want to test --kcsan or --kasan runs.
This commit therefore adds a --do-normal argument so that "--kcsan
--do-no-kasan --do-no-normal" runs only KCSAN runs. Note that specifying
"--do-no-kasan --do-no-kcsan --do-no-normal" gets normal runs, so you
should not try to use this as a synonym for --do-none.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
The torture.sh --do-rt command-line parameter is intended to mimic -rt
kernels. Now that CONFIG_PREEMPT_RT is upstream, this commit makes this
mimicking more precise.
Note that testing of RCU priority boosting is disabled in favor
of forward-progress testing of RCU callbacks. If it turns out to be
possible to make kernels built with CONFIG_PREEMPT_RT=y to tolerate
testing of both, both will be enabled.
[ paulmck: Apply Sebastian Siewior feedback. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Mixing different flavors of RCU readers is forbidden, for example, you
should not use srcu_read_lock() and srcu_read_lock_nmisafe() on the same
srcu_struct structure. There are checks for this, but these checks are
not tested on a regular basis. This commit therefore adds such tests
to srcu_lockdep.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
The srcu_lockdep.sh currently blindly trusts the rcutorture SRCU-P
scenario to build its kernel with lockdep enabled. Of course, this
dependency might not be obvious to someone rebalancing SRCU scenarios.
This commit therefore adds code to srcu_lockdep.sh that verifies that
the .config file has lockdep enabled.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
This commit tests lazy preemption by causing the TREE07 rcutorture
scenario to build its kernel with CONFIG_PREEMPT_LAZY=y.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
This commit tests lazy preemption by causing the TREE10 rcutorture
scenario to build its kernel with CONFIG_PREEMPT_LAZY=y.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Add extra parameters for rcutorture module. One is the "nfakewriters"
which is set -1. There will be created number of test-kthreads which
correspond to number of CPUs in a test system. Those threads randomly
invoke synchronize_rcu() call.
Apart of that "rcu_normal" is set to 1, because it is specifically for
a normal synchronize_rcu() testing, also a newly added parameter which
is "rcu_normal_wake_from_gp" is set to 1 also. That prevents interaction
with other callbacks in a system.
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Link: https://lore.kernel.org/r/20250227131613.52683-2-urezki@gmail.com
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Recent experience shows that the srcu_read_lock_nmisafe() and
srcu_read_unlock_nmisafe() functions are not sufficiently tested.
This commit therefore causes the torture.sh script's SRCU lockdep testing
to use these two functions. This will cause these two functions to
be regularly tested by several developers (myself included) who use
torture.sh as an RCU acceptance test.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
This commit causes the rcutorture SRCU-P scenario use the
srcu_read_lock_fast() and srcu_read_unlock_fast() functions. This will
cause these two functions to be regularly tested by several developers
(myself included), for example, those who use torture.sh as an RCU
acceptance test.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
This commit adds the rcutorture.preempt_duration module parameter to
rcutorture's TREE03.boot parameter list in order to better test preemption
of RCU read-side critical sections.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Currently, a system that stops responding at the wrong time will hang
kvm-remote.sh. This can happen when the system in question is forced
offline for maintenance, and there is currently no way for the user
to kick this script into moving ahead. This commit therefore causes
kvm-remote.sh to wait at most 15 minutes for a non-responsive system,
that is, a system for which ssh gives an exit code of 255.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
In performance tests, it can be counter-productive to spread torture-test
guest OSes across sockets. Plus the experimenter might have ideas about
what CPUs individual guest OSes are to run on. This commit therefore
adds a --no-affinity parameter to kvm.sh to prevent it from running
taskset on its guest OSes.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
This commit adds a TINY scenario in order to support tests of Tiny RCU
and Tiny SRCU.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
Some servers have limitations on the number of CPUs a given guest OS
can use. In my earlier experience, such limitations have been at least
half of the host's CPUs, but in a recent example, this limit is less
than 40%. This commit therefore adds a --guest-cpu-limit argument that
allows such low limits to be made known to torture.sh.
Signed-off-by: "Paul E. McKenney" <paulmck@kernel.org>
Signed-off-by: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
This commit adds the rcutree.nohz_full_patience_delay=1000 kernel boot
parameter to the TREE07 scenario, on the observation that "if it ain't
tested, it don't work".
Signed-off-by: "Paul E. McKenney" <paulmck@kernel.org>
Signed-off-by: Neeraj Upadhyay <neeraj.upadhyay@kernel.org>
Currently, the torture.sh --do-kvfree testing is hard-coded to ten
minutes, ignoring the --duration argument. This commit therefore scales
this test duration the same as for the rcutorture tests.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
The TREE09 rcutorture scenario exhausts memory from time to time, and
this is due to a reader being preempted and blocking grace periods,
thus preventing recycling of the memory used in callback-flooding tests.
This commit therefore enables RCU priority boosting and sets the boosting
delay to 100 milliseconds after grace-period start.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Now that the KPROBES, TRACING, BLK_DEV_IO_TRACE, and UPROBE_EVENTS
Kconfig options select the TASKS_TRACE_RCU option, the torture.sh tests
of enabling exactly one of the RCU Tasks flavors fail. This commit
therefore disables these options to allow this testing to succeed.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
The tradition, extending back almost a full year, has been 2GB plus an
additional number of GBs equal to the number of CPUs divided by sixteen.
This tradition has served scftorture well, even the CONFIG_PREEMPT=y
version running KASAN within guest OSes having 40 CPUs. However, this
test recently started OOMing on larger systems, and this commit therefore
gives this test an additional GB of memory.
It is quite possible that further testing on larger systems will show
a need to decrease the divisor from 16 to (say) 8, but that is a change
to make once it has been demonstrated to be required.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
On a 8-socket server the TSC is wrongly marked as 'unstable' and disabled
during boot time on about one out of 120 boot attempts:
clocksource: timekeeping watchdog on CPU227: wd-tsc-wd excessive read-back delay of 153560ns vs. limit of 125000ns,
wd-wd read-back delay only 11440ns, attempt 3, marking tsc unstable
tsc: Marking TSC unstable due to clocksource watchdog
TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
sched_clock: Marking unstable (119294969739, 159204297)<-(125446229205, -5992055152)
clocksource: Checking clocksource tsc synchronization from CPU 319 to CPUs 0,99,136,180,210,542,601,896.
clocksource: Switched to clocksource hpet
The reason is that for platform with a large number of CPUs, there are
sporadic big or huge read latencies while reading the watchog/clocksource
during boot or when system is under stress work load, and the frequency and
maximum value of the latency goes up with the number of online CPUs.
The cCurrent code already has logic to detect and filter such high latency
case by reading the watchdog twice and checking the two deltas. Due to the
randomness of the latency, there is a low probabilty that the first delta
(latency) is big, but the second delta is small and looks valid. The
watchdog code retries the readouts by default twice, which is not
necessarily sufficient for systems with a large number of CPUs.
There is a command line parameter 'max_cswd_read_retries' which allows to
increase the number of retries, but that's not user friendly as it needs to
be tweaked per system. As the number of required retries is proportional to
the number of online CPUs, this parameter can be calculated at runtime.
Scale and enlarge the number of retries according to the number of online
CPUs and remove the command line parameter completely.
[ tglx: Massaged change log and comments ]
Signed-off-by: Feng Tang <feng.tang@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Jin Wang <jin1.wang@intel.com>
Tested-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Link: https://lore.kernel.org/r/20240221060859.1027450-1-feng.tang@intel.com
There is code in rcu_implicit_dynticks_qs() that checks for the current
grace period being halfway to the RCU CPU stall timeout, but rcutorture
currently does not test this code. This commit therefore adds a 14-second
stall to the TREE07 scenario in order to test this code given the default
RCU CPU stall warning timeout of 21 seconds.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Neeraj Upadhyay (AMD) <neeraj.iitr10@gmail.com>
This commit does the long-overdue conversion of the parse-console.sh
file to use mktemp to create its temporary directory.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
This commit adds a --debug-info argument to kvm.sh in order to ease
interpretation of addresses printed on the console and the like.
This argument also disables KASLR.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
In torture.sh, the testing of refscale incorrectly used verbose_batched
as a kernel boot parameter, which causes this parameter to be passed
to the init process. This commit therefore prefixes it with refscale,
so that refscale.verbose_batched is passed to the kernel.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
When debugging, it can be difficult to quickly find the ftrace dump
within the console log, which in turn makes it difficult to process it
independent of the rest of the console output. This commit therefore
copies the contents of the buffers into its own file to make it easier
to locate and process the ftrace dump. The original ftrace dump is still
available in the console log in cases because it can be more convenient
to process it in situ, for example, for scripts that process console
output as well as ftrace-dump data.
Also handle the case of multiple ftrace dumps potentially showing up in the
log. Example for a file like [1], it will extract as [2].
[1]:
foo
foo
Dumping ftrace buffer:
---------------------------------
blah
blah
---------------------------------
more
bar
baz
Dumping ftrace buffer:
---------------------------------
blah2
blah2
---------------------------------
bleh
bleh
[2]:
Ftrace dump 1:
blah
blah
Ftrace dump 2:
blah2
blah2
[ paulmck: Fixed awk indentation, input up front. ]
Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
This commit adds CONFIG_DEBUG_OBJECTS=y to the TRACE02 rcutorture scenario
to catch any further RCU Tasks bugs involving this Kconfig option.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
This commit switches from the old "/tmp/kvm-recheck.sh.$$" approach to
the newer and now reliable "mktemp" approach.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Pull smp_call_function torture-test updates from Paul McKenney:
"This prevents some memory-exhaustion false-postitive failures in
scftorture testing"
* tag 'scftorture.2023.08.15a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu:
scftorture: Add CONFIG_PREEMPT_DYNAMIC=n to NOPREEMPT scenario
scftorture: Pause testing after memory-allocation failure
scftorture: Forgive memory-allocation failure if KASAN
torture: Scale scftorture memory based on number of CPUs
Currently, if the C program created by mkinitrd.sh has compile errors,
the errors are printed, but kvm.sh soldiers on, building kernels that
have init-less initrd setups. The kernels then fail on boot when they
attempt to mount non-existent root filesystems.
This commit therefore improves user friendliness by making mkinitrd.sh
return non-zero exit status on compile errors, which in turn causes kvm.sh
to take an early exit, with the compile errors still clearly visible.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit causes the init program generated by mkinitrd.sh dump out
its parameters. Although this is in some sense redundant given that
the kernel already dumps them out, confirmation can be a good thing.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit switches the qemu argument "-nographic" to "-display none",
aligning with the nolibc tests.
Cc: Willy Tarreau <w@1wt.eu>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds the __loongarch__, __loongarch_lp64, and
__loongarch_double_float targets to rcutorture's mkinitrd.sh
script in order to allow nolibc init programs for loongarch.
[ paulmck: Apply feedback from Feiyang Chen. ]
Cc: Feiyang Chen <chenfeiyang@loongson.cn>
Cc: Huacai Chen <chenhuacai@loongson.cn>
Cc: Willy Tarreau <w@1wt.eu>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, the various torture tests sometimes react to an early-boot
bug by rebooting. This is almost always counterproductive, needlessly
consuming CPU time and bloating the console log. This commit therefore
adds the "-no-reboot" argument to qemu so that reboot requests will
cause qemu to exit.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds srcu_lockdep.sh to torture.sh, thus exercizing the
extended SRCU-aware lockdep-RCU functionality on a regular basis.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
KCSAN enables some Kconfig options unilaterally and unconditionally,
including CONFIG_PROVE_LOCKING. This in turn enables CONFIG_PROVE_RCU
and CONFIG_PREEMPT_COUNT, which conflicts with constraints in SRCU-T,
TRACE01, and TREE10, which in turn causes rcutorture to emit spurious
configuration complaints. This commit therefore forgives configuration
complaints involving CONFIG_PROVE_RCU and CONFIG_PREEMPT_COUNT.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
If some of the torture.sh runs had config and/or build errors, but all
runs for which kernels were built ran successfully to completion, then
torture.sh will incorrectly claim that all errors were KCSAN errors.
This commit therefore makes torture.sh print the number of runs with
config and build errors, and to refrain from claiming that all bugs were
KCSAN bugs in that case.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, the kernel boot parameters specified by the kvm.sh --bootargs
parameter are placed near the beginning of the -append list that is
passed to qemu. This means that in the not-uncommon case of a kernel
boot parameter where the last argument wins, the --bootargs list overrides
neither the list in the .boot file nor the additional parameters supplied
by the rcutorture scripting.
This commit therefore places the kernel boot parameters specified by
the kvm.sh --bootargs parameter at the end of qemu's -append list.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
